[Pasig-discuss] FW: Digital repository storage benchmarking

Fri May 12 17:15:15 EDT 2017

Thank you to Tab, Randy, Sheila, Richard, et al. Very interesting and helpful responses!

Best,
Tim

- - -

Tim Walsh
Archiviste, Archives numériques
Archivist, Digital Archives

Centre Canadien d’Architecture
Canadian Centre for Architecture
T 514 939 7001 x 1532
www.cca.qc.ca <http://www.cca.qc.ca/>

On 2017-05-12, 2:43 PM, "Pasig-discuss on behalf of Butler, Tab" <pasig-discuss-bounces at asis.org on behalf of tab.butler at mlb.com> wrote:

    Tim,

    At Major League Baseball, we are focused mostly on archiving the broadcast game video feeds, along with pregame, postgame, and individual camera iso feeds for each game.  The content includes both the home and away team broadcasts, with and without graphics.  Essentially, we record 7 hours plus of content for every 1 hour of baseball played.   We also record and archive all the MLB Network content that is produced, which is between 12 - 18 hours of live content per day.  We will archive the entire broadcast show of record, and the individual elements that make up a show.

    All in, we are recording over 1,000 hours of content per day.  This equates to 50+ TB of content being added to our archive per day.

    We have both an active on-line disk tier (2 SAN's - each 2.88 PB) for recording, editing, and on-line storage, and a data tape archive that supports Partial File Restore (PFR) of video files.  We load balance recording content across the two SAN's... American League on one SAN, and National League on the other... and all edits (96 high performance / 54 desktop machines) access both SAN's.

    Once content is written to a SAN, it is auto archived to tape, as per our DIAMOND asset management system (home grown).  We started archiving on LTO-4 in 2008, and are currently on Oracle T10000-D.  We are migrating content from LTO-4 to T10K-D tape within a tape group...

    We have both an 'On-Site' tape sub-group, and an 'Off-Site' tape sub-group for each of our Tape Groups.  Tape Groups include "Games with Graphics" (Dirty) and "Games without Graphics" (Clean)... the Dirty off-site tapes go to a separate off site location than the Clean off-site tapes.  We break up all of our Off-Site Tape Groups between two geographically distributed locations, as well.

    We are using the Oracle DIVArichive middleware, which performs a checksum value that is compared to the stored database value, each time a file is copied, moved, or restored.  We are performing between 1,000 to 2,000 PFR / Restores per day.

    Currently we have over 45,000 LTO-4's and over 10,000 T10K-D tapes, growing at the rate of 125,000 hours of content per year.

    If you would like more details regarding archiving video content, feel free to reach out to me.

    Sincerely,

    Tab

    Tab Butler | Sr. Director - Media Management & Post Production| MLB Network | 40 Hartz Way, Suite 10 | Secaucus, NJ  07094
    (201) 520-6252 Office | (646) 498-1662 Cell

    tab.butler at mlb.com

    -----Original Message-----
    From: Pasig-discuss [mailto:pasig-discuss-bounces at asis.org] On Behalf Of Stern, Randy
    Sent: Friday, May 12, 2017 1:59 PM
    To: Sheila Morrissey <Sheila.Morrissey at ithaka.org>; pasig-discuss at asis.org
    Subject: Re: [Pasig-discuss] FW: Digital repository storage benchmarking

    Harvard is similar – 2 disk copies in geographically distributed sites on, and one tape copy in a third location. We also have a 4th copy on tape in a tape library that is creating the tapes we remove off site to the third location. We run fixity checks on the disk copies, but not the tape copy. We currently have in excess of 200TB for each copy.

    We currently store preservation and real-time access copies of files in the same storage system with the same storage policies. We expect that to change in the future, with likely delivery copy storage in the cloud.

    Randy

    On 5/12/17, 1:43 PM, "Sheila Morrissey" <Sheila.Morrissey at ithaka.org> wrote:

        Hello, Tim,

        At Portico (http://www.portico.org/digital-preservation/), we preserve e-journals, e-books, digitized historical collections, and other born-digital scholarly content.

        Currently, the Portico archive is comprised of roughly 77.7 million digital objects (we call them "Archival Units", or AUs); comprising over 400 TB; made up of 1.3 billion files.

        We maintain 3 copies of the archive:  2 on disk in geographically distributed data centers, and a 3rd copy in commercial cloud storage.  We create and maintain backups (including fixity checks) using our own custom-written software.

        I hope this helpful.

        Best regards,
        Sheila

        Sheila M. Morrissey
        Senior Researcher
        ITHAKA
        100 Campus Drive
        Suite 100
        Princeton NJ 08540
        609-986-2221
        sheila.morrissey at ithaka.org

        ITHAKA (www.ithaka.org) is a not-for-profit organization that helps the academic community use digital technologies to preserve the scholarly record and to advance research and teaching in sustainable ways.  We provide innovative services that benefit higher education, including Ithaka S+R, JSTOR, and Portico.

        -----Original Message-----
        From: Pasig-discuss [mailto:pasig-discuss-bounces at asis.org] On Behalf Of Tim Walsh
        Sent: Friday, May 12, 2017 10:16 AM
        To: pasig-discuss at asis.org
        Subject: [Pasig-discuss] Digital repository storage benchmarking

        Dear PASIG,

        I am currently in the process of benchmarking digital repository storage setups with our Director of IT, and am having trouble finding very much information about other institutions’ configurations online. It’s very possible that this question has been asked before on-list, but I wasn’t able to find anything in the list archives.

        For context, we are a research museum with significant born-digital archival holdings preparing to manage about 200 TB of digital objects over the next 3 years, replicated several times on various media. The question is what precisely those “various media” will be. Currently, our plan is to store one copy on disk on-site, one copy on disk in a managed off-site facility, and a third copy on LTO sent to a third facility. Before we commit, we’d like to benchmark our plans against other institutions.

        I have been able to find information about the storage configurations for MoMA and the Computer History Museum (who each wrote blog posts or presented on this topic), but not very many others. So my questions are:

        * Could you point me to published/available resources outlining other institutions’ digital repository storage configurations?
        * Or, if you work at an institution, would you be willing to share the details of your configuration on- or off-list? (any information sent off-list will be kept strictly confidential)

        Helpful details would include: amount of digital objects being stored; how many copies of data are being stored; which copies are online, nearline, or offline; which media are being used for which copies; and what services/software applications are you using to manage the creation and maintainance of backups.

        Thank you!
        Tim

        - - -

        Tim Walsh
        Archiviste, Archives numériques
        Archivist, Digital Archives

        Centre Canadien d’Architecture
        Canadian Centre for Architecture
        1920, rue Baile, Montréal, Québec  H3H 2S6 T 514 939 7001 x 1532 F 514 939 7020 www.cca.qc.ca<http://www.cca.qc.ca/>

        Pensez à l’environnement avant d’imprimer ce message Please consider the environment before printing this email Ce courriel peut contenir des renseignements confidentiels. Si vous n’êtes pas le destinataire prévu, veuillez nous en aviser immédiatement. Merci également de supprimer le présent courriel et d’en détruire toute copie.
        This email may contain confidential information. If you are not the intended recipient, please advise us immediately and delete this email as well as any other copy. Thank you.

        ----
        To subscribe, unsubscribe, or modify your subscription, please visit http://mail.asis.org/mailman/listinfo/pasig-discuss
        _______
        PASIG Webinars and conference material is at http://www.preservationandarchivingsig.org/index.html
        _______________________________________________
        Pasig-discuss mailing list
        Pasig-discuss at mail.asis.org
        http://mail.asis.org/mailman/listinfo/pasig-discuss

        ----
        To subscribe, unsubscribe, or modify your subscription, please visit
        http://mail.asis.org/mailman/listinfo/pasig-discuss
        _______
        PASIG Webinars and conference material is at http://www.preservationandarchivingsig.org/index.html
        _______________________________________________
        Pasig-discuss mailing list
        Pasig-discuss at mail.asis.org
        http://mail.asis.org/mailman/listinfo/pasig-discuss

    ----
    To subscribe, unsubscribe, or modify your subscription, please visit http://mail.asis.org/mailman/listinfo/pasig-discuss
    _______
    PASIG Webinars and conference material is at http://www.preservationandarchivingsig.org/index.html
    _______________________________________________
    Pasig-discuss mailing list
    Pasig-discuss at mail.asis.org
    http://mail.asis.org/mailman/listinfo/pasig-discuss

    ----
    To subscribe, unsubscribe, or modify your subscription, please visit
    http://mail.asis.org/mailman/listinfo/pasig-discuss
    _______
    PASIG Webinars and conference material is at http://www.preservationandarchivingsig.org/index.html
    _______________________________________________
    Pasig-discuss mailing list
    Pasig-discuss at mail.asis.org
    http://mail.asis.org/mailman/listinfo/pasig-discuss