[Pasig-discuss] Experiences with S3-like object store service providers?

[Julian M. Morley]

From: Pasig-discuss <pasig-discuss-bounces at asist.org<mailto:pasig-discuss-bounces at asist.org>> on behalf of Neil Jefferies <neil.jefferies at bodleian.ox.ac.uk<mailto:neil.jefferies at bodleian.ox.ac.uk>>
Date: Wednesday, November 15, 2017 at 8:52 AM
To: "pasig-discuss at mail.asis.org<mailto:pasig-discuss at mail.asis.org>" <pasig-discuss at mail.asis.org<mailto:pasig-discuss at mail.asis.org>>
Subject: Re: [Pasig-discuss] Experiences with S3-like object store service providers?

Interestingly, in most cases the mere act of reading data allows devices to detect and correct future errors as the storage medium becomes marginal so there is value in doing that.

Yes, exactly this. It’s fundamental to how erasure coding works. Being able to prove that the data you wrote to storage is unchanged is important, but the process you use to provide the proof doesn’t have to be by comparing md5 file hashes - certifying a particular storage environment to either store bits correctly or alert on failure should be acceptable. I wouldn’t trust a USB thumb drive for long term storage, but I would trust a well-run reed-solomon based disk cluster with periodic scrubbing.

We’ve had some discussions internally about how to fixity check content that we send to the cloud, which has been focusing more on *how* to do it rather than *why* we do it. We’re storing checksums (so many checksums!) external to our main preservation system so that we can prove chain of custody and eventually start signing them, but I’m really not worried about undetectable, uncorrectable bit flipping of content that we send to IaaS providers. I’m more concerned about human-induced data loss, whether it’s accidental (coding errors or manual mistakes) or malicious. Fixity checking definitely has a role to play there, but is only one part of the entire audit process.

--
Julian M. Morley
Technology Infrastructure Manager
Digital Library Systems & Services
Stanford University Libraries

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.asis.org/pipermail/pasig-discuss/attachments/20171115/f920bf98/attachment.html>