[Pasig-discuss] Experiences with S3-like object store service providers?

[Kyle Rimkus]

On the issue of fixity checking, I agree there is a great deal of misplaced
paranoia around "bit-flipping" in contemporary storage technologies, but at
the same time I've found regular fixity checks to be very useful in
protecting against various types of human error that can be made in
managing the storage itself or in scripted processes that interact with
stored data.

As an example, at my university we farm our storage out to a campus unit
which stores two copies locally while we push a third into Amazon Glacier.
Our preservation repository software's regular fixity checks of on-campus
data have helped us keep our storage providers and ourselves honest. We
have on more than one occasion discovered fixity errors that pointed to
questionable server management. Regular fixity checking was what flagged us
to these errors, and the storage of a third file copy off-site was what
saved us.

We are also looking into pushing more of our storage services into the
(most likely AWS) cloud, where greater guarantees are made against this
type of problem. Maybe in time we'll come to see regular fixity checks as
less critically important than we do now. Julian's comment that "certifying
a particular storage environment to either store bits correctly or alert on
failure should be acceptable" is interesting. I'm sure we'd all like to get
away from having to run constant fixity checks in our repositories, and
would like to see digital preservation management architecture evolve in
this direction. For now though I'd wager that most of us are constrained by
the fact that fixity checking remains essential to making sure our storage
is doing what it claims to do.

Kyle

-- 
Kyle R. Rimkus
Assistant Professor
Preservation Librarian
University of Illinois at Urbana-Champaign
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.asis.org/pipermail/pasig-discuss/attachments/20171115/bf469c10/attachment.html>