[Pasig-discuss] Experiences with S3-like object store service providers?

[Neil Jefferies]

Kyle,

Can you expand... because that doesn't actually sound like a "fixity" 
issue (which is kind of my point). Either the wrong thing has been 
copied or the wrong thing has been stored i.e. an off-storage process 
failure. For those cases checks should really be done at the time of 
whatever operation caused the issue. Waiting for a check to pick it up 
downstream at some later time is somewhat risky.

My point is, these are not "fixity" checks, they are transmission 
checks.

Neil

On 2017-11-15 22:59, Kyle Rimkus wrote:
> On the issue of fixity checking, I agree there is a great deal of
> misplaced paranoia around "bit-flipping" in contemporary storage
> technologies, but at the same time I've found regular fixity checks to
> be very useful in protecting against various types of human error that
> can be made in managing the storage itself or in scripted processes
> that interact with stored data.
> 
> As an example, at my university we farm our storage out to a campus
> unit which stores two copies locally while we push a third into Amazon
> Glacier. Our preservation repository software's regular fixity checks
> of on-campus data have helped us keep our storage providers and
> ourselves honest. We have on more than one occasion discovered fixity
> errors that pointed to questionable server management. Regular fixity
> checking was what flagged us to these errors, and the storage of a
> third file copy off-site was what saved us.
> 
> We are also looking into pushing more of our storage services into the
> (most likely AWS) cloud, where greater guarantees are made against
> this type of problem. Maybe in time we'll come to see regular fixity
> checks as less critically important than we do now. Julian's comment
> that "certifying a particular storage environment to either store bits
> correctly or alert on failure should be acceptable" is interesting.
> I'm sure we'd all like to get away from having to run constant fixity
> checks in our repositories, and would like to see digital preservation
> management architecture evolve in this direction. For now though I'd
> wager that most of us are constrained by the fact that fixity checking
> remains essential to making sure our storage is doing what it claims
> to do.
> 
> Kyle
> 
> --
> 
> Kyle R. Rimkus
> 
> Assistant Professor
> 
> Preservation Librarian
> 
> University of Illinois at Urbana-Champaign
> ----
> To subscribe, unsubscribe, or modify your subscription, please visit
> http://mail.asis.org/mailman/listinfo/pasig-discuss
> _______
> PASIG Webinars and conference material is at
> http://www.preservationandarchivingsig.org/index.html
> _______________________________________________
> Pasig-discuss mailing list
> Pasig-discuss at mail.asis.org
> http://mail.asis.org/mailman/listinfo/pasig-discuss