[Pasig-discuss] Experiences with S3-like object store service providers?

[Kyle Rimkus]

I see your point. We had an instance of a number of files failing their
regular fixity check, after having passed several times over the years.
When we investigated, the files were the same size on disk as they always
were, but were just strings of zeroes, and unreadable. Whatever caused this
could have been due to a transmission error behind the scenes during some
sort of server maintenance, but from where we stood as a repository that
had outsourced our storage we had no way of knowing one way or the other,
and the fixity checks saved us.

On Wed, Nov 15, 2017 at 6:15 PM Neil Jefferies <neil at jefferies.org> wrote:

> Kyle,
>
> Can you expand... because that doesn't actually sound like a "fixity"
> issue (which is kind of my point). Either the wrong thing has been
> copied or the wrong thing has been stored i.e. an off-storage process
> failure. For those cases checks should really be done at the time of
> whatever operation caused the issue. Waiting for a check to pick it up
> downstream at some later time is somewhat risky.
>
> My point is, these are not "fixity" checks, they are transmission
> checks.
>
> Neil
>
> On 2017-11-15 22:59, Kyle Rimkus wrote:
> > On the issue of fixity checking, I agree there is a great deal of
> > misplaced paranoia around "bit-flipping" in contemporary storage
> > technologies, but at the same time I've found regular fixity checks to
> > be very useful in protecting against various types of human error that
> > can be made in managing the storage itself or in scripted processes
> > that interact with stored data.
> >
> > As an example, at my university we farm our storage out to a campus
> > unit which stores two copies locally while we push a third into Amazon
> > Glacier. Our preservation repository software's regular fixity checks
> > of on-campus data have helped us keep our storage providers and
> > ourselves honest. We have on more than one occasion discovered fixity
> > errors that pointed to questionable server management. Regular fixity
> > checking was what flagged us to these errors, and the storage of a
> > third file copy off-site was what saved us.
> >
> > We are also looking into pushing more of our storage services into the
> > (most likely AWS) cloud, where greater guarantees are made against
> > this type of problem. Maybe in time we'll come to see regular fixity
> > checks as less critically important than we do now. Julian's comment
> > that "certifying a particular storage environment to either store bits
> > correctly or alert on failure should be acceptable" is interesting.
> > I'm sure we'd all like to get away from having to run constant fixity
> > checks in our repositories, and would like to see digital preservation
> > management architecture evolve in this direction. For now though I'd
> > wager that most of us are constrained by the fact that fixity checking
> > remains essential to making sure our storage is doing what it claims
> > to do.
> >
> > Kyle
> >
> > --
> >
> > Kyle R. Rimkus
> >
> > Assistant Professor
> >
> > Preservation Librarian
> >
> > University of Illinois at Urbana-Champaign
> > ----
> > To subscribe, unsubscribe, or modify your subscription, please visit
> > http://mail.asis.org/mailman/listinfo/pasig-discuss
> > _______
> > PASIG Webinars and conference material is at
> > http://www.preservationandarchivingsig.org/index.html
> > _______________________________________________
> > Pasig-discuss mailing list
> > Pasig-discuss at mail.asis.org
> > http://mail.asis.org/mailman/listinfo/pasig-discuss
>
> --
Kyle R. Rimkus
Assistant Professor
Preservation Librarian
University of Illinois at Urbana-Champaign
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.asis.org/pipermail/pasig-discuss/attachments/20171116/9c9856d6/attachment-0001.html>