[Pasig-discuss] Risks of encryption & compression built into storage options?
Chris Wood
lw85381 at yahoo.com
Fri Mar 17 12:48:40 EDT 2017
Jos:
I just knew somebody would ask this. Ha. Several years ago several of
us wrote a paper for the MPEG (Motion Pictures Expert Group) and a
mathematician named Jeff Bonwick figured out all the math. I haven't
found it yet in the junk heap of my PC, but did find a companion paper
written by by the same set of authors. It's not exactly, what you are
looking for, but close. It's more about Bit Error Rates at a rather low
level. I will continue to look for the MPEG paper. It's got to be
somewhere. The Internet "never forgets" Right?
Stay tuned as I keep looking.
CW
On 3/17/2017 12:48 AM, van Wezel, Jos (SCC) wrote:
> Chris,
> do you happen to have any reference to the mathatical correctness or
> computation that 3 copies is optimal. Is proof based on the standard
> ecc values that vendors list with their components (tapes, disks,
> transport lines, memory etc). I'm asking because its difficult to
> argue for the additional costs of a third copy without the math.
> Currently I can't tell my customers how much (as in percentage) extra
> security an addittional copy will bring, even theoretically.
>
> regards
>
> jos
>
> Sent from my Samsung Galaxy smartphone.
>
> -------- Original message --------
> From: Chris Wood <lw85381 at yahoo.com>
> Date: 17/03/2017 02:07 (GMT+01:00)
> To: "Raymond A. Clarke" <Raymond.Clarke1 at Verizon.net>,
> gail at trumantechnologies.com, 'Jeanne Kramer-Smyth'
> <jkramersmyth at worldbankgroup.org>, 'Robert Spindler'
> <rob.spindler at asu.edu>, pasig-discuss at mail.asis.org
> Subject: Re: [Pasig-discuss] Risks of encryption & compression built
> into storage options?
>
> Thanks Ray as always for a great summary. Now my three bits:
>
> Three (3) copies please. One of which is in a remote location on a
> different flood plane, Electric grid, fault line etc. for the obvious
> reasons. Mathematically, this has turned out to be the optimal number
> looked at with a cost/benefit mindset. Kind of like: 2 is better than
> one, buta local problem gets both copies. Three (remote) is more
> expensive but you get A LOT more data resilience/persistence. Four
> costs a bunch more, but delivers just a little bit more resilience.
> Four+ are all examples of ever diminishing returns.
>
> CW
>
> On 3/16/2017 4:40 PM, Raymond A. Clarke wrote:
>>
>> Hello All,
>>
>> A few years back, I did some research on bit-rot and data corruption,
>> as it relates to the various medium that data passes through, on its
>> way to and from the user. Consider this simple example; as data from
>> memory to HBA to cable to air to cable and so on, bits can be lost
>> along way at any one of, or several of the medium transit points.
>> This something that current technologies can help with, in part.
>> Back to the original question, :how do we insure against corruption,
>> either from compression, encryption” and/or transmission? Well disk
>> and tape(/data resting places/, if you will) have a come very long
>> way in reducing bit-error rates, compression and encryption. But the
>> “/resting places”/ are only part of a problem. In accordance with
>> Gail’s suggestion and as Dr. Rosenthal has coined, LOCKSS (“lot of
>> copies keep stuff safe”).
>>
>> Take good care,
>>
>> Raymond
>>
>> *From:*Pasig-discuss [mailto:pasig-discuss-bounces at asis.org] *On
>> Behalf Of *gail at trumantechnologies.com
>> *Sent:* Thursday, March 16, 2017 5:10 PM
>> *To:* Jeanne Kramer-Smyth <jkramersmyth at worldbankgroup.org>; Robert
>> Spindler <rob.spindler at asu.edu>; pasig-discuss at mail.asis.org
>> *Subject:* Re: [Pasig-discuss] Risks of encryption & compression
>> built into storage options?
>>
>> Hello again, Jeanne,
>>
>> I think you're hitting on something that needs to be raised to (and
>> pushed for with) vendors, and that is the need for "More
>> transparency" and the reporting to customers of "events" that are
>> part of the provenance of a digital object. The storage architectures
>> do a good job of error detection and self healing; however, they do
>> not report this out. I'd like to (this is my dream) have vendors
>> report back to customers (as part of their SLA) when a object (or
>> part of an object if it's been chunked) has been repaired/self-healed
>> - or lost forever. I could then record this as a PREMIS event. As you
>> know, vendors "design for" 11x9s or 13x9s durability, but their SLAs
>> do not require them to tell us if their durability and data
>> corruption starts to get really bad for whatever reason.
>>
>> I've not directly answered your question about whether the
>> encryption, dedupe, compression, and other things that can happen
>> inside a storage system is increasing the risk of corruption. I'll
>> look around. I am sure the disk vendors and storage solution and
>> cloud storage vendors have run the numbers, but am not sure if
>> they're made public.
>>
>> This alias has people from Oracle, Seagate and other storage
>> companies on it so I encourage them to please share any research they
>> have on this -
>>
>> Gail
>>
>> Gail Truman
>>
>> Truman Technologies, LLC
>>
>> Certified Digital Archives Specialist, Society of American Archivists
>>
>> /*Protecting the world's digital heritage for future generations*/
>>
>> www.trumantechnologies.com <http://www.trumantechnologies.com>
>>
>> facebook/TrumanTechnologies
>>
>> https://www.linkedin.com/in/gtruman
>>
>> +1 510 502 6497
>>
>> -------- Original Message --------
>> Subject: RE: [Pasig-discuss] Risks of encryption & compression built
>> into storage options?
>> From: Jeanne Kramer-Smyth <jkramersmyth at worldbankgroup.org
>> <mailto:jkramersmyth at worldbankgroup.org>>
>> Date: Thu, March 16, 2017 1:44 pm
>> To: "gail at trumantechnologies.com
>> <mailto:gail at trumantechnologies.com>"
>> <gail at trumantechnologies.com
>> <mailto:gail at trumantechnologies.com>>, "Robert
>> Spindler" <rob.spindler at asu.edu <mailto:rob.spindler at asu.edu>>,
>> "pasig-discuss at mail.asis.org <mailto:pasig-discuss at mail.asis.org>"
>> <pasig-discuss at mail.asis.org <mailto:pasig-discuss at mail.asis.org>>
>>
>> Thanks Gail & Rob for your replies.
>>
>> I am less worried about the scenario of someone stealing a drive
>> – as Rob pointed out, if that is happening we have bigger problems.
>>
>> I do wonder if there are increased risks of bit-rot/file
>> corruption with encryption, compression, and data deduplication.
>> Have there been any studies on this? Could pulling a file off a
>> drive that requires reversal of the auto-encryption and
>> auto-compression in place at the system level mean a greater risk
>> of bits flipping? I am trying to contrast the increased
>> “handling” and change required to get from the stored version to
>> the original version vs the decreased “handling” it would require
>> if what I am pulling off the storage device is exactly what I
>> sent to be stored.
>>
>> I am less worried about issues related to not being able to
>> decrypt content. The storage solutions we are contemplating would
>> remain under enough ongoing management that these issues should
>> be avoidable. Since ensuring that non-public records remain
>> secure is also very important, encryption gets some points in the
>> “pro” column. I agree that having multiple copies in different
>> storage architectures and with different vendors would also
>> decrease risk.
>>
>> I want to understand the risks related to the different storage
>> architectures and the ever increasing number of “automatic”
>> things being done to digital objects in the process of them being
>> stored and retrieved. Are there people doing work, independent of
>> vendor claims, to document these types of risks?
>>
>> Thank you,
>>
>> Jeanne
>>
>> *Jeanne Kramer-Smyth*
>>
>> *IT Officer, Information Management Services II*
>>
>> http://siteresources.worldbank.org/NEWS/Images/spacer.png
>>
>> *Information and Technology Solutions*
>>
>> *WBG Library & Archives of Development*
>>
>> T
>>
>>
>>
>> 202-473-9803
>>
>> E
>>
>>
>>
>> jkramersmyth at worldbankgroup.org
>> <mailto:jkramersmyth at worldbankgroup.org%20>
>>
>> W
>>
>>
>>
>> www.worldbank.org
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.worldbank.org&d=DQMFAg&c=AGbYxfJbXK67KfXyGqyv2Ejiz41FqQuZFk4A-1IxfAU&r=NJgCuYsVfzWCDaR17iRz_stYXCBl0BBUfunzpCgq3O4&m=6K-rNEvustg-w3KUuAEUFhjRVFmFu0yMAsazbeVm-lg&s=TkShGzs9qr7es714pkkxzLceCXcULADNIGs74_m1QKQ&e=>
>>
>> http://siteresources.worldbank.org/NEWS/Images/twitter_logo.jpg
>>
>>
>>
>> spellboundblog
>>
>> http://siteresources.worldbank.org/NEWS/Images/skype_logo.jpg
>>
>>
>>
>> jkramersmyth
>>
>> http://siteresources.worldbank.org/NEWS/Images/linkedin_logo.jpg
>>
>>
>>
>> jkramersmyth
>>
>> A
>>
>>
>>
>> 1818 H St NW Washington, DC 20433
>>
>> http://siteresources.worldbank.org/NEWS/Images/spacer.png
>>
>> http://siteresources.worldbank.org/NEWS/Images/WBG_Information_and_Technology_Solutions.png
>>
>> *From:*gail at trumantechnologies.com
>> <mailto:gail at trumantechnologies.com>
>> [mailto:gail at trumantechnologies.com]
>> *Sent:* Thursday, March 16, 2017 3:18 PM
>> *To:* Robert Spindler <rob.spindler at asu.edu
>> <mailto:rob.spindler at asu.edu>>; Jeanne Kramer-Smyth
>> <jkramersmyth at worldbankgroup.org
>> <mailto:jkramersmyth at worldbankgroup.org>>;
>> pasig-discuss at mail.asis.org <mailto:pasig-discuss at mail.asis.org>
>> *Subject:* RE: [Pasig-discuss] Risks of encryption & compression
>> built into storage options?
>>
>> Hi all, a good topic!
>>
>> There is new drive technology from Seagate (probably other
>> manufacturers) called "Self Encrypted Drives" (SEDs) which can be
>> used to solve the problem of a person stealing a drive and
>> running off with data.
>>
>> Most cloud services now automatically provide "server side
>> encryption" which means the vendor is doing the encryption for
>> all data at rest (as you point out Jeanne). This is required by
>> HIPAA for all health care data, and is now considered cloud best
>> practice for cloud vendors due to the very real risk of hacking.
>> So, for archival, we need to weigh the data security provided by
>> cloud storage services using server side encryption with the risk
>> of the vendor managing the encryption keys. Which IMO underscores
>> the importance of having multiple copies of all your archival
>> data -- with different vendors and storage architectures or media
>> types if possible.
>>
>> Gail
>>
>> Gail Truman
>>
>> Truman Technologies, LLC
>>
>> Certified Digital Archives Specialist, Society of American Archivists
>>
>> /*Protecting the world's digital heritage for future generations*/
>>
>> www.trumantechnologies.com <http://www.trumantechnologies.com>
>>
>> facebook/TrumanTechnologies
>>
>> https://www.linkedin.com/in/gtruman
>>
>> +1 510 502 6497
>>
>> -------- Original Message --------
>> Subject: Re: [Pasig-discuss] Risks of encryption &
>> compression built
>> into storage options?
>> From: Robert Spindler <rob.spindler at asu.edu
>> <mailto:rob.spindler at asu.edu>>
>> Date: Thu, March 16, 2017 9:06 am
>> To: Jeanne Kramer-Smyth <jkramersmyth at worldbankgroup.org
>> <mailto:jkramersmyth at worldbankgroup.org>>,
>> "pasig-discuss at mail.asis.org
>> <mailto:pasig-discuss at mail.asis.org>"
>> <pasig-discuss at mail.asis.org
>> <mailto:pasig-discuss at mail.asis.org>>
>>
>> At risk of starting a conversation, here are a couple basic
>> issues from an archival standpoint:
>>
>> Encryption: Who has the keys and what happens should a
>> provider go out of business?
>>
>> Compression: Lossy or Lossless and how does that compression
>> act on different file formats (video/audio). If this is
>> frequently accessed material it becomes more of an issue.
>>
>> Short story: At a CNI meeting perhaps 15 years ago in a
>> session about ebooks I asked a panel of vendors if they would
>> give up the keys to encrypted e-books when they reached
>> public domain. Crickets.
>>
>> Physical discs are not secure given the forensics software
>> widely available today, but if someone can grab a physical
>> disc the provider has more problems than forensics.
>>
>> Rob Spindler
>>
>> University Archivist and Head
>>
>> Archives and Special Collections
>>
>> Arizona State University Libraries
>>
>> Tempe AZ 85287-1006
>>
>> 480.965.9277
>>
>> http://www.asu.edu/lib/archives
>>
>> *From:*Pasig-discuss [mailto:pasig-discuss-bounces at asis.org]
>> *On Behalf Of *Jeanne Kramer-Smyth
>> *Sent:* Thursday, March 16, 2017 8:54 AM
>> *To:* pasig-discuss at mail.asis.org
>> <mailto:pasig-discuss at mail.asis.org>
>> *Subject:* [Pasig-discuss] Risks of encryption & compression
>> built into storage options?
>>
>> Is anyone aware of active research into the risks to digital
>> preservation that are posed by built in encryption and
>> compression in both cloud and on-prem storage options? Any
>> and all go-to sources for research and reading on these
>> topics would be very welcome.
>>
>> I am being told by the staff who source storage solutions for
>> my organization that encryption and compression are generally
>> included at the hardware level. That content is automatically
>> encrypted and compressed as it is written to disc – and then
>> un-encrypted and un-compressed as it is pulled off disc in
>> response to a request. It is advertised as both more secure
>> (someone stealing a physical disc could not, in theory,
>> extract its contents) and more cost efficient (taking up less
>> space).
>>
>> I want to be sure that as we make our choices for long-term
>> storage of permanent digital records that we take these risks
>> into accounts.
>>
>> Thank you!
>>
>> Jeanne
>>
>> *Jeanne Kramer-Smyth*
>>
>> *IT Officer, Information Management Services II*
>>
>> http://siteresources.worldbank.org/NEWS/Images/spacer.png
>>
>> *Information and Technology Solutions*
>>
>> *WBG Library & Archives of Development*
>>
>> T
>>
>>
>>
>> 202-473-9803
>>
>> E
>>
>>
>>
>> jkramersmyth at worldbankgroup.org
>> <mailto:jkramersmyth at worldbankgroup.org%20>
>>
>> W
>>
>>
>>
>> www.worldbank.org
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.worldbank.org&d=DQMFAg&c=AGbYxfJbXK67KfXyGqyv2Ejiz41FqQuZFk4A-1IxfAU&r=NJgCuYsVfzWCDaR17iRz_stYXCBl0BBUfunzpCgq3O4&m=6K-rNEvustg-w3KUuAEUFhjRVFmFu0yMAsazbeVm-lg&s=TkShGzs9qr7es714pkkxzLceCXcULADNIGs74_m1QKQ&e=>
>>
>> http://siteresources.worldbank.org/NEWS/Images/twitter_logo.jpg
>>
>>
>>
>> spellboundblog
>>
>> http://siteresources.worldbank.org/NEWS/Images/skype_logo.jpg
>>
>>
>>
>> jkramersmyth
>>
>> http://siteresources.worldbank.org/NEWS/Images/linkedin_logo.jpg
>>
>>
>>
>> jkramersmyth
>>
>> A
>>
>>
>>
>> 1818 H St NW Washington, DC 20433
>>
>> http://siteresources.worldbank.org/NEWS/Images/spacer.png
>>
>> http://siteresources.worldbank.org/NEWS/Images/WBG_Information_and_Technology_Solutions.png
>>
>> ------------------------------------------------------------------------
>>
>> ----
>> To subscribe, unsubscribe, or modify your subscription,
>> please visit
>> http://mail.asis.org/mailman/listinfo/pasig-discuss
>> _______
>> PASIG Webinars and conference material is at
>> http://www.preservationandarchivingsig.org/index.html
>> _______________________________________________
>> Pasig-discuss mailing list
>> Pasig-discuss at mail.asis.org <mailto:Pasig-discuss at mail.asis.org>
>> http://mail.asis.org/mailman/listinfo/pasig-discuss
>>
>>
>>
>> ----
>> To subscribe, unsubscribe, or modify your subscription, please visit
>> http://mail.asis.org/mailman/listinfo/pasig-discuss
>> _______
>> PASIG Webinars and conference material is athttp://www.preservationandarchivingsig.org/index.html
>> _______________________________________________
>> Pasig-discuss mailing list
>> Pasig-discuss at mail.asis.org
>> http://mail.asis.org/mailman/listinfo/pasig-discuss
>
> --
> ----------------------------------------------------
> Chris Wood
> Storage & Data Management
> Office: 408-782-2757 (Home Office)
> Office: 408-276-0730 (Work Office)
> Mobile: 408-218-7313 (Preferred)
> Email:lw85381 at yahoo.com
> ----------------------------------------------------
--
----------------------------------------------------
Chris Wood
Storage & Data Management
Office: 408-782-2757 (Home Office)
Office: 408-276-0730 (Work Office)
Mobile: 408-218-7313 (Preferred)
Email: lw85381 at yahoo.com
----------------------------------------------------
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.asis.org/pipermail/pasig-discuss/attachments/20170317/ac0012ce/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/png
Size: 170 bytes
Desc: not available
URL: <http://mail.asis.org/pipermail/pasig-discuss/attachments/20170317/ac0012ce/attachment-0003.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/png
Size: 170 bytes
Desc: not available
URL: <http://mail.asis.org/pipermail/pasig-discuss/attachments/20170317/ac0012ce/attachment-0004.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/png
Size: 6577 bytes
Desc: not available
URL: <http://mail.asis.org/pipermail/pasig-discuss/attachments/20170317/ac0012ce/attachment-0005.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Archiving_Movies_in_a_Digital_World.pdf
Type: application/pdf
Size: 266120 bytes
Desc: not available
URL: <http://mail.asis.org/pipermail/pasig-discuss/attachments/20170317/ac0012ce/attachment-0001.pdf>
More information about the Pasig-discuss
mailing list