[Pasig-discuss] Experiences with S3-like object store service providers?

Tue Nov 14 13:17:58 EST 2017

All,

Oracle does offer Archive Storage (Oracle Archive Cloud) to non-Oracle DB customers.  Oracle also offers Object Storage in the Cloud for faster access to content.  Both of these can be in support of Oracle based applications or non-Oracle based applications (which would be straight cloud storage) either using an API – or not.

>From a diversity or availably perspective, Oracle has built their platform to support SLAs specific to the service.  The Archive Cloud delivers 11 9s data durability by maintaining multiple copies of each object on different devices.  Object Storage carries the same SLA.

Hope this helps

Frank

Frank Boensch

Eastern US Sales Lead – Oracle Diva

HYPERLINK "mailto:frank.boensch at oracle.com"frank.boensch at oracle.com

646-303-5187

From: Julian M. Morley [mailto:jmorley at stanford.edu] 
Sent: Tuesday, November 14, 2017 12:49 PM
To: Lewis, Stuart <stuart.lewis at nls.uk>; gail at trumantechnologies.com; pasig-discuss at mail.asis.org
Subject: Re: [Pasig-discuss] Experiences with S3-like object store service providers?

Hi Stuart,

Thanks for sharing! 

I talked to the Oracle folks quite a bit last year - Oracle Cloud Storage Archive is offline tape. They don’t say/won’t publicly admit, but it’s two copies of data on two tapes, with ‘periodic’ CRC checks and recovery. However, a little birdie told me that they’re still trying to figure out how to do Cloud for non-Oracle Database customers, so I’m not convinced about the long-term sustainability of the service.

We’re moving to four copies - one local (online) , three offsite, either at our secondary site or in the cloud. Right now the plan is for at least two of the offsite copies to be ‘offline’ - Glacier and Oracle Cloud Storage Archive, with the third copy being either another cold/vault store or on an S3-IA object store. 

When you say you’re targeting a geo-replicated object store for your two online copies, do you mean that you’re counting the geo-replication as a copy mechanism? I’d be concerned about an error on one site propagating to the other replica - in our plan, whilst we intend to have a robust storage system that does EC and/or replication, possibly even to our second site, we’re only counting that as one logical copy. 

>> We’re also purposefully not going to use the object storage system’s in-built cloud connectors for replication.  We feel it might be safer for us to manage the replication to the cloud in our repository, rather than having a single vendor system manage all three copies at once.

I came to the same conclusion. Strength in diversity!

-- 

Julian M. Morley

Technology Infrastructure Manager

Digital Library Systems & Services

Stanford University Libraries

From: "Lewis, Stuart" <HYPERLINK "mailto:stuart.lewis at nls.uk"stuart.lewis at nls.uk>
Date: Tuesday, November 14, 2017 at 1:25 AM
To: Julian Morley <HYPERLINK "mailto:jmorley at stanford.edu"jmorley at stanford.edu>, "HYPERLINK "mailto:gail at trumantechnologies.com"gail at trumantechnologies.com" <HYPERLINK "mailto:gail at trumantechnologies.com"gail at trumantechnologies.com>, "HYPERLINK "mailto:pasig-discuss at mail.asis.org"pasig-discuss at mail.asis.org" <HYPERLINK "mailto:pasig-discuss at mail.asis.org"pasig-discuss at mail.asis.org>
Subject: RE: [Pasig-discuss] Experiences with S3-like object store service providers?

Hi Julian, Gail, all,

At the National Library of Scotland we are also in the middle of some procurements to revamp our storage infrastructure for holding our digitised content archive.

The approach historically taken here has been to use general purpose SANs, with a second copy placed on offline tape.  The SANs have never been built to scale (so they fill and we buy another), and they are general purpose, trying their best (but often failing!) to run a mixed workload of everything from VMs to data archive and everything in between.

We’re now wanting to move to three copies, two online and one offline (in the cloud if possible).

For the online copies we’re about to get to tender to buy a geo-replicated object storage system, to be hosted in our data centres in Edinburgh and Glasgow.  I suspect the likely candidates will be systems such as Dell EMC ECS, HPE+Scality, IBM ESS**, and Hitachi HPC.

(** ESS rather than CleverSafe, as I think that is predicated on three datacentres, but we only want two).

We’re also about to try a large-scale proof of concept with the Oracle Archive Cloud, but have an open question regarding its characteristics compared to local offline tape.  Due to lack of transparency about what is actually going on behind the scenes in a cloud environment, we don’t know whether this gives us the same offline protection that tape gives us (e.g. much harder to corrupt or accidentally delete).

We’re also purposefully not going to use the object storage system’s in-built cloud connectors for replication.  We feel it might be safer for us to manage the replication to the cloud in our repository, rather than having a single vendor system manage all three copies at once.

Critique of this plan is most welcome!

Also happy to join in any offline discussion about this.

Best wishes,

Stuart Lewis
Head of Digital

National Library of Scotland
George IV Bridge, Edinburgh EH1 1EW

Tel: +44 (0) 131 623 3704
Email: HYPERLINK "mailto:stuart.lewis at nls.uk"stuart.lewis at nls.uk
Website:HYPERLINK "http://www.nls.uk/"www.nls.uk 
Twitter: HYPERLINK "http://twitter.com/stuartlewis"@stuartlewis

From: Pasig-discuss [mailto:pasig-discuss-bounces at asist.org] On Behalf Of Julian M. Morley
Sent: 14 November 2017 04:28
To: HYPERLINK "mailto:gail at trumantechnologies.com"gail at trumantechnologies.com; HYPERLINK "mailto:pasig-discuss at mail.asis.org"pasig-discuss at mail.asis.org
Subject: Re: [Pasig-discuss] Experiences with S3-like object store service providers?

Hi Gail,

Sure - would be happy to chat with you.

I’ve got Scality in my list of contenders - didn’t mention it here because my first few use cases are explicitly ‘not on campus’, but I agree it’s definitely a fit for our main on prem system. As with any commercial software, ongoing licensing costs are a potential pain point for us.

-- 

Julian M. Morley

Technology Infrastructure Manager

Digital Library Systems & Services

Stanford University Libraries

From: "HYPERLINK "mailto:gail at trumantechnologies.com"gail at trumantechnologies.com" <HYPERLINK "mailto:gail at trumantechnologies.com"gail at trumantechnologies.com>
Date: Monday, November 13, 2017 at 4:06 PM
To: Julian Morley <HYPERLINK "mailto:jmorley at stanford.edu"jmorley at stanford.edu>, "HYPERLINK "mailto:pasig-discuss at mail.asis.org"pasig-discuss at mail.asis.org" <HYPERLINK "mailto:pasig-discuss at mail.asis.org"pasig-discuss at mail.asis.org>
Cc: "HYPERLINK "mailto:gail at trumantechnologies.com"gail at trumantechnologies.com" <HYPERLINK "mailto:gail at trumantechnologies.com"gail at trumantechnologies.com>
Subject: RE: [Pasig-discuss] Experiences with S3-like object store service providers?

Hi Julian, thanks for sharing your list and comments. Very thorough list. I'd love to chat (and I'm close by in Oakland).... I've quite a lot of experience in the cloud storage field and would suggest you also take a look at multi-cloud connector technologies that will allow you to standardize on S3, but write to non-S3-based public cloud vendors. And to tier or move data among private and public clouds and do federated search on metadata across a single namespace (across these clouds).

Check out a couple of interesting technologies:

Open Source HYPERLINK "http://Zenko.io"Zenko.io - offering S3 connect to AWS, Azure and Google (the latter 2 are coming shortly), and also

Scality Connect for Azure Blog Storage - translates S3 API calls to Azure blob storage API calls.

See the attached datasheet and also  https://www.zenko.io/

I'd add Scality to your list -- see the Gartner magic quadrant they're shown in the Upper Right Visionary quadrant and are close to you in San Francisco. They talk S3, File, NFS/SMB, REST (CDMI etc), can tier off to public clouds, and have lots of multi-PB size customer installs.  Gartner MQ is here: https://www.gartner.com/doc/reprints?id=1-4IE870C&ct=171017&st=sb

I'd be very interested in learning more about your use cases -- can we connect outside of this PASIG alias?

Gail

Gail Truman

Truman Technologies, LLC

Certified Digital Archives Specialist, Society of American Archivists

Protecting the world's digital heritage for future generations

HYPERLINK "http://www.trumantechnologies.com"www.trumantechnologies.com

facebook/TrumanTechnologies

https://www.linkedin.com/in/gtruman 

+1 510 502 6497

-------- Original Message --------
Subject: [Pasig-discuss] Experiences with S3-like object store service
providers?
From: "Julian M. Morley" <HYPERLINK "mailto:jmorley at stanford.edu"jmorley at stanford.edu>
Date: Mon, November 13, 2017 12:43 pm
To: "HYPERLINK "mailto:pasig-discuss at mail.asis.org"pasig-discuss at mail.asis.org" <HYPERLINK "mailto:pasig-discuss at mail.asis.org"pasig-discuss at mail.asis.org>

Hi everyone,

I’ve currently got at least four use cases for an S3-compatible object store, spanning everything from traditional S3 through infrequent access stores to cold vaults. As a result I’ve spent considerable time researching options and prices, and was wondering if anyone on this list has any similar experiences they’d like to share.

Our use cases range from hundreds of TB through to several PB, with different access patterns and comfort levels around redundancy and access. For most of them a 100% compatible S3 API is a requirement, but we can bend that a bit for the cold storage use case. We’re also considering local/on-prem object stores for one of the use cases - either rolling our own Ceph install, or using Dell/EMC ECS or SpectraLogic ArcticBlue/Blackpearl.

The vendors that I’m looking at are:

Amazon Web Services (S3, Infrequent Access S3 and S3-to-Glacier). 

This is the baseline. We have a direct connect pipe to AWS which reduces the pain of data egress considerably.

IBM Cloud Bluemix (formerly CleverSafe)

A good choice for multi-region redundancy, as they use erasure coding across regions - no ‘catch up’ replication - providing CRR at a cheaper price than AWS. If you only want to keep one copy of your data in the cloud, but have it be able to survive the loss of a region, this is the best choice (Google can also do this, but not with an S3 API or an infrequent access store).

Dell/EMC Virtustream (no cold storage option)

Uses EMC ECS hardware. Actually more expensive than AWS at retail pricing for standard object storage; their value add is tying Virtustream into on-prem ECS units.

Iron Mountain Iron Cloud (Infrequent Access only)

Also uses EMC ECS hardware. Designed primarily for backup/archive workloads (no big surprise there), but with no retrieval, egress or PUT/GET/POST charges.

Oracle Cloud (cheapest cold storage option, but not S3 API)

Uses Openstack Swift. Has the cheapest cloud-tape product (Oracle Cloud Storage Archive), but has recently increased prices to be closer to AWS Glacier.

Google Cloud Platform (not an S3 API)

Technically brilliant, but you have to be able to use their APIs. Their cold storage product is online (disk, not tape), but not as cheap as Glacier.

Microsoft Azure (not an S3 API)

Competitively priced, especially their Infrequent Access product, but again not an S3 API and their vault product is still in beta.

Backblaze B2 (not an S3 API)

Another backup/archive target, only slightly more expensive than Glacier, but online (no retrieval time or fees) and with significantly cheaper data egress rates than AWS.

Wasabi Cloud

Recently launched company from the team that brought you Carbonite. Ridiculously cheap S3 storage, but with a 90-day per-object minimum charge. It’s cheaper and faster than Glacier, both to store data and egress it, but there’s obvious concerns around company longevity. Would probably make a good second target if you have a multi-vendor requirement for your data.

If anyone is interested in hearing more, or has any experience with any of these vendors, please speak up!

-- 

Julian M. Morley

Technology Infrastructure Manager

Digital Library Systems & Services

Stanford University Libraries

  _____  

----
To subscribe, unsubscribe, or modify your subscription, please visit
http://mail.asis.org/mailman/listinfo/pasig-discuss
_______
PASIG Webinars and conference material is at http://www.preservationandarchivingsig.org/index.html
_______________________________________________
Pasig-discuss mailing list
HYPERLINK "mailto:Pasig-discuss at mail.asis.org"Pasig-discuss at mail.asis.org
http://mail.asis.org/mailman/listinfo/pasig-discuss

National Library of Scotland, Scottish Charity, No: SCO11086

This communication is intended for the addressee(s) only. If you are not the addressee please inform the sender and delete the email from your system. The statements and opinions expressed in this message are those of the author and do not necessarily reflect those of National Library of Scotland. This message is subject to the Data Protection Act 1998 and Freedom of Information (Scotland) Act 2002. No liability is accepted for any harm that may be caused to your systems or data by this message.

 Before you print please think about the ENVIRONMENT

HYPERLINK "http://www.nls.uk/"

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.asis.org/pipermail/pasig-discuss/attachments/20171114/d23a51dd/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.jpg
Type: image/jpeg
Size: 18754 bytes
Desc: not available
URL: <http://mail.asis.org/pipermail/pasig-discuss/attachments/20171114/d23a51dd/attachment-0001.jpg>