[Pasig-discuss] Experiences with S3-like object store service providers?

Mon Nov 13 23:27:46 EST 2017

Hi Gail,

Sure - would be happy to chat with you.

I’ve got Scality in my list of contenders - didn’t mention it here because my first few use cases are explicitly ‘not on campus’, but I agree it’s definitely a fit for our main on prem system. As with any commercial software, ongoing licensing costs are a potential pain point for us.

--
Julian M. Morley
Technology Infrastructure Manager
Digital Library Systems & Services
Stanford University Libraries

From: "gail at trumantechnologies.com<mailto:gail at trumantechnologies.com>" <gail at trumantechnologies.com<mailto:gail at trumantechnologies.com>>
Date: Monday, November 13, 2017 at 4:06 PM
To: Julian Morley <jmorley at stanford.edu<mailto:jmorley at stanford.edu>>, "pasig-discuss at mail.asis.org<mailto:pasig-discuss at mail.asis.org>" <pasig-discuss at mail.asis.org<mailto:pasig-discuss at mail.asis.org>>
Cc: "gail at trumantechnologies.com<mailto:gail at trumantechnologies.com>" <gail at trumantechnologies.com<mailto:gail at trumantechnologies.com>>
Subject: RE: [Pasig-discuss] Experiences with S3-like object store service providers?

Hi Julian, thanks for sharing your list and comments. Very thorough list. I'd love to chat (and I'm close by in Oakland).... I've quite a lot of experience in the cloud storage field and would suggest you also take a look at multi-cloud connector technologies that will allow you to standardize on S3, but write to non-S3-based public cloud vendors. And to tier or move data among private and public clouds and do federated search on metadata across a single namespace (across these clouds).

Check out a couple of interesting technologies:

Open Source Zenko.io<http://Zenko.io> - offering S3 connect to AWS, Azure and Google (the latter 2 are coming shortly), and also
Scality Connect for Azure Blog Storage - translates S3 API calls to Azure blob storage API calls.

See the attached datasheet and also  https://www.zenko.io/

I'd add Scality to your list -- see the Gartner magic quadrant they're shown in the Upper Right Visionary quadrant and are close to you in San Francisco. They talk S3, File, NFS/SMB, REST (CDMI etc), can tier off to public clouds, and have lots of multi-PB size customer installs.  Gartner MQ is here: https://www.gartner.com/doc/reprints?id=1-4IE870C&ct=171017&st=sb

I'd be very interested in learning more about your use cases -- can we connect outside of this PASIG alias?

Gail

Gail Truman
Truman Technologies, LLC
Certified Digital Archives Specialist, Society of American Archivists

Protecting the world's digital heritage for future generations
www.trumantechnologies.com<http://www.trumantechnologies.com>
facebook/TrumanTechnologies
https://www.linkedin.com/in/gtruman

+1 510 502 6497

-------- Original Message --------
Subject: [Pasig-discuss] Experiences with S3-like object store service
providers?
From: "Julian M. Morley" <jmorley at stanford.edu<mailto:jmorley at stanford.edu>>
Date: Mon, November 13, 2017 12:43 pm
To: "pasig-discuss at mail.asis.org<mailto:pasig-discuss at mail.asis.org>" <pasig-discuss at mail.asis.org<mailto:pasig-discuss at mail.asis.org>>

Hi everyone,

I’ve currently got at least four use cases for an S3-compatible object store, spanning everything from traditional S3 through infrequent access stores to cold vaults. As a result I’ve spent considerable time researching options and prices, and was wondering if anyone on this list has any similar experiences they’d like to share.

Our use cases range from hundreds of TB through to several PB, with different access patterns and comfort levels around redundancy and access. For most of them a 100% compatible S3 API is a requirement, but we can bend that a bit for the cold storage use case. We’re also considering local/on-prem object stores for one of the use cases - either rolling our own Ceph install, or using Dell/EMC ECS or SpectraLogic ArcticBlue/Blackpearl.

The vendors that I’m looking at are:

Amazon Web Services (S3, Infrequent Access S3 and S3-to-Glacier).
This is the baseline. We have a direct connect pipe to AWS which reduces the pain of data egress considerably.

IBM Cloud Bluemix (formerly CleverSafe)
A good choice for multi-region redundancy, as they use erasure coding across regions - no ‘catch up’ replication - providing CRR at a cheaper price than AWS. If you only want to keep one copy of your data in the cloud, but have it be able to survive the loss of a region, this is the best choice (Google can also do this, but not with an S3 API or an infrequent access store).

Dell/EMC Virtustream (no cold storage option)
Uses EMC ECS hardware. Actually more expensive than AWS at retail pricing for standard object storage; their value add is tying Virtustream into on-prem ECS units.

Iron Mountain Iron Cloud (Infrequent Access only)
Also uses EMC ECS hardware. Designed primarily for backup/archive workloads (no big surprise there), but with no retrieval, egress or PUT/GET/POST charges.

Oracle Cloud (cheapest cold storage option, but not S3 API)
Uses Openstack Swift. Has the cheapest cloud-tape product (Oracle Cloud Storage Archive), but has recently increased prices to be closer to AWS Glacier.

Google Cloud Platform (not an S3 API)
Technically brilliant, but you have to be able to use their APIs. Their cold storage product is online (disk, not tape), but not as cheap as Glacier.

Microsoft Azure (not an S3 API)
Competitively priced, especially their Infrequent Access product, but again not an S3 API and their vault product is still in beta.

Backblaze B2 (not an S3 API)
Another backup/archive target, only slightly more expensive than Glacier, but online (no retrieval time or fees) and with significantly cheaper data egress rates than AWS.

Wasabi Cloud
Recently launched company from the team that brought you Carbonite. Ridiculously cheap S3 storage, but with a 90-day per-object minimum charge. It’s cheaper and faster than Glacier, both to store data and egress it, but there’s obvious concerns around company longevity. Would probably make a good second target if you have a multi-vendor requirement for your data.

If anyone is interested in hearing more, or has any experience with any of these vendors, please speak up!

--
Julian M. Morley
Technology Infrastructure Manager
Digital Library Systems & Services
Stanford University Libraries
________________________________
----
To subscribe, unsubscribe, or modify your subscription, please visit
http://mail.asis.org/mailman/listinfo/pasig-discuss
_______
PASIG Webinars and conference material is at http://www.preservationandarchivingsig.org/index.html
_______________________________________________
Pasig-discuss mailing list
Pasig-discuss at mail.asis.org<mailto:Pasig-discuss at mail.asis.org>
http://mail.asis.org/mailman/listinfo/pasig-discuss
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.asis.org/pipermail/pasig-discuss/attachments/20171114/25a9e906/attachment-0001.html>