[Pasig-discuss] Experiences with S3-like object store service providers?

Thu Nov 16 14:19:57 EST 2017

Hi All
I'll clarify AWS position on Etags and fixity checking, but I'm hesitant to
hijack this thread into a "how S3 and Glacier works" thread. So feel free
to follow up with me at mikdav at amazon.com to connect for any sort of deeper
dialogue. I'll with with Art and Gail on possibly setting up a webinar for
the group.

To answer Julian, our S3 Etags are fixed at time of upload. They are never
modified, and never reflect any background tasks associated with fixity
checking, audit runs, media migration, or healing processes that occur over
time. I do recognize the desire of the preservation community (who has a
long history of bit-rot-related data-loss) to achieve more transparency on
those housekeeping functions. But rest assured, AWS is fully committed to
supporting "100-year archive" type workloads. S3 and Glacier's 11 9's of
durability says a single object will be protected across 3 facilities,
against concurrent media/host/power/network failures, across flood plains,
discrete power grids, and discrete networking providers. We ask you to
consider the histogram of things that cause data-loss: For those losses
related to *infrastructure*, we reduce the odds of data-loss to <near>
zero, leaving access control, application bugs, malicious insiders, catalog
loss, and workflow bugs as imposing far higher risks to data-integrity.

So storage fixity checks are no substitute for *end-to-end* data validation
checks, audit mechanisms, and DLP (see AWS Macie). Etag is not the best
solution for fixity checks given its' non-deterministic nature (it's not
necessarily a MD5) and the fact that it doesn't reflect internal integrity
housekeeping. To store a hash with the object, one approach would be to
append it as a S3 key-value tag (along with other shadow metadata that
could use 11 9's of protection). Although we wouldn't see the ROI of
running your own integrity crawl every N months, you could certainly do
it...Spot compute, Glacier Bulk restore, and $0 in-region transfers
*should* make this a relatively cheap operation.

-Mike

On Tue, Nov 14, 2017 at 7:34 PM, Julian M. Morley <jmorley at stanford.edu>
wrote:

> Hi David,
>
> We haven’t done it yet, but we plan to do something similar to what you
> describe below when we start storing SDR content in cloud providers.
>
> One open question (that I intend to ask Mike Davis about!) is how S3
> stores and/or updates ETag fields for uploaded objects. I think that when
> content is recovered from Glacier to S3-IA, the ETag/MD5 of the file is
> computed when the file is written to S3-IA. This means that a ‘good enough’
> fixity check can be done simply by recalling the data to S3-IA (relatively
> cheap! No data egress charges!) and performing a simple metadata check of
> the recovered object. Costs are easily projected/constrained simply by
> deciding what % of your total data corpus you want ‘in flight’ at any one
> time.
>
> This requires us to store checksums for all objects that we send to the
> cloud in a separate datastore - we’ll be using something called the Trusted
> Checksum Repository for this - a WORM-style database that stands to the
> side of the SDR.
>
> I’m also assuming (again, this is a question for Mike) that S3 and other
> cloud providers do perform periodic scrubs of their data, and use EC to
> correct for any bad blocks that they find. For example, Wasabi explicitly
> states that they validate md5 checksums of content every 90 days.
> Presumably when they do that they’ll update the ETag if it has changed,
> which again allows a metadata check to validate fixity.
>
> The same process for Glacier recovery to S3-IA works for Oracle Cloud
> Storage Archive to Oracle Cloud Storage - the recovered object has a
> freshly-generated ETag, which can then be compared against the stored
> checksum when the file was first uploaded. No cloud compute instance
> needed. Oracle has also told me (although this is not official) that they
> write content to two tapes, and perform occasional CRC checks / migrations
> of content to ensure that data on the tapes hasn’t gone bad, although
> that’s not on a fixed schedule.
>
> For GCP and other online/non-vault options, running a compute instance is
> probably still the best way to go. And if we want to unpack the object and
> checksum all the elements it’s pretty much the only game in town - we plan
> to do that for a random sampling of our content, adjusting our throughput
> by varying recovery request rate and EBS disk sizes until we settle on an
> acceptable rate.
>
> ( There *is* a wrinkle here with ETags and multi-part uploads that I’m not
> getting into. It’s still possible to get and store a useful MD5, you just
> need to do a little bit of extra legwork to get there.  )
>
> --
> Julian M. Morley
> Technology Infrastructure Manager
> Digital Library Systems & Services
> Stanford University Libraries
>
> From: Pasig-discuss <pasig-discuss-bounces at asist.org> on behalf of David
> Pcolar <dave at dpn.org>
> Date: Tuesday, November 14, 2017 at 1:33 PM
> To: "pasig-discuss at mail.asis.org" <pasig-discuss at mail.asis.org>
>
> Subject: Re: [Pasig-discuss] Experiences with S3-like object store
> service providers?
>
> Hi All,
>
> This is a great thread and it has surfaced a couple of new options for
> cloud storage.  Thanks to everyone who has contributed.
>
> I would like to address a common issue with the cloud storage vendors and
> IO/retrieval costs.  Since the likelihood of recovering a specific object
> is low, cloud storage is quite economical from a simple 'recover the object
> standpoint'. However, preservation repositories are touching those objects
> frequently to perform fixity checks. I am not aware of any cloud platform
> that will do fixity audits on demand, as detailed in NSDA Preservation
> level 3 (Check fixity of content at fixed intervals;Maintain logs of fixity
> info; supply audit on demand).
>
> A common method for providing these checks for content in S3 is to
> instantiate an EC2 instance, mount the S3 bucket, and run checksums on the
> objects. For Glacier, an additional step of staging the objects in an
> accessible area for the EC2 instance is required. This results in I/O and
> compute cycle fees that could dramatically inflate the cost of public cloud
> storage over time.
>
> For those utilizing public cloud storage for preservation, how are you
> addressing fixity checks and event audit capture?
>
> - Dave
>
>
> David Pcolar
> CTO, Digital Preservation Network
> dave at dpn.org
>
> On Nov 14, 2017, at 3:22 PM, Mike Davis <akropilot at gmail.com> wrote:
>
> Hi Gail
>
> I appreciate the fact that public cloud pricing can be complex; it's a
> function of the cost-following strategy. If the vendor incurs a cost,
> whether from media, IO, or networking, it's passed along as discrete
> charges to the customer. The alternative is opaquely bundling all the
> costs, which reduces transparency and flexibility to follow commodity
> curves downward. I believe it's publicly available data that S3 has dropped
> capacity pricing for example at an average 10% (ish) per year since launch.
>
> But the idea that transaction and I/O fees dramatically inflate the cost
> of public cloud storage is a myth, particularly for digital asset
> management and archival. It is certainly possible to design a wonky
> IO-heavy workload, place it on the wrong storage tier, and end up with
> unexpectedly high costs. But for archival-oriented workloads, the costs of
> *moving* data should never be greater than 10% of total or the situation
> needs to be examined more closely. For example, we might find that large
> objects are being inadvertently chunked into millions of 16KB objects by
> some third party gateway solution, that would inflate the transaction count.
>
> Happy to give you (and PASIG) a deeper dive on IAAS storage strategies,
> and to solve for long-term durability.
>
> -Mike Davis (AWS Storage)
>
>
> On Tue, Nov 14, 2017 at 9:05 AM, <gail at trumantechnologies.com> wrote:
>
>> Thanks for chiming in Jacob!  As always, great additional information. I
>> think it's worth emphasizing that having an open and native data format
>> independent of where the data lives - this is really what will enable
>> multi-cloud workflow management. And also having federated data search
>> of system and descriptive metadata across the namespace no matter where the
>> data is stored (including across public- and on-prem cloud storage). These
>> are what the newer cloud controller software, like Zenko, Starfish, Minio
>> and (and other sw within some cloud services) can enable.
>>
>> Public cloud storage prices are racing to the bottom, but (as David
>> Rosenthal and others have pointed out) often the "hidden" costs of pulling
>> the data back will usually result in costs greater than a private cloud.
>>
>> Stuart -
>> I just read a couple of Forrester papers on Total Economic Impact (TEI)
>> of public clouds -- the ones I have URLs to are posted below and make a
>> useful read:
>> https://www.emc.com/collateral/analyst-reports/dell-emc-ecs-
>> forrester-tei.pdf
>> https://whitepapers.theregister.co.uk/paper/view/5835/the-
>> total-economic-impact-of-scality-ring-with-dell-storage-servers
>> They talk about Dell hardware for building our on-prem clouds (ECS from
>> EMC and RING from Scality) and I believe you're working with HPE, but the
>> maths should be similar to show savings over public cloud. That said,
>> putting one or more copies in public cloud and managing them from one
>> namespace would be ideal... I envision use cases where multi-cloud
>> controller software will allow you to move data to the cloud service that
>> fits the data. [Even if it's for long-term archival, there are times when
>> preservation data services will need to be run (format migration, integrity
>> checks, creating access or derivatives of moving or still images, etc).]
>> Spin up some quick compute services or Hadoop (for other use case).
>>
>> This is a great topic - Julian and Stuart, all the best on your projects,
>> please do let this alias know what you decide to go with!
>>
>>
>> Gail
>>
>>
>> Gail Truman
>> Truman Technologies, LLC
>> Certified Digital Archives Specialist, Society of American Archivists
>>
>> Protecting the world's digital heritage for future generations
>> www.trumantechnologies.com
>> facebook/TrumanTechnologies
>> https://www.linkedin.com/in/gtruman
>>
>> +1 510 502 6497 <(510)%20502-6497>
>>
>>
>>
>>
>> -------- Original Message --------
>> Subject: RE: [Pasig-discuss] Experiences with S3-like object store
>> service providers?
>> From: Jacob Farmer <jfarmer at cambridgecomputer.com>
>> Date: Tue, November 14, 2017 7:03 am
>> To: "Lewis, Stuart" <stuart.lewis at nls.uk>, "Julian M. Morley"
>> <jmorley at stanford.edu>, gail at trumantechnologies.com,
>> pasig-discuss at mail.asis.org
>>
>> Hi, Stuart. I thought I would weigh in on your plans.  I’m a data storage
>> consultant with about 30 years in the game. I’m also the founder of
>> Starfish Storage which makes software products for virtualizing file
>> systems and object stores.
>>
>> ·       I think you are correct to manage multiple copies in the
>> application layer.  This gives you maximum control, ability to shift
>> vendors, etc.  Storage should always be thought of a stack that starts with
>> the application and ends in the storage media.  There can be multiple
>> layers and the system architect should pick the right layer of abstraction
>> for any given set of functionality.  The higher in the stack, the greater
>> your application awareness.
>>
>> ·       By handling replication and addressing in your application, you
>> should be able to switch object stores over time without much difficulty.
>> As such, it does not matter so much which object store you buy.  You could
>> simply chase price.
>>
>> ·       Oracle – They are mysterious about what they are doing under the
>> hood, but it does not matter.  It’s a “cloud”.  They are so inexpensive.
>> Use them as your second or third copy.  I know that Oracle people monitor
>> the news group.  Maybe one will offer to connect you to a product manager
>> who can describe the infrastructure. If not, I would be happy to connect
>> you to folks on the US side of the pond.
>>
>> ·       There is a particular vendor you did not list who might be
>> interesting for you.  They are called Caringo.  They have been in the
>> object storage business before S3 came to market, and thus they offer
>> alternative addressing to the S3 bucket paradigm.  They can emulate S3 just
>> like everyone else, but S3 was designed by Amazon for the purpose of
>> selling a storage service.  It is not necessarily, the logical way to store
>> objects for a digital library.  If you are going to address objects
>> directly from your application, they might have some unique value.  I am
>> happy to connect you to executives in the company.
>>
>> ·       The other vendor worth looking at is Minio.IO <http://minio.io/>.
>> I just pointed Julian to them the other day. They provide an object
>> interface in storage and could federate different cloud stores together.
>> You might consider them for one of your copies. I still like the idea of
>> doing your replication in the application.  They are similar in concept to
>> Zenko who Gail recommended earlier.
>>
>> ·       POSIX File System Gateway – My software company (Starfish
>> Storage) has a file system gateway under development (ready for early
>> adopters) that is ideal if you want a POSIX personality.  We can take the
>> contents of the object store and present it as a POSIX file system.
>>
>> o   We map files 1:1 to objects.  Most file system gateways on the
>> market break up files into smaller objects, akin to blocks.
>> o   We support Active Directory perfectly, SMB-2, SMB-3, NFS-3, NFS-4
>> o   We also work in-band or side-band to the object store.  That means
>> that you can use our POSIX interface simultaneously with S3.
>>
>> ·       You probably also have use cases for Starfish, maybe as a
>> migration tool from file to object or as an end-to-end fixity solution.  We
>> would be especially useful if you need to migrate files from your tape file
>> system.
>> o   Starfish is a metadata and rules engine for file systems and object
>> stores.  Too many concepts to put in an email!
>>
>> I hope that helps.  Message me offline if you want to discuss.  I’m at
>> the SuperComputer conference this week, so replies will be a bit slow.
>>
>>
>> Jacob Farmer  |  Chief Technology Officer  |  Cambridge Computer  |
>> "Artists In Data Storage"
>> Phone 781-250-3210 <(781)%20250-3210>  |  jfarmer at CambridgeComputer.com
>> |  www.CambridgeComputer.com <http://www.cambridgecomputer.com/>
>>
>>
>> From: Pasig-discuss [mailto:pasig-discuss-bounces at asist.org] On Behalf
>> Of Lewis, Stuart
>> Sent: Tuesday, November 14, 2017 4:26 AM
>> To: 'Julian M. Morley' <jmorley at stanford.edu>;
>> gail at trumantechnologies.com; pasig-discuss at mail.asis.org
>> Subject: Re: [Pasig-discuss] Experiences with S3-like object store
>> service providers?
>>
>> Hi Julian, Gail, all,
>>
>> At the National Library of Scotland we are also in the middle of some
>> procurements to revamp our storage infrastructure for holding our digitised
>> content archive.
>>
>> The approach historically taken here has been to use general purpose
>> SANs, with a second copy placed on offline tape.  The SANs have never been
>> built to scale (so they fill and we buy another), and they are general
>> purpose, trying their best (but often failing!) to run a mixed workload of
>> everything from VMs to data archive and everything in between.
>>
>> We’re now wanting to move to three copies, two online and one offline (in
>> the cloud if possible).
>>
>> For the online copies we’re about to get to tender to buy a
>> geo-replicated object storage system, to be hosted in our data centres in
>> Edinburgh and Glasgow.  I suspect the likely candidates will be systems
>> such as Dell EMC ECS, HPE+Scality, IBM ESS**, and Hitachi HPC.
>>
>> (** ESS rather than CleverSafe, as I think that is predicated on three
>> datacentres, but we only want two).
>>
>> We’re also about to try a large-scale proof of concept with the Oracle
>> Archive Cloud, but have an open question regarding its characteristics
>> compared to local offline tape.  Due to lack of transparency about what is
>> actually going on behind the scenes in a cloud environment, we don’t know
>> whether this gives us the same offline protection that tape gives us (e.g.
>> much harder to corrupt or accidentally delete).
>>
>> We’re also purposefully not going to use the object storage system’s
>> in-built cloud connectors for replication.  We feel it might be safer for
>> us to manage the replication to the cloud in our repository, rather than
>> having a single vendor system manage all three copies at once.
>>
>> Critique of this plan is most welcome!
>>
>> Also happy to join in any offline discussion about this.
>>
>> Best wishes,
>>
>>
>> Stuart Lewis
>> Head of Digital
>>
>> National Library of Scotland
>> George IV Bridge, Edinburgh EH1 1EW
>>
>> Tel: +44 (0) 131 623 3704 <+44%20131%20623%203704>
>> Email:stuart.lewis at nls.uk
>> Website:www.nls.uk
>> Twitter:@stuartlewis <http://twitter.com/stuartlewis>
>> <image001.jpg>
>>
>>
>>
>>
>> From: Pasig-discuss [mailto:pasig-discuss-bounces at asist.org
>> <pasig-discuss-bounces at asist.org>] On Behalf Of Julian M. Morley
>> Sent: 14 November 2017 04:28
>> To:gail at trumantechnologies.com; pasig-discuss at mail.asis.org
>> Subject: Re: [Pasig-discuss] Experiences with S3-like object store
>> service providers?
>>
>> Hi Gail,
>>
>> Sure - would be happy to chat with you.
>>
>> I’ve got Scality in my list of contenders - didn’t mention it here
>> because my first few use cases are explicitly ‘not on campus’, but I agree
>> it’s definitely a fit for our main on prem system. As with any commercial
>> software, ongoing licensing costs are a potential pain point for us.
>>
>> --
>> Julian M. Morley
>> Technology Infrastructure Manager
>> Digital Library Systems & Services
>> Stanford University Libraries
>>
>> From: "gail at trumantechnologies.com" <gail at trumantechnologies.com>
>> Date: Monday, November 13, 2017 at 4:06 PM
>> To: Julian Morley <jmorley at stanford.edu>, "pasig-discuss at mail.asis.org" <
>> pasig-discuss at mail.asis.org>
>> Cc: "gail at trumantechnologies.com" <gail at trumantechnologies.com>
>> Subject: RE: [Pasig-discuss] Experiences with S3-like object store
>> service providers?
>>
>>
>> Hi Julian, thanks for sharing your list and comments. Very thorough list.
>> I'd love to chat (and I'm close by in Oakland).... I've quite a lot of
>> experience in the cloud storage field and would suggest you also take a
>> look at multi-cloud connector technologies that will allow you to
>> standardize on S3, but write to non-S3-based public cloud vendors. And to
>> tier or move data among private and public clouds and do federated search
>> on metadata across a single namespace (across these clouds).
>>
>> Check out a couple of interesting technologies:
>>
>> Open Source Zenko.io <http://zenko.io/> - offering S3 connect to AWS,
>> Azure and Google (the latter 2 are coming shortly), and also
>> Scality Connect for Azure Blog Storage - translates S3 API calls to Azure
>> blob storage API calls.
>>
>> See the attached datasheet and also  https://www.zenko.io/
>>
>> I'd add Scality to your list -- see the Gartner magic quadrant they're
>> shown in the Upper Right Visionary quadrant and are close to you in San
>> Francisco. They talk S3, File, NFS/SMB, REST (CDMI etc), can tier off to
>> public clouds, and have lots of multi-PB size customer installs.  Gartner
>> MQ is here: https://www.gartner.com/doc/reprints?id=1-4IE870C&ct=1
>> 71017&st=sb
>>
>> I'd be very interested in learning more about your use cases -- can we
>> connect outside of this PASIG alias?
>>
>> Gail
>>
>>
>>
>> Gail Truman
>> Truman Technologies, LLC
>> Certified Digital Archives Specialist, Society of American Archivists
>>
>> *Protecting the world's digital heritage for future generations*
>> www.trumantechnologies.com
>> facebook/TrumanTechnologies
>> https://www.linkedin.com/in/gtruman
>>
>> +1 510 502 6497 <(510)%20502-6497>
>>
>>
>>
>>
>> -------- Original Message --------
>> Subject: [Pasig-discuss] Experiences with S3-like object store service
>> providers?
>> From: "Julian M. Morley" <jmorley at stanford.edu>
>> Date: Mon, November 13, 2017 12:43 pm
>> To: "pasig-discuss at mail.asis.org" <pasig-discuss at mail.asis.org>
>>
>> Hi everyone,
>>
>> I’ve currently got at least four use cases for an S3-compatible object
>> store, spanning everything from traditional S3 through infrequent access
>> stores to cold vaults. As a result I’ve spent considerable time researching
>> options and prices, and was wondering if anyone on this list has any
>> similar experiences they’d like to share.
>>
>> Our use cases range from hundreds of TB through to several PB, with
>> different access patterns and comfort levels around redundancy and access.
>> For most of them a 100% compatible S3 API is a requirement, but we can bend
>> that a bit for the cold storage use case. We’re also considering
>> local/on-prem object stores for one of the use cases - either rolling our
>> own Ceph install, or using Dell/EMC ECS or SpectraLogic
>> ArcticBlue/Blackpearl.
>>
>> The vendors that I’m looking at are:
>>
>> Amazon Web Services (S3, Infrequent Access S3 and S3-to-Glacier).
>> This is the baseline. We have a direct connect pipe to AWS which reduces
>> the pain of data egress considerably.
>>
>> IBM Cloud Bluemix (formerly CleverSafe)
>> A good choice for multi-region redundancy, as they use erasure coding
>> across regions - no ‘catch up’ replication - providing CRR at a cheaper
>> price than AWS. If you only want to keep one copy of your data in the
>> cloud, but have it be able to survive the loss of a region, this is the
>> best choice (Google can also do this, but not with an S3 API or an
>> infrequent access store).
>>
>> Dell/EMC Virtustream (no cold storage option)
>> Uses EMC ECS hardware. Actually more expensive than AWS at retail pricing
>> for standard object storage; their value add is tying Virtustream into
>> on-prem ECS units.
>>
>> Iron Mountain Iron Cloud (Infrequent Access only)
>> Also uses EMC ECS hardware. Designed primarily for backup/archive
>> workloads (no big surprise there), but with no retrieval, egress or
>> PUT/GET/POST charges.
>>
>> Oracle Cloud (cheapest cold storage option, but not S3 API)
>> Uses Openstack Swift. Has the cheapest cloud-tape product (Oracle Cloud
>> Storage Archive), but has recently increased prices to be closer to AWS
>> Glacier.
>>
>> Google Cloud Platform (not an S3 API)
>> Technically brilliant, but you have to be able to use their APIs. Their
>> cold storage product is online (disk, not tape), but not as cheap as
>> Glacier.
>>
>> Microsoft Azure (not an S3 API)
>> Competitively priced, especially their Infrequent Access product, but
>> again not an S3 API and their vault product is still in beta.
>>
>> Backblaze B2 (not an S3 API)
>> Another backup/archive target, only slightly more expensive than Glacier,
>> but online (no retrieval time or fees) and with significantly cheaper data
>> egress rates than AWS.
>>
>> Wasabi Cloud
>> Recently launched company from the team that brought you Carbonite.
>> Ridiculously cheap S3 storage, but with a 90-day per-object minimum charge. *It’s
>> cheaper and faster than Glacier*, both to store data and egress it, but
>> there’s obvious concerns around company longevity. Would probably make a
>> good second target if you have a multi-vendor requirement for your data.
>>
>> If anyone is interested in hearing more, or has any experience with any
>> of these vendors, please speak up!
>>
>> --
>> Julian M. Morley
>> Technology Infrastructure Manager
>> Digital Library Systems & Services
>> Stanford University Libraries
>> ------------------------------
>> ----
>> To subscribe, unsubscribe, or modify your subscription, please visit
>> http://mail.asis.org/mailman/listinfo/pasig-discuss
>> _______
>> PASIG Webinars and conference material is at
>> http://www.preservationandarchivingsig.org/index.html
>> _______________________________________________
>> Pasig-discuss mailing list
>> Pasig-discuss at mail.asis.org
>> http://mail.asis.org/mailman/listinfo/pasig-discuss
>>
>>
>> National Library of Scotland, Scottish Charity, No: SCO11086
>> This communication is intended for the addressee(s) only. If you are not
>> the addressee please inform the sender and delete the email from your
>> system. The statements and opinions expressed in this message are those of
>> the author and do not necessarily reflect those of National Library of
>> Scotland. This message is subject to the Data Protection Act 1998 and
>> Freedom of Information (Scotland) Act 2002. No liability is accepted for
>> any harm that may be caused to your systems or data by this message.
>>  Before you print please think about the ENVIRONMENT
>>
>> <http://www.nls.uk/>
>>
>>
>>
>> ----
>> To subscribe, unsubscribe, or modify your subscription, please visit
>> http://mail.asis.org/mailman/listinfo/pasig-discuss
>> _______
>> PASIG Webinars and conference material is at
>> http://www.preservationandarchivingsig.org/index.html
>> _______________________________________________
>> Pasig-discuss mailing list
>> Pasig-discuss at mail.asis.org
>> http://mail.asis.org/mailman/listinfo/pasig-discuss
>>
>>
>
>
> --
> Michael Davis  |  akropilot at gmail.com  |  mobile 408-464-0441
> <(408)%20464-0441>
> ----
> To subscribe, unsubscribe, or modify your subscription, please visit
> http://mail.asis.org/mailman/listinfo/pasig-discuss
> _______
> PASIG Webinars and conference material is at http://www.
> preservationandarchivingsig.org/index.html
> _______________________________________________
> Pasig-discuss mailing list
> Pasig-discuss at mail.asis.org
> http://mail.asis.org/mailman/listinfo/pasig-discuss
>
>
>
> ----
> To subscribe, unsubscribe, or modify your subscription, please visit
> http://mail.asis.org/mailman/listinfo/pasig-discuss
> _______
> PASIG Webinars and conference material is at http://www.
> preservationandarchivingsig.org/index.html
> _______________________________________________
> Pasig-discuss mailing list
> Pasig-discuss at mail.asis.org
> http://mail.asis.org/mailman/listinfo/pasig-discuss
>
>

-- 
Michael Davis  |  akropilot at gmail.com  |  mobile 408-464-0441
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.asis.org/pipermail/pasig-discuss/attachments/20171116/587b4bf1/attachment-0001.html>