Storage as a Service: Part 1: Object Storage Services

By | March 8, 2012
I’ve been doing some work on a plan to refresh and enhance the storage platforms used by a pan-European managed IT and cloud Service Provider over the last week. I thought it might be useful to put some of my thoughts down in a quick post and share where this SP is heading directionally, and where I think the market will move over the next year to 18 months.

I hope this will give an insight into the way in which Service Providers need to be thinking in order to create an efficient operating model for storage services and provide the breadth of different services that are being required to support today’s application architectures.

I’ve got quite a bit of content to cover so I’m going to break this up into a series of three posts to make this more digestible.

Picture for Object Storage Post.001

Firstly, here’s a chart that shows the typical service offerings that a service provider might want to offer and their characteristics.

In this first post, I’m going to focus on Scale out Object Storage, which is at the bottom left of the chart. Incidentally, the sizes of the bubbles on the chart indicate my view of the relative global deployment size of the different classes of storage over time. Obviously, this isn’t the situation today, but as Cloud becomes the de facto delivery model for appliations, more and more application data will migrate from block to file based storage and from file into Object-based storage.

Object storage presented via a web-services API is the immediate assumption of the service offer that purists will assume is at the core of any ‘Cloud-storage’ offer, but what’s the reason for this?  Two factors that are driving this perception in my view. Firstly, Amazon’s S3 service is often regarded as the ‘flagship’ storage a service offer within the AWS portfolio. Secondly, Amazon’s position as the largest ‘over-the-top’ (OTT) or internet-based cloud storage service provider gives them the opportunity to act as an anchor for customer’s expectations of the services and pricing that other SPs need to have in their portfolios. The first one of these points isn’t strictly true as Elastic Block Storage (EBS), or something with similar characteristics, is also a critically important component of AWS or any other cloud storage offer, but I’ll come back to this when I cover block in a subsequent post.

The key thing to remember about Object Storage Services is that they are accessed via a programmatic web-services interface over an IP network and that in most cases these APIs are directly accessible via the internet. The use of a web-services API for access means that they cannot be used with most existing enterprise (or legacy, if you prefer) applications that are expecting to deal with directories and files via an interface provided by the operating system or even directly with raw blocks of storage capacity in some cases. Whilst some gateway devices are appearing now that allow you bridge between operating system and cloud storage, these are very new and the performance characteristics of Object Storage might result in unexpected or unpredictable results with your applications.

There’s obviously an upside to the API presentation, which is that you can post metadata whenever you store an object via the API. This metadata can then be used to apply policies to the object whilst it is held in the platform. These policies might control features such as the expiration and deletion date of the object, the number of copies of the object to hold, and where to hold them, or the security policy applied to requests for access to the object. It doesn’t take much imagination to quickly realise how this might make managing extremely large numbers of objects and owned by very large numbers of tenants much less resource intensive that traditional storage management techniques – as long as you get the policy and metadata framework right in the first place of course!

Having said this, there are still a huge number of use cases that are a perfect fit for this type of platform. In fact, if you’re developing an application today that can tolerate the performance characteristics of Object storage, which for large files are typically going to be writing data at about 2 megabytes a second and reading it at about 10 megabytes a second, then you’d be crazy not to leverage this sort of storage infrastructure whenever you can as it potentially offers the lowest cost per GB stored and accessed of any platform, which brings us neatly back to Amazon S3.

Just one thing about costs and pricing for these services – if you plan to retain data for a significant length of time, say in excess of 12 months, or it access it frequently, you need to look carefully at the costs of using S3. And remember that IOs and data transfer from the Amazon service are charged in addition to the baseline data storage charges.  It is significantly more cost efficient to build a platform using either a software and commodity hardware platform, or an appliance-based platform like EMC’s Atmos, than it is lease capacity on Amazon S3. When you’re operating at scale, like most service providers do of course, you can achieve a sell price to your customers below that of Amazon’s and still gnerate significant margin from these services.

Service Provider selection criteria for object storage platforms? You need to be thinking about operations at scale from day one, so a robust multi-tenancy model and policy based management framework are absolutely critical. Also, look for simplicity and ease of scaling when adding nodes to the environment, integration with rating and billing systems and of course acquisition and operating costs.The ultimate objective for a Service Provider launching this type of service is generating the demand to ensure rapid adoption and scaling of the platform. This will in turn result in lower per-unit operating costs and improved profitability and the most successful SPs are those that have developed a portfolio of solutions that drive content onto the Object Storage platform – good examples are remote data archiving and backup services, sync and share applications like Oxygen Cloud or VMware’s Project Octopus and of course other software solutions from software developers that might be persuaded or encouraged to use an SP provided Object Storage Service – for example mobile applications, medical imaging applications and pretty much anything else you or more likely a 21 year old computer science student can think of :-)

Some Useful Links

Nasuni Report Comparing a Number of Cloud Storage Providers
EMC Atmos
The OpenStack Object Storage Project
Case Study of an Object Storage (Atmos) Implementation within an Enterprise

That’s a quick explanation of Object storage services and their increasingly important role in an overall storage architecture. I will cover scale out File and scale out Block in similar detail in a couple of subsequent posts over the next week or so.

Leave a Reply