In Shopping the Cloud: Performance Benchmarks I listed a number of services and reports that compare cloud provider performance results, but the truth is that in computing (cloud included) you can throw money at almost any performance and scale problem. It doesn't make any sense, therefore, to talk about performance alone, you want to compare price/performance.
But here's the rub: it is becoming increasingly difficult to compare the pricing of the various cloud providers.
Problem #1: Cloud providers use non-standard, obfuscated terminology
About a year and a half ago I wrote What Are Amazon EC2 Compute Units? in which I raised the issue of how difficult it is to know what it is you are actually getting for what you are paying in the cloud. Other vendors use their own terminology, such as Heroku's Dynos. I'm not just picking on these two, everyone has their own system.
Problem #2: Cloud providers use wildly varying pricing schemes
In addition, the pricing schemes by the various vendors include different components. Take storage as the simplest example, which clearly illustrates the point. Here's a screenshot from the Rackspace Cloud Files pricing page:
It is fairly straightforward, but also contains many elements that are extremely difficult to project (especially for a new application), such as the Bandwidth and Request Pricing. That's OK - you have to make some assumptions.
But here's my main point -- now compare it to Amazon S3 pricing:
Good luck with that.
Problem #3: Not all cloud offerings are created equal
To make things worse, not all cloud storage services were made equal. They have different features, different SLAs, varying levels of API richness, ease-of-use, compliance and on and on.
Problem #4: Cloud computing pricing is fluctuating rapidly
Another big problem with dealing with pricing is that the market is very dynamic and prices change rapidly. Fortunately, most of the movement right now is downwards, due to the increased competitiveness (especially in the IaaS space) and thanks to vendors benefiting from economies of scale and increased efficiency due to innovation.
Andrew Shafer from CloudScaling wrote a blog post a couple of weeks ago in which he shows how Amazon pricing is constantly shrinking. Check out this graphic he created:
What to do about it?
So what do you do in such a complex landscape? There seems to be no escape from creating a test application and running it on multiple services to see where the cost comes out. Then again, that may turn out to be a very time-consuming and expensive effort that may not be worth it -- at least not initially. So you should be prepared to have to move your app across cloud providers if and when the costs become prohibitive (which I am seeing happening to more and more companies).
Hopefully, the cloud benchmark services will also start paying attention to pricing and provide a comparison of price/performance and not just performance.
