Thinking Out Cloud

In the Game of Clouds, You Win Or You Die: CloudStack

Citrix is making a big announcement today. It has two parts. First, it's moving its CloudStack framework, which was owned by Citrix and distributed at CloudStack.org through a GPL license, to the Apache Foundation. Second, it is aligning CloudStack with Amazon Web Services' architecture and APIs.

This is big news that's sure to ruffle some feathers in the cloud computing space. So the questions I have are:

• Why are they doing this?
• What are the implications?

Why Is Citrix Moving CloudStack to the Apache Foundation?

The move appears straightforward. Citrix acquired CloudStack through its $200 million acquisition of Cloud.com in July 2011. From what I'm hearing, it has had good success with enterprise and provider adoption of the product, but it was far from being accepted as an industry de facto standard, which is what, I assume, they had hoped for. So it would make sense for them to move it to a respected open source foundation like Apache.

But wait. There was already a OpenStack vying for that de facto standard open source platform status, and Citrix announced its support for it two months before the Cloud.com acquisition. The company said it would continue to support both platforms after the acquisition, so what happened?

Citrix claims that the OpenStack foundation wasn't run well: it was dominated by Rackspace and had a "pay-to-play" model. The result was that the APIs were poorly designed and the product lacked stability and maturity (a claim I have heard from others). At the same time, while OpenStack is getting support from the likes of Cisco and HP, the community is somewhat fragmenting, with multiple distributions and extensions from the likes of startups Cloudscaling and Piston Cloud. This started creating a problem for enterprise customers and providers who were getting confused and having a bad experience with OpenStack.

In the meantime, Citrix is feeling competitive pressure from the company it views as its primary rival for actual customers (as opposed to winning the hearts and minds of the "community"): VMWare.

It simply couldn't wait and had to go on the attack with a bold move. This is it -- and it's a pretty good one.

Why Align CloudStack with the AWS APIs?

The final piece in making CloudStack the de facto standard cloud platform is the API. By aligning with the Amazon APIs, CloudStack gains a hghly-adopted, proven API with a massive ecosystem of integrated tools and services around it. They tell me they will have 100% compatibility by the end of this year.

But the catch is this comes on the heels of Eucalyptus's announcement that it is aligning with the AWS APIs. Eucalyptus, just a reminder, is another open source cloud platform that has been vying for leadership among enterprise customers.

Game of Clouds

With this move complete, Citrix has a production-proven stable product, which is now an open source platform managed by the widely-respected Apache Foundation, using a popular API with a massive ecosystem around it. Pretty clever, I think.

On the other hand, CloudStack is fighting on three fronts now: its traditional arch-enemy VMWare, the OpenStack camp with the dozens of vendors -- large and small -- behind it, and Eucalyptus. A regular game of thrones. And to paraphrase Game of Thrones: In the Game of Clouds, you either win or you die.

10 Predictions About Cloud Computing

Next week in Tel Aviv I'm going to participate in a panel about "the future of clouds", moderated by the legendary Yossi Vardi. In preperation, I wrote down a few of the concepts I've been thinking about for the past several years and I thought I would share them with my readers to get some feedback. Keep in mind these are long-term predicitions and trends (in no particular order).

1. PaaS Rules :IaaS becomes niche. In the long-run, IaaS doesn't make sense, except for a limited set of scenarios. All IaaS providers want to be PaaS when they grow up.


2. Public Rules: Internal clouds will be niche. In the long-run, Internal Clouds (clouds operated in a company's own data centers, aka "private clouds") don't make sense. The economies of scale, specialization (an aspect of economies of scale, really) and outsourcing benefits of public clouds are so overwhelming that it will not make sense for any one company to operate its own data centers. Sure, there need to be in place many security and isolation measures, and feel free to call them "private clouds" -- but they will be owned and operated by a few major public providers.


3. Specialized Clouds: There are many dimensions to an application: the pattern of its workload; the government regulations it must adhere to; the geographic access to it; the programmig language and framework it supports; the levels of security, performance and reliability it requires; and other more specialized requirements. It's not a one-size-fits-all world. At least, not always. There will be big generic clouds, and then, many specialized clouds. I've written about this in the past.


4. Government Regulation: The largest cloud providers will become nationally strategic infrastructure (like utilities, financials, telcos, airlines and shipping companies in the past). Given my "public rules" prediction above, cloud providers will become crucial infrastructure to the economy and the interests of their respective nations. They will become "too big to fail". Any change in their pricing will have a profound effect on the economy. And they will also hold the risk of a "cloud run" (similar to a "bank run", a sudden surge of demand they haven't anticipated. Not to mention the fact that they will maintain the sensitive data of consumers, corporations and government agencies. Any way you slice it, it spells regulation. But if history teaches us anything, this regulation will only come after "The Great Cloud Catastrophe" (use your imagination to figure out what that will look like). 


5. The Control vs. Freedom Debate: This sums up the story of cloud computing so far. Freedom is the catch-all phrase for drivers of cloud adoption (no upfront costs, on-demand, self-service, empowerment of the rank & file - e.g., developers), but control (or lack thereof) is the catch-all phrase of barriers to adoption by large enterprises. Every democratic country experiences this: there is sometimes a contradiction between the so-called sacred principles of rule-of-law and personal freedom. It's a matter of drawing the line -- and we're just in the beginning stages of understanding this issue when it comes to cloud computing. This debate will be with us for years to come and will shape the variety of enterprise cloud computing offerings.


6. Cloud Federations - While AWS has enjoyed tremendous international success, in any business that relies heavily on trust, such as IT, nothing beats a local brand. So people will flock to the cloud of their trusted national telco or big IT provider. But on the flip side, they will need to reach a global audience and will want servers around the world. As a result, we will see the formation of cloud federations, similar to what we see in airline alliances, such as Star, SkyTeam and Oneworld.


7. Financial Efficiency and Sophistication: Computing is a commodity, and every commodity ends up being traded, future-traded, brokered, arbitraged, speculated and manipulated with derivative instruments. The good: the market becomes very efficient. The bad: the market becomes complex and opaque. We are already seeing spot markets 


8. Cloud Standards: About two years ago there was a strong wave of interest and discussion about the need for cloud standards. I wrote then, and still believe, it is too soon. But it is also inevitable. We will, however, see multiple competing standards. At least one formal stanard specification from a standards body and several de facto standards from large commercial players such as Amazon and VMWare. 


9. The Ecosystem Wars: I've recently written about the importance of ecosystems in cloud computing. Success in building an ecosystem will be a determining factor in who wins and loses in the cloud. It is not just about the size and breadth of the ecosystem, but how well it all works together. In many ways, Amazon has done a poor job of this so far, but it has the one big compelling factor for an ecosystem: a very large install base. 


10. Horizontal and Vertical Consolidation: As with any industry, as cloud computing matures, it will consolidate. This will happen both horizontally, for example large IaaS players will roll-up regional and smaller IaaS and hosting providers, as well as vertically, for example IaaS providers will acquire cloud management system providers such as RightScale and enStratus.

I'd love to hear some feedback on these trends. Do you agree? Disagree? Have I left something out? Please let me know in the comments.

Shopping the Cloud: Pricing (or Apples and Oranges in the Cloud)

In Shopping the Cloud: Performance Benchmarks I listed a number of services and reports that compare cloud provider performance results, but the truth is that in computing (cloud included) you can throw money at almost any performance and scale problem. It doesn't make any sense, therefore, to talk about performance alone, you want to compare price/performance.

But here's the rub: it is becoming increasingly difficult to compare the pricing of the various cloud providers.

Problem #1: Cloud providers use non-standard, obfuscated terminology

About a year and a half ago I wrote What Are Amazon EC2 Compute Units? in which I raised the issue of how difficult it is to know what it is you are actually getting for what you are paying in the cloud. Other vendors use their own terminology, such as Heroku's Dynos. I'm not just picking on these two, everyone has their own system.

Problem #2: Cloud providers use wildly varying pricing schemes

In addition, the pricing schemes by the various vendors include different components. Take storage as the simplest example, which clearly illustrates the point. Here's a screenshot from the Rackspace Cloud Files pricing page:

 

It is fairly straightforward, but also contains many elements that are extremely difficult to project (especially for a new application), such as the Bandwidth and Request Pricing. That's OK - you have to make some assumptions. 

But here's my main point -- now compare it to Amazon S3 pricing:

Good luck with that.

Problem #3: Not all cloud offerings are created equal

To make things worse, not all cloud storage services were made equal. They have different features, different SLAs, varying levels of API richness, ease-of-use, compliance and on and on.

Problem #4: Cloud computing pricing is fluctuating rapidly

Another big problem with dealing with pricing is that the market is very dynamic and prices change rapidly. Fortunately, most of the movement right now is downwards, due to the increased competitiveness (especially in the IaaS space) and thanks to vendors benefiting from economies of scale and increased efficiency due to innovation.

Andrew Shafer from CloudScaling wrote a blog post a couple of weeks ago in which he shows how Amazon pricing is constantly shrinking. Check out this graphic he created:

What to do about it?

So what do you do in such a complex landscape? There seems to be no escape from creating a test application and running it on multiple services to see where the cost comes out. Then again, that may turn out to be a very time-consuming and expensive effort that may not be worth it -- at least not initially. So you should be prepared to have to move your app across cloud providers if and when the costs become prohibitive (which I am seeing happening to more and more companies).

Hopefully, the cloud benchmark services will also start paying attention to pricing and provide a comparison of price/performance and not just performance.

Have Google App Engine and Microsoft Azure Surpassed Rackspace?

Today Zenoss released a survey it conducted about cloud computing and virtualization. It has some interesting data and they created a very nice looking Infographic with the key findings.

Here's the data point I found most interesting:

 

I have followed many cloud surveys and reports that measure cloud traction of the different providers (see for example Guy Rosen's State of the Cloud). It has consistently been the case that Amazon is ranked #1 and Rackspace #2 (which is what prompted my Rackspace: The Avis of Cloud Computing post). The Zenoss survey suggests a different story with Google App Engine and Microsoft Azure coming in at #2 and #3 respectively, pushing Rackspace to #4.

Also, GoGrid's penetration, as well as RightScale's (which is a very different animal than the other players on the list) is very impressive.

Note that the wording of the question in the survey was a bit ambiguous: "What are your cloud computing plans for 2010?". I say ambiguous because the survey was conducted Q2 2010, so probably close to the middle of the year. But in any case, it has a forward looking element to it, which gives a little indication of the trends as they are happening.

Anyway, lots of interesting info on both cloud and virtualization. Check out the full survey results (requires registration).

Thoughts on Amazon EC2 Spot Instances

The innovation just keeps on coming from the good folks at Amazon Web Services. This week they announced a new pricing model for Amazon EC2 instances: spot pricing. Spot pricing is the third pricing model Amazon is offering for EC2 instances -- with On-Demand and Reserved being the other two -- and it brings us closer to an efficient and commoditized IT infrastructure market, and it got my mind racing on the various possibilities of it, and where it goes if taken to its logical conclusion.

You can read explanations of what it is and how it works from Jeff Barr, Werner Vogels, RightScale, Reuven Cohen and James Urquhart.

James has a very succinct explanation of the key tenets of the new offering:

• Each customer sets a maximum price he or she is willing to pay for "spot instances."

• Amazon sets a "spot price" for instances hour-by-hour, based on available supply and demand.

• Customers pay whatever the spot price is up to their maximum price. So, if someone bids $0.07/hour, and the spot price is $0.05/hour, the person pays $0.05/hour.

• If the spot price exceeds the customer's maximum price, the customer's instances are terminated.

I had to open my old finance textbook from business school and think of all sorts of possibilities: call options, put options, futures, and other forms of derivatives and hedging techniques. It will be interesting to see if any of those evolve over time. By the way, there already is a real-time ticker for Amazon spot pricing, called Cloud Exchange. But here are some thoughts on issues that are relevant in the shorter term.

Workloads

At first glance it would seem that Spot Instances are only relevant to a limited set of workloads, namely those that are not time-sensitive and can be easily stopped and re-started. The examples given by Jeff Barr are web crawling, data processing (presumably of the batch, scientific type, as his Pfizer case study implies) and data transformation (the example Jeff gives is media transcoding -- such as changing video formats).

But requesting a spot instance is accessible via API, as are other types of instance provisioning in Amazon, which means that you can programmatically write very interesting logic around which types of instances you want provisioned under what circumstances. I think this means that over time people will realize that spot instances are feasible for a wide variety of workloads. It will require large scale apps to be worth the effort, but those are increasingly commonplace on AWS.

As I said in regards to Reserved Instances, there will be a psychological barrier for a while, but it will be overcome eventually.

Bidding

I posted a question on Twitter: "If I bid for EC2 spot instances at list price, doesn't it mean I'll always have a full-time instance at the lowest possible price?" That's all I could explain within the 140 character limit, but let me put it another way here.

If my objective is to have the longest possible running ec2 instance at the lowest possible price, I always want to bid at the then-current ec2 on-demand price. Here's the logic:

If the current Amazon spot price is lower than my bid, I will be charged the current price regardless of my bid (that's how the system works). I am making an assumption that the spot price will never go above the on-demand price. I am making this assumption for a very simple reason, which is that no one will ever bid equal to or more than the on-demand price because then they can simply provision an on-demand instance (which does not have some limitations that spot instances do).

No one has yet to come up with an argument (at least to me on Twitter) that contradicts this logic. Several people said that technically it's possible that spot prices will rise above the on-demand price, but as I already said, that doesn't make sense. On-demand prices are the ceiling for spot prices (except for the caveat below). If you have different thoughts on this, please leave them in the comments below.

Here are two possible issues with this logic as I see them:

• On-Demand Instances are not always available: This was the argument made by Ezra Zygmuntowics (@ezmobius), co-founder of Engine Yard. This means that On-Demand is not a real ceiling for spot prices. Although I've seen some noise about this on Twitter, I don't think there is a real case to be made that there is a problem with getting On-Demand Instances "on-demand". If there is, please correct me in the comments. But more importantly, if there is, that's a big "whoa" moment. It is completely contradictory to the whole Amazon (and cloud computing) model of, er, on-demand computing. It would also raise the biggest fear possible about the cloud computing model of relying on an external provider for your basic computing needs (i.e., servers): that capacity is unavailable. It also contradicts the rationale that Amazon gave for spot pricing in the first place -- to take advantage of unused ec2 capacity. But I really don't think that it is an actual problem. I have not heard of anyone seriously complaining about lack of availability of on-demand Amazon instances. Finally, going back to theory, if there was such a problem, Amazon would simply raise prices for on-demand and we'd have a new ceiling for spot. We're actually seeing the opposite happening: Amazon keeps lowering prices for on-demand instances.
• There is a hard cap on the desired cost for the job: This makes a lot more sense to me, but I think it is a rare real-life case. This use case assumes that I have a time-insensitive compute problem that is only worth for me to solve at X cents per hour/per instance. To make it easier to digest let's say that X equals 5 cents and the current price of an on-demand small instance (and we'll use that as the unit) is 9 cents per hour. If that's the case, then obviously I want my spot bid to be 5 cents and not the on-demand instance price.

Barring these two arguments, there is no reason to bid for a spot instance at less than the current on-demand instance price. That's it. I said it, now skewer me in the comments. :-)

Shlomo Swidler suggested on Twitter that someone run a competition to see who can create the logic that keeps the longest running spot instance. That would be cool, but I would add the condition: at the lowest possible price.

It's going to be fun to see where these innovations take us. Any thoughts? Please share in the comments.

UPDATE: there is a great comment thread to this post with interesting insights from James Watters, Guy Rosen and Shlomo Swidler.

Amazon Reserved Instances Update

Back in March, when Amazon first came out with reserved instances for EC2 (instances you can pre-pay at a discount for a period of 1 or 3 years) I posted Amazon Reserved Instances: Do They Make Business Sense? In that post I also made available a public Zoho spreadsheet with a simple calculator that allows figuring out whether reserved instances make sense for you or not.

In August, Amazon announced lower pricing for Amazon EC2 reserved instances, so I created an updated spreadsheet (if this cuts off in your browser click here):

I also created a version for the European prices (full, editable version here):

The break-even point for 1-year instances is 3,250 hours, or about 4.5 months (assuming you are running the instances 24/7). The break-even point for 3-year reserved instances is 5,000 hours, or about 7 months (assuming you are running the instances 24/7).

Several of the companies I have been working with have a substantial pool of instances constantly running. And although they are accounting for reserved instances in their business plans to demonstrate the long-term benefits of the gross margins they allow, they seem hesitant to actually pay for reserved instances. These calculations demonstrate that they are probably being too risk averse as in their case, the platforms/apps are sure to be running for at least 7 months and there are substantial cost-savings involved.

Are you considering or already running reserved instances? What is the use case and what are the pros and cons? Please share in the comments.

Cloud Computing Security Podcast

Security is consistently brought up as one of the main barriers to the adoption of cloud computing and one of the big challenges it presents. Issues such as security in virtualized environments, multi-tenant architectures and the cloud in general are often discussed with very little real-world understanding of the problems and the potential solutions.

So it's nice to discuss it with someone who actually knows what he's talking about, as James Urquhart and I do with Chris Hoff, the well-respected security expert and blogger, in Show #8 of our cloud computing podcast, Overcast.

Listen to it here.

Amazon Reserved Instances: Do They Make Business Sense?

UPDATE (10/2009): Amazon has updated their reserved instance pricing as of August 2009. Please see my revised numbers and spreadsheets here: Amazon Reserved Instances Update.

Amazon just announced that they are giving an option to pay upfront for Reserved Instances on EC2. See Amazon CTO Werner Vogel's blog for more details.

I set out to understand the economics of Reserved Instances. For that purpose, I created a calculator that allows you to plug in the number of hours you expect to use the AMI during the year, and will then tell you how much money you will save or lose by using Reserved Instances versus pure On-Demand Instances. I share this calculator below as a embedded Zoho spreadsheet (if you cannot see it well, click on the full screen link on the top-right).

The way to use it is very simple. For each instance size, just plug in the number of hours you expect to use the instance during the year in the light blue cells. You will then get the results for both 1 year and 3 year contracts in the corresponding gray cells. If it's a positive number, that's how much you save by paying the upfront Reserved Instance fee. If it's a negative number, it's how much you'll lose.

The default number I put in the hours column, 8760, is 24x365. So if, for example, you run a Large AMI for the entire year non-stop, you will save $1,153 by using reserved instances.

If you are curious to know the break-even point, it is 4,643 hours annually for the 1-Year fee and 2,381 hours *annually* for the 3-Year fee. In other words, if you expect to run an instance for more than 4,643 hours during the coming 12 months (works out to an average of 12 hours a day), you're better off with Reserved, otherwise, stick to On-Demand.

Without further ado:

Feel free to download the spreadsheet, and if you find any errors in it, please let me know.

There are two more important considerations you should take into account in any ROI calculation for this:

• If there is any uncertainty with regards to the expected number of hours, you should take into account the probability that you will overshoot or undershoot the number of hours and create an Expected Value based on that. There are no refunds on the upfront fees!
• Consider the time-value-of-money of your upfront fees. In other words, what value could you get for that money by investing it in other areas of the business (or in U.S. T-Bonds for that matter). You should determine your "discount rate" and calculate a Net Present Value.

I admit I 'm somewhat obsessed with cloud pricing lately, but that's for two good reasons. First, as one of the main value propositions of cloud computing is cost savings, I think it is important that as an industry we examine what the vendors are doing and making sure we're getting it right. Second, for my own selfish reasons. I am working on several projects now where I either need to figure out the economics of using cloud for an end customer (i.e., a company that wants to run its apps in a cloud environment and needs to figure out ROI), as well as for Platform-as-a-Service vendors who are leveraging EC2 under the hood.

UPDATE: Amazon has updated their reserved instance pricing as of August 2009. Please see my revised numbers and spreadsheets here: Amazon Reserved Instances Update.

Hyperic and the Cloud Podcast

James Urquhart and I recorded show #7 of Overcast, our podcast series on cloud computing. It's up and available here for your listening pleasure.

In this episode we have a discussion with Javier Soltero, CEO of Hyperic. Reposting the show notes:

Show Notes:

• Javier gives us a general background on Hyperic and specifically its activities in the cloud, which include CloudStatus.com and the Hyperic HQ Amazon Machine Image (AMI).
• Javier reveals some of Hyperic's developments in the works, including a technology that allows easily instrumenting Java, Hadoop and Memcached applications and exposing the data  to CollectD. 
• We discuss at length Javier's recent blog post The Cloud Dilemma for Developers, in which Javier writes about the fact that in a cloud environment developers are required to address operational issues, such as monitoring and management. 
• Javier explains how application monitoring and management requirements are different in cloud computing, and why he believes traditional tools won't cut it in the cloud  
• and lots more...  
• Companies and products mentioned in this podcast include Mosso, GoGrid, Google App Engine, Heroku, Amazon Web Services, Salesforce.com, GigaSpaces, Cisco and Windows Azure.