Is cloud computing stable in bad weather?
Ian Rae (
@ianrae)
Abstract: Network effects are powerful and scary.
The clouds just keep rolling in. Microsoft has thrown their hat in the ring with Azure which deserves notice because Ray Ozzie is behind it. Amazon just released a CDN service. With recent coverage in The Economist and every major newspaper, even my local news tonight, cloud computing appears to be here to stay, judging by the marketing feeding frenzy.
There have been some recent well publicized outages that have generated some concerned opinion. Something struck a related chord when I read an excellent interview by Stephen Strogatz about network effects in Seed magazine:
I have this general concern about entering this networked era, which we’re clearly already in. For example, the power grid used to not be a grid. It was just a lot of isolated power stations. When there was trouble people would just close down the power plants and repair whatever the problem was.
But now that there’s a grid, when something bad happens at one point in the grid, and you use the defense strategy of just shutting down that plant, it can have propagating effects. It can put too much load on other plants, which may cause them to shut down. And this is exactly what we saw here in the northeast when we had the 2003 blackout. Or think about what is happening right now in the market, where there are all kinds of propagating, cascading failures in our market and financial systems. So, I’m just thinking that you may be opening a Pandora’s Box
We’ve seen it with electrical grids, financial markets and now cloud computing: network effects can kill. All that said, cloud computing has some compelling advantages. Good for the environment since more efficient data centers which can run at higher utilization, and on demand, although you could argue that there is a cost of centralization (data needs to travel further, and this is less efficient) and the ease of adoption could drive more usage and therefore have a net detrimental effect on the environment, especially if it is backed by dirty, non-renewable power. This may be why Amazon and Google are building cloud infrastructure in Oregon to take advantage of the Coumbia river’s hydro electric power.
Better code optimization is another potential advantage, as Alistair has pointed out, since code efficiency is more apparent to the bottom line when you pay for utility computing, instead of being hidden from the CFOs view in your monitoring system, if you have one. Cloud computing is shared infrastructure and platforms, which promise flexible usage and the theoretical capability to scale your application resource usage smoothly while paying for what it takes to maintain user experience, as Alistair discussed in The cloud’s most important equation. And therein lies the rub:Famous last words: “Everyone go see the whales on the starboard side of the boat…”Statistically cloud performance should stabilize as the number of customers become very large, but the potential problems also get bigger with size, if for some reason all clients move together in an accidental denial of service “attack,” as would also be exhibited in theory (a wide scale version of what we know as the Slashdot or more recently the Digg effect) if there was a run on the cloud (everyone trying to pull their code or upload it simultaneously.
The rise of Private Clouds
One of the more amusing comments about Microsoft’s entry into the cloud computing “market” was a warning on a mailing list to look out for the “Azure Sky of Death.” Its only funny because there is a grain of truth to it…publicly accesible cloud computing has lots of advantages, but reliability isn’t likely to be one of them in the short to medium term. Even long term the very nature of the shared on-demand infrastructure could lead to “tragedies of the common cloud” which is why this author expects to see rapid development of private cloud infrastructure that better balances the economic constraints and benefits for businesses that can’t afford to take the risk of betting on AWS.
Publicly accesible cloud computing is likely to suffer from fluctuations in availability, until cloud providers build in greater constraints or until we figure out how to control the human behaviour that, for example, underlies the wild financial markets…until then it will be like trying to control the weather.
2 Comments, Comment or Ping
Reuven Cohen, CTO Enomaly Inc
ian, your post reads like the doom and gloom forecasts we saw about cloud computing back in 2007. Cloud computing is as much about trust as it is about efficiency. The real question you should be asking is do I trust microsoft, amazon or even AT&T to mange my infrastructure better then an internal data center team, and in my rather bias opinion the answer for the most part is yes.
Also, about your comment on outages, the question isn’t will the cloud provider go down, because it certainly will. The real question is how do I enable a hybrid cloud environment that assumes failure and can do so gracefully. (This one of my main motivators for a unified cloud interface standard)
Lastly, as some one who has built a business on the so called private cloud platform. The idea of a quarantined shared internal cloud, aka private cloud is an oxymoron. Cloud computing is about using resources where ever and when ever, internally or externally.
Nov 26th, 2008
Ian Rae
My principal point was that network effects could affect large, relatively open clouds. This wasn’t intended to burst any bubbles as I’m a huge fan of cloud computing and both my business and customers rely heavily on it! Rather this post is an observation that the behavior of complex and interconnected systems is at odds with predictable performance, availability and scalability.
To switch to the “private cloud” question, this may a semantic disagreement, but I also believe that such on-demand utility computing is as useful in the enterprise data center as on the public “internets.” Hence the concept of a private cloud where I have my physical resources dedicated to a specially authorized group of users (i.e. my customers, who have some requirements not met by AWS) with a much more tightly controlled cloud architecture, allowing on demand computing, pay-per-use billing to various departments, granular SLA monitoring, etc..
As previously written here, we will likely see different cloud computing ecosystems, with vertically targeted clouds (i.e. “genome analysis cloud” or a HIPAA cloud) that are architected for specific workloads. And I think some will require private control (data sensitivity…or SLA requirements etc..) while benefiting from the same flexible technology as public clouds. To think that publicly available cloud computing will solve every computing requirement would be naive.
The hybrid model of public-private cloud computing is very interesting for certain kinds of applications (www.smugmug.com did a great implementation here from what I understand) but not very practical for most at this point. Certainly one of the top requests I get from customers these days is how to take advantage of the “promise” of virtually unlimited scalability, so they don’t have to over-provision internally. Its a nice theory but the world get real very fast when you set about running things this way.
Nov 26th, 2008
Reply to “Is cloud computing stable in bad weather?”