Today, we’re publishing that research. It’s an extensive study of cloud performance, involving hundreds of tests from many locations across Amazon, Google, Salesforce, Rackspace, and Terremark. We built and deployed custom test agents on each cloud, and crunched hundreds of megabytes of log data. The report measures service response, network performance, CPU, and internal I/O.

Here’s a quick summary of the results; you can download the full study — complete with detailed conclusions, test methodology, and even agent code –for free from Webmetrics.

At Bitcurrent, we try to remain nonpartisan. We want to contribute to the industry’s growth and understanding. So we were thrilled at Webmetrics/Neustar’s approach here: they financed the research into cloud performance, and let us use their monitoring tool to collect the data from our agents. They were supportive, and enthusiastic, but exercised no editorial control whatsoever over the content of the report. This kind of altruistic, good-for-the-market contribution is, in our opinion, the best kind of marketing, and we commend them for adopting it.

Here’s the slide deck, with speakers’ notes.

The first step of this transition is that from now on, all Cloud Computing related blog entries will appear on our new cloud-centric blog.

We call it Rented Metal. Head on over and read about what’s happening at Cloud Connect.

We’re pretty excited, because this is the first time Bitcurrent has helped build an event from scratch (unless you count Bitnorth, that is, but Cloud Connect is a beast of a different magnitude.) There are four days of content, built around three audiences: those who buy and finance cloud decisions; those who build cloud applications, and those who have to run the cloud platforms.

Getting here has been an interesting experience. Here’s what we did, plus an easter egg for reading all the way to the end.

1. We first defined the content and themes based on what had worked in past conferences and where we thought the industry was going.
2. We reached out to roughly 50 people we knew were heavily involved in cloud computing, across industry, analyst, and end user, making a technical advisory board (TAB).
3. We solicited presentation ideas and papers from the world, and put these into a voting system (called Pligg), asking the TAB to choose which presenters, topics, or sessions they wanted to see.
4. We brought on a number of track chairs who specialized in our chosen themes.
5. Those chairs used the winning sessions, plus their own networks, to define the sessions and panels

We really couldn’t have organized this without some exceptional track chairs and organizers.

• Shlomo Swidler is a cloud developer who’s one of the top-ranked participants in the Amazon Web Service forums.
• Joe Weinman is an authority on cloud economics.
• Drew Bartkiewicz specializes in computing governance and risk.
• Randy Bias helps people deploy and migrate to cloud environments.
• Bradford Cross pushes the envelopes of big data.
• Greg Ness lives in the world of new infrastructure needed to support the cloud.
• Dr. Robert Marcus coordinates many of the standards meetings around clouds.
• Hooman Beheshti, Hon Wong, and Imad Mouline know all about performance.
• Dan Koffler, John Willis, Jesse Robbins, James Urquhart and Jason Hoffman know about cloud operations first-hand.
• Dwight Merriman has built some of the fastest applications in the world, and is now focused on scalable high-performance data.
• Judith Hurwitz is an expert in making technical topics accessible, and has written many books on IT.

Choosing track chairs, and letting each of them find the right content and participants for the sessions, proved invaluable. I would never have found the calibre of presenters they did. We also invited a number of cloud providers to come and teach the audience how to build on their distinct clouds. We have Microsoft, Google, Amazon, Heroku, Adobe, Intuit, and Salesforce.com all presenting in a single day.

It’s tough to find the right balance for any event. Cloud computing is at once broad (since it affects every aspect of IT) and specialized (since cloud spending is still a tiny fraction of global IT budgets) but the interest in clouds is huge. Many people are just trying to separate the wheat from the chaff, in order to understand what’s legitimately new and what’s just “cloudwashing.”

Cloud Connect seems to have become the place where that will happen. Attention and enrollment has surpassed all our expectations for a first-time conference, and we have a hundred of the most influential thinkers and builders in cloud computing under one roof. I’m humbled and amazed at the amount of work everyone’s put into the event.

Oh, and if you’ve read this far, find me on Twitter in the next 2 days, and send me a note. I’ve got a free conference pass for someone. If you want to attend the full event, including the workshops, use the code CNJRCC06 for a 40% discount or a free Expo pass. Hopefully we’ll see one another in a few weeks.

One of the biggest things I realized was that “clouds” can mean “elastic, on-demand compute platforms” or just “stuff that runs on the web” depending on who you’re talking to. And while these seem like two separate definitions, ultimately, they’re the same thing.

The Q&A, below the fold.

Q: What are the challenges raised by the use of wireless and/or cloud computing as teaching/learning/classroom infrastructure

A: Right away I can tell this will be confusing. “Cloud Computing” is a, well, nebulous term. It means a lot of things to a lot of people. At the most vague and broad level, it means “computer stuff I don’t run, but it just works.” This definition has its roots in the idea of Ubiquitous Computing, or ubicomp (look it up) where the computer recedes into the background, and access to information is pervasive. This is, frankly, the most interesting part of cloud computing: how it affects humans when everything we see and do is available for access instantly.

A more narrow (but more accurate, from a technical perspective) definition is “computing platforms that are available as a utility model.” This implies several things:

• It’s a platform, i.e. you build atop it
• It’s elastic, i.e. you can have as much or as little as you want
• It’s billed in small increments ($.10 an hour)
• It’s a shared resource (many other people are using it right now)

A popular analogy for this, from author Nicholas Carr, is the electrical utility: Computers are like generators; cloud computing is like the electric company.

Q: How can cloud computing help AU become a more prominent research institution?

A: Okay, we need to look at the two models of cloud computing I described above. In the narrow one, cloud computing means it’s easier to experiment with technology — online video, searching massive amounts of information, etc. Both the New York Times and the Washington Post did great work using cloud platforms to analyze, for example, Hillary Clinton’s diary. This let them report on it in a fraction of the time that they otherwise would have. So access to elastic, on-demand computing eliminates many of the barriers to entry that previously existed (like car rental companies lower the barrier to driving.)

In the broad sense, though, there are more interesting things afoot. Clouds mean you can find experts quickly through sites like Linkedin and Twitter. They mean you can determine what’s interesting (Reddit, Digg, and open source tools like Pligg.) They compress the time from news to audience: Just as blogs hurt newspapers, so Twitter is hurting cable news. You only need to look at the failure of the movie Bruno — killed by opening-night Tweets — to see this.

The challenge, of course, is that most of the Internet is cats and fart jokes and porn. So you need a curator, a trusted editor. In many ways, the Internet has made librarians cool again. Think about it: Librarians used to spend their time on three things: Written words, search catalogs, and footnotes. Snore. Today, however, we’re writing more words than ever; search engines are awesome; and what is a hyperlink if not a footnote? As a result, many of the library science skills are now relevant to information architecture and cogent curation of information. IMHO this is the role of journalism going forward.

Q: I’m not an IT person; I’m the guy writing the check. Convince me that spending on cloud computing is worth the investment?

A: On the narrow definition, cloud platforms are economic when you’re not using something 100% of the time. Joe Weinman’s blog (cloudonomics) is a great resource for this. So if you need something temporarily (1000 computers for 1 hour) clouds make sense. If you need 1 computer for 100,000 hours, maybe not. Also, because the cloud operators (Amazon, Google, Rackspace, Joyent, Terremark, etc.) are focused on efficiency, they can probably get a better deal on power, cooling, and so on than you can from a machine at home. So eventually, I think we’ll put things into the utility by default.

On the broad definition, cloud computing — ubiquitous, pervasive access to information — makes people smarter, more connected, more agile, and more productive … for most jobs. There are some places (assembling a car? Brain surgery?) where being connected might be bad. But for many (journalism, knowledge work in general, insurance, real estate) connectedness is a good thing. This is a major ethical decision: how connected do humans want to be. The bottom line is that most businesses need to embrace this movement towards an always-connected world simply because their competitors are.

Q: What impact will cloud computing have on the teaching/learning/classroom infrastructure? What would this help us deliver/do better?

That’s a tough one. I think the content will be less important, and the way a teacher tailors content to individual audiences will matter more. So much course material is going online for free (look at MIT) and companies like The Learning Company are taking courses and selling them to the lifetime students of the world. But what’s lost is the individual interaction.

I recently saw a documentary called Two million minutes. It follows six students — two American, two Indian, and two Chinese — through high school. It’s a pretty depressing film (if you’re from North America). One of the points it made is that in the US, if a kid is good at baseball, or swimming, we get them a coach so they can be amazing. But if they’re bad at math, we get them a tutor. In India, if the kid shows promise in math, we get them a tutor. Think about that.

So maybe one of the things that will save the US learning experience is one-on-one tutoring to accelerate the talented, rather than teach-the-slowest in order to leave no child behind.

I have a friend who built a company that records classes at McGill and makes them available by iPod, etc., through the school’s learning system. Others (Echo360 is one) are doing this. But simply sending out data isn’t interesting — to me, it’s the opportunity for feedback loops, showing how student learning is working, which concepts need to be taught more, which students are struggling with what — that can really make the classroom experience better.

So:

• Personalized teaching
• Recording and distributing content for free
• Analytics

Q: Cloud computing is in its infancy, should the university take that route now or do we wait to see how the industry is going to shake out?

A: That’s too vague a question. If you mean, “should the school embrace ubiquitous computing?” then the answer is, it already is, because all the students are. If you mean, “should the school abandon traditional server rooms and data centers?” then the answer is:

• Turn some of the existing machines into a “private cloud” and give students and staff access to them. You do this by putting virtualization on top of the machines (so there is one or more “virtual machines” on each physical machine) and deploying some basic services (storage, authentication) on top of them.
• Get an account with an Infrastructure as a Service provider (such as Amazon’s AWS product) and use those machines for “bursty” (sudden, short-lived) tasks. For example, imagine you had to process a huge amount of survey data. You need it by tomorrow. Use Amazon, pay for 1,000 machines at $0.10 an hour, and 2 hours and $200 later you have your answer. You’d never do that internally; you don’t have 1,000 machines.
• For new projects, consider building them on cloud platforms. Google’s App Engine is free (yeah, free) for apps that have less than around 1,000,000 page views a month.

Q: What’s this going to cost me? Is it worth the investment?

A: That’s the point: There’s little or no investment. This is what clouds do — they turn upfront investment into recurring costs. There’s a learning curve, of course.

Q: How can the university benefit from using the cloud (or clouds)?

• Reduced upfront costs (because you pay only for what you use)
• Easier experimentation (the cost of a failed effort is negligible)
• Easier to make content available (if you consider Youtube a “cloud” then you have a free way to deliver short lessons, for example)
• Easier to have engagement between faculty and students (as a friend of mine put it, “Twitter lets you stalk without being creepy.”)

Q: What are the challenges of integrating cloud computing into AU’s existing IT structure?

Now that’s a good question. Lots of traditional apps won’t work in the clouds, or will require some form of rewriting. To really understand this, we need to talk about virtual machines and stuff, which is pretty boring, so I’m going to try and simplify it:

Most web apps work well in the cloud, and that’s why a lot of startups use clouds. But a lot of university apps (like payroll, or registration) are ancient fossils running on mainframes, using stuff like Fortran. The university has three choices:

• Keep running it and hope it doesn’t break
• Turn the “legacy” stuff into a service. Similar to the way that RSS is a “service” for news feeds, you can turn that old machine into something the end user never sees. Put a layer of user interface in front of it (i.e. a web server) and hide the old stuff behind it. That way the old machine doesn’t have to make everything look pretty — it just has to handle business logic and storage. Then you can put the beautiful web front-end wherever you like, and treat the legacy back-end machine with kid gloves.
• Rewrite the app. Many of these older apps can be rewritten pretty easily, particularly on “platform as a service” clouds like Salesforce’s force.com, Google’s App Engine, or Heroku. Instead of giving you a “virtual machine”, these clouds give you a place to write your code — you don’t worry about the machines underneath it.

The university will also have to ensure that it’s compliant — meaning it respects the storage of data according to privacy laws. Consider that data stored in the cloud can be accessed according to the Patriot Act without your knowledge. With a cloud, you give up control over what’s stored where.

Finally, there’s performance and availability. If the Internet breaks, you may not be able to get to the app from campus, whereas you would be able to reach local machines. In some cases, this means an application could be sluggish too.

Q: I’m a skeptic. What’s the downside? How difficult will the transition be?

Baby steps. It’s easy for some things, hard for others. Pick the easy ones. Look at the government of Washington, DC. They were about to spend millions on an Intranet. They changed their mind and rolled it out with Google Apps for a fraction of the cost. On the other hand, I’m sure there’s some stuff buried in a DMV office that won’t be in a cloud for 20 years.

The downside is that a lot of IT people will have to learn new jobs. This is disruptive stuff. IT won’t be unemployed — but those guys are fossils and don’t want to change. They have comfy relationships with suppliers, and they’ve trained the organization to expect things to take a certain amount of time. Now, all of a sudden, cloud computing is upsetting that détente, showing the rest of the organization that it can have things better and faster, and IT is defensive. The smart IT folks I know love clouds, because they understand that they can build things that help the company. But the Luddites are afraid.

Q: What are reasons why a university would not want to go with cloud computing?

I can’t think of any, other than the privacy, legacy integration, and performance described above.

Q: Any idea what colleges/universities in general are with cloud computing? Are we already behind our competition?

Well, you’re asking, which is a good thing. But I would say that this is where the “broad” and the “narrow” definition of clouds I outlined above come together. On the one hand, you have this set of on-demand platforms you can use to experiment, connected to the public Internet, and therefore, to your faculty and students. On the other hand, you have sweeping human change, transforming the way we meet, think, and play. It’s time to connect the two.

In other words, the way we learn as a species is changing. Rote learning is out when search works well. We’ll never forget anything when we instrument our digital lives. (Remember that once, teachers resisted the use of calculators in exams — how long until students can review lessons during the exam?) My mom, who’s a teacher, always said to focus on the learning moment — the moment when something mattered to the student. That moment can happen anywhere, any time. How does that change learning?

The good news about cloud computing platforms is that the systems are easy to get started with (by definition.) It’s like saying, “am I behind houses that use electricity?” Well, yes, but calling up the electric company isn’t hard. What’s hard is catching up to people who’ve embraced electricity, and have adapted to the appliances and conveniences of modern life.

Q: Where does the industry stand today?

I can’t answer that in less that 4 days. That’s why I’m doing the Cloud Connect thing.

Q: How does the university choose a service provider?

Decide what you need, then try some out. Pick an app that’s relatively self-contained, or a new project, and build it. Mistakes are cheap. Don’t sign long-term contracts.

Q: I’ve seen lots of material suggesting that security is a major concern. Why is security a concern and how does AU address security concerns.

A: Studies show that security is a concern. Other studies show security is a benefit of clouds (I trust Amazon and Google to know more about security than I do.) The reality is that “security” is an umbrella term, including:

• Governance, the set of policies and rules by which an IT platform is managed. This is the business of defining policies.
• Privacy, which is about treating data appropriately
• Mis-use, which may be harder to detect in clouds. This could be internal use (a student using a cloud machine to track torrents) or hackers
• Data theft, such as credit cards getting out or someone eavesdropping
• Compliance, which means respecting legislation

Each of these is a distinct topic. There’s no easy way to address them other than to build secure applications and patch them often, and to know what data goes where. Clouds aren’t a magic wand, or a curse, for security. Many security concerns come from a sense that nobody’s watching the app in the cloud (look at how Google just got hacked) but the reality is that nobody’s watching the app in the data center, either. At least when you get hacked in the cloud, your bill suddenly goes up and you know about it.

There are tons of cloud breach stories. One of the reasons for that is that when it happens in the cloud, the victim has someone to yell at. When it happens internally, it gets swept under the rug, so you don’t hear about it. Consider that the majority of computer crime is an inside job; cloud operators aren’t insiders.

The trick here is that every Google App Engine site has a name that ends in appspot.com. You can call your new site whatever you want (mysite.com), but there’s a default name for it (mysite.appspot.com). And that makes it searchable.

First of all — how many are there? Well, according to Google’s own search, around 221,000. But what’s most interesting is that the top-ranked ones aren’t in North America. Many of them are Chinese, or Indian. Some of them are phishing sites (such as this one that is a pixel-for-pixel copy of Twitter — don’t log in.) Looks like the free trial and easy enrolment in the service make it appealing for certain markets.

Where’s the word “Appspot” being searched for? Once again, we turn to Google — this time their Insight for Search service — and see that China and Japan are where most of the discussion’s going on about “appspot”; for “appspot.com”, however, it looks like it’s North America.

It’s not clear that App Engine is getting the broad adoption of other cloud platforms, like Amazon. For one thing, it’s a Platform, not Infrastructure, model, which makes it less attractive for enterprises. There are concerns over portability, particularly for people who code to Google’s APIs for storage. But if these searches are anything to go by, it’s getting attention in some parts of the world.

• Steve Rubinow, EVP and CIO of NYSE Euronext
• Yosi Shneck, CIO, Israeli Light Company
• Eyal Waldman, Co-founder & CEO, Mellanox
• Liam Lynch, Chief Security Strategist, eBay

Here are the slides, including panel questions.

• Hybrid isn’t one app in two places; it’s internal and external apps talking to one another
• Migrating for new apps is easy; for already deployed ones, it’s much harder
• In the new world, the developers are the admins and ops toolsets are changing
• The 2010 platform will be
  • Infrastructure-aware; parallel; split between dev and ops
  • Not really PaaS; but not IaaS either
  • Runs both in-house and externally
• Increased focus on making it easy for developers to transition to on-demand environments
• Portability becomes a bigger concern (in/out and between clouds)
• Where enterprises will initially embrace it:
  • Collaboration, messaging, things “just above” infrastructure
  • Areas that don’t add strategic value
  • Leverage utility model of what’s there now (apps with inherent burstability)
• Ideological battle in infrastructure
  • Bottoms-up focus on primitives (storage, queue, compute); we build things from easy-to-connect, RESTful functions
  • Top-down modelled approach, which we reduce down to the underlying patterns and can generate code from them (policies, etc.)
• Growth of 2 kinds of technologies
  • That make this easier for developers (Ruby on Rails) ➜ This will win
  • That help to migrate legacy systems into cloud-compatible containers
• Enterprises about 5-7 years behind consumer/public Internet (Web tech, Hadoop, enterprise mashups)
• Let us not forget: All big web businesses use a ton of Oracle
• Huge $ to be made solving enterprise migration: Discrete components or specific apps
• Standardization is 15% of the problem; standards bodies are still arguing taxonomies
• Internet standards are built on rough consensus and running code; whoever produces a useful product that is available to many people quickly will win

Let us know if you have questions or want to use the content somewhere.

Photo by pasotraspaso

Every IT decision of any real consequence starts with a shortlist of three competing offerings. One of the three is usually the incumbent provider—Cisco Systems, IBM, EMC, Microsoft, and so on. Along with this incumbent are a couple of alternate providers. As a buyer, you line the features and prices of each contender up in nice, clean columns where you can compare them. Sometimes these providers are simply “column fodder” designed to rein in the incumbent; but many IT companies have built healthy businesses by being the alternate.

It’s time to add a fourth column: a cloud-based offering. That means every Request for Proposals (RFP) that a company issues must have a cloud-based option, regardless of whether the company actually plans to adopt clouds.

Here’s why.

Clouds set the upper bar

A cloud-based solution is often cheaper than hosted software. That’s because it’s not as customized, and can amortize costs across many buyers. As such, it’s a good “high watermark” for any IT project. Clouds act as a sanity check on pricing.

Clouds make the “basic feature” set apparent

Cloud-based offerings, particularly SaaS applications, are based on mass production. They offer 80 percent of the features one might want, for 20 percent of the cost. This is how cloud vendors achieve economies of scale—they don’t do a lot of customization, and where they do, it’s usually a set of tools and scripting that let you tailor it yourself.

This means the cloud offering shows you what the core features are. They make it clear what’s an extra, and what’s “table stakes.” Then it’s up to the vendors in the first three columns to prove that they’re worth the premium pricing because of their special features.

Clouds force true cost accounting

Enterprise software pricing is confusing—sometimes intentionally so. It’s often tacked on to the price of a CPU, or a computer. There are add-ons, installation fees, per-seat licenses, and more. It can be hard to compare offerings cleanly. Then there’s the storage, networking, security, and processor power needed to run those offerings.

A cloud, on the other hand, has a relatively simple pricing model. There aren’t hidden costs. Despite Salesforce.com’s famous “no software” logo, they’re really all about no hardware. Because clouds are specific to users or usage, it’s easier to charge costs back to individuals or departments.

By including a cloud vendor in a competitive comparison, you force a true assessment of costs across all participants, and surface the hidden costs of software you run on your own.

Cloud pricing is easy to get

Cloud providers have a very different sales strategy from enterprise software. They seldom have direct sales teams; instead, they rely on word of mouth and easy trials to get adopted. That means they don’t guard their pricing jealously, since they have a one-size-fits-all mentality.

Clouds are all about self-service IT access, and their sales and marketing processes are also more self-service. As a result, it’s easier to get numbers and details for the fourth column: just visit their website.

Make clouds the fourth column

Even if you believe you’ll never use a cloud computing platform (you Luddite, you!) you need to treat a cloud offering as a fourth column when evaluating any IT solution. You’ll be better armed, and more likely to discover hidden costs.

Sidenote: If you’re into cloud computing and the enterprise, stay tuned for details on the Cloud Connect vent next March. We’ve assembled a top-tier advisory board, and we’re currently accepting submissions for presentations.