Cloud Connect – Cloudy Operations Session Live Blog

The Cloud Connect Conference in Santa Clara is getting underway. This will be my attempt to share interesting tidbits from todays sessions.

Cloudy Operations Session – 9:00am

Very excited for this day long session being MC’d by John Willis from OpsCode. Should be great to get deep into devops tech and procedures.

* Some big names in the audience today including Adobe, Paypal State of Nevada amongst them.¬†Apparently this”devops” thing has real legs ūüėČ

* Once you get over all the “Cloud” hype, devops is still just about managing systems in a datacenter.

* John Willis quote: “When you grow up you want to build a Facebook app”

* According to the Google Chief Economist, we are in a period of “Combinatorial Innovation”.

* Devops is about trying to build “bulletproof” infrastructure. There’s still a lot of cultural baggage in how we view operations. This requires a cultural shift to support future innovations.

* Operations can be a real competitive advantage and not just some comoditized function. As Tim O’reilly puts it “Operations is the elephant in the room”

* Big players like Amazon, Microsoft, Google understand the realities of operating large scale infrastructure and have then built their own tools.

* Who cares whether you’re working with IaaS, PaaS, SaaS or XaaS. They’re all services, whether they live in the public cloud or a private datacenter and you want to treat them similarly.

* Currently we compete on Scale and Velocity of Innovation. How do you get an idea from the whiteboard to a bulletproof production system in the shortest amount of time? You can’t do it without the proper operational culture.

* Devops is based on 3 things: 1) Cooperation between developers & operations 2) Renaissance of tools and API based infrastructure. 3) Global community of practice.

* The walls between developers and operations must be broken down. You need developers who think like operations folks and ops technicians that think like developers.

* Agile: Where you get stuff done by doing stuff. Instead of having meetings about how you’re going to do stuff.

* Who has had a two week devops agile sprint decimated by the pager? Hands-up all over the room. Great line from Andrew Shafer

* The security guys who say you can’t automate security, probably don’t understand what’s going on. Systems like Chef & Puppet allow you to know and dictate exactly what’s on one or thousands of systems. Try manually verifying 300 new systems before deployment in a timely manner.

* Test Driven systems are a requirement. When you create things like Chef or Puppet recipes to deploy services, those same recipes MUST include adding the service to monitoring, etc…

* “Cowboy” based operations is fun, but if you rely on cowboys you’ll be stepping in horse shit for the rest of your life.

* Being able to experiment in operations is a critical asset. There’s no teacher better than pain.

* In the devops world, you need to evaluate your team not only based on technical skills or abilities but also based on team cultural and team cohesion.

* Boundary Objects are key. You need data to drive devops and that means providing realy data in realtime on performance of systems as well as business based KPIs.

* Fault tolerance shouldn’t just apply to your infrastructure. It must apply to your staff as well. Knowledge can’t be isolated in silos which are “vulnerable”.

* Interesting discussion of Kanban as opposed to Scrum based Agile devops.

Configuration Management

Industry has changed:

1990: Systems are inventoried, packaged & files transfered

2005: Unattended bare metal servers “very very” hard. 7K nodes took 5 days w/90% success

2007: unattended bare metal in under 10 minutes, fully configured in 3 minutes.

2008: Unattended server in 2 minutes, 5K servers in a week

2010: 10K nodes in under 5 minutes

* Cloud means to operations that: demand is dynamic and developers are crucial to operations.

Config management has 3 components:

  1. Provisioning (turn on the system)
  2. Configuration (get the system to a certain state)
  3. Systems Integrations (get the service to an operational state, let systems know about each other)
  4. Orchestration

* Devops has forced operations to get better at what they do.

* Config management treats infrastructure as code. Using development tools, mindsets and techniques to manage operations.

* Infrastructure as code means the infrastructure is self-documenting.  Version control, process control and application control.

* In a cloudy world your prime constraint for recovery should be the time it takes to restore your application data.

* CM apps like Chef allow you to treat your infrastructure like a programming language or subroutines therein.