Cloud Computing Standards: Cloud Platform Reference Architecture

Almost a year ago I said that cloud standards were premature, noting that traditional standards bodies are not known for being "fertile grounds for agile innovation" (which is not necessarily a bad thing by the way). Back then I said "what we need are simple, rugged, market tested interfaces defined by the innovators in each area (virtualisation, storage, services, etc.)" and sure enough some of these APIs are now starting to gain some traction (e.g. AppEngine, EC2 and S3).

You may also have just read that over the weekend I got sick of waiting for the Cloud Computing Interoperability Forum (CCIF) to produce something (it has after all been ~6 months already) and declared it dead and doghoused. Rather than being one to complain without offering a solution, earlier today I kicked off a new cloud standards workgroup. No invites. No resumes. No dedicated websites. No moderation/censorship. No fancy graphics. No sponsors. No added salt. Just standards. For more see the:

• Google Group (cloud-standards)
  
• Google Code project (cloud-standards) and
• MediaWiki installation (ala Wikipedia)
  

This was all done under the Cloud Computing Community umbrella, with a view to getting some runs on the board fairly quickly. I'm pleased to annouce that after the first day of researching existing implementations and talking to platform vendors we already have a useful deliverable.

The majority of the attention around cloud standards has thus far been focused at the bottom of the cloud computing stack, in the physical and virtual infrastructure itself. That's understandable because that's where most of the pain is today, and with it most of the opportunity. Ultimately though the real value of cloud computing is further up the stack in the platform and application layers. Accordingly this is where I started: a generic reference architecture for cloud platforms like Google App Engine and Salesforce's Force.com.

Work is already underway on a Debian GNU/Linux based reference implementation but not being one to talk about unreleased software I'll tell you more about that later, for a start take a look at the diagram and the various implementations on which the consensus was based (copied and pasted from the Cloud Standards Wiki):

Platform

Reference Architecture

This cloud platform reference architecture is compatible with all of the existing platform deployments, allowing for developers to submit application code, data and metadata via Git and/or Subversion, WebDAV, or in the simplest case (per Google App Engine) native HTTP.
YAML is recommended for metadata as it is both human and machine readable, less fragile and complex than XML and supports complex data structures. There are also stable implementations in most common programming languages.
Runtime management should be available via REST over HTTP with an optional web interface.

Source Code Management (SCM)

Requirements

Uploading of application code (in source and/or object form) and data, as well as metadata required for the operation of the application (which can, for the sake of simplicity, be stored in file(s)).

Actors

• Developers need to upload code, data and metadata for execution on the platform. Version control support (e.g. for collaborative development, rollbacks, compliance, etc.) is optional but desirable.

Existing Implementations

Compatibility is sought with the following existing implementations:

• Google App Engine RPC over HTTP

The Google App Engine appcfg.py tool uses an RPC over HTTP interface (appengine_rpc.py) to upload code, data and metadata to the cloud.

• Force.com SOAP

The Eclipse based Force IDE retrieves and saves code in the cloud using the Metadata API (as opposed to the usual Web Services API. There are options for CRUD and file (zip) based transactions and SOAP, WSDL and WS-I are supported. See the FAQ for more information.

• Heroku API

Although Heroku has a web based source code editor, they also have a Git interface:

sudo gem install heroku heroku create cloud-standards git push heroku

• Reasonably Smart Platform Git

Joyent's Smart platform uses Git exclusively, but the repositories are independent of the management console (which "checks out" the latest version of the code pushed up from developer machines).

• Subversion Protocol

Subversion is a mature server-based version control system with obvious roots in traditional CVS. In addition to working with local file systems and with a dedicated 'svn' protocol (optionally over ssh for security), it also works over HTTP with WebDAV/Delta-V.

• Git Protocol

Git is a distributed version control system whereby every Git working directory is a full-fledged repository with complete history and full revision tracking capabilities, not dependent on network access or a central server. Although native Git relies on SSH for security (which requires additional key management tasks) there is also an option of using Git over HTTP/WebDAV (git-http-push) which would likely be preferable in a cloud environment.

• WebDAV Protocol

WebDAV is a set of extensions to HTTP that allows users to collaboratively edit and manage files on remote World Wide Web servers, essentially making the web read/write. Web servers including Apache has good WebDAV support built in (mod_dav).

• Others?

Management

Requirements

Runtime monitoring and management needs to be catered for by cloud platforms, allowing actors to access performance data, modify application configuration, access billing information, etc.

Actors

• Developers need to manage non-production environments (e.g. development, test, staging).
• System Administrators need to be able to monitor and modify the configuration of running applications as well as change control.
• Managers need to be able to monitor the performance of applications and manage billing and payment.
• Users may have an interest in monitoring the performance of the application over time, learning about outages, upgrades, etc.

Existing Implementations

Runtime management of applications deployed on cloud platforms is currently done via proprietary web interfaces with little automation. There are no known platform management APIs at this time.

• Any?

Update: Preempting any questions about intellectual property, there's no patents that I'm aware of for any of this and with a descriptive/generic name there should be no trademark problems too, so need to concern yourself with a re-run of today's CCIF "XMPP is too stuffy" debacle.