Cloud User Shell (cush) multi-call executable prototype released

After much discussion of what goes on below the water line (in the provider/enabler space), hopefully this prototype release will refocus energy and get the creative juices flowing above it (with the users themselves). The idea is to follow Unix’s example (as explained below) by seamlessly blending cloud resources into existing working environments. See examples below for some ideas of what is possible – my favourite is taking a local file and pipelining it through two web based services (the first is a rudimentary spam filter and the second appends legal notices), but being able to interact with HTTP in general from the command line is quite cool. Support for true bidirectional pipes (eg via CONNECT rather than simplex POSTs) and remote instantiation of pipelines is… well… in the pipeline. As is dealing with content types (verbs and nouns having already been catered for), mapping arguments to the query string and a flexible authentication infrastructure (all of which will require some thought and creation of static interfaces).

Cloud User Shell (cush) is a multi-call executable, bringing cushy, RESTful cloud control to the command line.

Philosophy

It follows the wildly successful Unix philosophy:

This is the Unix cloud computing philosophy:

  1. Write programs components that do one thing and do it well.
  2. Write programs components to work together.
  3. Write programs components to handle text HTTP streams, because that is a universal interface.

RESTful

It is also RESTful, clearly delineating the 3 sides of the REST Triangle:

  • Verbs (constrained) e.g. GET
  • Nouns (unconstrained) e.g. URIs (https://samj.net/])
  • Content Types (constrained, optional */*) e.g. HTML (text/html)

Installation

To install the python prototype cush.py

  • Unix
  • cush.py upgrade periodically to stay up-to-date

Usage

Usage: cush [verb] [noun]
  or: [verb] [noun]

eg: get http://samj.net

Cloud User SHell (cush) is a multi-call binary (like BusyBox) that combines
many useful cloud computing utilities into a single executable. Most people
will create a link to cush for each function they wish to use and cush will

act like whatever it was invoked as, but it can also be called directly and
passed the command as the first parameter.

Currently defined functions:
@, @@, delete, get, head, ls, post, propfind, put, rm

Examples

All of these work alongside your favourite shell, eg bash

Send files to/from the cloud and process them as if they were local:

$ echo "The quick brown fox jumps over the lazy dog." | put http://local/uploads/brown.txt
$ get http://localhost/uploads/brown.txt
The quick brown fox jumps over the lazy dog.

$ get http://local/uploads/brown.txt | sed -e ‘s/brown/red/’ | put http://local/uploads/red.txt

Even when they are:

$ get http://local/uploads/red.txt > /tmp/red.txt

$ get file:///tmp/red.txt
The quick red fox jumps over the lazy dog.

Dereference URIs with shorthand syntax (equivalent to GET command):

$ @ http://localhost/uploads/brown.txt

The quick brown fox jumps over the lazy dog.

Manipulate data in the cloud by listing (ls) and removing (rm) URIs like local files:

$ ls http://localhost/uploads/colours/

brown.txt
red.txt
$ rm http://localhost/uploads/colours/red.txt
$ ls http://localhost/uploads/colours/
brown.txt

Set up HttpPipelines (like Unix pipelines) for data processing:

The following shows an rfc822 email message being pipelined through two URIs for spam filtering (which adds a X-Spam-Flag: YES header if it finds ‘Viagra’) and a legal notice (which appends ‘This message is confidential.’).

$ cat rfc822-spam.txt |@ http://localhost/cgi-bin/spam-vapouriser.cgi |@ http://localhost/cgi-bin/legal-notifier.cgi

X-Spam-Flag: YES
Received: by 10.150.96.14 with HTTP; Tue, 5 Aug 2008 04:52:14 -0700 (PDT)
Message-ID: <21606dcf0808050452l74eee336g6bac459463c17f79@mail.gmail.com>

Date: Tue, 5 Aug 2008 13:52:14 +0200
From: “Sam Johnston” <samj@samj.net>
To: “Sam Johnston” <samj@samj.net>
Subject: Cloud Computing User Shell (cush)

MIME-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
Delivered-To: samj@samj.net

Check out Viagra – it rocks.

This message is confidential.

Obtain and manipulate structured data (eg titles of the items in a feed, weather in Sydney):

$ get http://feeds.feedburner.com/samj | xpath //rss/channel/item/title

Found 24 nodes:
— NODE —
<title>Cloud standards: not so fast…</title>– NODE —
<title>Cloud computing and Open Source software</title>– NODE —

$ get http://weather.yahooapis.com/forecastrss?p=ASXX0112 | xpath /rss/channel/item/yweather:condition

Found 1 nodes:
— NODE —

Obtain information about URIs and technical debugging:

$ head https://samj.net/
content-length: 0
server: GFE/1.3
last-modified: Mon, 25 Aug 2008 22:04:36 GMT
etag: "55ed4707-7cbc-4208-9cbc-cbc6105bc682"

cache-control: max-age=0 private
date: Thu, 28 Aug 2008 00:22:45 GMT
content-type: text/html; charset=UTF-8

Cloud computing and Open Source software

Here is a post I just made to the cloud computing group in the ‘Power of opensource!!!’ thread. I thought was relevant for blog readers as well as it asks whether IT is still nervous about Open Source software:

In terms of Open Source, I believe the vast majority of cloud computing solutions today are run on open source software stacks, or at least open source operating systems (eg Linux). On the client side a large and increasing number of users are using open source browsers (eg Firefox).

I don’t see anything that will change this on the server side – at these scales you just need to be able to get in and tweak the stack from the network driver through the database itself and you’ve got a snowflake’s chance in hell of doing that with most proprietary systems.

I also believe this trend will continue, particularly on the client side where the cost of the hardware will continue to drop to the point where software licensing cannot be carried on top of it – even the first generation of CherryPal, Eee PC and XO machines, at around $200, can’t support a $100 OS license (plus apps etc).

In terms of the actual licensing, now that software is merely ‘performed’ rather than ‘conveyed’ the ‘triggers’ for most licenses requiring release of modifications don’t fire so licenses like the Affero GPL are appearing to address this ‘loophole’.

Finally, vendors who have found themselves in the unfortunate position of being in competition with open source products ought to be very careful about calling into question the value that it provides – we’ve heard it all before and are using it extensively anyway.

In summary, Open Source is here and here to stay. Cloud computing is user-centric so users don’t need to worry so much about what software they are using, but the overwhelming majority of software on both server and client side will be Open Source before long (much of it already is).

A new risk for Open Source today is the ‘service provider loophole’, whereby changes can be made to server side software that is never distributed and thus kept secret. We’ve covered this ground before, for example when using databases like MySQL with proprietary software, but much of the innovation is now taking place in the applications of the software rather than the software itself. I hope to see licenses like the Affero GPL (which address this weakness) being widely adopted by purveyors of network based software. The best example I’ve seen of this so far is Identi.ca‘s assault on Twitter (which is incredibly useful but notoriously unreliable).

Single command Django installer for OS X

So you want to see what all the fuss around Django is about? To get the latest bleeding edge snapshot (as discussed here, here, here, here, here, here, here and here) you just need to run these commands (as root), per the official install instructions:

/usr/bin/svn co http://code.djangoproject.com/svn/django/trunk/ /usr/local/django-trunk
ln -s /usr/local/django-trunk/django /Library/Python/2.5/site-packages/django
ln -s /usr/local/django-trunk/django/bin/django-admin.py /usr/local/bin

For the lazy, get root (eg sudo -s) and run:

curl https://s3.amazonaws.com/media.samj.net/devel/django.sh | sh

If it’s already there then this script will update to the latest revision.

Making SSL work with Apache 2 on Mac OS X with CAcert

Getting SSL up and running on OS X is not too difficult these days. First you need to tell it to read the SSL config file (removing red lines, adding green lines):

— /etc/apache2/httpd.conf 2008-06-11 03:42:25.000000000 +0200
+++ /etc/apache2/httpd.conf.dist 2008-06-11 04:15:15.000000000 +0200
@@ -470,7 +470,7 @@
#Include /private/etc/apache2/extra/httpd-default.conf

# Secure (SSL/TLS) connections
-#Include /private/etc/apache2/extra/httpd-ssl.conf
+Include /private/etc/apache2/extra/httpd-ssl.conf
#
# Note: The following must must be present to support
# starting without SSL on platforms with no /dev/random equivalent

Then you need to fix this config file for your environment:

— /private/etc/apache2/extra/httpd-ssl.conf.dist 2008-06-11 03:43:21.000000000 +0200
+++ /private/etc/apache2/extra/httpd-ssl.conf 2008-06-11 04:03:50.000000000 +0200
@@ -22,9 +22,9 @@
# Manual for more details.
#
#SSLRandomSeed startup file:/dev/random 512
-#SSLRandomSeed startup file:/dev/urandom 512
+SSLRandomSeed startup file:/dev/urandom 512
#SSLRandomSeed connect file:/dev/random 512
-#SSLRandomSeed connect file:/dev/urandom 512
+SSLRandomSeed connect file:/dev/urandom 512

#
@@ -75,8 +75,8 @@

General setup for the virtual host

DocumentRoot “/Library/WebServer/Documents”
-ServerName http://www.example.com:443
-ServerAdmin you@example.com
+ServerName secure.samj.net:443
+ServerAdmin xxxx@samj.net
ErrorLog “/private/var/log/apache2/error_log”
TransferLog “/private/var/log/apache2/access_log”

@@ -125,6 +125,7 @@

Makefile to update the hash symlinks after changes.

#SSLCACertificatePath “/private/etc/apache2/ssl.crt”
#SSLCACertificateFile “/private/etc/apache2/ssl.crt/ca-bundle.crt”
+SSLCACertificateFile “/private/etc/apache2/server-ca.crt”

Certificate Revocation Lists (CRL):

Set the CA revocation path where to find CA CRLs for client

@@ -143,6 +144,8 @@

issuer chain before deciding the certificate is not valid.

#SSLVerifyClient require
#SSLVerifyDepth 10
+SSLVerifyClient require
+SSLVerifyDepth 2

Access Control:

With SSLRequire you can do per-directory access control based

Notice that I’m using client certificates for authentication but you can comment out the SSLCACertificateFile, SSLVerifyClient and SSLVerifyDepth options if you don’t need this. If you do you’ll want to grab the root from CAcert:

# curl -o server-ca.crt https://www.cacert.org/certs/root.crt

You’ll want to generate random nubmers (key) and a certificate signing request (csr) in order to get a certificate (crt) file, and despite most information on the topic this can be done in one command:

# openssl req -newkey rsa:2048 -nodes -keyout server.key -out server.csrGenerating a 2048 bit RSA private key
………+++
……………………………………………………………+++
writing new private key to ‘server.key’
—–
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter ‘.’, the field will be left blank.
—–
Country Name (2 letter code) [AU]:
State or Province Name (full name) [Some-State]:New South Wales
Locality Name (eg, city) []:Sydney
Organization Name (eg, company) [Internet Widgits Pty Ltd]:Australian Online Solutions Pty Ltd
Organizational Unit Name (eg, section) []:Security
Common Name (eg, YOUR name) []:secure.samj.net
Email Address []:xxxx@samj.net

Please enter the following ‘extra’ attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:

Actually in the case of CAcert.org everything except the common name is ignored so you can leave it as defaults.

For testing we’ll use a script which prints all the environment variables (this is what I was after for my certificate authentication anyway):

# cat /Library/WebServer/CGI-Executables/printenv
#!/bin/bash
echo “Content-type: text/plain”
echo “”
printenv

And when you browse to your machine (eg https://secure.samj.net/) you should see something like this:

SSL_SERVER_A_KEY=rsaEncryption
SSL_CLIENT_VERIFY=SUCCESS
SSL_SESSION_ID=A6C2F73FBFBB30AF927947D03B8E61AF26E0C4C68CB98F3B9CB7EB6E6ED78147
SERVER_SIGNATURE=
SSL_CLIENT_S_DN_Email=xxxx@debian.org
SSL_CLIENT_A_SIG=sha1WithRSAEncryption
SSL_CLIENT_I_DN_Email=support@cacert.org
HTTP_KEEP_ALIVE=300
SSL_VERSION_LIBRARY=OpenSSL/0.9.7l
HTTP_USER_AGENT=Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-GB; rv:1.9) Gecko/2008053008 Firefox/3.0
SERVER_PORT=443
HTTP_HOST=secure.samj.net
SSL_CIPHER_ALGKEYSIZE=256
SSL_SERVER_I_DN=/O=CAcert Inc./OU=http://www.CAcert.org/CN=CAcert Class 3 Root
SSL_CLIENT_M_VERSION=3
DOCUMENT_ROOT=/Library/WebServer/Documents
HTTP_ACCEPT_CHARSET=ISO-8859-1,utf-8;q=0.7,*;q=0.7
SCRIPT_FILENAME=/Library/WebServer/CGI-Executables/printenv
HTTPS=on
REQUEST_URI=/cgi-bin/printenv
SSL_SERVER_I_DN_OU=http://www.CAcert.org
SSL_CLIENT_A_KEY=rsaEncryption
SCRIPT_NAME=/cgi-bin/printenv
SSL_SERVER_S_DN=/CN=secure.samj.net
SSL_VERSION_INTERFACE=mod_ssl/2.2.8
SSL_CLIENT_I_DN_OU=http://www.cacert.org
SSL_CIPHER_EXPORT=false
HTTP_CONNECTION=keep-alive
SSL_SERVER_I_DN_O=CAcert Inc.
SSL_CLIENT_S_DN=/CN=Sam Johnston/emailAddress=xxxx@debian.org/emailAddress=xxxx@samj.net/emailAddress=66e1c629ca065f0cead0ac9bee8e4cb016f93cb7
SSL_COMPRESS_METHOD=NULL
REMOTE_PORT=50392
PATH=/usr/bin:/bin:/usr/sbin:/sbin
SSL_CLIENT_I_DN_O=Root CA
SSL_CLIENT_M_SERIAL=0551B7
SSL_CIPHER=DHE-RSA-AES256-SHA
PWD=/Library/WebServer/CGI-Executables
SERVER_ADMIN=xxxx@samj.net
SSL_SERVER_A_SIG=sha1WithRSAEncryption
SSL_CLIENT_V_START=Jun 11 01:38:03 2008 GMT
SSL_SERVER_M_SERIAL=56C8
SSL_PROTOCOL=SSLv3
SSL_CLIENT_I_DN_CN=CA Cert Signing Authority
HTTP_ACCEPT_LANGUAGE=en-gb,en;q=0.5
SSL_SERVER_S_DN_CN=secure.samj.net
HTTP_ACCEPT=text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
REMOTE_ADDR=::1
SHLVL=1
SERVER_NAME=secure.samj.net
SSL_SERVER_M_VERSION=3
HTTP_PRAGMA=no-cache
SSL_CLIENT_V_END=Jun 11 01:38:03 2010 GMT
SERVER_SOFTWARE=Apache/2.2.8 (Unix) mod_ssl/2.2.8 OpenSSL/0.9.7l DAV/2
QUERY_STRING=
SERVER_ADDR=::1
SSL_SERVER_V_END=Jun 11 01:48:07 2010 GMT
SSL_CLIENT_I_DN=/O=Root CA/OU=http://www.cacert.org/CN=CA Cert Signing Authority/emailAddress=support@cacert.org
SSL_CLIENT_S_DN_CN=Sam Johnston
GATEWAY_INTERFACE=CGI/1.1
SERVER_PROTOCOL=HTTP/1.1
HTTP_CACHE_CONTROL=no-cache
HTTP_ACCEPT_ENCODING=gzip,deflate
SSL_SERVER_I_DN_CN=CAcert Class 3 Root
REQUEST_METHOD=GET
HTTP_COOKIE=__utma=64622253.189158232302809900.1213012569.1213012569.1213087976.2; __utmz=64622253.1213012569.1.1.utmcsr=blogger.com|utmccn=(referral)|utmcmd=referral|utmcct=/rearrange
SSL_SERVER_V_START=Jun 11 01:48:07 2008 GMT
SSL_CLIENT_V_REMAIN=730
SSL_CIPHER_USEKEYSIZE=256
_=/usr/bin/printen

That’s it for this morning’s lesson.

Compiling bash-3.0 on Interix

So you’ve followed my instructions for updating config.guess for Interix 5.2 (the version shipping with Windows 2003 Server R2) and now you want to compile something. Interix ships with C Shell (csh) and Korn Shell (ksh) but lacks the Bourne Again Shell (bash) – the shell most Linux users will be familiar with, so why not start there? From Start->Programs->Subsystem for UNIX-based Applications start either ‘Korn Shell’ or ‘C Shell’. You’ll end up in ‘/dev/fs/C/Documents and Settings/’ (this is your home directory, ‘~’) and the root is ‘%SystemRoot%\SUA’. Download bash-3.0 and extract it somewhere sensible (like /usr/src). You’ll need to ‘gunzip bash-3.0.tar.gz’ first and then do ‘tar xf bash-3.0.tar’ as it’s not gtar so it doesn’t understand ‘z’ (gzip) and ‘j’ (bzip2) options. Change to the ‘bash-3.0’ directory and ‘./configure –prefix=/usr/local/bash-3.0’, then ‘make’ and ‘make install’. Now it’s just a case of creating a link to ‘%SystemRoot%\posix.exe /u /c /usr/local/bash-3.0/bin/bash -l’ in the start menu. When you click on this link you’ll end up with a window that looks and behaves like a command window, only with a red/yellow/blue logo.

You may get errors like ‘error retrieving current directory: getcwd: cannot access parent directories: Undefined error: 0’ – I suspect these are due to permissions problems, or issues with spaces in paths. I’d be interested if someone has a better explanation, especially if it came with a fix.

Compiling on Interix 5.2 (Windows 2003 Server R2 SUA)

The soon-to-be-released Windows 2003 Server R2 includes features that were previously shipped as Services for Unix (SFU) – perhaps the most interesting of which is Subsystem for UNIX-based Applications (SUA).

At the time of writing, if you want to see what R2 is all about you’ll need to download the Windows Server 2003 R2 Release Candidate 1 (RC1) Software, which can only be installed on a trial version of Windows 2003 Server (available as a download with R2). After you’ve installed the ‘Subsystem for UNIX-based Applications’ Windows component (Add/Remove Programs applet) you will need to download and install 200Mb or so of ‘Utilities and SDK for UNIX-based Applications’. See Installing and Using Utilities and SDK for UNIX-based Applications. Be sure to do a custom install as the default doesn’t install the GNU utilities (among other things).

Once you’ve got it all installed you’ll probably want to start compiling software (like bash), but when you run configure you’ll get a message like this:

 checking build system type... ./support/config.guess: unable to guess system type
    
    This script, last modified 2005-09-19, has failed to recognize
    the operating system you are using. It is advised that you
    download the most up to date version of the config scripts from
    
      http://savannah.gnu.org/cgi-bin/viewcvs/*checkout*/config/config/config.guess and
      http://savannah.gnu.org/cgi-bin/viewcvs/*checkout*/config/config/config.sub
    
    If the version you run (./support/config.guess) is already up to date, please
    send the following data and any information you think might be
    pertinent to  in order to provide the needed
    information to handle your system.
    
    config.guess timestamp = 2005-09-19
    
    uname -m = x86
    uname -r = 5.2
    uname -s = Interix
    uname -v = SP-9.0.3790.2049
    
    /usr/bin/uname -p = Intel_x86_Family15_Model4_Stepping8
    /bin/uname -X     =
    System = Interix
    Node = aosdubvsvr03
    Release = 5.2
    Version = SP-9.0.3790.2049
    Machine = x86
    Processor = Intel_x86_Family15_Model4_Stepping8
    HostSystem = Windows
    HostRelease = SP1
    HostVersion = 5.2
    
    hostinfo               =
    /bin/universe          =
    /usr/bin/arch -k       =
    /bin/arch              =
    /usr/bin/oslevel       =
    /usr/convex/getsysinfo =
    
    UNAME_MACHINE = x86
    UNAME_RELEASE = 5.2
    UNAME_SYSTEM  = Interix
    UNAME_VERSION = SP-9.0.3790.2049
    configure: error: cannot guess build type; you must specify one

This is because the config.guess script only knows about Interix versions 3 and 4 – you’ll have to tell it about version 5:

$ diff config.guess config.guess.new
     782c782
          ---
     >     x86:Interix*:[345]*)

Once you’ve done this you should be able to start compiling…