Getting started with OpenStack in your lab

Having recently finished building my new home lab I wanted to put the second server to good use by installing OpenStack (the first is running VMware ESXi 5.0 with Windows 7, Windows 8, Windows 8 Server and Ubuntu 12.04 LTS virtual machines). I figured many of you would benefit from a detailed walkthrough so here it is (without warranty, liability, support, etc).

The two black boxes on the left are HP Proliant MicroServer N36L’s with modest AMD Athlon(tm) II Neo 1.3GHz dual-core processors and 8GB RAM and the one on the right is an iomega ix4-200d NAS box providing 8TB of networked storage (including over iSCSI for ESXi should I run low on direct attached storage). There’s a 5 port gigabit switch stringing it all together and a 500Mbps CPL device connecting it back up the house. You should be able to set all this up inside 2 grand. Before you try to work out where I live, the safe is empty as I don’t trust electronic locks.

IMG 1198

Download Ubuntu Server (12.04 LTS or the latest long term support release) and write it to a USB key — if you’re a Mac OS X only shop then you’ll want to follow these instructions. Boot your server with the USB key inserted and it should drop you straight into the installer (if not you might need to tell the BIOS to boot from USB by pressing the appropriate key, usually F2 or F10, at the appropriate time). Most of the defaults are OK but you’ll probably want to select the “OpenSSH Server” option in tasksel unless you want to do everything from the console, but be sure to tighten up the default configuration if you care about security. Unless you like mundane admin tasks then you might want to enable automatic updates too. Even so let’s ensure any updates since release have been applied:

sudo apt-get update
sudo apt-get -u upgrade

Next you’ll want to install DevStack (“a documented shell script to build complete OpenStack development environments from RackSpace Cloud Builders“), but first you’ll need to get git:

sudo apt-get install git

Now grab the latest version of DevStack from GitHub:

git clone git://github.com/openstack-dev/devstack.git

And run the script:

cd devstack/; ./stack.sh

The first thing it will do is ask you for passwords for MySQL, Rabbit, a SERVICE_TOKEN and SERVICE_PASSWORD and finally a password for Horizon & Keystone. I used the (excellent) 1Password to generate passwords like “sEdvEuHNNeA7mYJ8Cjou” (the script doesn’t like special characters) and stored them in a secure note.

The script will then go and download dozens of dependencies, which are conveniently packaged by Ubuntu and/or the upstream Debian distribution, run setup.py for a few python packages, clone some repositories, etc. While you wait you may as well go read the script to understand what’s going on. At this point the script failed because /opt/stack/nova didn’t exist. I filed bug 995078 but the script succeeded when I ran it for a second time — looks like it may have been a glitch with GitHub.

You should end up with something like this:

Horizon is now available at http://10.0.1.10/
Keystone is serving at http://10.0.1.10:5000/v2.0/
Examples on using novaclient command line is in exercise.sh
The default users are: admin and demo
The password: qqG6YTChVLzEHfTDzm8k
This is your host ip: 10.0.1.10
stack.sh completed in 431 seconds.

If you browse to that address you’ll be able to log in to the console:

Openstack login

That will drop you into the Admin section of the OpenStack Desktop (Horizon) where you can get an overview and administer instances, services, flavours, images, projects, users and quotas. You can also download OpenStack and EC2 credentials from the “Settings” pages.

Openstack console

Switch over to the “Project” tab and “Create Keypair” under “Access & Security” (so you can access any instances you create):

Openstack keygen

The key pair will be created and downloaded as a .pem file (e.g. admin.pem).

Now select “Images & Snapshots” under “Manage Compute” you’ll be able to launch the cirros-0.3.0-x86_64-uec image which is included for testing. Simply click “Launch” under “Actions”:

Openstack project

Give it a name like “Test”, select the key pair you created above and click “Launch Instance”:

Openstack launch

You’ll see a few tasks executed and your instance should be up and running (Status: Active) in a few seconds:

Openstack spawning

Now what? First, try to ping the running instance from within the SSH session on the server (you won’t be able to ping it from your workstation):

$ ping 10.0.0.2
PING 10.0.0.2 (10.0.0.2) 56(84) bytes of data.
64 bytes from 10.0.0.2: icmp_req=1 ttl=64 time=0.734 ms
64 bytes from 10.0.0.2: icmp_req=2 ttl=64 time=0.585 ms
64 bytes from 10.0.0.2: icmp_req=3 ttl=64 time=0.588 ms

Next let’s copy some EC2 credentials over to our user account on the server so we can use the command line euca-* tools. Go to “Settings” in the top right and then the “EC2 Credentials” tab. Now “Download EC2 Credentials”, which come in the form of a ZIP archive containing an X.509 certificate (cert.pem) and key (pk.pem) pair as well as a CA certificate (cacert.pem) and an rc script (ec2rc.sh) to set various environment variables which tell the command line tools where to find these files:

Openstack ec2

While you’re at it you may as well grab your OpenStack Credentials which come in the form of an rc script (openrc.sh) only. It too sets environment variables which can be seen by tools running under that shell.

Openstack rc

Let’s copy them (and the key pair from above) over from our workstation to the server:

scp b34166e97765499b9a75f59eaff48b98-x509.zip openrc.sh admin.pem samj@10.0.1.10:~

Stash the EC2 credentials in ~/.euca:

mkdir ~/.euca; chmod 0700 ~/.euca; cd ~/.euca
cp ~/b34166e97765499b9a75f59eaff48b98-x509.zip ~/.euca; unzip *.zip

Finally let’s source the rc scripts:

source ~/.euca/ec2rc.sh
source ~/openrc.sh

You’ll see the openrc.sh script asks you for a password. Given this is a dev/test environment and we’ve used a complex password, let’s modify the script and hard code the password by commenting out the last 3 lines and adding a new one to export OS_PASSWORD:

# With Keystone you pass the keystone password.
#echo "Please enter your OpenStack Password: "
#read -s OS_PASSWORD_INPUT
#export OS_PASSWORD=$OS_PASSWORD_INPUT
export OS_PASSWORD=qqG6YTChVLzEHfTDzm8k

You probably don’t want anyone seeing your password or key pair so let’s lock down those files:

chmod 0600 ~/openrc.sh ~/admin.pem

Just make sure the environment variables are set correctly:

echo $EC2_USER_ID
42
echo $OS_USERNAME
admin

Finally we should be able to use the EC2 command line tools:

euca-describe-instances 
RESERVATION r-8wvdh1c7 b34166e97765499b9a75f59eaff48b98 default
INSTANCE i-00000001 ami-00000001 test test running None (b34166e97765499b9a75f59eaff48b98, ubuntu) 0 m1.tiny 2012-05-05T13:59:47.000Z nova aki-00000002 ari-00000003 monitoring-disabled 10.0.0.2 10.0.0.2 instance-store

As well as the openstack command:

openstack list server
+--------------------------------------+------+--------+------------------+
| ID | Name | Status | Networks |
+--------------------------------------+------+--------+------------------+
| 44a43355-7f95-4621-be61-d34fe53e50a8 | Test | ACTIVE | private=10.0.0.2 |
+--------------------------------------+------+--------+------------------+

You should be able to ssh to the running instance using the IP address and key pair from above:

ssh -i admin.pem -l cirros 10.0.0.2
$ uname -a
Linux cirros 3.0.0-12-virtual #20-Ubuntu SMP Fri Oct 7 18:19:02 UTC 2011 x86_64 GNU/Linux

That’s all for today — I hope you find the process as straightforward as I did and if you do follow these instructions then please leave a comment below (especially if you have any tips or solutions to problems you run into along the way).

HOWTO: Set up OpenVPN in a VPS

If, like me, you want to do any or all of the following things, you’ll want to tunnel your traffic over a VPN to a remote location:

  • Access media services restricted by geography (Hulu, FOX, BBX, etc.)
  • Bypass draconian censorship
  • Conceal your identity/location/etc.
  • Protect your machine from attackers
  • etc.

You could of course use a commercial service like AlwaysVPN in which case you typically pay ($5-10) per month or (~$1) per gigabyte, but many will prefer to run their own service. FWIW AlywaysVPN has worked very well for me but it’s time to move on.

First thing’s first you’ll want to find yourself a remote Linux server, and the easiest way to do so is to rent a virtual private server (VPS) from one of a myriad providers. No point spending more than 10 bucks a month on it as you don’t need much in the way of resources (only bandwidth). Check out lowendbox.com for VPS deals under $7/month or just run with a BurstNET VPS starting at $5.95/month for a very reasonable resource allocation (including a terabyte of bandwidth!).

Once you’ve placed your order and passed their fraud detection systems (which includes an automated callback on the number you supply) you’ll have to wait 12-24 hours for activation, upon which you’ll receive an email with details for accessing your vePortal control panel as well as the VPS itself (via SSH). You’ll get 2 IP addresses and I dedicated the second to both inbound and outbound traffic for VPS clients (which live on a 10.x RFC1918 subnet and access the Internet via SNAT).

If you didn’t already do so when signing up then choose a sensible OS in your control panel (“OS Reload”) like Ubuntu 8.04 – a Long Term Support release which means you’ll be getting security fixes for years to come – or better yet, 10.4 if it’s been released by the time you read this (it’s the next LTS release). Do an “apt-get install unattended-upgrades” and you ought to be fairly safe until 2015. You’re also going to need your TUN/TAP device(s) enabled which involves another trip to the control panel (“Enable Tun/Tap”) and/or a helpdesk ticket (http://support.burst.net/). If /dev/net/tun doesn’t exist then you can create it with “mknod /dev/net/tun c 10 200”.

To install OpenVPN it’s just a case of doing “apt-get install openvpn”… you could also download a free 2-user version of OpenVPN-AS from http://openvpn.net/ but I found it had problems trying to load netfilter modules that were already loaded so YMMV. If you want support or > 2 users you’ll be looking at a very reasonable $5/user – you’re on your own with the free/open source version but there’s no such limitations either.

OpenVPN uses PKI but rather than go to a certificate authority we’ll set one up ourselves. EasyRSA is included to simplify this process so it’s just a case of doing something like this:

cd /usr/share/doc/openvpn/examples/easy-rsa/2.0. ./vars./clean-all./build-ca./build-dhopenvpn --genkey --secret ta.key./build-key-server server./build-key client1./build-key client2./build-key client3

It’ll ask you a bunch of superflous information like your country, state, city, organisation, etc. but I just filled these out with ‘.’ (blank rather than the defaults) – mostly so as not to give away information unnecessarily to anyone who asks. The only field that matters is the Common Name which you probably want to leave as ‘server’, ‘client1’ (or some other username like ‘samj’), etc. When you’re done here you’ll want to “cp keys/* /etc/openvpn” so OpenVPN can see it.

Next you’ll want to configure the OpenVPN server and client(s) based on examples in /usr/share/doc/openvpn/examples/sample-config-files. I’m running two – one “Faster” one for the best performance when I’m on a “clean” connection (which uses udp/1194) and another “Compatible” one for when I’m on a restricted/corporate network (which shares tcp/443 with HTTPS). I did a “zcat server.conf.gz > /etc/openvpn/faster.conf” and edited it so it (when filtered with cat faster.conf | grep -v "^#" |grep -v "^;" | grep -v "^$") looks something like this:

local 173.212.x.xport 1194proto udpdev tunca ca.crtcert server.crtkey server.keydh dh1024.pemserver 10.9.0.0 255.255.255.0ifconfig-pool-persist faster-ipp.txtpush "redirect-gateway def1 bypass-dhcp"push "dhcp-option DNS 8.8.8.8"push "dhcp-option DNS 8.8.4.4"client-to-clientkeepalive 10 120tls-auth ta.key 0cipher BF-CBCcomp-lzouser nobodygroup nogrouppersist-keypersist-tunstatus /var/log/openvpn/faster-status.loglog-append /var/log/openvpn/faster.logverb 3mute 20

Noteworthy points:

  • local specifies which IP to bind to – I used the second (of two) that BurstNET had allocated to my VPS so as to keep the first for other servers, but you could just as easily use the first and then put clients behind the second, which would appear to be completely “clean”.
  • We’re using “tun” (tunneling/routing) rather than “tap” (ethernet briding) because BurstNET use venet interfaces which lack MAC addresses rather than veth. Wasn’t able to get bridging up and running, as originally intended.
  • There are various hardening options but to keep it simple I just run as nobody:nogroup and use tls-auth (having generated the optional ta.key with “openvpn –genkey –secret ta.key” above).
  • Pushing Google Public DNS addresses to clients as they won’t be able to use their local resolver addresses once connected. Also telling them to route all traffic over the VPN (which would otherwise only intercept traffic for a remote network).
  • Configured separate log files and subnets (10.8.0.0/24 and 10.9.0.0/24) for the “faster” and “compatible” instances.

The “compatible.conf” file varies only with the following lines:

port 443proto tcpserver 10.8.0.0 255.255.255.0status /var/log/openvpn/compatible-status.loglog-append /var/log/openvpn/compatible.log

Next you’ll want to copy over client.conf from /usr/share/doc/openvpn/examples/sample-config-files (but set ‘AUTOSTART=”compatible faster”‘ in /etc/default/openvpn so it’s ignored by the init scripts).

clientdev tunproto udpremote 173.212.x.x 1194resolv-retry infinitenobindpersist-keypersist-tunca burstnet-ca.crtcert burstnet-client.crtkey burstnet-client.keyns-cert-type servertls-auth burstnet-ta.key 1cipher tls-cipher DHE-RSA-AES256-SHA:DHE-DSS-AES256-SHAcipher BF-CBCcomp-lzoverb 3

As I’ve got a bunch of different connections on my clients I’ve prepended “burstnet-” to all the files and called the main config files “BurstNET-Faster.conf” and “BurstNET-Compatible.conf” (which appear in the Tunnelblick menu on OS X as “BurstNET-Faster” and “BurstNET-Compatible” respectively – thanks to AlwaysVPN for this idea). The only difference for BurstNET-Compatible.conf is:

proto tcpremote 173.212.x.x 443

You’re now almost ready for the smoke test (and indeed should be able to connect) but you’ll end up on a 10.x subnet and therefore unable to communicate with anyone. The fix is “iptables -t nat -A POSTROUTING -s 10.8.0.0/255.255.255.0 -j SNAT --to-source 173.212.x.x” (where the source IP is one of those allocated to you).

Being paranoid though I want to lock down my server with a firewall, which for Ubuntu typically means ufw (you’ll need to “apt-get install ufw” if you haven’t already). My ufw rules look something like this:

# ufw statusStatus: activeTo                         Action  From--                         ------  ----Anywhere                   ALLOW   1.2.3.41194/udp                   ALLOW   Anywhere443/tcp                    ALLOW   Anywhere

The first rule allows me to access the server from home via SSH and 1194/udp and 443/tcp allow VPN clients in. To allow the clients to access the outside world we’re going to have to rewrite their traffic to come from a public IP (which is called “SNAT”), but first you’ll want to enable forwarding by setting DEFAULT_FORWARD_POLICY="ACCEPT" in /etc/default/ufw. Then it’s just a case of adding something like this to /etc/ufw/before.rules:

# nat Table rules*nat:POSTROUTING ACCEPT [0:0]# SNAT traffic from VPN subnet.-A POSTROUTING -s 10.8.0.0/255.255.255.0 -j SNAT --to-source 173.212.x.x-A POSTROUTING -s 10.9.0.0/255.255.255.0 -j SNAT --to-source 173.212.x.x# don't delete the 'COMMIT' line or these nat table rules won't be processedCOMMIT

You may need to enable UFW (“ufw enable”) and if you lose access to your server you can always disable UFW (“ufw disable”) using the rudimentary “Console” function of vePortal.

On the client side you’ve got support for (at least) Linux (e.g. “openvpn --config /etc/openvpn/BurstNET-Faster.conf“), Mac and Windows and there’s various GUIs (including OpenVPN GUI for Windows and Tunnelblick for Mac OS X). I’m (only) using Tunnelblick, and after copying Tunnelblick.app to /Applications I just need to create a ~/Library/openvpn directory and drop these files in there:

  • BurstNET-Compatible.conf
  • BurstNET-Faster.conf
  • burstnet-ca.crt
  • burstnet-client.key
  • burstnet-client.crt
  • burstnet-ta.key

When Tunnelblick’s running I have a little black tunnel symbol in the top right corner of my screen from which I can connect & disconnect as necessary.

I think that’s about it – hopefully there’s nothing critical I’ve missed but feel free to follow up in the comments if you’ve anything to add. I’m now happily streaming from Hulu and Fox in the US, downloading Amazon MP3s (using my US credit card), and have a reasonable level of anonymity. If I was in Australia I’d have little to fear from censorship (and there’s virtually nothing they can do to stop me) and as my machine has a private IP I’m effectively firewalled.

Update: It seems that my VPS is occasionally restarted (which is not all that surprising) and forgets about its tun/tap device (which is). The device node itself is still visible in the filesystem, but with no driver to connect to in the kernel it doesn’t work and OpenVPN doesn’t start. You can test if your tun device is working using cat:

WORKING:

cat /dev/net/tun

cat: /dev/net/tun: File descriptor in bad state

NOT WORKING:

cat /dev/net/tun

cat: /dev/net/tun: No such device

I’ve also noticed that ufw may need to be manually started with a ‘ufw enable’. Hope that saves you some time diagnosing problems!

Installing VMware tools in Ubuntu 8.04 (hardy)

So like me you’ve been hanging out for another Long Term Support (LTS) Ubuntu release and having arrived last month (8.04) you’ve got it up and running in VMware (Fusion in my case).

To make VMware tools install you need to:

  • Virtual Machine->Install VMware Tools (that’s the easy part)
  • apt-get install build-essential linux-headers-$(uname -r)
  • mount /cdrom
  • cd /cdrom
  • cp VMwareTools*.tar.gz /tmp
  • cd /tmp
  • tar xzf VMwareTools*
  • cd vmware-tools-distrib
  • ./vmware-install.pl
  • Press enter for everything until it won’t go any further because it wants the real location of your kernel headers. Give it ‘/lib/modules/2.6.24-16-server/build/include’ and then keep pressing enter again until you get back to your prompt.

VMware Tools should start up (except perhaps for the advanced networking guff).

Update: Added extraction steps.

Update: The same process works for Debian 4.0 (etch)

SSHKeychain 0.8.2 Post Install Problem on Leopard

It seems SSHKeychain breaks on (recent?) Leopard builds because it wants to find a group for each user (eg samj:samj):

#!/bin/sh
chown -R $USER:$USER "$2/SSHKeychain.app"
#chown root:admin "$2/SSHKeychain.app/Contents/Resources/TunnelRunner"
#chmod u+s "$2/SSHKeychain.app/Contents/Resources/TunnelRunner"

You’ll want to change the second ‘$USER’ to id -gn so it picks up your group name (eg ‘staff’) by itself, and while you’re there you can comment out the two TunnelRunner lines if you want to set up tunnels on privileged ports and don’t care about the security implications of setuid root binaries. You can do this by copying SSHKeychain.pkg from the mounted disk image, and right clicking to ‘Show Package Contents’… then you can browse for Content->Resources->postinstall, or apply this diff:

--- SSHKeychain.pkg/Contents/Resources/postinstall 2008-06-09 09:25:03.000000000 +0200
+++ SSHKeychainFixed.pkg/Contents/Resources/postinstall 2008-06-09 09:19:47.000000000 +0200
@@ -1,4 +1,4 @@
#!/bin/sh
-chown -R $USER:$USER "$2/SSHKeychain.app"
+chown -R $USER:`id -gn` "$2/SSHKeychain.app"
#chown root:admin "$2/SSHKeychain.app/Contents/Resources/TunnelRunner"
#chmod u+s "$2/SSHKeychain.app/Contents/Resources/TunnelRunner"

Hope this saves someone some time.

Google Apps – On in 60 seconds

To celebrate today’s Google Apps launch here’s a screencast I whipped up showing just how easy it is with Google Apps to get ‘On in 60 seconds’:

Update: On advice from Dave Girouard (VP, Google Enterprise) we’ve changed the name to ‘On in 60 seconds’ from the somewhat less enthralling ‘Setting up Google Apps’.

Update: Thanks for the amazing response – 20,000 views! Let’s hope Google Apps is as popular as its video!

Update: The music in the video was that of Michael Johns, now one of the favourites in American Idol!

Compiling bash-3.0 on Interix

So you’ve followed my instructions for updating config.guess for Interix 5.2 (the version shipping with Windows 2003 Server R2) and now you want to compile something. Interix ships with C Shell (csh) and Korn Shell (ksh) but lacks the Bourne Again Shell (bash) – the shell most Linux users will be familiar with, so why not start there? From Start->Programs->Subsystem for UNIX-based Applications start either ‘Korn Shell’ or ‘C Shell’. You’ll end up in ‘/dev/fs/C/Documents and Settings/’ (this is your home directory, ‘~’) and the root is ‘%SystemRoot%\SUA’. Download bash-3.0 and extract it somewhere sensible (like /usr/src). You’ll need to ‘gunzip bash-3.0.tar.gz’ first and then do ‘tar xf bash-3.0.tar’ as it’s not gtar so it doesn’t understand ‘z’ (gzip) and ‘j’ (bzip2) options. Change to the ‘bash-3.0’ directory and ‘./configure –prefix=/usr/local/bash-3.0’, then ‘make’ and ‘make install’. Now it’s just a case of creating a link to ‘%SystemRoot%\posix.exe /u /c /usr/local/bash-3.0/bin/bash -l’ in the start menu. When you click on this link you’ll end up with a window that looks and behaves like a command window, only with a red/yellow/blue logo.

You may get errors like ‘error retrieving current directory: getcwd: cannot access parent directories: Undefined error: 0’ – I suspect these are due to permissions problems, or issues with spaces in paths. I’d be interested if someone has a better explanation, especially if it came with a fix.

Compiling on Interix 5.2 (Windows 2003 Server R2 SUA)

The soon-to-be-released Windows 2003 Server R2 includes features that were previously shipped as Services for Unix (SFU) – perhaps the most interesting of which is Subsystem for UNIX-based Applications (SUA).

At the time of writing, if you want to see what R2 is all about you’ll need to download the Windows Server 2003 R2 Release Candidate 1 (RC1) Software, which can only be installed on a trial version of Windows 2003 Server (available as a download with R2). After you’ve installed the ‘Subsystem for UNIX-based Applications’ Windows component (Add/Remove Programs applet) you will need to download and install 200Mb or so of ‘Utilities and SDK for UNIX-based Applications’. See Installing and Using Utilities and SDK for UNIX-based Applications. Be sure to do a custom install as the default doesn’t install the GNU utilities (among other things).

Once you’ve got it all installed you’ll probably want to start compiling software (like bash), but when you run configure you’ll get a message like this:

 checking build system type... ./support/config.guess: unable to guess system type
    
    This script, last modified 2005-09-19, has failed to recognize
    the operating system you are using. It is advised that you
    download the most up to date version of the config scripts from
    
      http://savannah.gnu.org/cgi-bin/viewcvs/*checkout*/config/config/config.guess and
      http://savannah.gnu.org/cgi-bin/viewcvs/*checkout*/config/config/config.sub
    
    If the version you run (./support/config.guess) is already up to date, please
    send the following data and any information you think might be
    pertinent to  in order to provide the needed
    information to handle your system.
    
    config.guess timestamp = 2005-09-19
    
    uname -m = x86
    uname -r = 5.2
    uname -s = Interix
    uname -v = SP-9.0.3790.2049
    
    /usr/bin/uname -p = Intel_x86_Family15_Model4_Stepping8
    /bin/uname -X     =
    System = Interix
    Node = aosdubvsvr03
    Release = 5.2
    Version = SP-9.0.3790.2049
    Machine = x86
    Processor = Intel_x86_Family15_Model4_Stepping8
    HostSystem = Windows
    HostRelease = SP1
    HostVersion = 5.2
    
    hostinfo               =
    /bin/universe          =
    /usr/bin/arch -k       =
    /bin/arch              =
    /usr/bin/oslevel       =
    /usr/convex/getsysinfo =
    
    UNAME_MACHINE = x86
    UNAME_RELEASE = 5.2
    UNAME_SYSTEM  = Interix
    UNAME_VERSION = SP-9.0.3790.2049
    configure: error: cannot guess build type; you must specify one

This is because the config.guess script only knows about Interix versions 3 and 4 – you’ll have to tell it about version 5:

$ diff config.guess config.guess.new
     782c782
          ---
     >     x86:Interix*:[345]*)

Once you’ve done this you should be able to start compiling…