03 February 2010

Private cloud security is no security at all

It's ironic that the purveyors of "Private Cloud" sell their wares on the premise of enhanced privacy and security - a totally unjustified claim which is too often accepted without question - and that they are quick to dismiss the huge benefit of the armies of security boffins employed by "public" cloud vendors (whose future is largely dependent on keeping customer data safe). It's also very convenient for them that the term itself is disparaging of "public" cloud in the same way that "Blog With Integrity" badges imply that the rest of us are somehow unethical (one of the main reasons I personally have and will always dislike[d] it).

It is with that in mind that I was intrigued by Reuven Cohen's announcement today regarding Enomaly, Inc. having recently joined the Intel Cloud Builder Program (whatever that is). It was these two quotes that I found particularly questionable regarding their Enomaly ECP product:
  1. Intel was among the first to full(sic) understand the opportunity in enabling a truly secure virtualized cloud computing environments(sic) for service providers and Telco's.
  2. Our work with the Intel Cloud Builder Program will help to accelerate our efforts to deliver a massively-scalable, highly-available, high-security cloud platform to our customers.
The reason I'm naturally suspicious of such claims is that I've already discovered a handful of critical security vulnerabilities in this product (and that's without even having to look beyond the startup script - a secure-by-default turbogears component that was made insecure through inexplicable modifications):
  1. CVE-2008-4990 Enomaly ECP/Enomalism: Insecure temporary file creation vulnerabilities
  2. CVE-2009-0390: Argument injection vulnerability in Enomaly Elastic Computing Platform (ECP)
  3. Enomaly ECP/Enomalism: Multiple vulnerabilities in enomalism2.sh (redux)
I had to dig a little (but not much) deeper for the silent update remote command execution vulnerability. I also inadvertently discovered another serious security vulnerability (sending corporate BestBuy credentials in the clear over the Internet to a 3rd party service), which as it turns out was also developed by Enomaly, Inc. It's only natural that I would be suspicious of any future security claims made by this company.

It doesn't help my sentiment either that every last trace of the Open Source ECP Community Edition was recently scrubbed from the Internet without notice, leaving angry customers high and dry, purportedly pending the "rejigging [of their] OSS strategy". While my previous attempts to fork the product as Freenomalism failed when we were unable to get the daemon to start, having the code in any condition is better than not having it at all. In my opinion this is little more than blatantly (and successfully I might add) taking advantage of the Open Source community for as long as necessary to get the product into the limelight. Had they not filled this void others would certainly have done so, and the Open Cloud would be better off today as a result.

As part of cloud standards work I was interested in taking a look at the "secure" mechanism they developed for distributing virtual machines:
VMcasting is an automatic virtual machine deployment mechanism based on RSS2.0 whereby virtual machine images are transferred from a server to a client which securely delivers files containing a technical specification and virtual disk image.
Another bold claim that initially appeared justified by a simple but relatively sensible embedding of crytpographically strong checksums into descriptor and manifest files that were in turn digitally signed using GPG. Unfortunately no consideration was given to the secure retrieval of the archive itself (nor the RSS feed listing the archives for that matter), nor were signatures actually required by the specification, meaning that it would be trivial for an attacker to insert their own unsigned packages and/or replace existing signed packages with modified, unsigned ones. Or replaying an older, signed version of an insecure workload for that matter.

Fortunately an attacker need not even go to these lengths as despite acknowledging the need for digital signatures in the VMcasting specification, none of the security features appear to have been implemented in Enomaly ECP itself. Worse still, it won't even let you use SSL if you're sensible enough to try:
if url[0].lower not in ("http", "ftp"):
raise E2UndefinedError(_("Unknown scheme in package URL."))
Think you're safe if you keep everything on your own network (that's the whole point, right?). Don't be so sure, as the vmfeed module quietly registers these HTTP URLs for you:
Sure enough if you retrieve the first URL you'll get a feed of "virtual appliances" like this one (delivered over HTTP from Amazon S3 no less) and as expected, if you untar it you'll see that there's no signatures whatsoever. Don't get me started on the myriad vulnerabilities no doubt present within the appliances themselves given their age - packaging applications as virtual machines is a notoriously bad idea and one that I hope will be overrun by containers/platforms in the not too distant future.

But wait, there's more - being able to run workloads of your choice (e.g. trojan horses, network scanners, etc.) within your victim's network is one thing, and being able to obtain and reverse engineer their existing workloads (given there's no catering for authentication) another, but taking over the management system itself is where there's real fun to be had. Fortunately all you need to do is set the MIME type to application/python-egg rather than application/enomalism2-xvm2 and this little chestnut gets invoked, quietly unzipping and forcibly installing the supplied python module:
elif self.get_mime()==EGG_MIME:
tx.update("Installing Python egg.", 90)
shutil.move(filename, target)
The vmcast_modules feed currently advertises the e2_drivemounter, e2_exception and e2_phone_home modules which are all available for download, again over HTTP, from http://enomaly.com/fileadmin/eggs/.

Anyway I'm sure there'll be backpedalling, downplaying, shooting-the-messenger, etc. which is why you're reading this here rather than in a vulnerability announcement. While the bugs are obviously unconfirmed this still illustrates my point nicely - don't take it for granted that private cloud offerings are secure, and in the unlikely event that the systems themselves are secure, don't assume you or your provider can run them in a more secure fashion than a "public" cloud provider could.

Incidents like this go a long way towards realising one of my predictions for 2010 (or should I say @philww's "considered prediction") in that Private clouds will be discredited by year end.

Update: Following Enomaly, Inc.'s CEO denying access to the source, a "Strategic Advisor and Board Member" downplayed the issues (below), once again claiming "many of the items above have been addressed in [other] editions" and once again failing to provide any details or code for verification. Finally, the CTO tweeted "Seriously, reviewing software you've never tried is like reviewing book you've never read or a movie you've never watched. #Fail" and promptly blocked me.

Given Enomaly claimed to have 15,000 users some 18 months ago and 15,000 organisations more recently (both officially and unofficially), if they're to be believed then that's a lot of people left high and dry by the outstanding vulnerabilities, not to mention their having pulled the source. It's also more than enough motivation to announce the release of OpenECP: Open Elastic Computing Platform.

Whether the community run with it is yet to be seen but in any case it fills the void left by Enomaly ECP, throws stranded customers a lifeline and may just coax the company into being better behaved with respect to security issues and the open source community. Time will tell.

Update: According to Secunia "The vendor disputes the problems: reportedly, the vulnerable module is not used in any of their current products and was only used in the now unsupported 'Community Edition'". This conflicts with their "VM Repository Management" screencast which clearly shows both the offending VMcasting protocol and the offending insecure URLs in use in their commercial product:


  1. Hi Sam,

    Firstly, thanks for taking so much time to have tried, used, bug tested and analyzed Enomaly's open source edition software. Our partnership with Intel is focused on our two new editions of the product - our Service Provider Edition and our soon to be release High Assurance Edition, each have significant new and enhanced functionality over the version you have used. Many of the items above have been addressed in those editions. We will review your comments above for future inclusion into our product road map. Regarding our open source activities, we are a small company and have put our focus, in the short term, into our two new Service Provider editions. We expect to come back to the community with new contributions in the future, but for now, we just aren't able to do it all. We will take your comments as a keen interest in seeing more enomaly open source contributions in the future.

    Lastly, regarding private clouds being discredited by year end. As with any ground breaking technology, cloud technology will need to fill many different needs in the market and therefore will need to provide different options - both private and public clouds. Even electricity has applications where it needs to be 'off grid' and independent of the main 'open' grid. Cloud computing will be the same.

  2. Sam,
    OpenECP is the one Enomaly will use in their partnership with Intel ? and also the one that Miriam talking about?

    which is more secure and cover all the vulnerabilities you discovered ?

  3. Hmm, I read another post (http://www.cloudave.com/link/openecp-an-enomaly-ecp-fork) and i noticed that you who forked the ECP and created the new brand of OpenECP ...


Note: only a member of this blog may post a comment.