NoSQL “movement” roadblocks HTML5 WebDB

Today’s rant is coming between me and a day of skiing so I’ll keep it brief. While trying to get to the bottom of why I can’t enjoy offline access to Google Apps & other web-based applications with Gears on Snow Leopard I came across a post noting Chrome, Opera to support html5 webdb, FF & IE won’t. This seemed curious as HTML5 is powering on towards last call and there are already multiple implementations of both applications and clients that run them. Here’s where we’re at:

  • Opera: “At opera, we implemented web db […] it’s likely we will [ship it] as people have built on it”
  • Google [Chrome]: “We’ve implemented WebDB … we’re about to ship it”
  • Microsoft [IE]: “We don’t think we’ll reasonably be able to ship an interoperable version of WebDB”
  • Mozilla [Firefox]: “We’ve talked to a lot of developers, the feedback we got is that we really don’t want SQL […] I don’t think mozilla plans to ship it.”

Of these, Microsoft’s argument (aside from being disproven by existing interoperable implementations) can be summarily dismissed because offline web applications are a direct competitor to desktop applications and therefore Windows itself. As if that’s not enough, they have their own horse in this race that they don’t have to share with anyone in the form of Silverlight. As such it’s completely understandable (however lame) for them to spread interoperability FUD about competing technology.

Mozilla’s argument that “we really don’t want SQL” is far more troublesome and posts like this follow an increasingly common pattern:

  1. Someone proposes SQL for something (given we’ve got 4 decades of experience with it)
  2. Religious zealots trash talk SQL, offering a dozen or so NoSQL alternatives (all of which are in varying stages of [early] development)
  3. “My NoSQL db is bigger/better/faster than yours” debate ensues
  4. Nobody does anything

Like it or not, SQL is a sensible database interface for web applications today. It’s used almost exclusively on the server side already (except perhaps for the largest of sites, and even these tend to use SQL for some components) so developers are very well equipped to deal with it. It has been proven to work (and work well) by demanding applications including Gmail, Google Docs and Google Calendar, and is anyway independent of the underlying database engine. Ironically work has already been done to provide SQL interfaces to “NoSQL” databases (which just goes to show the “movement” completely misses the point) so those who really don’t like SQLite (which happens to drive most implementations today) could conceivably create a drop-in replacement for it. Indeed power users like myself would likely appreciate a browser with embedded MySQL as a differentiating feature.

In any case the API [cs]hould be versioned so we can offer alternatives like WebSimpleDB in the future. Right now though the open web is being held back by outdated standards and proprietary offerings controlled by single vendors (e.g. Adobe’s AIR and Microsoft’s Silverlight) are lining up to fill in the gap. Those suggesting “it’s worth stepping back” because “there are other options that should be considered” which “might serve those needs better” would want to take a long, hard look at whether their proposed alternatives are really ready for prime time, or indeed even necessary. To an outsider trying to solve real business problems today a lot of it looks like academic wankery.

Towards a Flash free YouTube killer (was: Adobe Flash penetration more like 50%)

A couple of days ago I wrote about Why Adobe Flash penetration is more like 50% than 99%, which resulted in a bunch of comments as well as a fair bit of discussion elsewhere including commentary from Adobe’s own John Dowdell. It’s good to see some healthy discussion on this topic (though it’s a shame to see some branding it “more flash hate” and an AC poster asking “How much did M$ pay you for this”).

Anyway everyone likes a good demonstration so I figured why not create a proof-of-concept YouTube killer that uses HTML 5’s video tag?

Knowing that around 20% of my visitors already have a subset of HTML 5 support (either via Safari/WebKit or Firefox 3.1 beta), and that this figure will jump to over 50% shortly after Firefox 3.1 drops (over 50% of my readers use Firefox and over 90% of them run the most recent versions), I would already be considering taking advantage of the new VIDEO tag were I to add videos to the site (even though, as a Google Apps Premier Edition user I already have a white label version of YouTube at http://video.samj.net/).

Selecting the demo video was easy – my brother, Michael Johns, did a guest performance on American Idol last Wednesday and as per usual it’s already popped up on YouTube (complete with a HD version). Normally YouTube use Flash’s FLV codec but for HD they sensibly opted for H.264 which is supported by Safari (which supports anything QuickTime supports – including Ogg Vorbis for users with Perian installed). Getting the video file itself is just a case of browsing to the YouTube page, going to Window->Activity and double clicking the digitally signed link that looks something like ‘http://v4.cache.googlevideo.com/videoplayback‘ which should result in the video.mp4 file being downloaded (though now Google are offering paid downloads they’re working hard to stop unsanctioned downloading).

On the other hand Firefox 3.1 currently only supports Ogg Vorbis for licensing/patent reasons as even Reasonable and Non-Discriminatory (RAND) licensing is unreasonable and discriminatory for free and open source software. Unfortunately the W3C working group infamously removed a recommendation that implementors ‘should’ support Ogg Vorbis and Theora for audio and video respectively. Currently a codec recommendation is conspicuously absent from the HTML 5 working draft. So what’s a developer to do but make both Ogg and H.264 versions available? Fortunately transcoding MP4 to OGG (and vice versa) is easy enough with VLC, resulting in a similar quality but 10% smaller file (video.ogg).

The HTML code itself is quite straightforward. It demonstrates:

  • A body onLoad function to switch to Ogg for Firefox users
  • YouTube object fallback for unsupported browsers (which in turn falls back to embed)
  • A simple JavaScript Play/Pause control (which could easily be fleshed out to a slider, etc.)
  • A simple JavaScript event handler to show an alert when the video finishes playing
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN"
"http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">

<html xmlns=”http://www.w3.org/1999/xhtml&#8221; xml:lang=”en”>
<head>
<title>Towards a Flash free YouTube killer…</title>
</head>

<!– Basic test for Firefox switches to Ogg Theora –>
<!– TTest could be arbitrarily complex and/or run on the server side –>
<body onLoad=”if (/Firefox/.test(navigator.userAgent)){ document.getElementsByTagName(‘video’)[0].src = ‘video.ogg’; }”>
<h1>Michael Johns &amp; Carly Smithson – The Letter</h1>
<p>(Live At American Idol 02/18/2009) HD
(from <a href=”http://www.youtube.com/watch?v=LkTCFo8XfAc”>YouTube</a&gt;)</p>

<!– Supported browsers will use the video code and ignore the rest –>
<video src=”video.mp4″ autoplay=”true” width=”630″ height=”380″>
<!– If video tag is unsupported by your browser legacy code used –>

</video>

<!– Here’s a script to give some basic playback control –>
<script>
function playPause() {
var myVideo = document.getElementsByTagName(‘video’)[0];
if (myVideo.paused)
myVideo.play();
else
myVideo.pause();
}
</script>
<p><input type=button onclick=”playPause()” value=”Play/Pause”></p>

<!– Here’s an event handler which will tell us when the video finishes –>
<script>
myVideo.addEventListener(‘ended’, function () {
alert(‘video playback finished’)
} );
</script>
<p>By <a href=”https://samj.net/”>Sam Johnston</a> of
<a href=”http://www.aos.net.au/”>Australian Online Solutions</a></p>
</body>
</html>

This file (index.html) and the two video files above (video.mp4 and video.ogg) are then uploaded to Amazon S3 (at http://media.samj.net/) and made available via Amazon CloudFront content delivery network (at http://media.cdn.samj.net/). And finally you can see for yourself (bearing in mind that to keep the code clean no attempts were made to check the ready states so either download the files locally or be patient!):

Towards a Flash free YouTube killer…

Why Adobe Flash penetration is more like 50% than 99%

Slashdot discussed PC PRO’s “99% Flash Player Penetration – Too Good to be True?” article today which prompted me to explain why I have always been dubious of Adobe’s claim that “Flash content reaches 99.0% of Internet viewers“. Here’s the claim verbatim:

Adobe Flash Player is the world’s most pervasive software platform, used by over 2 million professionals and reaching 99.0% of Internet-enabled desktops in mature markets as well as a wide range of devices.

The methodology section asks more questions than it answers but fortunately we don’t even need to go into details about how 3-6% margins of error can lead to a 99% outcome. Here’s why the claim is bogus:

Mobile devices

“Internet viewers” clearly includes users of mobile devices like the iPhone and yet they are conveniently only counting “desktops in mature markets”. Fortunately this little chestnut which has just been removed from Adobe’s own Mobile & Devices Develoer Center is still available in the Google cache:

So by Adobe’s own reckoning there’s about 1 billion PCs to 3 billion mobile devices. That reconciles with other research, but the latest news is that we’ve just passed 4 billion (or 60% of the entire global population today). Let’s be conservative anyway and use Adobe’s own figures. So that’s 1 billion PCs to 3 billion mobiles.

99% of 25% is still less than 25% overall…

but that’s not really fair because many (most?) of those mobile devices aren’t yet Internet enabled. Well every iPhone sells with Internet and none of them have Flash so those 13 million Internet-enabled mobile devices already knock nearly a point and a half off their 99% claim. Google’s Android’s just getting started and its first incarnation (the T-Mobile G1) has already sold a million devices with many more to come this year.

The iPhone alone knocks it down to 97.5%

On the other hand, Flash Lite will have shipped on a billion devices by March so:

1 billion PCs + 1 billion mobiles = 2 billion out of 4 billion total = 50%

As if it’s not enough for Adobe that large segments of the mobile device market are currently out of their reach they have another two emerging technologies which will erode their penetration rates:

HTML 5

Features like the “video” and “canvas” tags, timed media playback, offline storage, messaging/networking, etc. (which have previously only been possible via 3rd party plugins) will soon be supported natively by the browser. I’m a W3C Invited Expert in the HTML 5 working group (primarily monitoring the web application developments) and it’s great to see demos like offline mobile Gmail already starting to appear. It will also be nice not having to download and install a plugin to view video content in sites like YouTube, Facebook and MySpace (arguably the main reason why anyone would install Flash in the first place).

Netbooks

Over 10 million of Netbooks (“Internet Notebooks”) shipped last year and another 35 million are expected to ship this year. So far Flash penetration hasn’t been too bad with Windows/Linux on x86 devices but the next generation will run a myriad operating systems on alternatives to x86 like ARM. Flash will certainly have trouble maintaining its usual penetration rates, and will have to do so with less incentive to install thanks to HTML 5 support in the browsers.

In summary, Flash has its place. It is constantly evolving and there are things that it does that can be difficult or impossible to do otherwise (e.g. video capture). However, if you choose to deploy Flash then you are choosing to exclude some potential users and it’s hard to say how many as it depends on many factors (your specific audience’s demographics, devices, locations, etc.). Adobe’s figures are not perfect so the only way to reliably know how many users you are turning away is to measure it after the fact. If you don’t need these advanced features then opt for a Native Web Application, or confine Flash objects to pages where it is absolutely required.

Update: There’s a handful of (mostly) developers discussing this article over at Y Combinator’s hacker news, generally in defense of Adobe’s figures (including a comment from Adobe’s own John Dowell). This is not “more flash hate“, rather an observation that things that seem “too good to be true” usually are. Take a quick glance at this screenshot from Adobe.com:

What does it tell you? Did you bother to read beyond the title? Would you have any doubts about using Flash for your site after seeing it? Should you?

Is an iPhone user really not an “Internet viewer” when iPhones ship with a full-featured browser and a data plan? Does “desktop” mean desktop computer or desktop metaphor? Are laptops and netbooks included? What happens if you drop the “mature markets” restriction? What does “over 2 million professionals” really mean anyway? How do the numbers for “real world” tests compare to the survey results?

Basically this clever marketing exercise (which no doubt overcomes the #1 objection to Flash in many instances) raises more questions than it answers.

Writing Valid XHTML 1.1

There’s a lot of good reasons to write valid XHTML (even if the vast majority of sites don’t bother):

  • Your site will render better, faster and more consistently across all browsers.
  • Your layout will be pushed from tables and tags to CSS, separating data from presentation and reducing maintenance costs.
  • Computers (most notably, search engines) will be able to parse and make sense of your content easier than they might otherwise have been able to.
  • You’re supporting standards compliance (which translates to freedom for you and your users) and you can advertise valid XHTML using the W3C logos:

Once you’ve gone to the effort of writing valid XHTML and CSS and the W3C Markup Validation Service (http://validator.w3.org/) is happy with your efforts you’ll still want to make sure you’re serving your content with the right mime-type: application/xhtml+xml, but only to browsers that support it (and ask for it via the HTTP Accept: header)… most notably not IE6 😐

It’s unfortunate those who care about standards compliance have to jump through hoops by implementing content negotiation, but it’s not too hard to do…. for example in PHP you can do something like this:

<?php
  header("Vary: Accept");
  if (stristr($_SERVER["HTTP_ACCEPT"], "application/xhtml+xml") || stristr($_SERVER["HTTP_USER_AGENT"], "W3C_Validator"))
    header("Content-Type: application/xhtml+xml; charset=utf-8");
  else
     header("Content-Type: text/html; charset=utf-8");
?>

Notice that the validator won’t send an Accept header by default. You can force it to, but I’m just checking for the user agent; if you don’t you’ll get a warning about the mime-type even if the document is valid (and you’re serving it correctly).

Note also that neither this blog nor the Australian Online Solutions blog are valid XHTML (and I’m not rewriting Blogger templates to make them compliant), but TrustSaaS.com is and validates cleanly.