Wombat Diet

The Cloud

If, like me, you have bought books from Amazon.com, and if your credit card details on Amazon are up to date, then you too can have a piece of the cloud.

The cloud, to a geek, is a place where magic happens. It’s where Google lives. It’s how Internet devices connect to each other. It’s how the Internet is represented on whiteboards the world over.

Nick Carr’s fine article on the The End of Corporate Computing recently became available on the web (from behind the MIT Sloan Management Review paywall) and is well worth a read. I never needed the history of consolidation in the electricity generation business to persuade me of the advantages of utilities, of outsourcing headaches or of virtual infrastructure but still, it’s an interesting parallel.

In 2006 Amazon decided to resell services it used internally in its global e-commerce network and it launched Amazon S3 (for Simple Storage Service). In the last couple of years some former colleagues have plugged in and used it for off-site backup for their organizations. The IT world seems divided between those going with it and those resisting; a replay of sorts of what happened when PCs displaced terminals on desks.

My personal tipping point happened just the other day with the release of a plugin for WordPress that automates backup to Amazon S3 storage. Since it backed up both files and database automatically this was something I had to take a look at. Within a few minutes I signed up for Amazon Simple Storage Service. Shortly afterwards a complete copy of Wombatdiet was backed up on Amazon storage and it looks as if my monthly bill for this will be of the order of $0.05.

My other reason for taking the plunge was to get my hands wet, so to speak.

Network storage in the cupboard

Or a bit wetter, perhaps. I’d already used Ubuntu One, but not very seriously. I hadn’t realized at the time that it’s based on Amazon S3, as is Dropbox.

More and more people are reselling cloud storage. Netgear, who make the “silver box” I keep in the cupboard in the hall — my own network storage solution — now sell online backup for later versions of this. The ReadyNAS Vault solution offered is expensive, so I was amused to discover just now that someone has posted information on how to use Amazon S3 directly instead.

Accidents have been known to happen with these boxes (here’s one; I have a spare power supply and memory stick). If the truth is known I have not been as dutiful in my offsite backups as I should be (I use a pair of external USB drives, rsync and a teacher and her locker).

I can now organize daily or weekly incremental backups to the cloud and not feel guilty when it’s pointed out to me that I haven’t swapped the offsite drive for a while. Yes, shamefully, that has happened.

So, what happens when you sign up for Amazon S3 is that you get two things:

Active key ID:
Secret Access Key:

that function like a username and password. You plug them into the software you use to connect to Amazon S3 and you forget them; both are far too long to remember.

And this brings me to the point of this post: Amazon S3 client software.

The fastest and best first way of getting data into Amazon storage, into “buckets”* of one’s own, is to install the Firefox plugin S3Fox and to play with it. Signing up for S3 and installing this can probably be done in 2 minutes, or not much more. To have access to cheap, unlimited storage for online backup as quickly and easily as that was, even now, a little surprising. I’ve paid my dues in organizing remote backups (using rsync) to my own storage over encrypted connections. Now anyone with a web browser and a credit card can easily avoid losing data, or make it shareable on the web, with or without security.

*A bucket is Amazonian, sort of, for top level directory.

So why isn’t everyone using this? I am certain that if I were to walk into my local university and business school one thing that will not have changed in the year since I graduated is: posters offering rewards for the return of lost memory sticks and USB drives, often with heart rending details of dissertation data and other irreplaceable data appended.

The simple answer I believe is that Amazon is missing a trick. S3 is implemented using proprietary technology and it’s not possible for a PC user to connect to Amazon storage by mapping a network drive (there’s some interesting discussion on the Amazon developer forum about why this is so here).

Amazon has left the job of providing interfaces to its storage to the market.

The most sophisticated tool for dealing with Amazon storage I found was Bucket Explorer, the team version of which is surely a must have for any IT department. But if you just want a simple mapped drive for a Windows PC it’s overkill.

There aren’t many simple solutions for that. I looked at three: Tntdrive, Webdrive and Gladinet Cloud Explorer Starter Edition. I didn’t try Jungle Disk as it’s subscription-based and I’d read of people having issues with it, but it’s definitely one to keep an eye on.

Webdrive

I’ve used Webdrive for many years to provide mapped-drive access to FTP and WebDAV sites, principally to use tools like Beyond Compare. Webdrive now supports Amazon S3 sites. I downloaded the latest version and tried it first and found it worked perfectly. I’d have stopped there but but for having tried Cloudberry Explorer, a terrific free application, but which doesn’t do drive mapping. The thought of paying $50 for what seemed like a modest update didn’t seem compelling. It seems South River Technologies likes you to keep paying for the software over and over. A cheaper upgrade from my v8.2 wasn’t on offer.

Gladinet Cloud Explorer

Next, I tried Cloud Explorer. It’s $39.99, a bit cheaper, and apparently more capable. The cute idea here is that the same program can connect you to a variety of different brands of cloud, not just Amazon’s, and all of them look like directories of a mapped drive, making copying between different cloud services straightforward. Not  until after the software is installed, however, do you get to discover the difference between the “always free” and the paid versions. The free version has a limit of 1000 transactions per task (= no serious backups for you, sunshine). More annoyingly, the software didn’t work properly.

One expects file sizes to differ when copying between Linux and PC systems, but one doesn’t expect directory differences between Linux and Amazon s3. I discovered that one needs to pay attention to whether dates and times are preserved (copying between buckets with Cloudberry Explorer I found that datestamps were not preserved). Cloud Explorer didn’t work as expected with dates and times, but worse still it considered 0 byte files with the name index.php as folders, as it did certain .js (javascript) files, thereby putting a spanner in the works of a side-by-side comparison with Beyond Compare. I tried v1.4. Version 2 is in the works and will be a free upgrade. If disk level access to Amazon S3 is all you want this is hardly a compelling proposition. If support for multiple cloud types is your thing then SMEStorage looks more impressive.

Tntdrive

Tntdrive is available in a 10 day trial version and a Pro version. The difference is that the latter doesn’t expire. It’s the first time I’ve encountered non-expiration as a Pro feature. Tntdrive is a paid add-on to a free tool called S3 Browser, which is quite nice but which isn’t a patch on Cloudberry Explorer. Cloudberry, unlike S3 Browser, permits split screen comparison of and synchronization of data sources, including two different buckets, and in the case of two buckets it’s not necessary for the data to be copied via one’s PC. Tntdrive alone required a computer restart when installed. A reboot was again needed when it was uninstalled, and it failed to remove Microsoft Visual C++ 2005 Redistributable when uninstalled. It was slow and suffered similar problems to Gladinet Cloud Explorer, this time with artifacts (files ending in “-$folder$”) cluttering up split-screen directory comparisons with PC data. It’s $39.95. It’s got the best logo though.

Conclusions

Personal cloud storage is a no brainer in princple; having something like S3 and even S3Fox access should make backup of working directories so easy that data loss should be inexcusable. Alas we’re not really there yet. In practice, there are still many pitfalls.

Only one of these solutions works fully and the marginal utility for me is not currently worth the asking price.

I set out wanting to use a mapped drive and one of my favourite file and directory synchronization tools, Beyond Compare. Along the way, I discovered that Superflexible File Synchronizer, already supports Amazon S3. I’ve been using Superflexible as a powerful adjunct to Beyond Compare (and vice versa) for some years. Until BC supports S3 I’ll just get by with Cloudberry Explorer for the side-by-side comparisons and syncs involving S3 storage.

Plug: Superflexible isn’t just a remarkably powerful tool. It’s got a great business model that is in stark contrast to that for Webdrive. Once you pay for the software updates are free forever. On top of which, the support from the author, Tobias Giesen, is outstanding. I’ve only had 3 or 4 questions over the years but the responses were immediate. The reason I didn’t discover sooner that Superflexible was supporting S3 was because I stopped upgrading it. It was working for the automated tasks I’d assigned it to do and I had stopped getting email about updates (previously sent by Betanews.com) and I never went looking.

AlternativeTo will be worth keeping an eye on to track new and improved alternatives to Gladinet, e.g. and similar tools.

Meanwhile, Google Gdrive and SkyDrive for more on Google and Microsoft’s catch-up efforts to S3. As of this month Google Docs now supports uploading of other file types and more changes are planned.

In the process of writing this post I came across myDisk: $5/year for 2Gb of WebDAV storage with a mapped-drive accessible across platforms (Windows, Mac, Linux, iPhone). You can map the drive by connecting to https://mydisk.se/demo, using the username demo and password demo, and access it via the web at www.demo.mydisk.se using the same credentials. MyDrive is similar but free for 2Gb. All in all, pretty much what I’d like but with relatively low storage limits. I am not sure how sustainable such individual services are, after all the point of cloud storage is risk avoidance, but I was reminded of a couple of things:

1. I’ve already got good WebDAV storage (6Gb for files for me alone) provided by my email provider, Fastmail.fm, and,
2. Since I converted to a Family account it’s possible for me to set up shared storage for use by family members (as well as a shared address book, distribution lists and mail folders); the main initial advantage was simpler administration.

I’ve used Fastmail’s file storage in a small way and should try to do more with it, perhaps to help synchronize contact and calendar information.

Related posts:

1. A Cloud In The Home The day is coming when we will not need elaborate backup arrangements, such as I described yesterday, for each individual computer. OSNews had a story...
2. WordPress To Amazon S3 Backup I posted recently about finally signing up for Amazon S3 (A Cloud Of One's Own: Amazon S3 Drive Mapping) to take advantage of a new...
3. Religion: Backup I've never filled in my religion on a form as Backup. Someday I will, just for fun. Real backup is automated, redundant and rotated off-site...
4. Microsoft Worried: Cheap Dogfood, $300m Promotion Who seriously believes Vista is as good as it ought to be? Bill Gates thinks the I'm a Mac, and I'm a PC ads are...
5. World Daylight Map A review of some world time applications that display a map of the world showing daytime and nightime....

Related posts brought to you by Yet Another Related Posts Plugin.

Tags: Amazon, Amazon S3, cloud storage, drive mapping, fastmail, gladinet, network storage, ReadyNAS, superflexible, tntdrive, Ubuntu, webdrive, Wordpress