Recently we have been doing work with Amazon S3 for scalable storage of assets we use in some of our applications. These files have been sitting on SAN disks and served out via web servers. It’s high performance storage dedicated to mass file storage. In all there is around 18TB of data and billions of files.
The inertia of moving these files to he cloud has been overcome by Amazons AWSImport. It’s a pretty simple concept, ship them a drive with your files and then have them import these files into Amazon S3 Buckets. It’s snail mail – at terabyte scale.
Firstly to cover the service, it’s simple and it works. If you grab S3Fox and install it with Firefox the whole process can be done in the browser. Step 1 is to create a manifest.txt which is basically a set of attributes for the ‘job’. The manifest defines the destination bucket, device ID (we used the device serial number), access control and also where to ship the drive back to.
With the manifest file you email AWS Import with a subject line of ‘CREATE JOB’ and the manifest.txt attached to the email. In generally less than a days time AWSImport comes back with a Job Number. You load this job number into S3FOX and you can create a SIGNATURE. Place the SIGNATURE on the hard drive you are sending to AWS and then ship it to the address. Very simple stuff.
Simple things still fail 
As we were sending alot of drives to AWSImport there was one we (read I) mixed up the manifest.txt and SIGNATURE. Luckly AWSImport has a CHANGE JOB functionality. You can email AWSImport with the job id and subject line CHANGE JOB and you can apply new manifests and signatures to the job. This saves you having one shot at goal.
We also found the AWS team were flexible on modifications to manifests and import processing. A good example is we needed Content-Encoding set to gzip for all imported files (as the majority were gzipped) and then we wanted to use the API to change it for a subset of files later. Within about a couple of weeks the AWS team added setContentEncodingForGzFiles:Yes attribute for manifest files (Thanks Eric@AWS).
We are based in Australia so when were last in the US some of the team went to Best Buy and bought a number of 1.5TB esata drives. This gave us the US power plugs that Amazon AWS required. The second mistake we made was forgetting to pack the power cord (duh!). Now this is where Amazon customer support starts to scare me with regards to how good it is. Put yourself in this situation with any online/offline company you deal with. You courier a drive to them for processing and you forgot to pack the power cable and they don’t have a converter. You then then get an email that looks like this.
“DELAYED JOB” – That’s not good. I was getting visions that the drive is coming back to me, I have lost cash on the courier and then lost time on the turn-around for getting data ready for production (worse than the cash). But then check out how Eric handles the situation.
Very Classy. It’s like I am dealing with a supplier down the road and when we (not I ) messed up instead of shooting me he saved me. I especially like “please let me know if this is unacceptable”. Eric even gives me the option if I don’t want to be saved!!
I wonder how any of the other cloud providers would have responded (even if they had a service similar to AWSImport)? I wonder how many of them have found employees like Eric.
These guys (through Eric) are killing it.
