Cloud services provide excellent value but it’s easy to underestimate the challenge of getting large quantities of data to the cloud. When moving very large quantities of data, even the fastest networks are surprisingly slow. And, many companies have incredibly slow internet connections. Back in 1996 MInix author and networking expert, Andrew Tanenbaum said “Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway”. For large data transfers, it’s faster (and often cheaper) to write to local media and ship the media via courier.
This morning the Beta release Amazon Web Services Import/Export was announced. This service essentially implements sneakernet allowing the efficient transfer of very large quantities of data into or out of the AWS Simple Storage Service. This initial beta release only supports import but the announcement reports that “the service will be expanded to include export in the coming months”.
To use the service, the data is copied to a portable storage device formatted using NTFS, FAT, ext2, or ext3 file systems. The manifest that describes the data load job is digitally signed using the sending users AWS access secret key and shipped to Amazon for loading. Load charges are:
Device Handling
· $80.00 per storage device handled.
Data Loading Time
· $2.49 per data-loading-hour. Partial data-loading-hours are billed as full hours.
Amazon S3 Charges
· Standard Amazon S3 Request and Storage pricing applies.
· Data transferred between AWS Import/Export and Amazon S3 is free of charge (i.e. $0.00 per GB).
In addition to allowing much faster data ingestion, AWS Import/Export reduces networking costs since there is no charge for the transfer of data from the Import/Export service and S3. A calculator is provided to compare estimated electronic transfer costs vs import/export costs. It’s a clear win for larger data sets.
–jrh
James Hamilton, Amazon Web Services
1200, 12th Ave. S., Seattle, WA, 98144
W:+1(425)703-9972 | C:+1(206)910-4692 | H:+1(206)201-1859 | james@amazon.com
H:mvdirona.com | W:mvdirona.com/jrh/work | blog:http://perspectives.mvdirona.com
Thanks u r information
Hi James,
There are attempts to use a lot of existing B/W to do such transfers – the key is to
schedule your transfers when there is lot more ‘available’ b/w on backbones.
For instance some work can be found here:
http://research.tid.es/nikos/index.php?option=com_content&task=view&id=21&Itemid=
So it’s not entirely clear what is cheaper – to send via Internet, albeit with smart scheduling or send physical media.
Thanks!
v
I agree Andrew. It’s all about the data.
–jrh
jrh@mvdirona.com
I love it. No more roach motel (data goes in but doesn’t come out). Total game changer.