Neil Turner's Blog

Blogging about technology and randomness since 2002

Uploading a literal truckload of data

Lizzie's truck

Imagine you work for a company that has 10 petabytes (10 million gigabytes) of data in its data centre. Now, imagine your company wants to move that to a cloud storage service like Amazon Web Services.

You’ve got a fast 1 Gbps link to the internet, which, at maximum throughput, will deliver 125 megabytes each second. That means it’ll take 80 million seconds to upload all the data. Distilled down, that’s 1.3 million minutes, 22,222 hours, 925 days, or just over two and a half years. And that’s assuming maximum throughput with no breaks in connection or downtime. Yeowch.

Thankfully Amazon has a solution, called the Snowmobile. It’s a lorry with a data centre inside a shipping container, sent to your workplace. You hook it up to your existing data centre using a fibre-optic link, and transfer your data across. When you’re done, the lorry takes the container back to Amazon’s own massive data centre and plugs it in, for you to use. At worst, you’d probably be done within the week.

As the Guardian article says, it’s an old solution to an ongoing problem. Internet speeds are increasing, but so is storage capacity, and the rise of big data means that we need more and more bandwidth to move it around.

Back in the days before my parents got broadband internet, I wanted to share around 250 megabytes of data with a friend. Nowadays I’d probably stick it in Dropbox, send a sharing link and be done with it, but uploading 250 megabytes on a 56k modem connection would have taken hours. Over 15 hours in fact, based on a 4.5 kilobytes per second throughput. And at a penny a minute, it would have cost my parents about £9 in call charges, because back in the 1990s we paid for our internet by the minute.

Instead, I burned a CD-R and posted it. I think I sent it at about 4pm on a Thursday, and it arrived at 9:30am on Friday. At around 18 hours, it was marginally slower than sending it over the internet, but cheaper – the combined cost of the CD-R, padded envelope and postage was less than £9. And it avoided having to explain the extra large phone bill to my parents.

It’s funny how old solutions come back around. And despite what Ted Stevens said, sometimes the internet can be both a big truck, and a series of tubes.

Yes, that is a photo of one of Lizzie’s toys in the header.

One Comment

  1. Although not a truckload, we used the snowball where they ship you a mini nas server that came in a huge protected box that we plugged in, transferred the data over and then sent back. Worked really well.