Offline disk import and the OmNomNom machine
By Benjamin Bechtolsheim, Product Marketing Manager
Cross-posted from the Google
Cloud Platform Blog
Yesterday, we announced that we are
expanding
our offline disk import service to better serve users globally. With new disk upload
centers in Switzerland, Japan and India, as well as our US center, it’s easier for people
around the world to import large data sets by mailing hard drives to us rather than sending
hundreds of terabytes over their slow, expensive or unreliable Internet connection.
But importing large amounts of data at scale isn’t simple. Our engineers have been working on
the challenge for years. Originally, offline disk import was handled at our
data centers as a way to
efficiently import large amounts of data from the hard drives in our
Street View
cars - vehicles that capture terabytes of photographs and information about the landscape as
they build a navigable, visual database of the world.
And although it’s technically challenging, the system we built for rapidly ingesting and
processing these large data sources has a playful name. We call it
OmNomNom. Here is one of the
test OmNomNom machines that the disk import team has at its offices in Mountain View:
But rapidly ingesting and processing large data sets and making them usefully available is a
bit more complex than gobbling down a cookie. As we’ve improved our ability to quickly import
these drives, we dramatically reduced the time between capturing these images and making them
available to users around the world.
Now, we are helping people across the world take advantage of the speed, scale and global
availability of Google Cloud Storage as well as this rapid disk-upload technology. Even though
it might sound like something out of Sesame Street, this is another example of how Google
Cloud Platform is making the advantages of Google-sized scalable infrastructure available to
you. All you need to do is send us your EncFS encrypted hard drives, and we will let you know
once your encrypted bytes are imported to your designated GCS bucket. Once uploaded, we can
mail your drives back to you, or if you prefer, safely and securely handle disk destruction
free of charge. Check out our website for how you can be part of the
Limited Preview of
International Offline Disk Import.
(Oh, and for those of you who want to use Google Cloud Storage but don’t need offline disk
import, you always have quick access Google Cloud Storage from the command line using
gsutil).
Benjamin Bechtolsheim is a Product Marketing Manager for Google Cloud Platform. When
not getting developers to code on Cloud Platform, he's probably riding his bike, trying to
keep his container garden alive, or playing guitar, piano or the shaky egg.
Posted by Scott Knaster,
Editor
