Danger in the Clouds
Do you own a Sidekick mobile phone (AKA the Danger Hiptop)? Then please accept my condolences while I describe the pain and suffering you’ve experienced over the past few days.
The Sidekick is made by Danger, a company acquired by Microsoft in 2008. As one of T-Mobile’s flagship mobile phones, the Sidekick was one of the first and most popular consumer smartphones. Featuring a real keyboard, it offered an instant messaging application at a time when many phones were still figuring out SMS. For IM and SMS addicts, the Sidekick was THE phone to own.
One of the Sidekick’s key features was that it kept all of your important stuff “in the cloud.” That meant it stored all of your contacts, messages, photos, and just about everything else on a server managed by Danger. This made it easy to recover your data in case your phone lost power or failed. What nobody anticipated was the cloud server itself failing.
In what the BBC calls “the biggest disaster yet for the whole concept of cloud computing,” that very thing happened this past weekend. A failed upgrade to the server managing the data for all Sidekicks resulted in the loss of everyone’s data at once. Microsoft is now warning Sidekick owners not to turn off their devices, thus permanently deleting what little data they might have cached locally.
Anyone building an app on the cloud should be worried, because what happened to Microsoft could just as easily happen to you too. With that in mind, here are a few lessons for cloud computing app developers:
The Cloud is not redundant.
One of the biggest mistakes anyone can make about the cloud is to assume that it’s all magically redundant and will keep your data safe no matter what. It’s easy to think the best of a technology when all of its pieces are hidden away from you. Out of site, out of mind – right? What you should never forget is that cloud infrastructure is just like any other infrastructure. Cloud servers live in a datacenter and use the same servers and storage as any other kind of server. That means that they can fail, eat data, drop offline, and make you miserable just like any other kind of server. The only difference is that you must rely on your cloud vendor to make everything right.
Know your vendor.
Remember, cloud computing was originally invented as a way to rapidly scale with demand while keeping prices low using improved economies of scale. Some cloud vendors are really good at that – to the detriment of other features like storage and persistence. If 99.999% uptime is critical to you, then factor that into your cloud decisions and vendor selection. Think carefully about what will happen to your data if a cloud server crashes or reboots. How is it backed up? Who backs it up? How long would it take to restore? If your vendor can’t answer those questions, then either keep looking or be prepared to build that infrastructure for yourself.
Keep your options open.
If the worst should happen to your product or data in the cloud, how long would it take to bring your information back online? One of the nice things about cloud computing is that you can easily replicate to other cloud environments, even if you don’t plan on using them. (Sadly, this is completely untrue for data hosted within cloud applications.) So if you use Amazon EC2, then you can replicate your information to Rackspace. Or vice versa. That way, if one should fail the other is on standby ready to go.
The only good backup is an offsite backup.
You’re feeling good because you make regular backups, but you keep those backups in another cloud instance that happens to be located in the same datacenter. Oh, and it’s not really a backup but a live mirror of your database that’s updated every few minutes. That’s ok, right? Wrong! A backup is only good it’s kept someplace completely separate from the rest of your data. That means storing your data someplace absolutely separate from the rest of your infrastructure. The more offline and “cold” that data is, the better. Even if you can only afford to go to this trouble weekly or even monthly, you’re still better off than not having the information at all.
What other best practices have I missed?
Other best practices you have missed? How about not using a device that doesn’t store anything locally?? Storage is so unbelievably inexpensive these days that there is no reason for a mobile device to not have the capability of storing all content on the device itself, or at the very least a large capacity memory card.
Other than that, anyone not doing any kind of regular full backups is a fool (yes, even you Microsoft! Apparently they had a Hitachi SAN upgrade failure AND NO WORKING BACKUP OF THE DATA!!!) And never trust a single site for backup purposes, use a a desktop manager or intellisync application to back your stuff up onto your desktop/laptop.
The level of Microsoft’s latest fumble absolutely boggles my mind!
Randall – Back in the day, the cloud storage feature was probably a cheap way to make the Sidekick a smarter phone than it actually was.
As storage prices dropped, I suspect it became a handy way to keep customers locked-in to the phone. You could upgrade to a new Sidekick pretty easily, but buying a competitor would require you to start over.