Predictable Black Swans in Cloudy Skies

There is a lot of Sturm und Drang about Amazon's EC2 "Cloud" system collapse over the last 2 days. the main thing it shows me is that most people fundamentally misunderstand networked system reliability. Allow me to make a quick calculation:

- there are c 356 days in a year, 24 hours a day, that is 8544 hours in a year
- a "Five Nines" reliable system like a voice telephone network, with massive investment and decades of operating experience will be up 99.999%, ie you need to expect about 0.09 hours or c 5 minutes of total system failure a year.
- Cloud systems are nowhere near Five Nines reliable, even Broadband isn't there. In my 10 years experience with web hosting/ASPs I'd say that 99.9% reliability would be a great target to aim for still, ie you need to assume c 8 - 9 hours of total failure every year.
- You can back this up by parallelling systems, but to be truly redundant this needs to be in a different datacentre, preferably in a different country, ideally with a different operator so there is no problem bleed and error buildup. In such a perfect case, you can assume you will get two systems of 99.9% reliability ensuring one of them is up 99.99% of the time, ie only one hour a year when both have crashed at the same time.

Also, the sorts of datacentres operating consistently (ie are prepared to sign a service level agreement - SLA - with penalties) at even "Three Nines" are pretty high end, your average cash starved startup sitting on a standrad deal on one cluster of a virtual shared hosting service like Amazon (as far as I recall they don't have an SLA, just Best Endeavours to hit 99.95%*) is probably not going to see even that level of reliability. Assuming they see something like 99.5% reliability, that means you must assume c 2 days of outage a year. Even a fully redundant system like I have described above, at 99.5% will give about 1 days complete outage a year.

In other words, these swans are going to be Black, and they are entirely predictable.

I'm afraid the Cloud industry tends to oversell it's capability for reliability somewhat (what's .499% between friends - well, about 2 days outage a year actually) , you will know its got to Telkco utility levels of reliability when it's provided by massive telecom utility type providers, not run off the back of companies' whose main business is doing something else. If you ever read Nick Carr's book The Big Switch you will see we are in the equivalent of where the 1920's was for electricity, where the industry is moving from fairly unreliable, non standardised services, run by people selling off spare capacity, on to dedicated service providers. No reason not to have Cloud now, (just as no reason not to have electricity in 1920) but you wouldn't bet an entire livelihood (or the farm) on it.

The one thing that Amazon needs to learn though, is telling customers what is going on early and honestly - survey after survey over the last 20 years shows that customers are much more confident, and ultimately happier with you if you tell them - even if it is just that something is wrong - and put in place a comms update plan. Amazon's Cloud apostles dropped off the Twitter radar for hours as this happened, and the initially company clammed up, except for rather legalese communications it seems. No datacentre operator worth their salt wouldn't have system healthcheckers in place so that at least they could tell customers where faults were very quickly, and what their disaster recovery procedures were going to be.

(For the record - we have an Amazon instance, but we don't host the blog or website on it. Our advice remains to use the cloud by all means but also to have your own systems running in hot backup and able to do short term switchover and be able to run the system at normal mean loadings - and ensure any data that you need if it goes down is mirrored)

* I was wrong, one can apply for credits now - but it's money off the rental cost, not business damage.

Update - and here come the Cloud snow jobs - this one is just too (sadly) funny for words but fortunately it's petard is well hoisted on Slshdot - it's on O'Reilly, which does surprise me.