[Quinn]

I had a little bit of that thunder over here, but it was worse last night.

I've heard that Hershey was a few street lights last night. Tonight, the only big thunder clap that I heard almost spilled my drink.

[TheEmpty]

Yes Heroku is killing me! Aww man, woot too!

[Daniel15]

It's funny that "the cloud" was brought down by actual clouds.

[ianonavy]

Daniel15, on 30 June 2012 - 02:58 AM, said:

It's funny that "the cloud" was bought down by actual clouds.

but I lol'd.

[TheMaster]

Hahahaha! Nice, Daniel

Hmmm. Shouldn't datacentres have backup generator's? Or like...UPS backups or something? Surely a major datacentre would have measures against.....thunderstorms?

[derTechniker]

Just some days ago i read an article on how a lot of those startup-web2.0 companies rely on amazon and how all of them would go down if amazon makes crap

[AwesomezGuy]

Yet another reason to host your sites in Ireland. No major weather events. Some, but not much copyright law. Cold most of the year, so cooling systems don't have to use much energy.

[Daniel15]

Quote

Hmmm. Shouldn't datacentres have backup generator's? Or like...UPS backups or something?

Backup generators and UPSes can't power the whole data centre for a long time (I think it's normally about enough power to safely shut down most servers). Only the most integral components (core routers and such) get lots of backup power.

[TheMaster]

Daniel15, on 30 June 2012 - 06:59 AM, said:

Backup generators and UPSes can't power the whole data centre for a long time (I think it's normally about enough power to safely shut down most servers). Only the most integral components (core routers and such) get lots of backup power.

Oh really? I thought it was something more like powering it for a few hours....

I guess that'd be an ENORMOUS amount of power though. Thousands of servers probably cost thousands each month to run....I'd hate to see that electricity bill....

[NeilHanlon]

Daniel15, on 30 June 2012 - 06:59 AM, said:

Backup generators and UPSes can't power the whole data centre for a long time (I think it's normally about enough power to safely shut down most servers). Only the most integral components (core routers and such) get lots of backup power.

This. At work, though not a data center, we host quite a few servers in one of the rooms and they're all on UPSes. The UPSes kick in when the power goes out, and give the systems time to shut down. We also have it on critical systems like all the computers we fix and stuff, so there isn't any hard drive failure, etc.

[Ruku]

Lol @ all the idiot startups thinking that more than one server in the same damn datacentre counts as redundancy. No sympathy; they should have started running instances in different AZs as Amazon intended. Evidently nobody learned anything the last time this happened?

TheMaster, on 30 June 2012 - 08:16 AM, said:

Oh really? I thought it was something more like powering it for a few hours....

I guess that'd be an ENORMOUS amount of power though. Thousands of servers probably cost thousands each month to run....I'd hate to see that electricity bill....

Nope; a typical datacentre will have a UPS system that'll power the entire site just long enough for all the generators to fire (seconds/minutes), then the generators can run only as long as they have fuel (hours/minutes). Fuel's expensive!

[@Tom]

AwesomezGuy, on 30 June 2012 - 06:34 AM, said:

Yet another reason to host your sites in Ireland. No major weather events. Some, but not much copyright law. Cold most of the year, so cooling systems don't have to use much energy.

But then your prone to a mad leprechaun getting inside the servers and sprinkling lucky charms. That can cause huge issues to scalability.

[Kyek]

Ruku, on 30 June 2012 - 11:47 AM, said:

Lol @ all the idiot startups thinking that more than one server in the same damn datacentre counts as redundancy. No sympathy; they should have started running instances in different AZs as Amazon intended. Evidently nobody learned anything the last time this happened?

It didn't matter. My company's stack is redundant in 3 zones, with exceptions for the ELBs (obviously) and an RDS instance that's set up with the managed multi-AZ failover.

I'm still working on my post-mortem, but it appears that the service responsible for detecting the failure of the primary RDS itself failed -- either that or the routing table wasn't reachable to actually re-point the hostname at the secondary, because the failover didn't happen. I had to work with support to manually re-point the hostname this morning. Even once that was done, there were connectivity issues between the AZs causing some of our services behind ELBs not to connect, so even though many of our servers were up and running, few of them could talk to each other.

It's a popular response to say "You can't blame Amazon, they made redundancy possible" and often times I agree with that sentiment. But in this case, the entire region's APIs went down, the failovers already in place didn't work, and once they restored power, it took 12 hours until they had the majority of services back up and they're still not finished. That is *insane*. Don't get me wrong, I still love the service, but clearly they need to revisit some of their failure plans.

[Daniel15]

Quote

I guess that'd be an ENORMOUS amount of power though. Thousands of servers probably cost thousands each month to run....I'd hate to see that electricity bill....

It depends on what electricity costs in the area, as this varies a LOT. One server would be anywhere between $10-25 for an average server, and even more than that if it's very powerful (top-of-the-line processor, huge amount of hard drives, etc).

A lot of the time, getting a VPS is actually cheaper than using a home server.

[arronhunt]

itom07, on 30 June 2012 - 12:24 PM, said:

But then your prone to a mad leprechaun getting inside the servers and sprinkling lucky charms. That can cause huge issues to scalability.

Sir you are the first person to make me piss myself laughing. Kudos.

[Fike]

AwesomezGuy, on 30 June 2012 - 06:34 AM, said:

Yet another reason to host your sites in Ireland. No major weather events. Some, but not much copyright law. Cold most of the year, so cooling systems don't have to use much energy.

Wish I knew of some decent providers over here. Blacknight is the leader here I guess but for the money I have they're too expensive.

[Ruku]

Just to be clear, my concerns were more angled towards healthcare providers hosting mission critical (as in life or death!) operations in single AZs. Hearing about all the foolish companies placing their customers' lives at risk purely out of selfishness and/or an insane lack of foresight was chilling.

Kyek, on 30 June 2012 - 12:56 PM, said:

It didn't matter. My company's stack is redundant in 3 zones, with exceptions for the ELBs (obviously) and an RDS instance that's set up with the managed multi-AZ failover.

I have positively no idea how Amazon's implementation works (I'm still not convinced that EC2 is cost-effective enough to migrate to), but surely you can just configure multiple A records on the domains which point to different load balancers? It's not particularly sophisticated or elegant, but it's a primitive way to ensure connectivity when a load balancer fails. I still see every individual IP address as a potential disaster zone.

Kyek, on 30 June 2012 - 12:56 PM, said:

I'm still working on my post-mortem, but it appears that the service responsible for detecting the failure of the primary RDS itself failed -- either that or the routing table wasn't reachable to actually re-point the hostname at the secondary, because the failover didn't happen. I had to work with support to manually re-point the hostname this morning. Even once that was done, there were connectivity issues between the AZs causing some of our services behind ELBs not to connect, so even though many of our servers were up and running, few of them could talk to each other.

Well that really sucks, and it may not exactly something anyone can change, at least from a technical standpoint. The only way they could possibly avoid issues like this in the future would be to preempt emergency situations and make the changes before an outage actually occurs, which would give their provisioning systems the necessary times to make the changes. Of course, they could always providie reliable UPS and generator systems (like every other damn datacentre) to failover onto when events like these occur, but I don't think we should really place blame until they've released their analysis.

Kyek, on 30 June 2012 - 12:56 PM, said:

It's a popular response to say "You can't blame Amazon, they made redundancy possible" and often times I agree with that sentiment. But in this case, the entire region's APIs went down, the failovers already in place didn't work, and once they restored power, it took 12 hours until they had the majority of services back up and they're still not finished. That is *insane*. Don't get me wrong, I still love the service, but clearly they need to revisit some of their failure plans.

I guess it's time to consider multiple hosting providers as well as availability zones? This is a huge argument for interopability between cloud hosting services, and a phenomenal part of the reason our company would rather go about building its "cloud" on the likes of OpenStack and Eucalyptus than some proprietary platform spearheaded by such a huge private company. I love the premise, but their sheer scale of operations is somewhat intimidating :s

[Kyek]

Ruku, on 01 July 2012 - 01:38 PM, said:

Just to be clear, my concerns were more angled towards healthcare providers hosting mission critical (as in life or death!) operations in single AZs. Hearing about all the foolish companies placing their customers' lives at risk purely out of selfishness and/or an insane lack of foresight was chilling.

Oh my, agreed. And screw multiple AZs -- if you're running literal life-or-death services and you're not in multiple regions, then your company should not exist.

Quote

I have positively no idea how Amazon's implementation works (I'm still not convinced that EC2 is cost-effective enough to migrate to), but surely you can just configure multiple A records on the domains which point to different load balancers? It's not particularly sophisticated or elegant, but it's a primitive way to ensure connectivity when a load balancer fails. I still see every individual IP address as a potential disaster zone.

That's just it -- that's super easy and even fairly cheap for things like webservers, especially when your failover endpoints use auto-scaling so you only need to pay for very minimal instances unless they're needed. But EC2 wasn't even our problem this time. Being stretched across multiple AZs, each of our services were durable enough to deal with it. I could have had us back online in half the time if it weren't for the fact that RDS (managed MySQL, ironically set up with failover) failed.

Quote

Well that really sucks, and it may not exactly something anyone can change, at least from a technical standpoint. The only way they could possibly avoid issues like this in the future would be to preempt emergency situations and make the changes before an outage actually occurs, which would give their provisioning systems the necessary times to make the changes. Of course, they could always providie reliable UPS and generator systems (like every other damn datacentre) to failover onto when events like these occur, but I don't think we should really place blame until they've released their analysis.

Agreed agreed agreed. I hope they release their postmortem.

Quote

I guess it's time to consider multiple hosting providers as well as availability zones? This is a huge argument for interopability between cloud hosting services, and a phenomenal part of the reason our company would rather go about building its "cloud" on the likes of OpenStack and Eucalyptus than some proprietary platform spearheaded by such a huge private company. I love the premise, but their sheer scale of operations is somewhat intimidating :s

Really, we could stay 100% with Amazon and be fine. There's availability zones, which are physically separate datacenters within a region, and then the regions, of which there are 8 -- and 3 of those are in the US. So we could go multi-region and be fine. The issue is that, while Amazon has amazing support for spanning their services across AZs, there is absolutely zero support for spanning them between regions. Obviously for webservers this is a non-issue, as they're not interconnected and there's a million ways to scale them. But I'd have to stop using DynamoDB and RDS and SQS and SimpleDB and others, because awesome as they are, there's no multiregion sync on them. To take a Dynamo-based DB multi-region, I'd be spending $20k on Riak and that's BEFORE server costs and data costs (since you pay WAN rates between regions). Taking MySQL multi-region is just a shit show, and I'd probably have to run MySQL Cluster somewhere. Extra servers, an admin workload that I'd be looking at either hiring or outsourcing for, and even then failure recovery is a joke. I'd be better off looking at something like Oracle for our relational needs, which is more software cost. SQS is something we'd probably have to build in-house but even then we'd be running 3-4 more servers for durability and eating the associated cost.

What it boils down to is, at a certain point it's more cost-effective to hang up a "Please excuse our maintenance" sign for a few hours and cut our losses on the revenue for that period than it would be to undertake all that extra work and pay out all those software costs and server costs. All the while, this could be resolved almost entirely if AWS would support multi-region syncing/multi-region services. And every time they have an outage like this, there's more and more of an outcry for it -- so hopefully they'll come to their senses soon!