Hey guys,
Tom brought the issue to my attention Sunday morning and I notified the appropriate teams. Unfortunately, the data center where our login services are hosted suffered a power outage. When the power was restored, the infrastructure team discovered that many virtual machine images had become corrupted, including ones that are required for the login service. Adding fuel to the fire, much of the networking infrastructure also didn't boot properly. In total, there were 88 affected images and 2 affected routers.
The infrastructure team, working with all the other affected service teams, Cisco, and VMWare worked from about April 26th @ 8:15am EST until April 27th @ 00:15 EST to resolve the issues.
That's about all I know at this time, the whole thing was out of our hands - I trust there will be a post-mortem discussion regarding what happened, but I doubt I'll be privy to that conversation. Still, I'm very grateful for their dedication and hard work. It was a catastrophic outage and they managed to get everything working again, which is no easy task.
Thanks for your patience during the outage. If you notice anything acting out of the ordinary, please email support and we'll take a look.
Tom brought the issue to my attention Sunday morning and I notified the appropriate teams. Unfortunately, the data center where our login services are hosted suffered a power outage. When the power was restored, the infrastructure team discovered that many virtual machine images had become corrupted, including ones that are required for the login service. Adding fuel to the fire, much of the networking infrastructure also didn't boot properly. In total, there were 88 affected images and 2 affected routers.
The infrastructure team, working with all the other affected service teams, Cisco, and VMWare worked from about April 26th @ 8:15am EST until April 27th @ 00:15 EST to resolve the issues.
That's about all I know at this time, the whole thing was out of our hands - I trust there will be a post-mortem discussion regarding what happened, but I doubt I'll be privy to that conversation. Still, I'm very grateful for their dedication and hard work. It was a catastrophic outage and they managed to get everything working again, which is no easy task.
Thanks for your patience during the outage. If you notice anything acting out of the ordinary, please email support and we'll take a look.