Site Comments & Suggestions

What happened? Unable to login.

  • Last Updated:
  • Apr 28th, 2015 2:02 pm
Tags:
None
Member
User avatar
Sep 26, 2010
422 posts
130 upvotes
Toronto
Hey guys,

Tom brought the issue to my attention Sunday morning and I notified the appropriate teams. Unfortunately, the data center where our login services are hosted suffered a power outage. When the power was restored, the infrastructure team discovered that many virtual machine images had become corrupted, including ones that are required for the login service. Adding fuel to the fire, much of the networking infrastructure also didn't boot properly. In total, there were 88 affected images and 2 affected routers.

The infrastructure team, working with all the other affected service teams, Cisco, and VMWare worked from about April 26th @ 8:15am EST until April 27th @ 00:15 EST to resolve the issues.

That's about all I know at this time, the whole thing was out of our hands - I trust there will be a post-mortem discussion regarding what happened, but I doubt I'll be privy to that conversation. Still, I'm very grateful for their dedication and hard work. It was a catastrophic outage and they managed to get everything working again, which is no easy task.

Thanks for your patience during the outage. If you notice anything acting out of the ordinary, please email support and we'll take a look.
Deal Expert
Jan 7, 2002
29710 posts
29129 upvotes
Waterloo, ON
tylerversion2 wrote: I trust there will be a post-mortem discussion regarding what happened, but I doubt I'll be privy to that conversation. Still, I'm very grateful for their dedication and hard work. It was a catastrophic outage and they managed to get everything working again, which is no easy task.
Thanks for the update and y'all's dedication. I have just one minor suggestion: In future if possible please update the login page with a message to the effect that there is a system-wide service issue, possible ETA (if known), etc. so that users understand that it's not just their login that's broken.
veni, vidi, Visa
Deal Guru
User avatar
Sep 19, 2006
11664 posts
6564 upvotes
Toronto
sent a message to support.. need help ahhhh
Member
User avatar
Sep 26, 2010
422 posts
130 upvotes
Toronto
bylo wrote: Thanks for the update and y'all's dedication. I have just one minor suggestion: In future if possible please update the login page with a message to the effect that there is a system-wide service issue, possible ETA (if known), etc. so that users understand that it's not just their login that's broken.
That's a good idea - I should have thought to do that. Right now, putting up a message like that would require pushing a whole new version of the code to our web servers. I'm going to create a feature request in our ticketing system to implement an error messaging system so we can notify users when things go wrong.
Deal Expert
Jan 7, 2002
29710 posts
29129 upvotes
Waterloo, ON
tylerversion2 wrote: That's a good idea - I should have thought to do that. Right now, putting up a message like that would require pushing a whole new version of the code to our web servers. I'm going to create a feature request in our ticketing system to implement an error messaging system so we can notify users when things go wrong.
Another possibility is to post to RFD's Twitter feed. That's completely independent of the system that's in trouble and can be done without bothering the people who are busy trying to resolve the issue.
veni, vidi, Visa
Deal Fanatic
User avatar
Jan 16, 2011
7205 posts
9238 upvotes
The NORTH
bylo wrote: Another possibility is to post to RFD's Twitter feed. That's completely independent of the system that's in trouble and can be done without bothering the people who are busy trying to resolve the issue.
This!
Deal Expert
May 30, 2005
49009 posts
10302 upvotes
Richmond Hill
bylo wrote: Another possibility is to post to RFD's Twitter feed. That's completely independent of the system that's in trouble and can be done without bothering the people who are busy trying to resolve the issue.
Not everyone uses Twitter. A separate system is best.
Tons of things for sale!
Silver Coins and Numismatics | Heatware
Deal Expert
Jan 7, 2002
29710 posts
29129 upvotes
Waterloo, ON
Jon Lai wrote: Not everyone uses Twitter. A separate system is best.
You don't need to "use" Twitter, i.e. no need to register or login. It also doesn't have to be Twitter. It could be any other website that's easy for RFD staff to update and RFD users to access. Just something that people can set a bookmark to and reference if they need status info on the main RFD site.
veni, vidi, Visa
Deal Expert
User avatar
Jul 5, 2004
27438 posts
6807 upvotes
Well a large banner on the home page would be nice, the reality is you can just check Hot Deals. If there's no activity in Hot Deals, the site is down.
Deal Expert
Jan 7, 2002
29710 posts
29129 upvotes
Waterloo, ON
Shaner wrote: Well a large banner on the home page would be nice, the reality is you can just check Hot Deals. If there's no activity in Hot Deals, the site is down.
That presumes the site is up. If it's down then what? Is it really that hard for RFD staff to keep people apprised of significant outages in any number of ways like Twitter?

BTW on Sunday I spent the better part of an hour clearing cookies, trying different browsers, trying a different username, trying different PCs, etc. I suppose I could have determined that there was a problem with logins by looking at post activity in Hot Deals. But that didn't occur to me, especially given the constant login issues that others have reported in the previous several months. A simple notice on the login page or a tweet etc. would have saved a lot of confusion. I doubt I'm the only one who would have benefited from that.
veni, vidi, Visa
Administrator
User avatar
Jun 17, 2013
18226 posts
36617 upvotes
Scarborough
We've started looking into a better on-site notification system to alert users of any service disruptions. Typically outages like these only last minutes but the one this past Sunday was particularly bad.
RFD Staff - Got a question? Send me a private message.
[Forum Rules] [Deal Alerts]
Deal Expert
Jan 7, 2002
29710 posts
29129 upvotes
Waterloo, ON
TomRFD wrote: We've started looking into a better on-site notification system to alert users of any service disruptions.
Thank you. May I humbly suggest you keep it simple, e.g. Twitter. There's nothing to "look into." There's no code to write. There's nothing but a simple policy to implement. It's far more important that the solution is easily-accessible and kept up-to-date when a serious service disruption occurs than it is that you spend the next several months designing the "perfect solution."
veni, vidi, Visa
Administrator
User avatar
Jun 17, 2013
18226 posts
36617 upvotes
Scarborough
bylo wrote: Thank you. May I humbly suggest you keep it simple, e.g. Twitter. There's nothing to "look into." There's no code to write. There's nothing but a simple policy to implement. It's far more important that the solution is easily-accessible and kept up-to-date when a serious service disruption occurs than it is that you spend the next several months designing the "perfect solution."
We will definitely be more on the ball regarding posting these updates to social media moving forward but as mentioned by a few previous posters, not everyone is on social media. It shouldn't be too hard for us to add something to the website itself. Here's hoping it can be created quickly but not have to be used for a very long time.
RFD Staff - Got a question? Send me a private message.
[Forum Rules] [Deal Alerts]

Top

Thread Information

There is currently 1 user viewing this thread. (0 members and 1 guest)