The Blog

A week-long nightmare thanks to provider issues

Posted by christian on Thursday, January 26, 2012 · 2 comments

When I returned from work last Thursday night numerous mails from the Frisbee community were waiting for me (thanks a million Bruno, Bommie, and others), indicating that something is wrong with FFindr!

A quick check revealed that the database connection was broken. I connect to the database: nothing, all data seems to be gone! In search for the emergency service hotline number of provider OVH I stumble upon an automated e-mail sent by OVH on Thursday 19 January 2012 at 16h17:

Your database has been CLOSED due to heavy database queries, exceeding the acceptable time limit! Indeed, the queries (used for the vicinity calculation on FFindr) took a far-above average time to execute. But strangely they performed just fine for almost 3 years.

I call the service hotline in Germany and file an incident ticket straight away, i.e. Thursday at ten to eight PM. Now the Odyssey starts. Naively I believed in a quick solution, all I need is read access or a database dump to restore order on FFindr. The support assures me that the data is safe and that only the access has been taken away. Neither Friday nor Monday (hey, the weekend is for playing Frisbee so I won't bother or blame them for not responding) I regain access to my data, four long days of helpless waiting! By Monday night, again on the phone with the support guys from OVH —who know me by name now— I learn that the ticket still hasn't been treated: It is not priority, your script caused the problem so wait.... What they don't know or understand: with every minute the work of merging all data back becomes more tedious, and by the time I got my access back almost 5 days elapsed (Tuesday 24 January 2012 11h11)!

I won't bother you with the annoying and time-consuming job of merging all changes back into the database (two long nights and everything was fine), I should have logged all database activity but didn't manage to quickly set this up last Thursday. But there's one lesson I learned: never trust your provider! It's not only the incredible response time, but first and foremost the decision to close the database! I made my inquiries and learned that OVH improved the connectivity of the database servers that day, which probably means that they also adapted the rules. Unfortunately my script didn't play following these new, never communicated, rules (by the way, also the old rules are stated nowhere). What about announcing such kind of changes? What about sending out some warnings?

No hard feelings! This nightmare is over after precisely one week, and all I need now is a dedicated server to finally stop playing games with unknown rules.

2 comments

  1. The site is growing. The community is growing. You'd need a dedicated server sooner or later...

    katon says on February 1, 2012 5:53 PM

  2. I'm working on it :) Thanks for your advise, Victor.

    christian says on February 1, 2012 6:35 PM

Comments are closed.

Recent blog postsRSS Feed for blog posts

Blog archive