Dear Members,
Introduction
We would like to offer you our sincere apologies for the interruption in service many of you experienced last weekend. During the outage our forum remained online and we attempted to keep all of our members up-to-date via our Service Status announcements.
What happened?
On Saturday 31 May at approximately 11pm GMT there was an explosion in The Planet Data Center in Houston Texas. Electrical gear shorted, creating an explosion and fire that knocked down three walls. Thankfully there were no human casualties.
On the instructions of the Fire Department, The Planet then turned off all power to the Data Center resulting in 9,000 servers being knocked offline.
How did the outage at The Planet affect StatCounter?
This affected StatCounter in a number of ways:
- Some of our database servers went down
- Our dns servers were temporarily offline
- Some of our incoming mail servers went down
- Our blog was unavailable
- Some of our web servers went down
- Some of our partitions were knocked offline
How did this mean for StatCounter members?
Different members were affected in different ways.
Members with projects on the following partitions were most seriously affected:
c1 (PN 0), c7 (PN 6) , c8 (PN 7), c14 (PN 13), c16 (PN 15), c17 (PN 16)
These members lost between 24 and 30 hours of stats over Sunday GMT and part of Monday morning.
Members with projects on the following partitions were unable to log into their accounts for a number of hours following the outage but stats continued to be recorded:
c4 (PN 3), c5 (PN 4), c6 (PN 5), c12 (PN 11)
New members and people who had just created new projects with StatCounter in the hours immediately prior to the outage temporarily “lost” their accounts/projects. This is because, since these projects were not on our last back-up, restoring the back-up did not “bring up” their projects. In this case, our advice is to generate a new project and re-install the StatCounter code on your site.
All other members lost about 3-5 hours of stats from approximately 11pm GMT on Saturday night. In addition members experienced difficulties reaching the StatCounter site and logging into their accounts.
As servers at The Planet come back online we continue to work to try to recover as many stats as possible to minimise the loss of information experienced.
Why doesn’t StatCounter have its own Data Center?
By outsourcing our server technology, we can keep costs down, minimize downtime and devote more resources to our members.
Why was StatCounter using The Planet?
StatCounter is powered by over 130 servers. These are located in a number of Data Centers in the United States and in Ireland and are spread among a number of hosting providers although our main hosting partner is The Planet.
We chose The Planet as our main hosting partner as they are the largest dedicated hosting service in the world and due to the apparent reliability of the service they provide.
We believed The Planet to be one of the most reliable and redundant data center providers in the world, particularly as they host servers in multiple centers in Houston and Dallas.
From The Planet website:
With multiple state-of-the art data centers located in Dallas and Houston, Texas, The Planet provides On Demand IT Infrastructure backed by complete redundancy in power, HVAC, fire suppression, network connectivity, and security. So if any of our data centers experiences a disruption for any reason, your eggs (or servers) are never in one basket.
The Planet have let us down, and in turn, we have let you down. For this we are truly sorry.
What did The Planet do wrong?
Accidents are a fact of life, however, we believe that, had The Planet operated in the professional manner we expected from an organisation of its standing, the disruption experienced could have been substantially lessened.
For example, The Planet have hosted our DNS for a number of years, however, it was only this weekend we discovered that, although our DNS servers are on different subnets within The Planet, the servers are actually all in the same location. We will be submitting a complaint to The Planet in this regard. We fully expected that The Planet would have implemented a geographical spread in our DNS servers – this was not something that we thought we would have to request or confirm – particularly since we have servers spread through all The Planet data centers. We have now secured the services of a new geographically spread, redundant DNS provider.
We also feel that the extent of the damage could have been acknowledged and communicated by The Planet in a more timely fashion. While we decided to implement our back-up plans early on, others waited many hours in the hope that The Planet would come back online, only to find that restoration of service was continually delayed.
In addition, we found that our efforts to communicate with The Planet were largely ignored or dismissed with a “template” response. This was particularly galling as we received a presentation glass globe (see below) and a letter from The Planet CEO on FRIDAY thanking us for being one of their largest customers… then Saturday… THIS!
Why couldn’t The Planet get the Data Center back more quickly?
We don’t know. Hundreds of angry customers have been asking this question.
What action did StatCounter take when this accident happened?
We immediately began work to restore full service as soon as possible.
- Initially, and in the absence of any official information from The Planet, we worked to establish exactly what was causing the problems.
- We started a thread in our Service Status forum to advise our members of the situation – this thread has been updated continuously.
- We added a notice to out homepage to advise members that service was limited.
- We procured the services of a properly redundant and geographically spread dns service and re-routed all our servers immediately.
- We prioritised the restoration of all our affected partitions from our latest back-up taken in the hours before the outage in order to resume tracking stats.
- We configured new servers.
- We redirected web servers which were temporarily down due to the outage.
- We responded to as many tickets as possible to try to explain the situation to our members.
- We migrated our affected mail servers to a new data center.
How will StatCounter prevent this happening in the future?
The bitter irony of this recent episode is that we have been working on our new beta system since September last year. We decided to develop this new StatCounter system for a number of reasons, one of the major motivations being to improve the architecture of our system so as to insulate it against major outages such as the one just experienced. Considering that we have never before experienced an outage of this magnitude, we are bitterly disappointed that our new system was not up and running before this episode.
Once “normal” service is restored, work will continue on the beta project as planned. The sooner we launch the beta, the sooner we can minimise our vulnerability to this kind of outage.
I’m not happy – how do I complain?
We completely understand why you feel aggrieved. Should you wish to submit a complaint to us please do so by logging into your StatCounter account and clicking the “support” link in the top menu bar. Within this area you will be able to submit a ticket to us – we will endeavour to respond to you as soon as possible.
How do StatCounter feel about what happened?
We are so desperately sorry that any of our customers had to experience this outage. We also feel so humbled by the numerous messages of support that we have received. At a time when we feel so terribly for the interruption in service suffered by some of our members, we have been just bowled over to receive so many of messages of encouragement. While we always knew that we had a great bunch of members, your support and patience throughout this episode has been nothing short of incredible and served to help maintain team morale in a very difficult situation. We are so grateful.
Conclusion
We hope this blog post has gone some way towards summarising the main issues relating to the recent outage. Work continues to restore full service. If you have any comments or queries, please do post them below.
Hey – the outage is no big deal – don’t worry about it.
If customers are angry – tell them to go elsewhere !
Accidents happen……
Have a cold beer on me, kick your boots up and relax.
Life is short, Break the rules, Forgive quickly,
Respect your elders, Kiss slowly, Love Truly,
Laugh uncontrollably, Love yourself, and be the
best person you can be. YOU…
Never regret anything that made you Smile.
Life may not be the party we hoped for, but while
we’re here we should Dance . . . .
bye
I appreciate your timely response and full explanation of what happened. You guys are great! Things happen: often they are strange, baffling , maddening things. It’s part of life! I’m glad nobody was hurt. Thanks for your terrific service.
Similar stuff has happened to me – hosting companies promise everything, the let you down, threatening the viability of your business. When I checked Statcounter at the weekend and got ‘server not found’ I knew that something server-related was wrong, probably beyond Statcounter’s control. I like the service, the transparency and directness of the communications… Long live Statcounter! Dishonest hosting companies, go take a running jump…
I used to work for theplanet, I have many friends that still work there. Having said that, I know for a fact that they have datacenters in Houston and Dallas. I also know for a fact that they have backup generators, and ups to cover everything imaginable. However from talking to my friends that work in the Houston datacenter, this was not something that anyone could imagine happening. I also think that ThePlanet is doing an excellent job of getting customers back online. While I understand downtime sucks…you have to realize that if electricity cannot get into the building, there isn’t much all the redundancy in the world can do to help make things better.
Thanks for all you do. I don’t know how you could have done anything differently. You ugys are fab. Thanks, and thank goodness no one was injured.
Hi guys
Thanks for the info, I think i only lost one days stats in the end, which isnt bad, as you said on ur blog accidents are a fact of life, and afterall statscounter is free after all !
Woc
I kind of missed the whole thing 🙂 well thank God no one was hurt. Thanks for the detailed information. All the best to Statcounter.
All we can ask of you guys is your best, and I am satisfied that you’re doing that. The only thing worse than an accident, is a lack of information about what happened. That is apparently what The Planet did to you. Thank you for not repeating the same mistake with your own customers.
Best of luck.
It’s all good Statcounter is the best by far!
Keep up the good work!
John D
http://www.JohnDennerRocks.com
http://www.YodaVanHalen.com
Are there more people among you who have problems with their stats? My counter doesn’t recieve data since yesterday, today is even worse…
It was nice to know about “The Planet”, and it was not a big problem.
Sc4 is working very well.
Thanks
To Dave & anyone else with the same sentiment, I would like to say that SC has every right to get upset at The Planet. SC is a client of The Planet, and that itself speaks volume. What makes it even more aggravating is that The Planet has specifically promised “…your eggs (or servers) are never in one basket” in order to avoid mishaps exactly like this. Furthermore, communication efforts with The Planet were tackled in a most unsatisfactory manner.
While we are all glad that there was no human casualty at The Planet, it is rather appalling to note that The Planet’s most essential promise was not duly executed.
Keep on trudging through. Well done, SC.
Great job during the recent Planet outage. I really appreciated your professional response to the situation, especially the fact that you kept us informed of the ongoing problem. I am very pleased that I selected StatCounter – you are the best!
I fully support you and feel that, for a service which you provide for free, you are making a 200% effort to keep us happy.
My websites were amongst the thousands which were offline and I’ve had to transfer to another server but only after waiting many days hoping that the promises of repairs “in the next few hours” would materialise.
Thanks for free tools
Only one of my sites were affected, so it’s not a big deal. What’s more, I only added the counter a day before the crash. So what I did was delete it and add it again, so it got assigned to another (working) partition from the affected ones.
As far as I know many sites got affected by this fire at The Planet’s server farm, and I’m as always impressed by StatCounter’s very fast response.
The fact that The Planet didn’t spread your servers over different locations proves that they are not fully aware of Murphy’s law. This must be the hardest lesson for them!
I’ve been using Statcounter for the past couple of years and that you provide this service free is just amazing. I have been using SC for my blog – now I’ve added it as one of my business counters I’m going to pay once my log file reaches its limit. I consider it a cost of doing business and I’d like to think I’m supporting a service I really like.
I think if one isn’t paying for this service then really they should have little to complain about in the big scheme of things. And if one is paying… I think it’s still true to say that accidents happen, and with the best plans and prep in the world things like this will happen. Servers go down. So I lose some stats… big deal.
The way you handled this throughout is stellar… well done.
De loin le meilleur compteur gratuit du net, et le plus performant… Désolée pour vos déboires mais personne ne peut vous en vouloir. Merci de ces explications claires et que l’on sent sincères… Continuez et merci ! 🙂