Outage - Explosion at The Planet
Dear Members,
Introduction
We would like to offer you our sincere apologies for the interruption in service many of you experienced last weekend. During the outage our forum remained online and we attempted to keep all of our members up-to-date via our Service Status announcements.
What happened?
On Saturday 31 May at approximately 11pm GMT there was an explosion in The Planet Data Center in Houston Texas. Electrical gear shorted, creating an explosion and fire that knocked down three walls. Thankfully there were no human casualties.
On the instructions of the Fire Department, The Planet then turned off all power to the Data Center resulting in 9,000 servers being knocked offline.
How did the outage at The Planet affect StatCounter?
This affected StatCounter in a number of ways:
How did this mean for StatCounter members?
Different members were affected in different ways.
Members with projects on the following partitions were most seriously affected:
c1 (PN 0), c7 (PN 6) , c8 (PN 7), c14 (PN 13), c16 (PN 15), c17 (PN 16)
These members lost between 24 and 30 hours of stats over Sunday GMT and part of Monday morning.
Members with projects on the following partitions were unable to log into their accounts for a number of hours following the outage but stats continued to be recorded:
c4 (PN 3), c5 (PN 4), c6 (PN 5), c12 (PN 11)
New members and people who had just created new projects with StatCounter in the hours immediately prior to the outage temporarily “lost” their accounts/projects. This is because, since these projects were not on our last back-up, restoring the back-up did not “bring up” their projects. In this case, our advice is to generate a new project and re-install the StatCounter code on your site.
All other members lost about 3-5 hours of stats from approximately 11pm GMT on Saturday night. In addition members experienced difficulties reaching the StatCounter site and logging into their accounts.
As servers at The Planet come back online we continue to work to try to recover as many stats as possible to minimise the loss of information experienced.
Why doesn’t StatCounter have its own Data Center?
By outsourcing our server technology, we can keep costs down, minimize downtime and devote more resources to our members.
Why was StatCounter using The Planet?
StatCounter is powered by over 130 servers. These are located in a number of Data Centers in the United States and in Ireland and are spread among a number of hosting providers although our main hosting partner is The Planet.
We chose The Planet as our main hosting partner as they are the largest dedicated hosting service in the world and due to the apparent reliability of the service they provide.
We believed The Planet to be one of the most reliable and redundant data center providers in the world, particularly as they host servers in multiple centers in Houston and Dallas.
From The Planet website:
With multiple state-of-the art data centers located in Dallas and Houston, Texas, The Planet provides On Demand IT Infrastructure backed by complete redundancy in power, HVAC, fire suppression, network connectivity, and security. So if any of our data centers experiences a disruption for any reason, your eggs (or servers) are never in one basket.
The Planet have let us down, and in turn, we have let you down. For this we are truly sorry.
What did The Planet do wrong?
Accidents are a fact of life, however, we believe that, had The Planet operated in the professional manner we expected from an organisation of its standing, the disruption experienced could have been substantially lessened.
For example, The Planet have hosted our DNS for a number of years, however, it was only this weekend we discovered that, although our DNS servers are on different subnets within The Planet, the servers are actually all in the same location. We will be submitting a complaint to The Planet in this regard. We fully expected that The Planet would have implemented a geographical spread in our DNS servers – this was not something that we thought we would have to request or confirm - particularly since we have servers spread through all The Planet data centers. We have now secured the services of a new geographically spread, redundant DNS provider.
We also feel that the extent of the damage could have been acknowledged and communicated by The Planet in a more timely fashion. While we decided to implement our back-up plans early on, others waited many hours in the hope that The Planet would come back online, only to find that restoration of service was continually delayed.
In addition, we found that our efforts to communicate with The Planet were largely ignored or dismissed with a “template” response. This was particularly galling as we received a presentation glass globe (see below) and a letter from The Planet CEO on FRIDAY thanking us for being one of their largest customers… then Saturday… THIS!
Why couldn’t The Planet get the Data Center back more quickly?
We don’t know. Hundreds of angry customers have been asking this question.
What action did StatCounter take when this accident happened?
We immediately began work to restore full service as soon as possible.
- Initially, and in the absence of any official information from The Planet, we worked to establish exactly what was causing the problems.
- We started a thread in our Service Status forum to advise our members of the situation – this thread has been updated continuously.
- We added a notice to out homepage to advise members that service was limited.
- We procured the services of a properly redundant and geographically spread dns service and re-routed all our servers immediately.
- We prioritised the restoration of all our affected partitions from our latest back-up taken in the hours before the outage in order to resume tracking stats.
- We configured new servers.
- We redirected web servers which were temporarily down due to the outage.
- We responded to as many tickets as possible to try to explain the situation to our members.
- We migrated our affected mail servers to a new data center.
How will StatCounter prevent this happening in the future?
The bitter irony of this recent episode is that we have been working on our new beta system since September last year. We decided to develop this new StatCounter system for a number of reasons, one of the major motivations being to improve the architecture of our system so as to insulate it against major outages such as the one just experienced. Considering that we have never before experienced an outage of this magnitude, we are bitterly disappointed that our new system was not up and running before this episode.
Once “normal” service is restored, work will continue on the beta project as planned. The sooner we launch the beta, the sooner we can minimise our vulnerability to this kind of outage.
I’m not happy – how do I complain?
We completely understand why you feel aggrieved. Should you wish to submit a complaint to us please do so by logging into your StatCounter account and clicking the “support” link in the top menu bar. Within this area you will be able to submit a ticket to us – we will endeavour to respond to you as soon as possible.
How do StatCounter feel about what happened?
We are so desperately sorry that any of our customers had to experience this outage. We also feel so humbled by the numerous messages of support that we have received. At a time when we feel so terribly for the interruption in service suffered by some of our members, we have been just bowled over to receive so many of messages of encouragement. While we always knew that we had a great bunch of members, your support and patience throughout this episode has been nothing short of incredible and served to help maintain team morale in a very difficult situation. We are so grateful.
Conclusion
We hope this blog post has gone some way towards summarising the main issues relating to the recent outage. Work continues to restore full service. If you have any comments or queries, please do post them below.
June 3rd, 2008 at 1:59 am
Guys, I know you were doing your best. Don’t sweat it.
June 3rd, 2008 at 2:01 am
Great response. all that cam go wrong will go wrong
June 3rd, 2008 at 2:10 am
As usual, you (Statcounter) are doing a fantastic job at a very difficult task - for free for a lot of people, and extremely low cost for those of us with paid subscriptions.
As usual, the level of professionalism at Statcounter is complete and commendable. Accidents occur, gremlins invade, stuff happens, and it’s not always under the control of anyone human.
Thank you, every one of you, for shouldering on, explaining the problem, picking up the pieces and getting back to it - and for letting us know what’s up. Transparency and honesty are coveted qualities in an internet entity. Keep up the great work.
June 3rd, 2008 at 2:11 am
Life is life…
I love this counter.
Thanks for the explanation.
June 3rd, 2008 at 2:25 am
What a horror story! I appreciate the effort to thoroughly explain this situation, which was inconveniencing me while you worked to get back online (my pages with counters were loading very slowly); and it made it impossible to change a counter when I needed to. I then went to your website but could not even get the Forum working, so this is the explanation I was seeking (though of course it’s infinitely worse than I imagined, and not your ultimate physical responsibility.) Keep up the good work, and thanks so much for your invaluable service; it’s superior to all the other counters I’ve tried.
June 3rd, 2008 at 2:44 am
Statcounter provides a tremendous service. Their are certain accidents you can not prepare for, but Statcounter did its best to keep users informed and updated. Thank you.
June 3rd, 2008 at 2:48 am
Excellent job guys to keep us up-to-date. If only companies had the same level of professionalism and customer service.
As for “The Planet”, if having a spatial spread of the servers servicing a single customer is a common tactic in this industry, they are partially at fault and must compensate you for your troubles.
At any case I hope you have a trouble-free recovery and year.
June 3rd, 2008 at 2:55 am
You guys ROCK! And I mean ROCK. Even if I was paying for an upgrade to service (and I’m here admitting that I’m not) I’d still say you rock.
Your service is invaluable. And friggin fun. So don’t sweat it b/c in the grand scheme of things this is small. And you’re so much bigger than that.
Now, those things being said, bad things in threes and all…paypal, this planet crap, and…please tell us someone over there had a bag hangnail or got their hair cut badly or something like that.
Keep up the great work over there, folks. And thanks for the great updates of everything!!
June 3rd, 2008 at 2:56 am
Where is Superman when you need him? (ok, somebody had to say it)
June 3rd, 2008 at 2:59 am
Hope everything is going well for you people there. Take care
June 3rd, 2008 at 3:07 am
Bah! So we lost a little bit of data, maybe. Who cares..? The way I figger (about this and life in general…) there’s no blood and everyone’s still alive, this too, shall pass. Oh and BTW can I just say MERCURY IS RETROGRADE! WHat does that mean..? You can see the info I have, and my experiences, here:
http://witchswonderfulworld.blogspot.com/2008/05/mercury-going-retrograde.html
Good luck & keep up the good work!
bb
dawtch
June 3rd, 2008 at 3:11 am
Hey, you guys run an excellent service, and you’ve done
an great job dealing with this difficult situation.
Keep up the great work!
June 3rd, 2008 at 3:12 am
Good job guys. Glad no one was hurt. Keep up the good work!
June 3rd, 2008 at 3:33 am
Hey,
You are one of the best. Take it easy. It’s not your fault. And thanks a lot for this post.
June 3rd, 2008 at 3:45 am
I agree.
You do provide a great service, for no charge. That’s not an easy feat.
June 3rd, 2008 at 3:57 am
thanks for your complete explanation. My heart will go on with statcounter.
June 3rd, 2008 at 4:11 am
Very much appreciate the update. It’s hard to find companies that care enough to provide great customer service and communication like you do. I can’t imagine having to deal with another company (and hope I never have to.)
June 3rd, 2008 at 4:16 am
Thanks for your honest efforts in explaining the outage. Your transparency is more valuable than losing 24 hrs of access data. Accidents are a fact of life and I hope that Planet Datacenter takes some lesson from this outage…particularly when they are handling 9000 servers…and contrary to their claims, they were keeping most of their eggs in one basket while claiming otherwise.
June 3rd, 2008 at 4:34 am
Your candor and responsible attitude more than make up for what little responsibility you had in this unfortunate accident. Keep up the good work. I’ll forever recommend statcounter.
June 3rd, 2008 at 4:40 am
It’s all good, dudes! Obviously this is pretty much the definition of a freak accident (save for like, meteors or something), and clearly very frustrating for you guys since the safe guards you thought were in place… weren’t. I appreciate how hard you’ve worked to keep everyone informed of what’s been happening, (as someone else said, IF ONLY other companies put that much effort into communication!) and I, for one, understand that sometimes shit just happens! Hope everything goes well with getting things back to working order.
June 3rd, 2008 at 5:15 am
Thanks Guys, This service is simply great. Although we missed our stats and missed Statcounter also but we are happy to see it working again.
You guys are doing great. Thanks for sharing all the problems with us.
June 3rd, 2008 at 5:22 am
Wow, a thorough and complete explanation beyond and above the call of duty.
I know that in this day and age everyone wants everything 24/7 and they want it yesterday. But it’s no big deal.
The outage didn’t bother me much because it happens — and even then it happens only very rarely. In all my years of using statcounter this is the only major outage I’ve experienced.
Some other websites I frequent got knocked off-line too, so I figured it was possibly a server farm outage.
June 3rd, 2008 at 5:24 am
Wow, a thorough and complete explanation beyond and above the call of duty.
I know that in this day and age everyone wants everything 24/7 and they want it yesterday. But it’s no big deal.
The outage didn’t bother me much because it happens — and even then it happens only very rarely. In all my years of using statcounter this is the only major outage I’ve experienced.
June 3rd, 2008 at 5:26 am
You guys are awesome. We are all grateful for your wonderful service.
Cheers,
micheailin o’cinnsealach
June 3rd, 2008 at 5:26 am
You guys have a great service. The interuption was quite minor for me. And compared to the product you deliver each day, it was nothing.
June 3rd, 2008 at 5:39 am
I was horrified to read of the explosion and relieved to find that no one was injured. My best wishes are with the company and all its employees to recover from this incident.
June 3rd, 2008 at 5:40 am
Hi there.
Thanx a lot for updating us, i actually found more about this outage from statscounter than i did from theplanet website updates.
It seems many companies use this dedicated hosting provider because i noticed that scriptlance.com also was offline for the same reason.
As it seems “big” hosting providers are not giving back as much as they should.
June 3rd, 2008 at 5:43 am
Statcounter is so far the best for me. I’ve been using your service for a long time, and i feel really satisfied with your service. Accidents like this could happen but it will not affect the excellent service that you give me.
You guys give us a wonderful and excellent service. Keep on going statcounter.
June 3rd, 2008 at 5:53 am
Boy, when you guys say an explosion, you mean it! I’m amazed you go everything back up so fast.
June 3rd, 2008 at 6:05 am
Thanks for all you do on our behalf. Everything’s cool. No harm done. Good luck and hope it works out for you without too many problems to solve.
June 3rd, 2008 at 6:08 am
Great detailed explanation, thank you. Unfortunately my stats were briefly up today (Monday) but are now gone again and have been reset to zero. No big deal, just curious why they were there for a few hours and now they’re gone again?
Anyone else on partition 13 see their stats now? (it’s Monday 11:07 PST).
StatCounter Team Reponse:
Hi Kansas,
Your stats for Monday should definitely be visible in your account. If you are continuing to have trouble please submit a support ticket to us by logging into your account and clicking the “support” link in the top menu bar. We will then be able to look into your account directly.
Thanks for your patience.
June 3rd, 2008 at 6:19 am
No worries…
It’s not your fault to begin with so no reason to feel so badly about it all.
Thanks for the great service.
June 3rd, 2008 at 6:26 am
Accidents happen. Also, it’s not your fault. You are doing a great service to us all. Keep up your good work. God bless you all.
June 3rd, 2008 at 6:45 am
Thanks for the clear and transparent explanation.
NASA has reported that it is safe from crazy Rasberry ants, but perhaps The Planet is not. Houston, we have a problem…
June 3rd, 2008 at 7:12 am
I now understand why it took a long time for statcounter to be back. Things happen, so, I am fine with that. I appreciate your efforts to get the service back at your best timely manner.
Throughout the years I have been using statcounter, it is the best I have found. So, despite this outage, I will keep using it.
June 3rd, 2008 at 7:45 am
Hey,
Don’t sweat it guys. Doing all you can do is the best service anyone can every give.
Thanks for the excellent work.
June 3rd, 2008 at 7:56 am
Don’t worry about that.
We personally don’t our stats on Sunday anyway.
Keep up the good work !!
June 3rd, 2008 at 8:38 am
A world of difference ….
Every other bulletin from Planet bleated on about them “working through the night”. I know you guys worked through the night also, but how many times do you mention it in your announcements? None.
You did a fantastic job …. now go get some sleep.
June 3rd, 2008 at 8:38 am
This couldnot really have happened at a more awkward moment for us, adding a small project which went live on Friday, a project we hope will lead to a bigger partnership. The project relies heavily on stats to show our potential client the value of our services.
All that said because we were able to keep the client informed, due to the efforts of your team.we got over the glitch. I have to say that we are happy to read that statcounter are already working on a new contigency plan so hopefully outages of this scale cannot happen again.
These things happen,and are to an extent expected, that they happen is not the main problem, it is how the problem is addressed and communicated to allow your clients to work around that makes the difference.In this regard Statcounter excelled !!
June 3rd, 2008 at 8:49 am
Thanks for the explanation. Statcounter clearly acted comeptetly and professionally throughout (with the sole exception of assuming, rather than verifying, that your DNS hosting was geographically spread).
I hope this will not make you feel that your should move your new system out of beta any quicker - day to day reliability is much more important than the risk of an outage like this.
June 3rd, 2008 at 9:03 am
At least you got it sorted quicker than it would take to get a reply from an email to PayPal
btw, typo: “On Saturday 31 May at approximately 11pm GMT there was an explosion in The Planet Date* Center…”
StatCounter Team Response:
Thanks Dan - all fixed. We’ll blame that on the sleep deprivation…!
June 3rd, 2008 at 9:21 am
A thoroughly understandable explanation - I don’t think it would be at all fair to blame Statcounter for the outage, especially given that you already had recognised the risks and were hoping to improve redundancy in the new version. Sometimes, these things just happen, and all you can do is communicate with your customers, which you did.
Having said that, just one minor comment. I bookmark my project so only visit the main stats page through this. This meant that I was never sure whether the error connecting was at my end or yours, and it was only by chance that I thought to visit the Statcounter front page through the address bar, where I found full details. Might it not be an idea to set up an automatic redirect on occasions of mass outage, so that all bookmarks are sent to a central explanation page (e.g. this blog). On the other hand, perhaps because of the nature of the problem with the DNS this would not have been possible anyway. But it is just a thought for next time…
June 3rd, 2008 at 9:34 am
You guys have handled the situation very well. When I was not able to view my statcounter for hours, I immediately understood, it may be due to some technical glitch. But I was not aware about the issue created from the Planet side. Anyways Cheers to the entire Statcounter team for working day and night to sort out this problem. All Good Wishes
June 3rd, 2008 at 10:07 am
Hi,
Take it easy.
Life is like that.
Best wishes for statcounter and the team.
June 3rd, 2008 at 10:12 am
Hey StatCounter Guys:
Since I set my project page as one of my quick boot-up buttons, I did not understand why my project did not show for a couple of days. But after seeing your notice I knew.
Next time I wonder if you can show your big breaking news on own project pages also.
Thank you for the hard work again, and please keep it up!
;D
StatCounter Team Response:
Thanks for the suggestion Kyoko - it’s been noted!
June 3rd, 2008 at 10:25 am
Your response was both thorough and quick. I appreciate your dedication to your users. This is good customer service!
June 3rd, 2008 at 10:27 am
All cool dudes!
Excellent service you guys provide anyway. This is the first time it went down since I started using it, for over a year and a half, so all good!
June 3rd, 2008 at 10:33 am
Thanks for the detailed information………you guys are #1……….keep up the great work.
June 3rd, 2008 at 10:38 am
StatCounter is an excelent service and I am impressed by the detailed and compreensive answer you provided. Outstanding. I wish I could get such a transparent response from payed services I know off. Keep up the good job.
June 3rd, 2008 at 11:29 am
I’m sorry to hear about this trouble and I hope it’s resolved completely soon.
However I still am unable to login on your site with my account that has been in place for years. I did get in on occasion yesterday but not at all today.
StatCounter Team Response:
Hi Kemi,
Sorry to hear that you continue to have trouble. Are you using Internet Explorer as your browser?
We have identified that some members are getting a “page not found” error when trying to log into their StatCounter accounts using MS Internet Explorer.
The solution to this problem is to clear your cache - after this you should again be able to log in.
To clear your cache go to Tools>Internet Options>General Tab> Delete Files. These instructions may not be exact for every version of MSIE but they will all be similar.
Hope this helps!
June 3rd, 2008 at 11:40 am
you really need some rest.
the red lettered comment above my projects spells depsite in stead of despite.
StatCounter Team Response:
Apologies Liesbeth! You’re right - that’s another typo that we will blame on sleep deprivation - thank you for reporting!
June 3rd, 2008 at 11:51 am
Having been in the IT industry for over 25 years I have to say that Statcounter is a first class operation. You were prepared to the max, you responded brilliantly and you communicated the problem to your users without hesitation. You went above and beyond the call of duty. Great job!!!
Sincerely
Michael Relfe
http://www.KinesiologyAffiliates.com
June 3rd, 2008 at 12:09 pm
Another very satisfied StatCounter customer echoing the sentiments expressed by many here - a message of support to a wonderful service. All the best and thank you for the way in which you handled this very difficult situation.
June 3rd, 2008 at 12:11 pm
Hey guys you are the best. So my attitude is when you have the best why mess with the rest. Keep up the great work.
June 3rd, 2008 at 12:11 pm
Stop being so European, and chill.
You guys have zero to apologize for, and all us who benefit from your service are grateful.
Cheers,
Annette
(a Brit who has lived in the States for 27 years, and who learned to calm down a bit in California, but who admittedly still has her European Moments…!)
StatCounter Team Response:
LOL! Thanks Annette.
June 3rd, 2008 at 12:24 pm
We had the same issues with the planet for several sites. Our complaints are largely ignored as the planets customer service has gone WAY downhill over the past few years.
As a former customer, I highly recommend looking into another hosting company (Rackspace and Midphase are 2 companies that I am using).
Great work on keeping us updated though!
June 3rd, 2008 at 12:33 pm
I’ve been using Statcounter for some time and believe you provide a tremendous service. Accidents unfortunately happen and you handled the situation like the professionals you are.
Thank you for the detailed explanation.
Job well done guys - keep up the good work.
Rob
June 3rd, 2008 at 12:41 pm
It’s only when such things happen that we beneficiaries of the services your dedication has provided appreciate just how much effort has been put in.
Thanks for everything guys.
June 3rd, 2008 at 12:46 pm
You guys are fantastic! I love everything about you and your service! Having said that, I have two small pieces of humble criticism…
1. Bad things will always happen (PayPal, The Planet). It’s how you handle it that counts. You don’t have to chuck those companies under the bus! Sure, it’s not your dirty laundry - it’s theirs - but you’re airing it! Rise above. Show some class.
2. In this day of leet speak and texted Twitter blogs, good writing is rare. This blog is well written, just one very small grammar tip to point out… subject-verb agreements. “The Planet” is a singular entity and requires a plural verb, as in “The Planet has,” not “The Planet have.”
Collective nouns can go either way depending on how they’re used in the sentence. Consider these examples:
“The band My Chemical Romance *is* coming in concert.”
“The members of My Chemical Romance *are* coming in concert.”
June 3rd, 2008 at 1:00 pm
Go ahead and continue your fantastic job. I’m sure the StatCounter team is doing his best to continue their project, congratulations !!!
Conclusion : Incidents may happen … no one is responsible …
June 3rd, 2008 at 1:28 pm
Oh well, life sucks. I think you handled it as good as can be expected. If a site is super serious about their stats, they’ll have dual counters anyway for verification purposes.
June 3rd, 2008 at 1:32 pm
To Aodhan Cullen….and all of your employees. You have done an outstanding job on keeping all of your customers fully aware of what has happened at “The Planet”.
Many other types of business, corporations etc… would do their best to keep all accidents, downtimes, etc… from reaching the public.
Paul
June 3rd, 2008 at 1:36 pm
Great communication. Good job.
And Stephen Logic… did you seriously just try to teach StatCounter some grammar? I’m sure that wasn’t the first thing on their minds…
June 3rd, 2008 at 1:55 pm
I’ve read all the posts above. Not one single rant, nobody yelling, “We’re never going to use you again.”
The largest complaint above is from a member of the grammar police.
You hold yourselves to high standards - higher standards than your clients hold you to - and you have pride in your service and product. For that we salute and admire you.
Can you name me one other entity out there that could be down hard for hours and hours on end, yet retain a 100% customer satisfaction rating like you are doing? This speaks volumes.
Thanks for doing what you do.
StatCounter Team Response:
Hi Jim,
We are almost speechless.
We sincerely try to be an open, honest and responsive company and provide the best service possible for our members… however, we would completely understand if our members were very angry with us following the recent outage. WE are angry with us.
So we are genuinely humbled by the loyalty displayed and the support we have received.
Thank you kindly for taking the time to comment.
June 3rd, 2008 at 2:05 pm
Look, accidents happen, and that whole fiasco was completely out of your control. Yeah, I may have lost one day of stats, but that’s really no big deal in the grand scheme of life anyway! I believe that your product is excellent and your commitment to your customers is evident.
June 3rd, 2008 at 2:41 pm
You folks are the best!
I think there are lessons for everyone in this incident.
June 3rd, 2008 at 2:42 pm
No big deal, life goes on. Anyone really bent out of shape about this probably needs to get their blood pressure checked.
June 3rd, 2008 at 3:11 pm
Am really appreciative to Statcounter for their professionalism
June 3rd, 2008 at 3:13 pm
Hi Guys,
Excellent message alerts. If anything the Planet has done, violated
typical reliability standards, then I support you in pursuing a resolution.
The next message alert should be regarding how much data has been
lost so that everyone can understand the breadth of what happened
with respect to their own little IT worlds.
This counter provides up-to-date information for my retail business because web searches are extremely indicative of the business activity that I will see by the following weekend. So the information is
valuable to many who have learned to use it as a business tool.
It lets me know when i can take time off. So, getting zero searches told
me to chill-out and relax and take a day or two off.
Thanks again,
Alan
June 3rd, 2008 at 3:15 pm
Its great to see some honesty guys! Not at all your fault, sure we lost 1.5 days worth of data, but that why we have more than one stat software, however statcounter has always been the best and easiest to use.
From all the way in Australia, don’t worry about it, mate!
June 3rd, 2008 at 3:25 pm
These things happen, and most of all I appreciate Stat Counter quickly adding a message to the site so I knew that the problem was being addressed! Just another example of your great service.
June 3rd, 2008 at 3:44 pm
Thank you for the clear and gracious explanation. No apologies required — you’re doing your best and keeping your users informed.
June 3rd, 2008 at 4:11 pm
I agree with all the above- very professional response to something that’s not your fault. Accidents happen, and you guys did a great job reacting to this one.
June 3rd, 2008 at 4:26 pm
It’s all good!! Thanks for keeping us to date.. I avoid using ThePlanet.com, their customer support is horrible!
June 3rd, 2008 at 5:06 pm
You guys have acted above and beyond the call of duty. Keep up the good work!
June 3rd, 2008 at 5:35 pm
Top notch service guys. Vendor issues can be an absolute bitch. Keep up the good work! And thanks for filling us in with the details so that we were all kept in the loop. You guys are a true class act. Can’t wait to see the new beta service you mentioned!
June 3rd, 2008 at 5:37 pm
It certainly made me realize that out of all the work I do on my websites, I rely on your software the most! It made me feel guilty that such a great service is free so I am going start paying for the upgrade. You have a terrific company and it’s the most valuable tool that I use. Thank you!
June 3rd, 2008 at 5:54 pm
Great service. I’ve been using you guys for 4 years and I wouldn’t expect any less quality service.
I actually live in Houston and used to see ThePlanet.com’s Data Center. Interestingly, I haven’t read much press of it in Houston media.
June 3rd, 2008 at 6:14 pm
[...] Due to an explosion in The Planet Datacenter at the weekend, service on this partition was interrupted. Depsite our best efforts 24 to 30 hours of stats spread across Sunday GMT and some of Monday morning have been lost. We sincerely apologise for this. More information is available from our blog. [...]
June 3rd, 2008 at 6:31 pm
Thank you for all the info..I must say this is the best help ever and I appreciate all you do..
Not sure doi I need new codes now for my Index page.
thank you once again
Stay safe
Catherine
June 3rd, 2008 at 6:41 pm
Thank you for your fantastic service through the years, you have huge integrity, and thank you for keeping us up to date, you are No 1. All the best.
June 3rd, 2008 at 7:03 pm
Hi,
I’m not being able to open the Statcounter website at all for the past twelve hours or so. I’m getting a Problem loading page error on Firefox whenever I try. Is there another/still some problem going on?
StatCounter Team Response:
Hi there,
You *should* be able to log in - are you using a bookmark to log in? Please try logging in directly from http://www.statcounter.com or my9.statcounter.com.
June 3rd, 2008 at 7:10 pm
I just want to say that StatCounter is a model of what customer service should be - to the point that I have been known to keep you in mind when dealing with my own customer service issues. I wish all my vendors had this level of integrity!
StatCounter Team Response:
WOW John,
High praise indeed - we really appreciate it - thank you!
June 3rd, 2008 at 7:26 pm
As a long-time paid subscriber I appreciate your hard work to get things back up and running quickly, and to evaluate the situation. That being said, in the four years I’ve been with you I think this is about the only thing I’ve ever seen go wrong, and it was definitely beyond your control. Don’t sweat it. You guys definitely ock in my books!
Cheers,
Connie
June 3rd, 2008 at 7:27 pm
….that should be, You guys definitely rock in my books! I guess you’re not the only ones who need sleep.
June 3rd, 2008 at 8:10 pm
I think StatCounter’s response to this matter is childish and irresponsible, and playing “heroes” to satisfy customers is beyond a joke.
It is clear there was an explosion, fire, and an evacuation. Where fire and electric are concerned people must come before data.. regardless of its importance - web stats are not vital to daily life - they are just numbers.
Without wanting to sound too judgemental, the 9/11 incident resulted in the same thing - fire, damage, loss of data. Were the companies “hosted” in those towers expected to give their clients the run around and apologise for that? Forget the loss of life - fire is the most destructive thing known, and I don’t see the need for any complaint, or to have expected The Planet to have rushed its response.
If The Planet were ordered to cut power by the Fire Brigade then it is a legal duty to respond to them - for the safety of everyone. The Planet would not have known the extent of the damage or loss until the situation was known. As I said - lives come first.
Statcounter needs to get off its high-horse. The Paypal incident was possibly a reason to be angry, but this is not. There’s no excuse to keep the ball rolling just because Paypal cost Statcounter a few customers - making The Planet a scaregoat is pathetic behaviour. No one cares about a few lost stats if the main concern is whether anyone was hurt. Burns hurt a person a more than a machine.
I’m sure everyone respects Statcounter for dealing with things how it could, but give The Planet a break. They thank you for your custom and yet you stick it up their backside for an accident.. for shame! If your offices burned down would you go to the efforts of custom replies to 100’s of customers, or would a template suffice for the initial concerns until the facts were known and a resolution prepared.
I expect The Planet are going to have to make insurance claims, repairs, etc. Thats assuming there is no long-lasting investigation into the cause. Do you expect them to return to 100% efficiency overnight?
I appreciate Statcounters service and dedication to its clients, but I think it needs to be more considerate towards real-life concerns and a little less self-centered in such situations in future. Accidents happen, but the “What did the Planet do wrong?” section is petty - end of the day, they’re running a bigger business than Statcounter.
If Statcounter is so upset and has been finaincially affected, why not take legal action against The Planet instead of flaming them in blogs?
Marcus
StatCounter Team Response
Hi Marcus,
We appreciate your views and your taking the time to comment, but we stand by our criticisms of The Planet. They claim to provide a fully redundant service, able to withstand even a total power outage - but this is not the case. For example, The Planet claimed redundant power… how can this be with only one power room? While, obviously, we are very grateful that there was no loss of life or serious injury, this does not excuse the fact that The Planet did not provide the service upon which they market themselves. Perhaps if you re-read our post, our stance may become clearer.
June 3rd, 2008 at 8:11 pm
Thank you for the detailed, and technical explanation.
June 3rd, 2008 at 8:16 pm
[...] Planet lost more than a few customers, and Statcounter.com may certainly gain some with a response like this, to a fiery service outage over the [...]
June 3rd, 2008 at 8:44 pm
Thanks for the full explaination. Having had outage time from my web provider in the last year, with no apology whatsoever, your refreshing honesty is much appreciated.
Have you ever thought of branching out
June 3rd, 2008 at 8:48 pm
I see SC is “moderating” by removing comments that disagree with their abuse towards The Planet. Such a biased stance is typical Americanism, as SC seems incapable of dealing with critisism.
StatCounter Team Response:
We’re confused Marcus… the only comments that get held in our moderation queue are those that appear to be spam or contain abusive words… If you disagree with us about anything, you are more than welcome to comment!
Typical Americanism…? We’re Irish!
Edit: NOW we understand - you were referring to the fact that your previous comment was not published. It has been released now and was held because we previously received numerous spam posts on our blog from the IP address from which you posted.
June 3rd, 2008 at 8:50 pm
‘Tis pretty much all good on my end, especially since I was lucky enough to not be on one of the partitions that went Tango Uniform with the data center. Yes, things were a bit slow yesterday and Sunday, but considering things were still jury-rigged, understandable.
Thanks for letting me know the lengthier explanation is out here. It is much appreciated.
June 3rd, 2008 at 9:14 pm
Spam? This is the first time I’ve ever posted in SC Blogs!
I know you’re Irish - but censorship isn’t something common to UK/Irish sites.. I realise also that I had run CCleaner and my comment had “vanished” - once I reposted it was showing again. Cookie?
Marcus
StatCounter Team Response:
Hi Marcus,
Apologies - we are not saying that you posted spam, rather that spam was posted by someone from the IP address to which you are currently linked. Also, there is a delay with comments being published due to the use of caching. This is necessary due to the high volume of traffic our blog receives. Hope this explains.
June 3rd, 2008 at 9:35 pm
Hi Guys,
I love statcounter and check it every day. You have handled this crisis so professionally, and we all wish you the very best! What is important is that none of you were hurt. Everything else is secondary.
Thanks for all you do year-round for us.
June 3rd, 2008 at 9:38 pm
I’ll truly say I was going ‘oh my gosh!’, but hey life goes on … there are bigger problems in the world.
Glad things are back to normal.
June 3rd, 2008 at 10:01 pm
The outage wasn’t a huge problem for me, I appreciate even with a high level of uptime (say 99%), in a given year that might mean a number of days when there’s no service.
That said, I can kind of see where Marcus is coming from. But I also understand Statcounter’s desire to explain exactly what happened.
I do, however, think that Statcounter should in a future post detail if and how they’re going to set things up to avoid problems like this in the future. Maybe you need redundancy across providers. Perhaps even just a backup service using Amazon’s cloud computing, that you can switch on temporarily if things go belly up with your usual provider, and switch your DNS to point to that (you could set up virtual servers with amazon to match all your servers with your main provider). I was kind of surprised to hear you also had your DNS management with The Planet. Most big sites outsource DNS management to a large dedicated DNS management service independent of their server hosting (which I gather is what you’ve done now).
I can see mentions that the new system that’s in beta now might help here, but no details of how exactly.
As for The Planet, I guess if anything it might make them better prepared for the future, so that they hopefully will going forward be among the most reliable in a catastrophe. Nothing beats real experience, and they’ve had some now.
June 3rd, 2008 at 11:00 pm
All is cool and thanks for being upfront. One or two days of stats lost is not bad
Keep up the good work!
June 3rd, 2008 at 11:11 pm
I am a VERY appreciative customer … more that you were willing to be so responsive to us than the fact that you got the servers up and running again! Your apology on your blog was so eloquent that I wish more companies would follow your example and remember that the customers are the people keeping them in business. Few companies are willing to be so apologetic when something goes wrong. Often, customers are left holding the bag and companies act like they’re doing a favor offering a product or service.
It looks to me like you couldn’t help what happened, and you took the steps necessary to rectify the situation. It can happen to anyone, and I for one appreciate your quick resolution!
Thanks so much for taking time to write to us and for offering such a detailed explanation!
June 3rd, 2008 at 11:13 pm
Thank you for the candid and detailed communication. Events will occur and all companies will have problems with service from time to time. Great companies differentiate themselves by how they handle and deal with their customers. It forms the basis of trust. Yours is a great company and your actions are consistent with that measure.
I have faith that you will continue to refine the fault resilience of your systems to improve the outcome when the next event occurs. Frankly, given the severity of the hishap at the DC, I’m impressed that you got back online as soon as you did and kept the data loss so minimal.
Thank you!
June 3rd, 2008 at 11:18 pm
I’ve been using StatCounter for — wow, nearly exactly 3 years now — and this is my first bit of lost logging *ever* as far as I know. 99.8% uptime for a completely free service, with the .2% being due to an explosion blowing out three walls of a data centre, is pretty a darn good record as far as I’m concerned, and I fully expect to be upgrading to a paid subscription in the very near future.
June 4th, 2008 at 12:30 am
Negative perspiiration here. I’m new onstatcounter, and it’s no big deal to lose a couple of days. Looks like mycounts are going up… (: I hope that all is well with you, and with the folks at planet as well. I’m sure that this was a terrible experience for them. All the best to you…G:
June 4th, 2008 at 12:39 am
Not a problem; these things happen, and no harm was done as far as I’m concerned.
June 4th, 2008 at 1:17 am
Sucks that service was down for 2 days for me, however good thing is that it’s back up. Thanks for keeping us informed, and good job on the recovery!
June 4th, 2008 at 2:32 am
Don’t feel bad, I lost all 7 of my websites from the same explosion, and 1 still comes and goes. It’s not your fault. I wonder if the “crazy rasberry ants” that have invaded Houston and love to eat electrical wiring had anything to do with it…
June 4th, 2008 at 3:06 am
Hey Statcounter;
I love your site. I love coming here and seeing who’s been to my blog and where they are from.
That being said, I’d like to agree with the Marcus comments. I understand your response to him stating that The Planet didn’t perform as marketed (i.e. being able to withstand a total power outage), but I’d like to point out in my opinion, a total power outage is completely different from a power outage caused by an explosion. A total power outage occurs…and it’s probable the data is still in the servers…maybe lose a teeny bit a info…but an explosion occurs, and that data is LOST. Blown up.
So please take that into consideration when you compose your complaint letter to The Planet. Again — the template response? Totally understandable given that they are scrambling with trying to regroup right now. I bet you a nickel you get a more personal response once things have settled down over there.
I would have to say that the overwhelming response to the lost data statcounter has is this: “bah…no one died, and we’re just happy your back”
No one is pointing the finger at statcounter for the lost data and your explanation of the occurrence was greatly appreciated. But I do ask again to reconsider a strongly worded complaint letter at this time.
Thanks for letting me differ in opinion!
I remain a loyal statcounter user…keep up the good work!
“norman”
June 4th, 2008 at 4:51 am
No, for some reason the site wasn’t showing up at all… Everytime I typed in the URL http://www.statcounter.com, I got the ‘Problem loading page’ error… and this happened on all three machines in my house!
I just tried my9.statcounter.com though… that seems tobe opening up, so yay!
June 4th, 2008 at 5:21 am
You are a very conscientious and dedicated site, thanks for everything.
June 4th, 2008 at 5:30 am
As always, you guys and gals have superseded the term used for customer service. You have always done an above excellent job and any criticism received shouldn’t be taken with a grain of salt as you weren’t at fault. In regards to “The Planet”, perhaps they should revise their procedures and handling of situations like this the way you all do. In my opinion the services you have provided are far, far above what any other company can provide and perhaps they should look to your company for guidance and handling procedures.
Well done.
From the staff at NMData ICS LLC.
June 4th, 2008 at 5:44 am
No problema Stat counter guys, glad you are back up and keep on rockin.
June 4th, 2008 at 7:15 am
We are with you guys.
June 4th, 2008 at 7:55 am
Guys you are a great team and give a fabulous service . As the saying goes ‘S—t ‘ happens . Sorry it’s the only way to express that things CAN and DO go wrong once in while . It happened when Skype crashed .Then Blogger had a mega problem only a short time ago . So you see, things go in threes ..What I’m glad about is, A) we have statcounter B) no person was hurt . C) in what is a very short time you terrific people have all sorted . So a few stats are lost . In a world full of terrible problems these are as nothing … Good work Stacounter team .
June 4th, 2008 at 11:44 am
thanks guy… the pro act like pro… keep up
June 4th, 2008 at 12:35 pm
The loss of 1.5 days worth of data was a bit unfortunate, but you guys have handled this as well as possible. One suggest - to put in something in the data charts for Sunday-Monday that mention the incident and loss of data, since that’s really the main thing that slightly bothers me - seeing the huge dip of “1 visitor” on Sunday on the summary charts.
StatCounter Team Response:
Hi there,
There IS a notice in all accounts affected by the outage - it’s in red writing just over the summary stats!
Hope this helps.
June 4th, 2008 at 12:35 pm
Great job guys! Thank you 1000 times.
June 4th, 2008 at 1:11 pm
Great recap, no worries. You guys are doing a great job.
June 4th, 2008 at 2:19 pm
Thanks! I know you were doing the best, thanks for the explanation, statcounter is the best free counter, Thank you for providing us a good quality free service.
June 4th, 2008 at 2:33 pm
Great service, great product.
I am just a small user of your service but I have to say I recommend it to anyone who asks. It did not really affect me however I hope all goes well with the new plans.
June 4th, 2008 at 2:52 pm
Great work with the recovery, guys! Appreciated your disaster recovery plans. Keep up the excellent work at Statcounter.
June 4th, 2008 at 3:07 pm
Appreciate the explanation. You’ve clearly learned valuable lessons about vendor disclosure and risk reduction through portfolio diversification. I’d like to make a suggestion. For the future, please consider a more PROACTIVE, rather than REACTIVE response. Put all of your customers’ email addresses on a backup file. Send your customers a proactive email when there is an event. This will reduce trouble tickets as well as tamp customer frustration with lack of communication when there is service disruption. Thanks!
June 4th, 2008 at 3:58 pm
i have had very many counters over the years (and i don’t leave comments AT ALL) but here i am now saying thanks! and telling you that to this day, i only use statcounter
June 4th, 2008 at 4:21 pm
Wow, that’s a very thorough explanation! So- I don’t mind and I agree with the others who said that Statcounter rocks!
June 4th, 2008 at 4:31 pm
Thank you so much for the comprehensive explanation. Boy, you give far better service than many of the big guys!!
June 4th, 2008 at 6:38 pm
Thanks. Far more visibility into the outage than I could reasonably expect for a free service. I can only hope other (paid) vendors will live up to your standard.
Nice job.
June 4th, 2008 at 6:39 pm
Thanks. Far more visibility into the outage than I could reasonably expect from a free service. I can only hope other (paid) vendors will live up to your standard.
Nice job.
June 4th, 2008 at 6:42 pm
A note to those criticizing StatCounter’s response:
ThePlanet has advertised “With multiple state-of-the art data centers located in Dallas and Houston, Texas, The Planet provides On Demand IT Infrastructure backed by complete redundancy in power, HVAC, fire suppression, network connectivity, and security. So if any of our data centers experiences a disruption for any reason, your eggs (or servers) are never in one basket.”
This was obviously not the case, as there was no redundancy on the DNS servers that tell the rest of the internet where to go, nor was there true data redundancy. According to ThePlanet’s boilerplate and industry best-practice, there should have been backup servers in multiple, separate locations and there should have been a backup DNS server at a separate location so that no matter what happened, normal failover processes would kick in and downtime would be minimized.
One poster made a remark about 9-11. It happens that I was working on 9-11 for a Very Large Multinational Telecom who had a server room in Building 7. However, they also had true redundancy - the kind that ThePlanet claims to have - and downtime on 9-11 was less than one minute.
That’s right. NO downtime to speak of. Why? Because those of us who were tasked, partially or completely, with keeping that data up and running knew where those servers were, what was on them, and where the backups were, and we the failover switches were thrown - from North Carolina, in the case of one small bank of machines that wasn’t set to automatically fall back to another location on fail - about 15 seconds after the second plane hit.
That’s how a redundant mission-critical data network is supposed to run. As much as I sympathize with ThePlanet for their loss and celebrate the fact that their losses did not include life or injury to human beings, the fact remains: They failed to deliver what they promised. With all due respect, anyone who thinks SC doesn’t have a legitimate gripe really has no idea how large data centers are configured and operated. There should be a backup of every single bit - literally - of data that is 100% synchronized, geographically separated (ideally by several hundred miles at least), and ready to go into action automatically, or as close to it as possible, when the primary machine 404s for any reason.
That is what TP promised, and they did not deliver. Do they have my sympathy? Absolutely, and I’m sure the SC team feels the same way. Do they have an excuse for their failure?
No. Not in the least. Any competent IT organization that size should have these systems in place and automated as a matter of common sense. It’s the IT equivalent of keeping a spare tire and a jack in the car…except in this case, you’ve got several thousand people in the car and relying on your ability to change a tire if you get a flat.
StatCounter Team Response:
We’re of the same mind it seems LowGenius - we couldn’t have said it better ourselves! We appreciate your taking the time to post - thank you… and best of luck with the site redesign!
June 4th, 2008 at 6:55 pm
StatCounter rocks regardless!
I’m very happy with your service, so I’m not upset or unhappy at all.
In addition, I’m glad no one was hurt at The Planet’s facility.
June 4th, 2008 at 8:32 pm
Amen LowGenius, I read those posts yesterday and wanted to respond but could only assume that data centers worked the way you just indicated so I had no real information to back my claim. Thanks for taking the time to post.
As for StatCounter staff, for a free service (sorry, I have not upgraded), you’ve definitely amazed me with your response. If only businesses that actually collect my money would be half as informative as you, this world would be a better place.
Keep up the amazing work!
June 4th, 2008 at 10:04 pm
Hi
As someone who just absolutley dispises poor customer service and badly run businesses (coming from the UK you have to just put up with it for some reason) I want to say how nice it was to read your report on the recent ‘outage’ we all had.
It was written well and with plenty of detail as to the cause and future preventative measures to be taken and most of all an APOLOGY, even though it wasn’t your fault.
As the saying goes ‘what a refreshing change’ !
Just how hard is it to say Sorry and keep people informed about what is going on, not very.
You put to shame the VAST majority of companies who Charge for services and are run by Idiots with customer service departments that are put there purely to fob people off.
So Thankyou very much for the Free service you give me and the superb Customer Service, Honesty and fast responses you give to all your ‘Members’
Very kind regards
Andy
June 5th, 2008 at 12:42 am
I’m a Sysadmin/DBA and lived the awful experience of a Datacenter loss, that was a nightmare don’t wish to anybody. Thanks for your great service, and the effort/sacrifice you did this weekend showed again the compromise with us. After this I’m seriously considering to upgrade my account…
Regards
OracleDisect
June 5th, 2008 at 3:42 am
I launched a new page that weekend and loosing data about how it did is a definite loss. A huge loss. If The Planet implied something they didn’t carry through, it’s definitely worth pursuing.
But you know what, fire happens, no one died, life goes on. Good work through the easy times and the hard times.
June 5th, 2008 at 7:23 am
Thanks for the detailed explanation.
Andrew
June 5th, 2008 at 7:23 am
I guess we should all recognise and acknowledge the fact that most of us use statcounter under a free registration to help us track our websites effectively. Even if there are users that have paid that services, we see that the problem is fully rectify in one day, which is really fantastic. Besides, this problem is not to much extent such that it happens very frequently or often that it will affect users detriementally.
On the bright side, it is pleasing to see that though we are registering under a free user, the statcounter team was really enthusiastic in solving the matter calmly and maturely, which really needed, render to us the correct form of services.
i am glad that we have such a such a great team of technicians powering statcounter.
great job
June 5th, 2008 at 10:05 am
These things happen. Don’t be so hard on The Planet. Don’t know the company, but a fire can always cause serious damage. We are still alive and counting …
PS: love the counter.
June 5th, 2008 at 12:17 pm
It would be hard for me to put a price tag on the value of the information SC has provided me with for several years.
While s*** does happen in life, planning ahead can avoid much of it. Thus, I’m wondering if SC has looked at hosting with 1and1.com whom I’ve used & trusted with all my websites since 2005. With banks and finance companies relying on my hosting services, I believe they offer the safest locations, back-ups, etc. Based on weather patterns in recent & projected years, I would not be comfortable with servers in Houston or Dallas.
June 5th, 2008 at 12:38 pm
I have been using StatCounter’s services for over five years and have never had reason to switch. Other stat and counter providers–free and paid–have come and gone but SC has gotten better and better over the years. I love the service and I love how there are real people behind it who care enough to inform members of matters that concern us all. Next to the great service, it’s their thoughtfulness that sets SC apart.
Go, Aodhan and the whole StatCounter Team! You’ll always have my vote!
June 5th, 2008 at 3:56 pm
Jen in particular needs to be commended for constantly updating the situation via the forum. You guys have my vote, you are always on top of things.
June 5th, 2008 at 4:29 pm
Wow, it seems like you have something against the planet. “The Planet have let us down, and in turn, we have let you down. For this we are truly sorry.” I dont think any of your free customers will think that is the case.
I am about to leave statcounter as I need all my counters to be in house from now on. This isnt because of your service of anything else, I have simply outgrown your free plan. My time at statcounter has been amazing. Your support is brilliant and the product is ideal to every website owner.
But I wouldnt be too pissed at the planet. I mean its not like they can prepare very well for something like this. At least none of their servers were lost.
June 5th, 2008 at 6:12 pm
You guys have done a great job of explaining and fixing things. Thanks for all the effort and great serivce.
June 5th, 2008 at 6:12 pm
I have a free account, so losing my stats for one weekend afters YEARS of perfect service is not going to get me worked up.
June 5th, 2008 at 7:45 pm
There’s a lot of interesting comments on here. You can tell who makes money (Or lost some) with their websites and who doesn’t. I’m stuck between sides. On one hand, I was ticked I lost my server since it caused me a fair bit of problems.
On the other hand, what can you do at this point, I had to explain to my customers the same thing, that there was an unfortunate incident where my server is, and I could do nothing. Most of my customers understood, while a few responded by being upset and didn’t except any excuses.
So which one should I be? The one who excepts it for what it is, or doesn’t except any excuses? I’ve decided to just leave it at that and be glad this doesn’t happen that often
June 5th, 2008 at 8:01 pm
How about not being so cheap and keeping online, offsite backups?
We do it and weren’t affected by the outage even though out servers reside in H1.
Not shame on The Planet. SHAME ON YOU!
June 5th, 2008 at 10:21 pm
De loin le meilleur compteur gratuit du net, et le plus performant… Désolée pour vos déboires mais personne ne peut vous en vouloir. Merci de ces explications claires et que l’on sent sincères… Continuez et merci !
June 5th, 2008 at 11:42 pm
I’ve been using Statcounter for the past couple of years and that you provide this service free is just amazing. I have been using SC for my blog - now I’ve added it as one of my business counters I’m going to pay once my log file reaches its limit. I consider it a cost of doing business and I’d like to think I’m supporting a service I really like.
I think if one isn’t paying for this service then really they should have little to complain about in the big scheme of things. And if one is paying… I think it’s still true to say that accidents happen, and with the best plans and prep in the world things like this will happen. Servers go down. So I lose some stats… big deal.
The way you handled this throughout is stellar… well done.
June 6th, 2008 at 12:16 am
The fact that The Planet didn’t spread your servers over different locations proves that they are not fully aware of Murphy’s law. This must be the hardest lesson for them!
June 6th, 2008 at 12:27 am
Only one of my sites were affected, so it’s not a big deal. What’s more, I only added the counter a day before the crash. So what I did was delete it and add it again, so it got assigned to another (working) partition from the affected ones.
As far as I know many sites got affected by this fire at The Planet’s server farm, and I’m as always impressed by StatCounter’s very fast response.
June 6th, 2008 at 2:51 am
Thanks for free tools
June 6th, 2008 at 7:54 am
I fully support you and feel that, for a service which you provide for free, you are making a 200% effort to keep us happy.
My websites were amongst the thousands which were offline and I’ve had to transfer to another server but only after waiting many days hoping that the promises of repairs “in the next few hours” would materialise.
June 6th, 2008 at 10:05 am
Great job during the recent Planet outage. I really appreciated your professional response to the situation, especially the fact that you kept us informed of the ongoing problem. I am very pleased that I selected StatCounter - you are the best!
June 6th, 2008 at 10:21 am
To Dave & anyone else with the same sentiment, I would like to say that SC has every right to get upset at The Planet. SC is a client of The Planet, and that itself speaks volume. What makes it even more aggravating is that The Planet has specifically promised “…your eggs (or servers) are never in one basket” in order to avoid mishaps exactly like this. Furthermore, communication efforts with The Planet were tackled in a most unsatisfactory manner.
While we are all glad that there was no human casualty at The Planet, it is rather appalling to note that The Planet’s most essential promise was not duly executed.
Keep on trudging through. Well done, SC.
June 6th, 2008 at 12:26 pm
It was nice to know about “The Planet”, and it was not a big problem.
Sc4 is working very well.
Thanks
June 6th, 2008 at 1:55 pm
Are there more people among you who have problems with their stats? My counter doesn’t recieve data since yesterday, today is even worse…
June 6th, 2008 at 4:58 pm
It’s all good Statcounter is the best by far!
Keep up the good work!
John D
http://www.JohnDennerRocks.com
http://www.YodaVanHalen.com
June 6th, 2008 at 6:27 pm
All we can ask of you guys is your best, and I am satisfied that you’re doing that. The only thing worse than an accident, is a lack of information about what happened. That is apparently what The Planet did to you. Thank you for not repeating the same mistake with your own customers.
Best of luck.
June 6th, 2008 at 7:29 pm
I kind of missed the whole thing
well thank God no one was hurt. Thanks for the detailed information. All the best to Statcounter.
June 6th, 2008 at 7:43 pm
Hi guys
Thanks for the info, I think i only lost one days stats in the end, which isnt bad, as you said on ur blog accidents are a fact of life, and afterall statscounter is free after all !
Woc
June 6th, 2008 at 8:40 pm
Thanks for all you do. I don’t know how you could have done anything differently. You ugys are fab. Thanks, and thank goodness no one was injured.
June 6th, 2008 at 10:03 pm
I used to work for theplanet, I have many friends that still work there. Having said that, I know for a fact that they have datacenters in Houston and Dallas. I also know for a fact that they have backup generators, and ups to cover everything imaginable. However from talking to my friends that work in the Houston datacenter, this was not something that anyone could imagine happening. I also think that ThePlanet is doing an excellent job of getting customers back online. While I understand downtime sucks…you have to realize that if electricity cannot get into the building, there isn’t much all the redundancy in the world can do to help make things better.
June 6th, 2008 at 11:36 pm
Similar stuff has happened to me - hosting companies promise everything, the let you down, threatening the viability of your business. When I checked Statcounter at the weekend and got ’server not found’ I knew that something server-related was wrong, probably beyond Statcounter’s control. I like the service, the transparency and directness of the communications… Long live Statcounter! Dishonest hosting companies, go take a running jump…
June 6th, 2008 at 11:51 pm
I appreciate your timely response and full explanation of what happened. You guys are great! Things happen: often they are strange, baffling , maddening things. It’s part of life! I’m glad nobody was hurt. Thanks for your terrific service.
June 6th, 2008 at 11:56 pm
Have a cold beer on me, kick your boots up and relax.
Life is short, Break the rules, Forgive quickly,
Respect your elders, Kiss slowly, Love Truly,
Laugh uncontrollably, Love yourself, and be the
best person you can be. YOU…
Never regret anything that made you Smile.
Life may not be the party we hoped for, but while
we’re here we should Dance . . . .
bye
June 7th, 2008 at 4:17 am
Hey - the outage is no big deal - don’t worry about it.
If customers are angry - tell them to go elsewhere !
Accidents happen……
June 7th, 2008 at 5:36 am
Great job guys! Thank you for your fantastic service through the years, you have huge integrity, and thank you for keeping us up to date, you are No 1. Thank you 1000 times All the best.
June 7th, 2008 at 7:10 am
Still a fanatical fan! Can’t say enough good about you and your service–and your efforts to keep us in the loop. You’re an inspiration and a fantastic example of how companies should strive to behave.
June 7th, 2008 at 8:41 am
Thanks for the full and frank explnation of this issue. We will be scrutinising the back up and DNS services our partners provide us even more carefully from now on!
June 7th, 2008 at 8:17 pm
haha, damn… 24 to 30hrs of downtime — after over 5 years of uninterrupted, FREE, amazing stats. I wont be looking for another stat company any time soon!
Thanks for the great service! Never said it before, but statcount is amazing
June 7th, 2008 at 10:22 pm
No problem guys! Sh** happens. I completely understand. I appreciate your concern for your clients.
June 7th, 2008 at 11:06 pm
I’ve been voluntarily paying you folks for years… and I’ll continue to voluntarily pay for your free service for many more years.
June 8th, 2008 at 3:05 am
Thanks for such a thorough explanation however I really don’t have any complaints - I find Statcounter to be very reliable and the power outage really was out of your hands. Thanks for the great service you give us!
June 8th, 2008 at 9:43 am
Thanks for this stat counter team. I also was experiencing very few or no increment in the counter. I was thinking of just too many things that might had happened. Things like this does happen from time to time. The only thing that we can do is learn from our previous mistakes and be prepared in the future.
The most important thing in this world for me and I believe most of the others is “Data”, which is safe.
Cheers to stat counter team and the ppl @ the planet.
June 8th, 2008 at 10:07 am
A little late to the game, but I want to say that I am bummed this happened but Stat Counter is awesome regardless of thisrandom accident. Keep up the goor work!!!
June 8th, 2008 at 1:47 pm
Thanks for the detailed explanation.
June 8th, 2008 at 2:52 pm
As usual, you guys have been honest and up front with us and are working hard to rectify a difficult situation. I am a loyal StatCounter fan and I commend you on your actions. Cheers!
June 8th, 2008 at 4:05 pm
thanks
June 8th, 2008 at 4:31 pm
No problem!
It’s all part of the wonderful world of computers!
Keep up the good work!
Ted
June 8th, 2008 at 4:43 pm
Thanks for doing everything humanly possible from your end and, keeping us up to date… You and your team are to be commended for your professionalism… have a pint on us
June 9th, 2008 at 4:31 am
You guys have done a great job. I would have never known anything happened without seeing this post. Obviously Statcounter did everything in its reach to keep service up and everyone informed. There are unfortunate things that will always happen. Mistakes are made whether its overloading a circuit or catching a roof on fire with a torch like with what happened at Universal Studios. People need to be understanding and see that you guys did a wonderful job informing us of this event beyond your control and getting things back to normal. No one could have seen this happen and no one could blame you for doing your best.
Thank You!
June 9th, 2008 at 4:35 am
No worries. You guys do an excellent job. Just sorry that you had to go through this…
June 9th, 2008 at 5:19 am
no sweat! u guys are always going a great job. no explanations needed. thanks for the amazing service.
June 9th, 2008 at 5:28 am
Seriously , you provide such a consistent service that we knew this must be something big and beyond your control. I’m sure everyone using your service supports your effort.
Thank you
Michael Kerr
June 9th, 2008 at 11:06 am
Too bad - sooner or later you have or want to work with others - you can’t do everything yourself\in house - when your ‘partners’ let you down, not delivering what you pay them for, you have to reconsider your cooperation - look for alternatives. And, don’t put all eggs in one basket, as it can strangle you.
Also, good you expose them - too often they try to cover it up, just throwing in more PR bloat - same goes for Paypal, another company letting their customers down big time.
Based on your detailed explanation, I’m not blaming you at all - let’s hope you can prevent it in the future - a lesson learned from this unfortunate experience.
June 9th, 2008 at 3:15 pm
No big deal as long as no one was hurt then its cool!
June 9th, 2008 at 8:56 pm
Hey, shit happens. Your service is the best!!!!
June 9th, 2008 at 10:09 pm
I agree statcounter is running and counting, it is alive.
The coments on this blog show that.
The best
June 10th, 2008 at 3:03 am
Don’t worry, guys. You have done a great job. Thanks for the information on the explosion. Glad no one got hurt.
June 10th, 2008 at 3:48 am
A nicely worded, intelligent and heartfelt explanation, guys. Thanks for taking the time to write (and rewrite) your blog up there to get into words your frustrations and your eagerness to be up front, while also not shifting the blame, but still holding The Planet accountable for their actions, inaction and reactions. I was already a fan of your company, but now I’m a fan of you guys, as well.
All the best,
Joey
June 10th, 2008 at 1:30 pm
like they all said … twice.
June 11th, 2008 at 12:38 am
Thanks for the thorough explanation of what happened, things can and will go wrong.. luckily this was dealt with in (what to me seemed) a very timely fashion, no sweat at all.
June 11th, 2008 at 12:49 pm
Thanks for all you do. guy
June 11th, 2008 at 3:53 pm
An excellent job of incident management here guys. Your service provider let you down and as you say, you’ll take that issue with them.
This communication goes above and beyond normal expectations and you should be commended for doing such. A search of The Planets website doesnt even give a hint that anything happened, let alone any word of apology.
For what I guess the majority of users are taking advantage of a free service, we can’t ask anymore of you.
May I wish you all the success and good fortune for the rest of the year ! I will certainly not hestitate to recommend Statcounter to friends & colleagues.
June 11th, 2008 at 3:58 pm
What was the initial source of the Explosion?
That sounds a tad more than just a circuit being overloaded and arcing…
This might make a good urban legend in a few years…
June 11th, 2008 at 9:05 pm
Sorry to hear that your company was affected, I can see the major drop in reporting for only one day it looks like you guys had things back up very quickly as normal.
Keep up the great work
June 12th, 2008 at 6:29 am
Great job! Hope everything is going fine soon.
June 13th, 2008 at 2:00 am
[...] web business, come away from the YUI docs and put down the Symfony manual, and read the comments on StatCounter’s blog. Then go explore the rest of their site, and their forums, and learn why they got that reaction. [...]
June 13th, 2008 at 2:05 am
Well, I thought it was my site, but now I know better. You guys are great! I always recommend you to anyone I know with a website. You’re the best counter out there.
June 14th, 2008 at 2:46 am
Thanks for the detailed respond. Life is life, good things bad things do happen. You people already did a wonderful job. Get well soon. I need your counter.
June 14th, 2008 at 1:00 pm
You are a very conscientious and dedicated site, thanks for everything.
June 14th, 2008 at 10:54 pm
No worries. Life goes on…
Thanks for the detailed explanation about what happened, but I’ll still be using moving forward.
Cheers,
Brandon
June 15th, 2008 at 2:13 am
Thank’s for statcounter, so happy to my friend all…
June 15th, 2008 at 2:42 am
Your customer service is second to none, and your communication is exceeded by no one. Thank you for caring. We are okay — losing a few days worth of stats is nothing compared to the service that Statcounter provides.
Thanks,
Mike
June 15th, 2008 at 2:11 pm
Thanks for your professionalism in reproting and handling and good luck in the future! How the response team gets some time off soon!
Chris
June 15th, 2008 at 9:27 pm
No apologies needed for me: less downtime with your FREE Statcounter than with my PAYING Internet provider
Thanks so much for taking care of all of us!
June 15th, 2008 at 10:35 pm
I totally understand, mainly because some of my sites were down because they are at the same data center with two different hosts.
Your service and support is great, thank you!
June 16th, 2008 at 1:36 am
As always you guys have done your best, these things happen. The Planet could have done a better job but you are not responsible for their performance. The important thing is no one got hurt and you at Statcounter have been up front about the whole situation.
As always thanks for your great service!
June 16th, 2008 at 4:22 am
I have nothing but good about Statcounter. You guys have done your best. No worries. The good thing is no one was hurt and some good lessons learned fom all of us. We also from G5 Business Directory and Zeroservers.net have learned from this experience. Drive on Statcounter and continue to lead the way.
June 16th, 2008 at 1:09 pm
Your honesty and professionalism is outstanding and commendable.
Thank you very much for all you do.
June 18th, 2008 at 1:27 am
GODDAMMIT! I HATE YOU GUYS! WHAT THE FREAK?
Just kidding. I probably got shit for visitors the days affected anyway.
June 18th, 2008 at 11:04 pm
thank you
June 19th, 2008 at 4:41 am
I thought it was a small server problem, but it is a scary strory.
June 20th, 2008 at 10:04 am
Now stand up, lift your hand and give yourself a big pat on the back! Thanks for keeping us updated. You did your best and so far, I’m a member for life!
June 21st, 2008 at 5:51 pm
Oh! not good
June 22nd, 2008 at 9:25 pm
[...] ganze Geschichte gibt es im Statcounter Blog. permalink | trackback uri [...]
June 24th, 2008 at 2:50 am
Thanks for the blog post to explain it. I don’t think it’s StatCounter’s fault.
June 24th, 2008 at 5:00 am
How can people get angry at something that wasn’t your fault? Gahh! So what they lost a single day of who visited their web sites. They’ll make that up.
Thanks for explaining, and I am very glad that no one was hurt. I am very happy and thankful for the service you provide. Kudos to Statcounter and may you live long and prosper!
June 24th, 2008 at 2:42 pm
Accidents do occur and I’m sure the inconvenience you’ve faced was in greater proportions than us on the user-end. I’m impressed with the detailed explanation and I see no reason to complain.
I’ve enjoyed the services of statcounter and this is, but a mere blip. Keep doing what you do best! =)
June 24th, 2008 at 9:44 pm
no problem , it is good that you annouced us
June 25th, 2008 at 7:59 am
Sounds like a nightmare, glad it is all sorted…
June 25th, 2008 at 10:36 pm
I love this counter…
July 14th, 2008 at 5:45 am
Again, thanks for keeping us all posted - above and beyond duty by supplying all the extra info.
My hosting company could stand to learn a few things from you guys regarding customer service etiquette
July 18th, 2008 at 9:41 pm
Some great person once said;
Do not search to be a person of success, search to be a person of integrity.
You people at Statcounter are doing a great job.
Best regards.
August 8th, 2008 at 9:09 am
Thank you for sharing…