Browse > Home / Blog / Blog article: Houston Data Center Explosion and Fire

Houston Data Center Explosion and Fire

June 1, 2008

Update: Interview with Yvonne Donaldson of The Planet.

Update from a third-party #6: According to my latest source, the FBI shut down the facility to confiscate servers. One customer tried to walk into a restricted area and was shown an FBI badge and told to turn around. The stories swarming around this outage are pretty far out. Who knows what is going on? Anyone have pictures? The real story?

Update from a third-party #5: The third-party who gave me the information for update number four has recanted his account. He did so after realizing I had posted his update to this site. He did provide several photos of the power cables powering the site on a temporary basis:

Update from a third-party #4: The ‘unexplained’ data center explosion and fire at The Planet wasn’t so unexplained. According to third-party reports from engineers who have been on site at The Planet’s facility, the incident was caused by an upgrade they were making to their power systems. The good news is that it WASN’T a random, crazy event, but a foreseeable accident.

Anytime a data center provider is working on their power systems they really need to open a maintenance window, notifying customers of possible downtime even if they don’t expect downtime. Adding generator is a GOOD thing, but connecting two live feeds at the same time is likely to cause a surge that will result in an explosion. Evidently this is what happened cause the explosion and fire at the Houston facility.

When I was in the data center business I learned this lesson the hard way. We were adding customers to a PDU and instead of opening a maintenance window we thought we could add more circuits without a problem. Turns out we didn’t count on the contractor dropping his metal wrench into the PDU and shorting out all the connect customers. It wasn’t pretty. We never did that again. Even simple upgrades deserve client notification - from what I understand The Planet made no such notification to their customers.

Update from The Planet #3: Power has been restored as of 2:45 according to The Planet.

Update #2: Early reports from the Houston Fire Department were that they had no record of a fire or explosion at the facility. We called to confirm (we felt like real reporters) and were finally told that someone would call us back today. No one ever called. We have been asking people with specific knowledge of the fire to post their photos and/or stories. We have not been able to independently confirm that a fire or explosion occurred; however, it seems as though the fire marshal was present as much of the conversation in the forum refers to the ‘marshal’ allowing The Planet employees back on site. Seems like the early reports of a conspiracy were simply conspiracy theories.

Update from The Planet #1: We continue to pursue our plans to provide initial power to our H1 data center this evening. It will take several hours to assure power can be safely restored to the facility. Based on how the initial work goes, we will have more information to provide you in the upcoming hours. We will post another update by 7:30 pm tonight. In the meantime, we have further rerouted both old and new IP addresses for our name servers previously housed in H1. This means that the servers can start resolving IP addresses on both their former and new addresses, and this will alleviate the issues we have been seeing with propagation delay from this address change.

Original Post: The Planet’s Houston data center fell victim to a electrical transformer explosion and subsequent fire. The explosion knocked down three walls and more than 7,500 businesses are affected (more than 9,000 servers are down). The company had hoped to get the power restored by 5PM; however, as of 5:46PM there has been no update. Will keep you posted as more information is available. Thanks to Allen Stern for the best information available.

Written by Alexander Muse · Filed Under Blog 

Comments

6 Responses to “Houston Data Center Explosion and Fire”

  1. allen Says:
    June 1st, 2008 at 5:49 pm

    hey thanks for the compliment - i appreciate it!

  2. martin Says:
    June 1st, 2008 at 9:22 pm

    thats why zetaboards and invision are down aswell

  3. Brian_D Says:
    June 2nd, 2008 at 2:02 am

    Blank Label Comics was affected, one I visit on weekdays, Waspi Square by Paul Taylor is down and gets redirected to the notice page

  4. Ferodynamics Says:
    June 2nd, 2008 at 6:30 pm

    It never fails to amaze me how many problems these datacenters have. No matter how many precautions they take, it’s always something bizarre like this. Fires, floods, explosions, robberies, generators that don’t work, power grid problems, disk failures, disgruntled employees, it’s always something. It’s like life, you’re breathing one minute and dead the next. Which reminds me, I should probably backup my data ;-)

  5. CodyLoco Says:
    June 4th, 2008 at 3:18 pm

    Nobody can honestly complain about the servers going down-

    This could have happened to any datacenter and had it not been for the firefighters stopping them the power would have been back up almost immediately!

    On the flipside, it did stall the fixing of their order form bug that was giving users 3x bandwidth: http://www.web-hosting-coupon-deals.com/dedicated-servers/the-planet/34-the-planet/52-theplanet-pay-296-for-a-799-server-huge-server-sale.html

  6. SomeDude Says:
    June 5th, 2008 at 6:05 am

    It was just a transformer that blew up in the building, what’s with all the crazy conspiracy theories and rumors? Seriously, it’s whacked when a company actually tries to be upfront and honest with its customers and then some douche starts spreading rumors that might drive customers to actually dishonest companies.
    There was no FBI, no upgrade… the power company’s transformer just went boom.
    No one got hurt and all is well.
    You’ll get over it.

Leave a Reply