Interview with Yvonne Donaldson of The Planet
June 5, 2008
Many of you know I have been following the explosion and electrical fire at The Planet’s H1 data center in Houston. I originally posted a link from Allen Stern who had reported the following:
- Electrical transformer explosion at ThePlanet’s Houston datacenter.
- 3 walls knocked down - but servers are ok
- Backup generators cannot be started due to fire safety concerns
- 9000 servers affected - over 7,500 customers.
- Countless third party services affected by outage
Subsequent to that post I received tweets, IM and email from customers initially asking me questions and eventually giving me information. Quickly I began hearing crazy speculation (i.e. it was the FBI, it was aliens or something even more sinister). Earlier today Yvonne Donaldson, The Planet’s director of public relations, called me to talk about my ‘coverage’ of the incident. Yvonne is a Louisiana State University graduate with a degree in Sociology. She worked for Compaq from 91-97 in the PR department and got back in the PR game in 2006 when she began working for The Planet. I asked Yvonne if she used Twitter, but she indicated she did not. Yvonne explained she hadn’t slept in 24+ hours and I was appreciative she took the time to give me a call. I had no idea she read my blog!
Yvonne wasn’t too pleased with me, suggesting that she would have expected better. I suggested that I suspected some of the rumors and speculation surrounding the explosion and subsequent electrical fire were perhaps caused by their various competitors. She suggested that Softlayer might be the cause of more than a few, but I told her I wasn’t willing to disclose by sources (I need to get one of those reporter hats). I tried to get her to clear up a few things, including:
WHY NO PHOTOS?: My advice to her was that by publishing a few pictures of the affected area she could dis-spell 99% of the of the rumors and speculation. Yyonne simply said she didn’t know why The Planet was unwilling to publish photos of the damage. I asked her if she could find out why they were unwilling and she told me “all I can say is that I don’t know.” She then told me with 80 people working there wasn’t room in the facility for someone with a camera to even take photos. She concluded that “there really isn’t that much to see.” I repeated the question, “Well, if there isn’t much to see, what would be the harm in publishing a few pictures?” She responded, “I don’t know.” Many of the customers I have talked to can’t understand what The Planet is hiding by keeping them out of the facility and refusing to publish video or pictures.
80 EMPLOYEES ONSITE, REALLY?: I then asked her how many employees were working to get the facility back up and running. She indicated 80 people were onsite. I then suggested that customers (with video cameras) had reported that only 3 or 4 cars were in the H1 parking lot overnight and only 2 or 3 in the H2 parking lot (and they weren’t clown cars). She sounded surprised and said, “Well, that is what I have been told.” Many of the frustrated customers have used this 80 employee number as proof The Planet hasn’t been honest with them.
CONCLUSION: There was an electrical fire. The employees of The Planet are tired, stressed out and upset. The PR folks don’t know why management won’t allow them to publish photos. The PR folks are just reporting what management is telling them and have no control over whether or not they are being told the truth. Are there 8 people or 80 people working round the clock? It doesn’t really matter does it? Whether or not clients are being told the truth is the only thing that really matters. I think that is the lesson to be learned here. My previous coverage below:
Houston Data Center Fire and Explosion Update 6/5/08 8:47PM
Houston Data Center Fire and Explosion Update 6/4/08 5:01PM
Houston Data Center Fire and Explosion Update 6/4/08 10:00AM
Houston Data Center Fire and Explosion Update 6/3/08
Houston Data Center Fire and Explosion Update 6/1/08
Yvonne suggested that anyone with specific questions should simple call her at 281.714.3602.

Local
June 5th, 2008 at 10:25 pm
I’m the one who started the Where are the Photos Thread on the forums, and I have to say when I first asked it, it was more out of curiousity than suspicion. The more and more they deny the photos to us, the ones who PAY for the that building, the more it gets fishy. When a client of mine showed me this site, my tin foil hat went on tight! An FBI raid seems very plausible, as does a MAJOR hack that they just decided to cut power too.
Talking to Tomy over at the forums, he mentions that he’s back in Dallas. I’m not sure if ANY single person on the forums has seen the damage with their own eyes. Its feels like a cover up.
I’m off to ask, again, where the photos are. Its been over 5 days.
Oh, and Tomy is not acting very polite, he sent me this:
“Warning level has been increased for trolling. There is no conspiracy. ”
–Tomy Durden
Data Center Manager — Dllstx2 / Dllstx6
June 5th, 2008 at 10:26 pm
my last post had my website address spelled wrong… please fix to idreamincode.com - thanks
June 6th, 2008 at 2:24 am
This women is The Planet’s Director of Public Relations? What was the purpose of her call?
She couldn’t give any info about why no pictures have been published to back their claims, she couldn’t answer any questions regarding the amount of workers on site to fix things or why there were no cars in the parking lot, and she gave no informative information.
Sounds like all she did was call you to lecture you and then bash a competitor and spread a rumor herself. Great public relations department they have over there. No wonder this company is going down the tubes. I’m glad I moved my servers just hours after this happened.
GOODBYE PLANET!
June 6th, 2008 at 5:09 am
I think we can estimate how the conversation would go if we did call her.
Scott: “Could I speak with Yvonne?”
Yvonne: “I don’t know.”
;)
June 6th, 2008 at 7:00 am
bullshit
June 6th, 2008 at 11:37 am
You have to realize tech support staff works from their downtown office, not H1.
Just FYI.
June 6th, 2008 at 12:12 pm
Mark - I get that, but the PR department is telling people that the crowding - i.e. 80 people on-site - is the reason they can’t take photos.
June 6th, 2008 at 12:38 pm
Now day 7 and I am the one Tomy banned for scolding him for his calling the customers trolls in open forum. I have sent copies to about 10 of the “bosses” at TP without so much as a “we’ll look into it” So now it is being sent to London… I did find out who owns this company and now they have more details of the events that have lead to me pulling my server, not because of the outage, but this boys treatment of customers. Someone should have told him his paycheck comes from the money we send in for services, not to be called trolls and as anyone who uses the Internet know to call someone a troll is very close to using the “N” word to someone.
June 6th, 2008 at 4:51 pm
As a fire dept. photographer myself (in another state) I can tell you that the fire department’s records of the call, including photos! (if taken), should be a matter of public record once the fire department’s investigation is complete. I suggest more stalking of this aspect of the “incident” … if you can’t get a straight answer out of Houston Fire, bark up the chain of command: Houston police, county sheriff, county emergency management, right on up to the Texas Attorney General’s office. Public offices are obligated to give you an answer one way or another. Was there a fire call at such-and-such address, on such-and-such date? Yes or no?
Oh, and get everything in writing. The paper kind. Certified mail, return receipt. ;) When something needs to be swept under the carpet, it is easy enough to claim “gosh we never got the email” or “so-and-so is out of the office right now.” However a snail mail paper trail is undeniable ………..
Good luck!!
June 6th, 2008 at 6:31 pm
the fire suppression system has an alarm that gives a signal to the fire dept. and they show up. That is why there is no record of a phone call…
June 6th, 2008 at 8:03 pm
The fire suppression system didn’t go off, no flames. It was an electrical fire.
June 7th, 2008 at 4:15 am
I notice there have been numerous misconceptions on the web about this incident - either from people who didn’t clearly read the service updates on the Planet website, or just from an outright lack of knowledge, so I will clear a few things up.
The transformer that caught fire - that took place within the data center. But in a different room than where the servers are located. Rooms are separated for these such reasons - to mitigate damage to servers should such an event occur.
There were three walls effected by this explosion, just as the report said. And yes, they really were knocked down. Or rather, chunks of each. This wasn’t a minor event - these were thick walls, and the transformer is huge.
There have been no direct damage to servers. In other words, they were not damaged by an explosion or fire. However, many of them had their power supplies blown because of the abrupt power outage. The Planet allocated hundreds of them for replacement ASAP, and multiple technicians were assigned to the task when they were allowed back in the building (for safety reasons).
As far as specific rumors or claims, some of them I just, quite frankly, don’t get. An FBI raid? They don’t blow down walls or take an entire data center down, and intermittently shut down the first floor of a data center over the course of a week.
Regarding the confusion of whether or not a fire department was used, you don’t have a large fire that knocks down 3 walls and decide to hold off calling the fire department. If you do call them, they aren’t going to let you back in until they declare it safe.
As far as pictures of the blast, most personnel within the Planet do not have the authority to take pictures. This is the digital age, and many could probably have taken pictures, but there are strict policies against that, and nobody wants to risk their job, even if it makes some people outside the Planet happy to see those pictures, or to fend off conspiracy theories.
There were dozens to scores of extra technicians for the first few days after the incident, working on this issue, from multiple data centers from the Planet. They came in on their days off, or worked extra shifts. And more than just technicians came to help, but people from all types of departments. It’s easy for some people on the outside of the situation to not take this event seriously, but for those customers affected, and those working at the Planet - it was as serious as it gets.
June 7th, 2008 at 10:11 am
To: Anonymous on June 7th, 2008 4:15 am
“…numerous misconceptions on the web about this incident - either from people who didn’t clearly read the service updates on the Planet website…”
Service updates from TP are shady. We read the info, have you ever had that feeling that you are being lied to?
“There were three walls effected by this explosion, just as the report said. And yes, they really were knocked down. Or rather, chunks of each.”
How do you know? No pictures were provided. You could tell me the moon is green, but without seeing it first I would not believe you.
“However, many of them had their power supplies blown because of the abrupt power outage.”
I remember reading that NO servers were damaged due to the “fire.” Untruth again!
“If you do call them, they aren’t going to let you back in until they declare it safe.”
If there was a fire how did they vent out the smoke. As the electrical room is inside the building.
**Anyone in Houston who has a server at TP should get in their car and drive over to the DC and pop in. Tell them you are there to do an on site inspection to ensure your investment is secure. If they deny you then move to a company who enjoys working for their customers. I would pop in if I didn’t have to drive over 375 miles, however I am starting to feel like I should.”***
June 7th, 2008 at 1:50 pm
Alexander Muse,
You have no idea what you’re talking about and clearly, if you think the FBI had to blow a hole in the wall when all they had to do was flash a badge to the dc techs, then it’s pretty obvious you’re just starting rumors to get more web traffic.
And yes, I know how the fire suppression system works because I used to work for EV1.
That is all.
June 7th, 2008 at 1:51 pm
Except, that someone burned popcorn in the microwave and the fire dept. showed up and they wouldn’t let us have microwave popcorn anymore. It’s THAT sensitive.
June 7th, 2008 at 3:17 pm
Anonymouse Rex - I never suggested the FBI was involved or that walls were blown open. I still don’t know what happened. I think the most likely story is that they had a serious electrical fire. The fire suppression system didn’t go off from what I have been told by The Planet. The only question that seems VERY reasonable, is why they won’t release the pictures they have of the affected area. I have my own suspicions, but I have not expressed them - simply reporting what others have said here.
June 9th, 2008 at 4:02 pm
We fortunately moved 6 of our 7 servers from The Planet a couple weeks ago. Our production cluster to a new datacenter. But our bugzilla server was still there. It _still_ isn’t back online. We can get absolutely no idea from them. That’s been frustrating, but it could have killed us. As of yesterday, we’re a hosted fogbugz customer. buh bye.
June 9th, 2008 at 4:23 pm
The Planet did a great job of convincing people they handled the outage well, but I have heard a lot of people who would argue they didn’t do that great of a job. My own conversation convinced me they weren’t telling the truth…