Corey Quinn's profile picture
Corey Quinn
@QuinnyPig
View on Twitter

This cloud thing is a fad. I'm building my next thing in a datacenter instead.

I fly to a city to tour DCs, because I'm a responsible grown-up.

The first one is staffed by super friendly folks. They wave me in. I'm a prospective customer, "they don't need to see my ID." Cross them off the list.

The second one shows me their electrical (or "one-line") diagram. It shows four transformers. We tour the site. There are two transformers and two concrete pads. NEXT.

The third one seems fine. We shake hands. I can move in starting Q2. I sign a contract committing me to pay them for three years.

I order servers and have them drop-shipped to the datacenter. I'm careful to schedule arrival so I don't have to pay storage charges. Dell fucks it up.

I order networking gear and have it shipped out. I grab the wrong part number and get a GBIC I don't need. I file a return. Cisco fucks it up.

The cage build-out will take me two days. I book four days because this isn't my first rodeo and I know how this works.

I order spare hard drives to leave in the cage for when my hardware inevitably dies.

I remember to stock a full tool-kit that will remain on-site so I don't have to remember to bring parts I need at 2AM.

I put the rails in the racks. My integrator cheaped out; I need to use screws instead of quick snap rails.

The cage nuts and the screws look like the same size; they are not. Fry's fucked it up.

Build-out day arrives. The PSUs are for 120v; the cage is wired for three-phase. I begin swearing.

I rack eight identical servers. Seven work. I begin diagnosing.

I planned ahead and have a cable tester, so I only lose two hours finding the bad cable.

The cross-connect is--fiber?! I need that GBIC after all. I reread the networking order. Of course CenturyLink fucks it up.

I'm too impatient to enable Spanning Tree. I inadvertently create a bridging loop. I bodily rip the cable out of the switch. TCP now terminates on the floor.

I call it a day and go to the hotel. At 2AM the pager goes off as my site starts dying. I rush back to the datacenter. A "helpful" NOC employee has been racheting down the "loose" fiber runs. We exchange words and almost blows.

One of the switches has issues and I can't configure it. Everything supports MDI/MDIX autosense except for networking gear. I make a crossover cable while swearing.

I rack a ninth server. It doesn't work. It turns out my assistant "helpfully" put the bad cable into the spares bin. I refrain from whipping them with the cable and cut it this time.

I configure screen on my Macbook to talk to the serial-to-usb adapter I plug into my borrowed rollover cable.

Things are finally working. Should that stop I can either come back, or pay $75 an hour for remote hands to poke around on my servers for me.

I fly home and begin writing my app. I'm sure glad I'm not locked in to a cloud provider!

(Nothing in this thread was fictional. Every single thing here happened to me in my datacenter days.)

"Hey I just woke you And this is crazy... But your DC's on fire. Fix it maybe?" --@pagerduty, 3AM

What, you thought it would be that easy to end this thread?

A machine isn't speaking to the network. DRAC is non-responsive, so I pay remote hands to power cycle the box.

Remote hands power cycles the wrong box despite me labeling the servers clearly and correctly.

Help us raise more money for charities by sharing this content ♥️
Wait! Before you go...
Grab Exclusive Deals for Books, Courses, Software.
100% of Profits Are Donated To Research-Backed Charities.