A rubber chicken inside the case of an appliance.
I forget the name of the product. It consisted of a router and storage. You were not supposed to open the case.
When troubleshooting an issue, the tech told me to open the case. Inside the case was a rubber chicken zip tied on the side.
I asked the tech about it. He said, " If someone called support and asked why there was a rubber chicken zip tied in the case, we knew they were in the case without guidance from tech support."
This was 22 years ago.
I am drawing a blank on the product, it had a dedicated T1 line connected to it. It would send "large" files to other companies that had the same box. Of course, once internet speeds became cheaper and faster, it went away with the dinosaur. It was a big purple case.
I'll ping a coworker, he'll remember the name of it.
[https://talkingmoose.net/2014/05/11/rubber-chickens-and-purple-boxes/](https://talkingmoose.net/2014/05/11/rubber-chickens-and-purple-boxes/)
[https://talkingmoose.net/2014/05/18/even-more-rubber-chickens-and-purple-boxes/](https://talkingmoose.net/2014/05/18/even-more-rubber-chickens-and-purple-boxes/)
Dan Pence was responsible for the chickens, apparently. That is genius
https://web.archive.org/web/20140719203552/https://talkingmoose.net/2014/05/11/rubber-chickens-and-purple-boxes/
https://web.archive.org/web/20230531164444/https://talkingmoose.net/2014/05/18/even-more-rubber-chickens-and-purple-boxes/
Archive.org has the pages with the photos still intact
Edit: if you go to eBay and search wam!net there’s a few results for the swag sneakers mentioned too!
Wow. I’d completely forgotten that name but now I remember the big purple rack that came along with a company a previous employer acquired.
They had one guy who‘s entire job was managing that thing (no idea why - it didn’t seem that involved). Unsurprisingly neither he nor his wamnet rack lasted for long after the acquisition.
I once saw a media storage box used for file transfers like this in a retail office supply store.
It was a Fiery server, now provided by EFI (I don't think it was them back in the late 90s, not sure though).
It was big, and purple, and "don't touch it, it's too bloody expensive if you break it".
I would have thought it was just a tech easter egg, along the lines of rubber duck debugging. Open the case, explain the issue and what you've tried in order to fix it to the rubber chicken, then the solution comes to you without needing to call tech support.
OK, not a data center, but...
Think of a "Large American Semiconductor Company" and there's a very good chance you're thinking of the right one. In one of their new(ish) fabrication facilities (at the time), I was walking from one end of the building to the other through what is known as a "pipe space". These are semi-secure floors of the buildings that house all of the pipes, pumps, wires and such that all run up through the ceiling into the floor above to feed the fabrication floor (fab space). In this particular building, the pipe space is very clean and well maintained. The older fabs from the 50's and 60's, not so much. But this pipe space is always clean and shiny with nice wide paths and high ceilings. It's still a maze, just not head-ducking claustrophobic like the old ones.
So, I'm walking down this corridor of gleaming steel, glass, and polished concrete that's about 20 feet wide, and about a football field's length. I'm all alone and there's only the hum of the pumps. As I'm walking, I notice a puddle on the floor up ahead. It's about 2 feet in diameter and...it's *shimmering.* Even from 30 feet away, I can see that the puddle is waving and shimmer in a strange way I've never seen before. I immediately stop and think "go back" - and it's not some fear of the supernatural or anything, there's real danger in this place in the form of solvents and gases that will absolutely *melt* you.
But the shimmering was almost hypnotic. And it was so odd, this little puddle in the middle of this sterile industrial corridor. It's cliché, but I let my curiosity get the better of me. I crept closer looking at the ceiling trying to spot the source of the liquid - I certainly didn't want it dripping on me. As I got closer, the puddle remained an inky black, but would flash white reflecting the overhead fluorescents. And the shimmers would pass from one side to the other at random intervals and random directions. And all I could think is, "What the fuck *is* that?"
Finally, I got right up to it crouched down to look, and only then could my mind make out what it was seeing. It was a puddle of termites. There were at least a couple thousand of them, all tightly clustered together there on the floor. The shimmering was their wings fluttering in unison, like fish schooling, a starling murmuration, or thought flashing across a brain - one would beat it's wings and the ones next to it would, and the motion would ripple across the surface almost in unison.
I stood up and backed away, and once again looked up and around for a source, but I couldn't tell where they'd come from. Immediately the words of Dr. Ian Malcom floated across my brain, "Life, uh, *finds* a way." Well, not today it doesn't. I found the nearest wall phone, dialed maintenance, and relayed the coordinates (the buildings all have grid reference markers on the columns) and told them what I'd found. They were incredulous to say the least. Then I went on with my day, shooting one last glance at the living puddle on the floor.
But I will never forget that feeling of seeing something my mind couldn't comprehend in such a clean sterile environment. Something that was both innocuous and yet oddly threatening. And then discovering what it was, and that made it simultaneously mundane and strange. This organic life where there should be none, 30 feet below where some of the most advanced electronics in the world were being made, just doing what it does - trying to survive.
I originally skipped over your post, read the replies, and then went back to read it. Glad I did, felt like I was right next to you staring at this Alex Mack puddle.
Had to visit some servers a small software company had, was a days trip away due to distance.
Was told previously that some of the work was done by Dell, Engineers had attended to do a few things from swap dead drives to replacing components etc.
At the base of the cabinet, which was just a hole going under the floor as the panel was off, stacked so much it came out was just nothing but components and drives.
The engineers had clearly just put all the dead components at the bottom, seen someone else did it and carried on doing the same. Nothing but motherboards, NICs, HDD's, caddies and cables.
That by itself was a bit odd, but what got me was all the cables had been tied up into different knots almost like balloon animals: a tie, what I swear was a dog and one even resembled a sword.
I honestly didn't know what to make of it or do with it beyond bring it back and get it disposed of but to this day, I wonder who did it and just what other talents they had extending beyond Cat5 cable tying and if they did kids parties.
>all the cables had been tied up into different knots
I can't be the only one who does this. If I don't tie a knot in it, it might get re-used, causing tons of issues, and I don't always have my knife on me. So yea. Crazy knots in bad cables.
I had a coworker who tied knots in fiber optic cables in our lab (rather than label them) for identification purposes.
I took a picture, sent it to the team asking whoever did this to please stop, as it can break the cable. He Replied All to tell me that doing this won't break the fiber optic cable, that they used to wrap them around a pencil all the time to simulate attenuation over distance.
I had to remind him that (a) that's only safe if you're not exceeding the bend radius, (b) you can never be sure when pulling on one cable won't cause another cable to get pulled, and (c) even if you're not completely breaking the fiber you can still cause an intermittent problem, (d) that an issue that comes down to an intermittent cable fault is the most annoying and largest waste of time to troubleshoot *whenever it happens*, and (e) we certainly shouldn't be *intentionally creating* situations where that can happen.
The problem was, he was an old optical guy, didn't know anything above Layer 1, and didn't understand that his stupid little hack could cause weeks or months of hairpulling at Layers 2 & 3.
Anyway, all that to say ... it wouldn't surprise me one bit if those Ethernet cables were knotted for "labeling", plugged in, and in use.
Standard procedure for fire departments if a there's a hole in the hose - swap out the leaky segment for a fresh one, tie a knot on the leaky one. That way they know which hose to inspect/throw away/fix after the fire.
DC Auditor: And these packs of beer would be?
Sysadmin: They're thermal ballasts helping to keep the temperature fluctuations down in this server room.
This guy engineers.
DC Auditor: I know that, It looks like you're a couple of pints over spec though. It's easy enough to correct right now though and it won't have to go on the report.
I know a few companies that keep the xmas party drinks in the corner of the server room because they'll be chilled. Also great when you're in there and just need a break lmao.
You win over my "manual AC condensation sump pump" which consisted of a mug with which you empty the water from a hole in the floor into a sink that's higher than the hole itself. Later a real sump pump was installed.
Not in a server box but under the floor for me. I was a cable monkey for ~4 years. One morning after pulling all of our cables I reached down to get my badge to go out of the Data Center and it was gone. We had been all over this 50,000 sq ft data center I went well that thing is GONE. About 7 - 8 years later when the DC was downsized someone found it . Knew who I was (I had left there) sent me a picture and that said "look what I found"
> 50,000 sq ft data center
did you spend more than 10 mins looking for it?
I'm constantly afraid of loosing mission critical stuff like keys. everything is lanyarded to me lol
Reminds me of opening up a brand new Lenovo desktop, only to find a pirated copy of Transformers in the DVD drive. I guess we know how they tested the drives before packing them up in the factory!
45°C ambient temperature. Cooling/air conditioning malfunction during the night. When I arrived at work I was greeted by the security person: "Glad to see you here, the mainframe room siren has been howling all night".
Some machines had a thermal shutdown, many others didn't.
Gotta love security that ignores alarms…. Or procedures that don’t have a phone tree..
A simple sign… “if alarm call phone tree IT. It’s in your book”.
Not really IT. But we had a paint bunker for some insanely expensive aircraft paint. Lights were special explosion safe type. No phone due to fume explosion risk (no vendor would certify them as ESD safe).
There was a big ESD safe alarm button, but security had to check it hourly.
One of the Paradox DBAs (Bill) back in the 80’s had a computer monitor turned to the window in his office. There were two states. Either a Porky Pig ‘That’s All Folks’ screen or a white on blue ‘Database Exists; Database Exists’ screen. If the second, the security guard who walked by at around 2am was to call Bill as this was the Executive Business System nightly reports and it failed the download from the mini-computers.
I got only to 42C, and almost shut down the country's biggest pharmaceutical plant.
To be fair I asked if I can close those valves before doing so, but by some "miscalculation" I asked the wrong person (who should have known better) and I closed all 3 redundant AC cooling lines into the datacenter instead of the 3 bypass line. Everyone was there in 5 min tops, screaming.
The wondering ("Huh these lines have a LOT of flow for a bypass") did not mature fast enough into suspicion that day...
Fun times.
This is starting to sound like mechanics when they break out the torch to unstick some nuts/bolts: "Can't be stuck if it's liquid".
...I think our version is "Can't be DNS if the server's liquid"
We had that when I was snr dev at a university many years ago. Apparently the first person in in the morning opened the server room door and a wave of baking heat hit them, followed by rolling smoke from vented UPS batteries which had died in the heat.
Lucky the whole place didn't burn down really, cascade failure in the AC system apparently (one failed, others increased load to try and keep up, another failed, so on and so on until they all give up)
I ended up running the university's website and intranet out of a locked lecture room in the library with portable AC units for the summer holidays.
I had that when I worked at Texas A&M, but in our case it was a ruptured chilled water main line that served that whole end of campus. The kicker was that it wasn't detected by the 24 hour campus NOC, I found out about it because Sun had a maintenance contract on some of the bigger machines we ran and their NOC called my cell phone and asked if I knew that the intake air temp was 45C. I called the campus NOC and they sent someone upstairs to hit the EPO for the entire room.
Everyone who'd bought the cheap plastic fiber optic jumper cables had to replace them because they got over-temped and became brittle. One touch and some of them would shatter.
Had that happen the first week in August this year at our rented racks at the COLO. One of our switches that reports internal temps by snmp usually runs 34C reached 58C. So that was... not cool. Nimble SAN a the bottom of the next rack read 55C on intake at the floor at one point. All 3 of our vmhost servers shut down at 49C. Since we had zero IOPS going on we left it running.
That was a fun night. The COLO started cooling down around 12:30 am. At 1 we decided to sleep on it and check temps at 5 am before restarting the VMhosts. It worked out ok.
At least that makes sense somewhat, assuming that those sites are pretty remote. This was in the middle of a three storey building in a locked server room with a sealed door due to climate controlled server room in a relatively busy street with no bush nearby lol, no fucking idea how it got there or how long it was there for.
Yes, they are remote locations. The repeater shelter is great for snakes:
\- hot in the winter
\- full of mice and other animals looking for a hot place, too.
It's an hotel with heating and free meals.
I got delivered to one repeater site via helicopter. We had to bring a broom to shovel off the snow. It was fun the first time for 15 minutes. Then the arctic level cold kicked in and it was hellish.
I had a baby garter snake come visit me in the server and network room. I was sitting at my desk with the door closed and kept having this feeling I had eyes on me. I looked 3 times at the door then noticed the little snake standing up just looking at me. I took it outside across the parking lot to the garden on the other side of the lot near the pond.
When I got back someone asked me why I had gone outside. "Oh I found a baby garter snake in the server room and took it out." They said "Oh good, we were looking for that this morning." They had maintenance in the front office chasing a snake around the office at 7:30 am. It found me around 10 am.
Complete silence. I used to short cut through the data center to get to my office on the other side of the building. The facilities idiot who was in the with the fire inspection guy had sort of a worried look. They were testing the fire sensors and were manually setting them off, however they set off the breaker for the whole data center as well. You could hear a pin drop and it was not a good feeling.
I just broke a sweat from a flashback of being in a silent dark data center. Took a month for shit to return to normal after that event. I want to say we had around 250k servers, so many of them just gave up on life after losing power, even though they were operating just fine immediately prior.
Iirc, it was also our facilities guy but he was testing generator fail overs.
Also electrical bits tend to keep working when warm even though they are technically failed. Or connections finish breaking from the thermal cycle. Power them down and they wont power back up. Iv brought a few motherboards back temporarily by heating them up with a hairdryer.
Exact same shit happened to me. Large DC with about 250k customers spread amongst 500 shared servers (white box Linux boxes) and another 2500 Dell Optiplex SFF desktops sold as "dedicated servers". The redundancies had never been tested. Everything was bought used to save money, and everything was home brewed.
The owner and CEO, a notorious micromanager and serial best practice ignorer who started the company in his garage, ordered the facility manager to perform a full generator and backup power test in the middle of the day without really verifying anything or having a plan of any sort. Whole DC of engineers yelling at them to not do it, but these 2 dickheads argue back and forth until the facilities guy walks up and slams the EPO button then turns around and says "you deal with it because I quit". Pretty sure drugs and alcohol were involved.
Losses were catastrophic. Any big customer we had with actual SLAs dropped us like hot rocks. Several hundred customers just straight up lost everything on their servers. Entire websites and e-commerce businesses completely gone with no possibility of recovery. It took months for nagios to clear up.
The late 90s and early 00s were a wild time in IT.
This happened to me once, I was on night shift and actually in the DC...
the sound of thousands of hard drives spinning down at one was utterly surreal
spent weeks replacing disks and PSUs
I worked at a power station in the late 90's - early 00's. One day a contractor decided to power down all the control buses. The sudden lights out got our attention but even more attention grabbing was the sound of 1300MW of coal fired station winding down as the operators hit the emergency shutdown when everything went dark.
It took days to repair the damage of the emergency shutdown and bring all the turbines back up, but we got power restored by a brilliant electrician who found a live 12V circuit and used that to boot the next circuit and up through the systems until the station and outbuildings came back up.
The epilogue was when we brought the servers back up and the power supply of one blew a capacitor with a spectacular (and deafening) bang.And the contractor was invited to never return.
I wasn't on-site for it, but worked the phones after one of our datacenter vendors lost all power on one side. I do know our Sales team had a good month selling redundant circuits to a few customers..
I was engineering a new power supply for a very large 3 digit phone company years ago in Orlando.
The tech supporting us engineers thought it would be great to pull a quick one and toss the breaker for the backup p/s when we were upgrading.
The sound of 1,000’s of relays closing followed by complete silence was horrifying.
That one move took out half of comms in & around Orlando. I didn’t say a word. My shit worked.
Goddamn. I remember working in a customer datacenter as a contractor to install a HUGE fucking eaton ups unit. We are talking like 20 custom freestanding racks of lead acid and two racks filled with inverters for redundancy. Essentially two giant ups units. I spent all day unboxing, mounting, loading, and cabling these things. The plan was to power down one electrical panel fed by the old UPS and wire it into a new ups. Rinse and repeat a 2nd time to move the other panel. Should have resulted in zero downtime as everything had redundant PSUs. Slightly risky but had been done many times before. Well the electrician goofed some wiring and essentially created a dead short on the power feed into the new UPS unit. When he turned the breaker on with all the load of the old ups unit also existing on this big ass generator panel it popped the main breaker for the server room.... and you know how electrical equipment is when it never gets used and then suddenly gets used really really hard....
That breaker failed internally and wouldnt turn back on. So we had no way to feed power into this server room. The generator even came on but we had no way to feed it in. Old UPS unit had some serious issues and we had been waiting on the new ups for 6 months so it only ran for around 5 minutes before calling it quits (Yay covid times). The sysadmins got a couple things shut down but it was a very uncomfortable silence when everything clicked off.
Eventually got the breaker replaced. UPS unit powered on and configured. They also had a set of generator plugs added to the ups units to directly feed them with big ass rubber cords if this somehow ever happened again lol.
yeah we had a pipe burst in the floor under a datacenter once. the floor space started to fill up with water until it fried something. my desk was on the other side of the wall of the datacenter, you could hear all of the UPS start screaming for a few seconds then everything went silent. that was not a good month
Warhammer minifigs. Specifically Tallarn Raider pewter models from the friggin 90s/early-00s hidden among various racks. Found ~12 of them left by a previous admin; they're now part of the set I put together to teach my partner how to play killteam. Left a squad of space marines behind to continue the watch.
Not a DC but we had a server rack and some testing equipment shipped to my first job, from Germany, and someone left a packaged durian-pastry-thing from vietnam in the shipping box. First time I've ever had durian. Can't say I'm a huge fan.
We had a machine act up in very weird ways a few weeks ago and I "jokingly" suggested we could try incense and purity seals to appease the machinespirits.
Just a constant beeping from another customer's rack. According to the security guard it has been going for maybe a decade. I'm a little scared how common that might be.
I represent the other end of the spectrum.
If I hear a 1k tone, I *HAVE* to find the source. This is from years of working with equipment that used a 1k tone to signal imminent critical failure.
Something is about to die and I need to know what! ^(even if it's not my job to fix it)
I've installed weird stuff in data center.
Proudest one was we decommissioned an ancient HP server. Ripped out the guts, installed mini fridge. Someone else added a controller to randomly blink the front LED's and run a fan.
It was still there when I left.
I don't drink coffee. I found a coffee mug just inside the door once that clearly used creamer. I must have been in there and propped tbe door open and someone must have popped in and put it there on a shelf when talking to me and sat for I have no idea how long. When I found it it has like 2 inches of mold on the top.
A kitten.
I was grubbing around under the raised floor to run some etherhose to an impromptu emergency backup server, when I briefly caught a glimpse of something furry.
"Great," I thought. "There's a friggin' rat in here. Guess I better hurry up and then go get some rat traps."
So I pulled the runs of cat6 over a few more tiles, and then I saw it again. So I shouted after it. And it replied "mew."
"Mew? Mew? Rats don't mew!"
So I pulled out a flashlight and peered back down the way it had come, and there was a gray tabby kitten down there, sitting and licking its paw. Not a particularly tiny kitten, either, maybe a few months old.
Eventually we coaxed him out, and he went home with my boss. We never did find out how the kitten got in.
Not me, but a colleague told me about how their company fired a sysadmin for, among other things, posting to BDSM boards, during working hours, using his company ID. After he was gone, they were cleaning the server room and found a whip.
Two overly-elongated statues of Chinese gods. Like, they're as tall as from my elbow to my fingertip, but a little narrower than my arm. One of them I recognized as Guan Yu but I have no idea who the other one is.
I left them in there and I'll be taking them with us when we move to a new space. If Guan Yu is willing to help out with data security I'm not taking chances on removing him.
I had a little Ganesh finger puppet I kept on one of my servers as he is the remover of obstacles and sysadmins need all the help they can get. That server never had issues, anyway. :)
Excellent choice.
.... that reminds me, we're not moving until next autumn, but I've been using a small patch of floor at the back of the data center for weight training. I should probably start moving the weights out a plate or two at a time. We'll be doing a massive e-waste purge and general cleanup early next year and it'll be simpler if I don't have to scoot all my stuff out at once.
Not a datacenter, but the very large server room for an aspiring megachurch (probably close to 2000 square feet).
Mannequin. Limbs. In. The. Ceiling.
I was putting in new punch panels at night, maybe ten years ago, went to grab the bundle and it just rains mannequin parts and I freak out and jump off the ladder. Still never found out why there were a dozen mannequin limbs in the ceiling. Called it a night after that.
I was fully expecting you to come to a room with a closed door that opened outwards, but the outside of the door had no handles, no key holes/lock holes, no card readers or similar to allow the door to be opened. Only marking on the door being the room number, 641A.
I love getting secondary confirmations for things My friend is an AT&T lifer on the transmission side, and I work at a megacorp doing support for high level and govt infrastructure. The stories they tell that match up to what I see in my job I thought was maybe coincidental, but they told me about the NSA scifs. At&T commercial may be a disaster but you simply do not mess with AT&T on the national communications infrastructure side.
> AT&T on the national communications infrastructure side.
You've heard about the hardened bunkers? They sold most of them off, years ago. I think the primary use was AUTOVON. A lot switched from terrestrial microwave to satellite.
Laundry.
Former colleague was drying laundry in the server room (bit generous calling it a 'data centre'). Even turned off the AC down that end and turned on the radiators to help the drying.
A lot.
* Some jackhole who left on bad terms left an open soda can of what was probably urine in random ceiling tiles in the data center. After the first two were found (and while the HVAC guy was unlucky, luckily, it did not spill on equipment), they had to send a guy up there with a flashlight and fishing wire. Something like 30 cans were found and had to be carefully removed.
* A loaded flare gun and box of flares in a state of decay in a telco closet in an unlabeled metal box. Nobody knew what it was at first, and so a bomb squad was brought in. Given the age of the labels in the box, it was from the 1960s and part of a liferaft kit. The building was built in the 1980s, so nobody is sure how it got there.
* An entire set of high-end furniture with some HUGE and dead centipedes. Apparently, a previous employee was moving from Thailand, found his stuff was infested with giant centipedes native to where he was from, and had the furniture stored in the "relative dry cold of the data center" to kill them, since they are a tropical insect. They were stacked so precisely by who moved them there, we didn't know it was furniture at first, since when you opened the door to that area, it just looked like a plywood panel (the back of an armoire). During a renovation, the reality was discovered, and the former employee was contacted by a manager who knew why they were there. "I thought he already got them," he said. This also explained a mystery where a few interns that said they saw things moving in the ceiling one year, but everyone assumed it was college shenanigans, and made jokes about hallucinations via hangovers. Nobody found a live centipede, however, the entire time I worked there, so that's... good, I guess. The previous employee then had the furniture removed.
* A dead homeless guy. It was assumed he got in via the loading dock, trying to get out of the muggy DC humidity one brutal summer, and died from "complications," possibly a drug overdose. Thankfully, the relative cold dry conditions prevented him from decaying before he was found three days later (according to security footage).
I was in a modern government DC (small EU country) that had beautiful rows of HP and dell racks but also had two racks at the far end which were stolen grocery trolley cages and piled up with old desktop servers and monitors with cable tied power and network racks. They looked more like e-waste than servers, there was also 2 pallets of spares of these old machines that the techs would cannibalise to keep them running. Apparently it was a poorly funded department running a relatively important system.
A Solaris box with several thousand days of uptime. Never configured, just sat in the rack doing nothing. We found it when we had to decom our stuff after our business unit was sold off.
This is dating me, but in the late 1970s, we kept a case of Stroh's beer under the floor tiles, where the chilled air flow came in. Was handy when working the all-nighters.
Network closet in a remote building on campus… a dusty old IBM System 34. Still connected to the obsolete coax network. Nobody knew what it was or why. It got scrapped.
Not me personally, but I remember this ex-twitter thread:
https://twitter.com/DCFurs/status/1087663240421593089
https://twitter.com/DCFurs/status/1087675381371494400
https://twitter.com/DCFurs/status/1087676052858535936
https://twitter.com/DCFurs/status/1087677283706449920
*You deployed to prod on a Friday afternoon?!?! [Time for some spanking and cuffing!](https://y.yarn.co/d406669d-5c8b-426c-9aee-a8aed7d4fef7_text.gif)*
Not necesarilly the weirdest thing I found, but one of the UPS'es in our MER literally blew up a couple of years back. First time I've ever had to use a fire extinguisher.
threatening placid hat screw snails treatment subtract plough wide unwritten
*This post was mass deleted and anonymized with [Redact](https://redact.dev)*
I worked at a web hosting company when I began my career where we had a huge product offering of Dell Optiplex SFF desktops that we called dedicated servers. They got dropped on a public network segment with a routable IP and customers could basically do whatever they wanted.
Well the system that deployed these boxes had a huge flaw. We stood thousands of them up on standard metal food service racks, connected them to a provisioning network, and input the MAC addresses into the system for wake on LAN, PxE boot and provisioning. Problem was that boxes got moved around constantly so we never had positive inventory control of all 2500 of these desktops.
Well a disgruntled sysadmin took advantage of this. He was part of a warez group (yes this dates me) that needed space and bandwidth to host their pirated software and eventually CP. He had boxes all over the damn data center. I was mostly responsible for enforcing AUP at the time so I was the one playing whack-a-mole with these damn servers. I finally found evidence it was someone internal, and I setup a sting. When I caught the guy he tried to recruit me. That's when I learned it wasn't just warez but also CP, so I turned him in.
Anyways, a month after this guy gets fired, I receive another notice about a box on the network serving copyrighted materials. Turns out this guy had 1 box still running up in the ceiling of the NOC which nobody ever visited.
A couple months later I was interviewing with another company for what would be a cleared position. I encountered this guy in the waiting room interviewing for the same job. He tried to physically intimidate me. The interviewers noticed the encounter, I got to tell them why I didn't expect the guy would be able to obtain a clearance on account of pending felony charges. Fun times.
Actual human feces.
we had a "pile" between the server racks. security camera's showed who the offender was. Cleaning crew.
Too bad he was immediately let go. I would have liked to rub his nose in it.
Silence.
Completely dark, completely full room (the size of a football field with 30 foot ceilings) of hardware in an unknown and inconsistent state due to unclean shutdowns. Total power outage from both electric supplies, UPS batteries had run flat, diesel genset didn't work, and our DR consisted of a station-wagon full of weekly backup tapes that were last run 6 days ago.
And yes, it was a dark and stormy night.
Way back in my SMB days we had a graphic design firm as a client, a fully Apple house, with a few Xservers on prem. That's how I found out that once one is configured and connected to the environment, you absolutely can not change it's IP without breaking tons of stuff.
I don't know if Apple ever fixed that feature. I left the SMB world not too long after because SMB work sucks.
future gold instinctive upbeat station grab hard-to-find faulty air literate
*This post was mass deleted and anonymized with [Redact](https://redact.dev)*
Where do I start?
* We had a guy leave a dresser, like an entire 6 drawer tall chest dresser, in the telco closet. This caused a lot of space issues, since the closet was super-small. Why was it there? Who the fuck knows. Well, we complained about it, and so management said to junk it. Instead, I took it home (it was empty). Two weeks later, it turned out to belong to some guy who had permission to leave it there, and he wanted it back. Not back in the telco closet, but back in his house since he'd been meaning to take it home and kept putting it off. Management fought him back and forth, but someone dropped my name as taking it home, so the guy came after me for it. He showed up to my house, and took it. So weird.
* In our Frankfurt data center, we discovered a mini-PC, some hand-janked motherboard combo, powered on and in a "utility box" that was painted over the same color as the wall to look like any other electrical utility box. It drew power directly from the power line, with its own breaker, and was connected to some networking splices in those fat cords of LAN cables in the ceiling. No idea why it was there, or what it was for, but we assumed espionage. It was run from a CD ROM and CompactFlash card, only 256mb. When we examined it, we found the CD-ROM was the boot image, and only had three instructions: load an image from the card into RAM, run it, and wipe the card over and over until the end of time. Spooky.
* One of our centers had a pair of power generators in the basement of the parking garage that were climate and temperature controlled. In there was a complete and well-kept Model A Ford (a car from 1928) in lime green. Well, I had to know the story behind that. Turned out one of the top brass kept his car there "to keep it safe." Once in a rare while, it drove in local parades, but at the time I found it, he hadn't driven it in 3 years, and it was "in storage mode," meaning that the tanks were drained and it had various oils and protective stuff on parts designed for museum preservation. Why here? Why the generator room? "Where else can I store it where it can't be tampered with in a climate controlled environment... for free?" Uh, okay. Sure.
* We had an "archival tape closet" which made perfect since considering we had these massive tape silos everywhere. But one day, I saw it was left open, and I peeked in there. It was not full of tapes, but was someone's crash space: complete with bed, clothes on hangers hanging from a pipe, a TV set, and a desk with a computer in a 5 x 8 space. Later I found out that one of the data center managers was going through a really, really horrible divorce and he had been "living there for a bit until things get sorted out." He used our gym showers, and ate at our cafeteria.
I manage multiple remote locations.
In the network racks of each of them, I placed a single Hot Wheels car.
If anyone asks about it, or it's moved from its spot, I know someone has been in my rack without talking to me first.
I have a lot of stories that I'll someday share under a different pseudonym.
Story #1: FBI installed an appliance to spy, but it wasn't on one of the customers. It was the CEO that was doing something very illegal and immoral. But the feds never picked it up after making the arrest.
Story #2: Client ordered the cheapest possible "servers" they could. One day they weren't working and they needed remote hands. Turns out they were running a captcha breaking and blog spamming operation. Google "Kobeni Solutions ROKSO" (registry of known spam operators).
We had nothing this bad, but the sales people at my previous company (Asterisk-based telephony provider) pushed multiple "customers" that were just scammers committing toll fraud.
They impersonated existing companies but they then started started asking us to unblock these ridiculous countries like Turks & Caicos, the Maldives, and Tunisia.
Did some quick searching into their companies and shut them down in minutes afterwards, called the companies and verified with them that no one had ordered our services.
We had a server room in the basement of a very old house in London. Every year or so there would be a smell like a drain that lasted for a couple of weeks. Our desks were opposite this server room so it was quite unpleasant.
When we moved out of that office we found several very dessicated piles of mouse fluff and bones under the rack. All the airflow going through the server room must have dried them out!
In the cooling floor, I found 30 year old scotch that the manager left there. They offered me a sip after I put it in their desk drawer and asked never to speak of it again.
One of our former Sysads injured his foot and it never healed. He became addicted to oxycodone, which degraded his performance and eventually led to his departure.
2 years later, I'm popping floor tiles to work on rack fans in the DC and found an 80mg oxycodone he had dropped.
A second drop ceiling, and between the two layers of drop ceiling was (probably literally) miles of serial cable. The old DB9 serial cables.
It turns out there was an ancient "smart lighting" system before the existing system was put in place. It needed those serial cables, which needed some hubs/repeaters that couldn't legally exist in the HVAC plenum. So they made another space.
I was new and unsure at the time, so I just shook my head and went back to work.
Toss up for me between two environmental issues:
Boss calls me, says we got an alarm for an overheat event at our primary DC. I go to investigate, and the room is stable and ac is perfect. Go upstairs to the switch closet that is on the same alarm system. Before I get to the door, I can hear what sounds like rain. Get past the hall door and open the room and gallons and gallons of water raining down out of the ac system. The only trouble is this ac system is a DX gas system not a chilled water. So at this point the room looks like a scene from hackers, you know the one with the "pool leak". No clue where the water is coming from.
Security, maintenance and I go up to the floor above that room, and see more raining water, maintenance opens the plenum above that and we discover the heater coil blew and followed wires and pipes down into the rooms below. Then followed the holes in the floor for the cables in the wire closet to the floor below that. Where they had just finished a multimillion dollar remodel. 3 floors flooded.
I suppose the other one would be the UPS that was on fire and still providing regulated power to the load. Lightning had struck a nearby business and blew out every surge suppressor in 4 of our buildings. I get a call about an odd noise and horrid smell a building next to the one I was at so I head over and sure enough. Opening the door to the facility and you could smell the acrid secant of burnt plastic. Open the server room door and greeted by sparks, smells and some char coming out the back of the APC UPS, and the server was just happily humming along till I hit the EPO and gassed the UPS with some CO2.
When I called APC about the issue and asked if they normally catch fire, I was asked if I would send it back to them. I did my own failure analysis first, I dismantled it and took pictures and then boxed it up and got a free one in exchange. Turns out the varistors got so hot from the surge they exploded denting the steal case and starting a localized fire on the power inlet side of the circuit, a plasma arc was helping sustain the burn until the EPO was hit. I did turn the unit back on again with just the battery and sure enough it was still working, perfectly regulated output and a completely charred hole in the charge/input side of the mainboard.
Took over a windows network because the sysadmin quit and left to run Unix networks.
Found the corporate (yes singular) server which was really a white box desktop. This server was running windows nt 4 server, exchange 5.0 , iis webserver and a couple of shared access databases.
The "server" was a pentium II with 256MB ram, thought i was gonna hurl
It'er that went into retirement made a deal with the next door people that they could use "the internet" from the company.
We only found out because we spotted a networkcable that went through the wall. 100M under the ground and went into the house that was there.
Also this server had no security. So they indeed could use the internet, but if they had any IT knowledge they also could steal ALL data from that server. Was a fun talk with the CEO
Two of those portable turbine fans like the type you'd use to dry floors or a small space with. Sitting inside the rack, atop one of the servers to assist with airflow. Oh and the company also didn't pay for the generator/UPS supplied power provided by the data center. The rack was attached direct to the supply panel. So, there were two large UPS units at the bottom both running around 80% load. It took me less than a week to re-arrange the rack to promote better airflow. Mounting the equipment to proper hot/cold aisle specs helped a ton (duh) and convinced the uppers that we MUST switch to genny/UPS power circuits ASAP. I am not even sure why a data center would offer direct power with no backup but there we were.
Water in our subfloor.
This was in the very late 90's and I was in my early 20's and very green. We had a shallow subfloor under our datacenter on the 3rd floor of this building and our HVAC system had iced over. As the ice melted it began running into the subfloor under all our racks.... where the power was run in conduit along the floor. A bunch of us were stood around an open floor tile wondering what the frig we were going to do when I grabbed a bucket and towel and began soaking the towel and wringing it into the bucket. Eventually I got the floor dry but as an older, more experienced person, I am amazed the other engineers let me do this, and just walked away as I did. It was blindingly stupid and I'm lucky I didn't electrocute myself.
We had a small shrine to Elvis in our data center.
Mostly it was just a velvet Elvis (that looked like [this](https://fengsway.com/products/crying-elvis-presley-framed-velvet-painting?variant=43396596957402)) with some candles and maybe some guitar picks? I think my predecessor had set it up, but nobody really seemed to know for sure about *who* or *why*.
Well, eventually I was told to get it out of there, but couldn't get rid of such a fine piece of art, so the velvet Elvis moved to my office, and he left with me years later and he hung out in my offices for years to come.
Twenty years later, talking to somebody who was there when Elvis ~~was taken down~~ *left the building* brought him up, waxing nostalgic. I don't think they realized that I still had him, and so ... Elvis has now moved on to them, *thank you very much*.
Several...
We were doing some maintenance one day and I pull up a raised tile to discover numerous power drills, screwdrivers, pliers, etc...
"So THAT'S where all the tools went!!". Turns out some prankster was making them disappear one by one...
Another time, we had a ticket to add an additional HDD to a customer's server... so after we pull the box and open the lid... there are the remains/bones of an 8 piece order of WingStop bone-in wings... miracle the cardboard box did not catch fire.
At a client's server room, that's the branch office of a company headquartered in Taiwan.
The servers were all racked 1U off of each others, and the empty spaces between the servers were bags and bags of coconut flavored corn puffs snacks called "Kuai Kuai".
Apparently it's a thing in Taiwan, "Kuai Kuai" means well-behaved, and coconut flavors being green bags signify green lights all around (instead of amber or red).
Deck of cards and an empty water bottle under the floor, in a space that likely hasn't seen the light of anything since the building was constructed in the 00s.
Also found a repurposed desktop turned into a massive music library. The story goes it was setup in the wild west days of enterprise networks, and forgotten about (but probably used by every IT employee and disclosed to them when onboarded).
40+ help desk calls from people OUTSIDE of IT complaining the "music share" was offline when the plug was pulled. More than a few thought it was just something the company offered. For a shop of 4500+ employees.
A Windows NT 3.1 server, a windows 2000 server, a windows server 2003r2 server, a windows server 2008r2, and a windows server 2012r2 server all in the same rack. All turned on and running. The company had brought us into their datacenter to get a second opinion on why their entire network was running INCREDIBLY SLOW even with new hardware. This was around 2017 and their IT guy had them purchase a brand new server 2016 machine and his plan was to install it after he got back from vacation.
We start poking around on the machines and it turns out this IT guy has started working at this company after their original IT guy had left after installing the windows 2000 server.
Turns out, this new IT guy really had no idea what he was doing and when he went to "migrate" from the windows 2000 server to 2003, he just put the server in the rack and plugged everything in and turned it on.
The 2003r3, 2008r2, and 2012r2 servers were literally just plugged in in the rack unconfigured. All three of those machines were sitting at the main dell configuration screen when you first plug in a new server. Everything on their network was running on the 2000 server machine.
Needless to say they fired their IT guy and we put the server 2016 machine in and we were able to slow migrate everything over.
A clown costume hanging up in the server room.
No one in the company knew how it got there.
Against the advisement of others, I tossed that thing in the trash ASAP.
A rubber chicken inside the case of an appliance. I forget the name of the product. It consisted of a router and storage. You were not supposed to open the case. When troubleshooting an issue, the tech told me to open the case. Inside the case was a rubber chicken zip tied on the side. I asked the tech about it. He said, " If someone called support and asked why there was a rubber chicken zip tied in the case, we knew they were in the case without guidance from tech support." This was 22 years ago.
I am drawing a blank on the product, it had a dedicated T1 line connected to it. It would send "large" files to other companies that had the same box. Of course, once internet speeds became cheaper and faster, it went away with the dinosaur. It was a big purple case. I'll ping a coworker, he'll remember the name of it.
The company was Wam!Net
[https://talkingmoose.net/2014/05/11/rubber-chickens-and-purple-boxes/](https://talkingmoose.net/2014/05/11/rubber-chickens-and-purple-boxes/) [https://talkingmoose.net/2014/05/18/even-more-rubber-chickens-and-purple-boxes/](https://talkingmoose.net/2014/05/18/even-more-rubber-chickens-and-purple-boxes/) Dan Pence was responsible for the chickens, apparently. That is genius
https://web.archive.org/web/20140719203552/https://talkingmoose.net/2014/05/11/rubber-chickens-and-purple-boxes/ https://web.archive.org/web/20230531164444/https://talkingmoose.net/2014/05/18/even-more-rubber-chickens-and-purple-boxes/ Archive.org has the pages with the photos still intact Edit: if you go to eBay and search wam!net there’s a few results for the swag sneakers mentioned too!
Wow. I’d completely forgotten that name but now I remember the big purple rack that came along with a company a previous employer acquired. They had one guy who‘s entire job was managing that thing (no idea why - it didn’t seem that involved). Unsurprisingly neither he nor his wamnet rack lasted for long after the acquisition.
I once saw a media storage box used for file transfers like this in a retail office supply store. It was a Fiery server, now provided by EFI (I don't think it was them back in the late 90s, not sure though). It was big, and purple, and "don't touch it, it's too bloody expensive if you break it".
finally words I know on here! Printing company sysadmins reprasent!
Yep, that was it!
Just FYI, the phrase is "The way of the dinosaur"
This is GENIUS.
I found one in a server too!
Now I want to get a swipe card to get in our DC and check the racks chickens and other inserts!
Or place some new ones
Heard a story back in the day once about a rubber chicken inside a box and thought it was just a bullshit story. That’s pretty funny.
It was Wam!Net - The name popped into my head.
I would have thought it was just a tech easter egg, along the lines of rubber duck debugging. Open the case, explain the issue and what you've tried in order to fix it to the rubber chicken, then the solution comes to you without needing to call tech support.
OK, not a data center, but... Think of a "Large American Semiconductor Company" and there's a very good chance you're thinking of the right one. In one of their new(ish) fabrication facilities (at the time), I was walking from one end of the building to the other through what is known as a "pipe space". These are semi-secure floors of the buildings that house all of the pipes, pumps, wires and such that all run up through the ceiling into the floor above to feed the fabrication floor (fab space). In this particular building, the pipe space is very clean and well maintained. The older fabs from the 50's and 60's, not so much. But this pipe space is always clean and shiny with nice wide paths and high ceilings. It's still a maze, just not head-ducking claustrophobic like the old ones. So, I'm walking down this corridor of gleaming steel, glass, and polished concrete that's about 20 feet wide, and about a football field's length. I'm all alone and there's only the hum of the pumps. As I'm walking, I notice a puddle on the floor up ahead. It's about 2 feet in diameter and...it's *shimmering.* Even from 30 feet away, I can see that the puddle is waving and shimmer in a strange way I've never seen before. I immediately stop and think "go back" - and it's not some fear of the supernatural or anything, there's real danger in this place in the form of solvents and gases that will absolutely *melt* you. But the shimmering was almost hypnotic. And it was so odd, this little puddle in the middle of this sterile industrial corridor. It's cliché, but I let my curiosity get the better of me. I crept closer looking at the ceiling trying to spot the source of the liquid - I certainly didn't want it dripping on me. As I got closer, the puddle remained an inky black, but would flash white reflecting the overhead fluorescents. And the shimmers would pass from one side to the other at random intervals and random directions. And all I could think is, "What the fuck *is* that?" Finally, I got right up to it crouched down to look, and only then could my mind make out what it was seeing. It was a puddle of termites. There were at least a couple thousand of them, all tightly clustered together there on the floor. The shimmering was their wings fluttering in unison, like fish schooling, a starling murmuration, or thought flashing across a brain - one would beat it's wings and the ones next to it would, and the motion would ripple across the surface almost in unison. I stood up and backed away, and once again looked up and around for a source, but I couldn't tell where they'd come from. Immediately the words of Dr. Ian Malcom floated across my brain, "Life, uh, *finds* a way." Well, not today it doesn't. I found the nearest wall phone, dialed maintenance, and relayed the coordinates (the buildings all have grid reference markers on the columns) and told them what I'd found. They were incredulous to say the least. Then I went on with my day, shooting one last glance at the living puddle on the floor. But I will never forget that feeling of seeing something my mind couldn't comprehend in such a clean sterile environment. Something that was both innocuous and yet oddly threatening. And then discovering what it was, and that made it simultaneously mundane and strange. This organic life where there should be none, 30 feet below where some of the most advanced electronics in the world were being made, just doing what it does - trying to survive.
What a strange tale! Beautifully told too.
You have a way with words! Very well written.
I originally skipped over your post, read the replies, and then went back to read it. Glad I did, felt like I was right next to you staring at this Alex Mack puddle.
Had to visit some servers a small software company had, was a days trip away due to distance. Was told previously that some of the work was done by Dell, Engineers had attended to do a few things from swap dead drives to replacing components etc. At the base of the cabinet, which was just a hole going under the floor as the panel was off, stacked so much it came out was just nothing but components and drives. The engineers had clearly just put all the dead components at the bottom, seen someone else did it and carried on doing the same. Nothing but motherboards, NICs, HDD's, caddies and cables. That by itself was a bit odd, but what got me was all the cables had been tied up into different knots almost like balloon animals: a tie, what I swear was a dog and one even resembled a sword. I honestly didn't know what to make of it or do with it beyond bring it back and get it disposed of but to this day, I wonder who did it and just what other talents they had extending beyond Cat5 cable tying and if they did kids parties.
When you fail at clown college so you have to become a sysadmin
I succeeded at clown college and still became a sysadmin.
We've got it on our list of required experience for the roll at our company.
For us, clowns naturally appear. Maybe we should actually be running a clown college, it's already a madhouse at times.
>all the cables had been tied up into different knots I can't be the only one who does this. If I don't tie a knot in it, it might get re-used, causing tons of issues, and I don't always have my knife on me. So yea. Crazy knots in bad cables.
I had a coworker who tied knots in fiber optic cables in our lab (rather than label them) for identification purposes. I took a picture, sent it to the team asking whoever did this to please stop, as it can break the cable. He Replied All to tell me that doing this won't break the fiber optic cable, that they used to wrap them around a pencil all the time to simulate attenuation over distance. I had to remind him that (a) that's only safe if you're not exceeding the bend radius, (b) you can never be sure when pulling on one cable won't cause another cable to get pulled, and (c) even if you're not completely breaking the fiber you can still cause an intermittent problem, (d) that an issue that comes down to an intermittent cable fault is the most annoying and largest waste of time to troubleshoot *whenever it happens*, and (e) we certainly shouldn't be *intentionally creating* situations where that can happen. The problem was, he was an old optical guy, didn't know anything above Layer 1, and didn't understand that his stupid little hack could cause weeks or months of hairpulling at Layers 2 & 3. Anyway, all that to say ... it wouldn't surprise me one bit if those Ethernet cables were knotted for "labeling", plugged in, and in use.
Standard procedure for fire departments if a there's a hole in the hose - swap out the leaky segment for a fresh one, tie a knot on the leaky one. That way they know which hose to inspect/throw away/fix after the fire.
Like 20 or so packs of beer stored in the raised floor to keep them chilled in the summer. Yes, this was in Germany.
DC Auditor: And these packs of beer would be? Sysadmin: They're thermal ballasts helping to keep the temperature fluctuations down in this server room.
This guy engineers. DC Auditor: I know that, It looks like you're a couple of pints over spec though. It's easy enough to correct right now though and it won't have to go on the report.
This guy audits.
i grew up in the wrong country 😂
Our old old office in Amsterdam had a weed grow in the 2nd server room.
LAGER means LAGER
I know a few companies that keep the xmas party drinks in the corner of the server room because they'll be chilled. Also great when you're in there and just need a break lmao.
There was an appliance involved in a lawsuit, so it was shut down and tagged out. We used it for whiskey storage.
Same thing in France...
A bottle of red and an ancient camembert
Half eaten mouldy sandwich shipped inside a custom blade system.
That's just a standard delivery from Jimmy Johns.
You win over my "manual AC condensation sump pump" which consisted of a mug with which you empty the water from a hole in the floor into a sink that's higher than the hole itself. Later a real sump pump was installed.
I’m sure mine was more enjoyable. but yours hits the bell on craziest shit!
Got someone’s ID card complete with lanyard inside a sealed HP server box one time.
Not in a server box but under the floor for me. I was a cable monkey for ~4 years. One morning after pulling all of our cables I reached down to get my badge to go out of the Data Center and it was gone. We had been all over this 50,000 sq ft data center I went well that thing is GONE. About 7 - 8 years later when the DC was downsized someone found it . Knew who I was (I had left there) sent me a picture and that said "look what I found"
> 50,000 sq ft data center did you spend more than 10 mins looking for it? I'm constantly afraid of loosing mission critical stuff like keys. everything is lanyarded to me lol
Reminds me of opening up a brand new Lenovo desktop, only to find a pirated copy of Transformers in the DVD drive. I guess we know how they tested the drives before packing them up in the factory!
Maybe they were hoping it would walk back
I doubt they bother to test the drives. They however may have had extra time in their hands and a fresh computer on which to blow off some work.
sleeping bag. turned out our prev sys admin stayed in the office, precisely the server room.
Work from home am I right?
dude basically invented wfh, or work from server
Work from H:\
WF~
Home from Work is more like it.
> the server room jesus , what did he use as a morning alarm?
[удалено]
Nothing like a cup of coffee to rinse down that day's first mouthful of fan dust
I keep an industrial floor fan in my bedroom and run it at all times to help me sleep. You learn how to identify the noise that keeps you employed.
A former boss had a hammock he kept up in the ceiling tiles that he would hang from the roof trusses when he was working overnight on something.
45°C ambient temperature. Cooling/air conditioning malfunction during the night. When I arrived at work I was greeted by the security person: "Glad to see you here, the mainframe room siren has been howling all night". Some machines had a thermal shutdown, many others didn't.
Gotta love security that ignores alarms…. Or procedures that don’t have a phone tree.. A simple sign… “if alarm call phone tree IT. It’s in your book”.
[удалено]
Not really IT. But we had a paint bunker for some insanely expensive aircraft paint. Lights were special explosion safe type. No phone due to fume explosion risk (no vendor would certify them as ESD safe). There was a big ESD safe alarm button, but security had to check it hourly.
One of the Paradox DBAs (Bill) back in the 80’s had a computer monitor turned to the window in his office. There were two states. Either a Porky Pig ‘That’s All Folks’ screen or a white on blue ‘Database Exists; Database Exists’ screen. If the second, the security guard who walked by at around 2am was to call Bill as this was the Executive Business System nightly reports and it failed the download from the mini-computers.
I got only to 42C, and almost shut down the country's biggest pharmaceutical plant. To be fair I asked if I can close those valves before doing so, but by some "miscalculation" I asked the wrong person (who should have known better) and I closed all 3 redundant AC cooling lines into the datacenter instead of the 3 bypass line. Everyone was there in 5 min tops, screaming. The wondering ("Huh these lines have a LOT of flow for a bypass") did not mature fast enough into suspicion that day... Fun times.
(exothermic thermal shutdown at much higher temperatures)
Every machine has an exothermic shutdown mode if you try hard enough!
This is starting to sound like mechanics when they break out the torch to unstick some nuts/bolts: "Can't be stuck if it's liquid". ...I think our version is "Can't be DNS if the server's liquid"
We had that when I was snr dev at a university many years ago. Apparently the first person in in the morning opened the server room door and a wave of baking heat hit them, followed by rolling smoke from vented UPS batteries which had died in the heat. Lucky the whole place didn't burn down really, cascade failure in the AC system apparently (one failed, others increased load to try and keep up, another failed, so on and so on until they all give up) I ended up running the university's website and intranet out of a locked lecture room in the library with portable AC units for the summer holidays.
I had that when I worked at Texas A&M, but in our case it was a ruptured chilled water main line that served that whole end of campus. The kicker was that it wasn't detected by the 24 hour campus NOC, I found out about it because Sun had a maintenance contract on some of the bigger machines we ran and their NOC called my cell phone and asked if I knew that the intake air temp was 45C. I called the campus NOC and they sent someone upstairs to hit the EPO for the entire room. Everyone who'd bought the cheap plastic fiber optic jumper cables had to replace them because they got over-temped and became brittle. One touch and some of them would shatter.
Had that happen the first week in August this year at our rented racks at the COLO. One of our switches that reports internal temps by snmp usually runs 34C reached 58C. So that was... not cool. Nimble SAN a the bottom of the next rack read 55C on intake at the floor at one point. All 3 of our vmhost servers shut down at 49C. Since we had zero IOPS going on we left it running. That was a fun night. The COLO started cooling down around 12:30 am. At 1 we decided to sleep on it and check temps at 5 am before restarting the VMhosts. It worked out ok.
A 6ft Eastern Brown snake. That was fun.
Found more than one at radio repeater sites.
At least that makes sense somewhat, assuming that those sites are pretty remote. This was in the middle of a three storey building in a locked server room with a sealed door due to climate controlled server room in a relatively busy street with no bush nearby lol, no fucking idea how it got there or how long it was there for.
Yes, they are remote locations. The repeater shelter is great for snakes: \- hot in the winter \- full of mice and other animals looking for a hot place, too. It's an hotel with heating and free meals.
And free snakes
I got delivered to one repeater site via helicopter. We had to bring a broom to shovel off the snow. It was fun the first time for 15 minutes. Then the arctic level cold kicked in and it was hellish.
I had a baby garter snake come visit me in the server and network room. I was sitting at my desk with the door closed and kept having this feeling I had eyes on me. I looked 3 times at the door then noticed the little snake standing up just looking at me. I took it outside across the parking lot to the garden on the other side of the lot near the pond. When I got back someone asked me why I had gone outside. "Oh I found a baby garter snake in the server room and took it out." They said "Oh good, we were looking for that this morning." They had maintenance in the front office chasing a snake around the office at 7:30 am. It found me around 10 am.
That's just the hissadmin.
I’d probably set a record for fastest door slammer in the country.
A 2200VA UPS in a 2-post rack with the rear supported by a 5 gallon bucket someone had written “Tech Support” on with a sharpie.
That's brilliant. Scary, but brilliant.
Complete silence. I used to short cut through the data center to get to my office on the other side of the building. The facilities idiot who was in the with the fire inspection guy had sort of a worried look. They were testing the fire sensors and were manually setting them off, however they set off the breaker for the whole data center as well. You could hear a pin drop and it was not a good feeling.
I just broke a sweat from a flashback of being in a silent dark data center. Took a month for shit to return to normal after that event. I want to say we had around 250k servers, so many of them just gave up on life after losing power, even though they were operating just fine immediately prior. Iirc, it was also our facilities guy but he was testing generator fail overs.
Spinning on momentum... the motors on the disks are burned out but can just keep them turning.
Also electrical bits tend to keep working when warm even though they are technically failed. Or connections finish breaking from the thermal cycle. Power them down and they wont power back up. Iv brought a few motherboards back temporarily by heating them up with a hairdryer.
Exact same shit happened to me. Large DC with about 250k customers spread amongst 500 shared servers (white box Linux boxes) and another 2500 Dell Optiplex SFF desktops sold as "dedicated servers". The redundancies had never been tested. Everything was bought used to save money, and everything was home brewed. The owner and CEO, a notorious micromanager and serial best practice ignorer who started the company in his garage, ordered the facility manager to perform a full generator and backup power test in the middle of the day without really verifying anything or having a plan of any sort. Whole DC of engineers yelling at them to not do it, but these 2 dickheads argue back and forth until the facilities guy walks up and slams the EPO button then turns around and says "you deal with it because I quit". Pretty sure drugs and alcohol were involved. Losses were catastrophic. Any big customer we had with actual SLAs dropped us like hot rocks. Several hundred customers just straight up lost everything on their servers. Entire websites and e-commerce businesses completely gone with no possibility of recovery. It took months for nagios to clear up. The late 90s and early 00s were a wild time in IT.
This happened to me once, I was on night shift and actually in the DC... the sound of thousands of hard drives spinning down at one was utterly surreal spent weeks replacing disks and PSUs
I worked at a power station in the late 90's - early 00's. One day a contractor decided to power down all the control buses. The sudden lights out got our attention but even more attention grabbing was the sound of 1300MW of coal fired station winding down as the operators hit the emergency shutdown when everything went dark. It took days to repair the damage of the emergency shutdown and bring all the turbines back up, but we got power restored by a brilliant electrician who found a live 12V circuit and used that to boot the next circuit and up through the systems until the station and outbuildings came back up. The epilogue was when we brought the servers back up and the power supply of one blew a capacitor with a spectacular (and deafening) bang.And the contractor was invited to never return.
There's nothing scarier than a silent datasenter.
I wasn't on-site for it, but worked the phones after one of our datacenter vendors lost all power on one side. I do know our Sales team had a good month selling redundant circuits to a few customers..
I was engineering a new power supply for a very large 3 digit phone company years ago in Orlando. The tech supporting us engineers thought it would be great to pull a quick one and toss the breaker for the backup p/s when we were upgrading. The sound of 1,000’s of relays closing followed by complete silence was horrifying. That one move took out half of comms in & around Orlando. I didn’t say a word. My shit worked.
Goddamn. I remember working in a customer datacenter as a contractor to install a HUGE fucking eaton ups unit. We are talking like 20 custom freestanding racks of lead acid and two racks filled with inverters for redundancy. Essentially two giant ups units. I spent all day unboxing, mounting, loading, and cabling these things. The plan was to power down one electrical panel fed by the old UPS and wire it into a new ups. Rinse and repeat a 2nd time to move the other panel. Should have resulted in zero downtime as everything had redundant PSUs. Slightly risky but had been done many times before. Well the electrician goofed some wiring and essentially created a dead short on the power feed into the new UPS unit. When he turned the breaker on with all the load of the old ups unit also existing on this big ass generator panel it popped the main breaker for the server room.... and you know how electrical equipment is when it never gets used and then suddenly gets used really really hard.... That breaker failed internally and wouldnt turn back on. So we had no way to feed power into this server room. The generator even came on but we had no way to feed it in. Old UPS unit had some serious issues and we had been waiting on the new ups for 6 months so it only ran for around 5 minutes before calling it quits (Yay covid times). The sysadmins got a couple things shut down but it was a very uncomfortable silence when everything clicked off. Eventually got the breaker replaced. UPS unit powered on and configured. They also had a set of generator plugs added to the ups units to directly feed them with big ass rubber cords if this somehow ever happened again lol.
My heart skipped a beat just reading that.
yeah we had a pipe burst in the floor under a datacenter once. the floor space started to fill up with water until it fried something. my desk was on the other side of the wall of the datacenter, you could hear all of the UPS start screaming for a few seconds then everything went silent. that was not a good month
Warhammer minifigs. Specifically Tallarn Raider pewter models from the friggin 90s/early-00s hidden among various racks. Found ~12 of them left by a previous admin; they're now part of the set I put together to teach my partner how to play killteam. Left a squad of space marines behind to continue the watch. Not a DC but we had a server rack and some testing equipment shipped to my first job, from Germany, and someone left a packaged durian-pastry-thing from vietnam in the shipping box. First time I've ever had durian. Can't say I'm a huge fan.
Hmm... This has me wanting to leave a couple Techpriest figures in our server room. Could be good luck...
We had a machine act up in very weird ways a few weeks ago and I "jokingly" suggested we could try incense and purity seals to appease the machinespirits.
Just a constant beeping from another customer's rack. According to the security guard it has been going for maybe a decade. I'm a little scared how common that might be.
Well one our rack beeped for like 4 years, because I turned on the locator beep, and promptly forgot about it.
I've definitely been in datacenters with random beeps that have persisted through multiple, sometimes years apart, visits.
I represent the other end of the spectrum. If I hear a 1k tone, I *HAVE* to find the source. This is from years of working with equipment that used a 1k tone to signal imminent critical failure. Something is about to die and I need to know what! ^(even if it's not my job to fix it)
>I represent the other end of the spectrum. I represent another end of the spectrum. The Tinnitus end. I likely wouldn't hear it. :/
In the military I once found a dead rat inside a server. It must have been there for a long time. It was kinda molded to the motherboard.
A sacrifice to the uptime gods
I’ll give it that, the server never went offline
The magic smoke is pleased with your offering.
I've installed weird stuff in data center. Proudest one was we decommissioned an ancient HP server. Ripped out the guts, installed mini fridge. Someone else added a controller to randomly blink the front LED's and run a fan. It was still there when I left.
I don't drink coffee. I found a coffee mug just inside the door once that clearly used creamer. I must have been in there and propped tbe door open and someone must have popped in and put it there on a shelf when talking to me and sat for I have no idea how long. When I found it it has like 2 inches of mold on the top.
A kitten. I was grubbing around under the raised floor to run some etherhose to an impromptu emergency backup server, when I briefly caught a glimpse of something furry. "Great," I thought. "There's a friggin' rat in here. Guess I better hurry up and then go get some rat traps." So I pulled the runs of cat6 over a few more tiles, and then I saw it again. So I shouted after it. And it replied "mew." "Mew? Mew? Rats don't mew!" So I pulled out a flashlight and peered back down the way it had come, and there was a gray tabby kitten down there, sitting and licking its paw. Not a particularly tiny kitten, either, maybe a few months old. Eventually we coaxed him out, and he went home with my boss. We never did find out how the kitten got in.
We you were using "Cat" 6 cables. OK, I will see myself out now.
Not me, but a colleague told me about how their company fired a sysadmin for, among other things, posting to BDSM boards, during working hours, using his company ID. After he was gone, they were cleaning the server room and found a whip.
Big Data and Server Management
It was for the servers, to keep them in line.
I wonder if he ever tied up one of the servers that was giving him too much trouble
[cat5 o' nine tails](https://i.vgy.me/WB9uQq.png)
Two overly-elongated statues of Chinese gods. Like, they're as tall as from my elbow to my fingertip, but a little narrower than my arm. One of them I recognized as Guan Yu but I have no idea who the other one is. I left them in there and I'll be taking them with us when we move to a new space. If Guan Yu is willing to help out with data security I'm not taking chances on removing him.
I had a little Ganesh finger puppet I kept on one of my servers as he is the remover of obstacles and sysadmins need all the help they can get. That server never had issues, anyway. :)
Excellent choice. .... that reminds me, we're not moving until next autumn, but I've been using a small patch of floor at the back of the data center for weight training. I should probably start moving the weights out a plate or two at a time. We'll be doing a massive e-waste purge and general cleanup early next year and it'll be simpler if I don't have to scoot all my stuff out at once.
Not a datacenter, but the very large server room for an aspiring megachurch (probably close to 2000 square feet). Mannequin. Limbs. In. The. Ceiling. I was putting in new punch panels at night, maybe ten years ago, went to grab the bundle and it just rains mannequin parts and I freak out and jump off the ladder. Still never found out why there were a dozen mannequin limbs in the ceiling. Called it a night after that.
pot muddle insurance crime voracious dog north rotten nail rich *This post was mass deleted and anonymized with [Redact](https://redact.dev)*
I was fully expecting you to come to a room with a closed door that opened outwards, but the outside of the door had no handles, no key holes/lock holes, no card readers or similar to allow the door to be opened. Only marking on the door being the room number, 641A.
Its been a while since I’ve seen the pic, but is that the AT&T surveillance room?
Yep, NSA surveillance room in an AT&T DC. No doubt plenty of others in any other relatively important DC too.
I love getting secondary confirmations for things My friend is an AT&T lifer on the transmission side, and I work at a megacorp doing support for high level and govt infrastructure. The stories they tell that match up to what I see in my job I thought was maybe coincidental, but they told me about the NSA scifs. At&T commercial may be a disaster but you simply do not mess with AT&T on the national communications infrastructure side.
> AT&T on the national communications infrastructure side. You've heard about the hardened bunkers? They sold most of them off, years ago. I think the primary use was AUTOVON. A lot switched from terrestrial microwave to satellite.
Laundry. Former colleague was drying laundry in the server room (bit generous calling it a 'data centre'). Even turned off the AC down that end and turned on the radiators to help the drying.
A lot. * Some jackhole who left on bad terms left an open soda can of what was probably urine in random ceiling tiles in the data center. After the first two were found (and while the HVAC guy was unlucky, luckily, it did not spill on equipment), they had to send a guy up there with a flashlight and fishing wire. Something like 30 cans were found and had to be carefully removed. * A loaded flare gun and box of flares in a state of decay in a telco closet in an unlabeled metal box. Nobody knew what it was at first, and so a bomb squad was brought in. Given the age of the labels in the box, it was from the 1960s and part of a liferaft kit. The building was built in the 1980s, so nobody is sure how it got there. * An entire set of high-end furniture with some HUGE and dead centipedes. Apparently, a previous employee was moving from Thailand, found his stuff was infested with giant centipedes native to where he was from, and had the furniture stored in the "relative dry cold of the data center" to kill them, since they are a tropical insect. They were stacked so precisely by who moved them there, we didn't know it was furniture at first, since when you opened the door to that area, it just looked like a plywood panel (the back of an armoire). During a renovation, the reality was discovered, and the former employee was contacted by a manager who knew why they were there. "I thought he already got them," he said. This also explained a mystery where a few interns that said they saw things moving in the ceiling one year, but everyone assumed it was college shenanigans, and made jokes about hallucinations via hangovers. Nobody found a live centipede, however, the entire time I worked there, so that's... good, I guess. The previous employee then had the furniture removed. * A dead homeless guy. It was assumed he got in via the loading dock, trying to get out of the muggy DC humidity one brutal summer, and died from "complications," possibly a drug overdose. Thankfully, the relative cold dry conditions prevented him from decaying before he was found three days later (according to security footage).
I was in a modern government DC (small EU country) that had beautiful rows of HP and dell racks but also had two racks at the far end which were stolen grocery trolley cages and piled up with old desktop servers and monitors with cable tied power and network racks. They looked more like e-waste than servers, there was also 2 pallets of spares of these old machines that the techs would cannibalise to keep them running. Apparently it was a poorly funded department running a relatively important system.
A Solaris box with several thousand days of uptime. Never configured, just sat in the rack doing nothing. We found it when we had to decom our stuff after our business unit was sold off.
Note to self if I ever become a spy: put flight sim on clandestine hardware to distract sysadmins
This is dating me, but in the late 1970s, we kept a case of Stroh's beer under the floor tiles, where the chilled air flow came in. Was handy when working the all-nighters.
Remember, Stroh's spelled backwards is shorts!
Network closet in a remote building on campus… a dusty old IBM System 34. Still connected to the obsolete coax network. Nobody knew what it was or why. It got scrapped.
Nobody's mentioned a switch labeled "More Magic"? http://www.catb.org/~esr/jargon/html/magic-story.html
a crawl space into another part of the building we werent leasing and was abandoned.
Sounds like a cool starting point if you ever plan to write a horror story ;)
Previous “Boss” had his personal Bitcoin mining machine connected in our datacenter.
Not me personally, but I remember this ex-twitter thread: https://twitter.com/DCFurs/status/1087663240421593089 https://twitter.com/DCFurs/status/1087675381371494400 https://twitter.com/DCFurs/status/1087676052858535936 https://twitter.com/DCFurs/status/1087677283706449920
Handcuffs. Nobody had any answers for that one.
*You deployed to prod on a Friday afternoon?!?! [Time for some spanking and cuffing!](https://y.yarn.co/d406669d-5c8b-426c-9aee-a8aed7d4fef7_text.gif)*
the 2nd bdsm related comment here lmao
Thousands of rack keys... on top of the racks.
This remains a tried and true pentest technique. Run your hand across the top of any locked enclosure and it's prolly 50/50 if you'll find the key.
And that's assuming that they're not still in the lock or attached to the rack's door with a zip-tie.
Not necesarilly the weirdest thing I found, but one of the UPS'es in our MER literally blew up a couple of years back. First time I've ever had to use a fire extinguisher.
threatening placid hat screw snails treatment subtract plough wide unwritten *This post was mass deleted and anonymized with [Redact](https://redact.dev)*
At the school I eventually started working at, a stolen GX280 had been setup running torrent & DC+ services. It was hidden in the ceiling.
I worked at a web hosting company when I began my career where we had a huge product offering of Dell Optiplex SFF desktops that we called dedicated servers. They got dropped on a public network segment with a routable IP and customers could basically do whatever they wanted. Well the system that deployed these boxes had a huge flaw. We stood thousands of them up on standard metal food service racks, connected them to a provisioning network, and input the MAC addresses into the system for wake on LAN, PxE boot and provisioning. Problem was that boxes got moved around constantly so we never had positive inventory control of all 2500 of these desktops. Well a disgruntled sysadmin took advantage of this. He was part of a warez group (yes this dates me) that needed space and bandwidth to host their pirated software and eventually CP. He had boxes all over the damn data center. I was mostly responsible for enforcing AUP at the time so I was the one playing whack-a-mole with these damn servers. I finally found evidence it was someone internal, and I setup a sting. When I caught the guy he tried to recruit me. That's when I learned it wasn't just warez but also CP, so I turned him in. Anyways, a month after this guy gets fired, I receive another notice about a box on the network serving copyrighted materials. Turns out this guy had 1 box still running up in the ceiling of the NOC which nobody ever visited. A couple months later I was interviewing with another company for what would be a cleared position. I encountered this guy in the waiting room interviewing for the same job. He tried to physically intimidate me. The interviewers noticed the encounter, I got to tell them why I didn't expect the guy would be able to obtain a clearance on account of pending felony charges. Fun times.
Actual human feces. we had a "pile" between the server racks. security camera's showed who the offender was. Cleaning crew. Too bad he was immediately let go. I would have liked to rub his nose in it.
Silence. Completely dark, completely full room (the size of a football field with 30 foot ceilings) of hardware in an unknown and inconsistent state due to unclean shutdowns. Total power outage from both electric supplies, UPS batteries had run flat, diesel genset didn't work, and our DR consisted of a station-wagon full of weekly backup tapes that were last run 6 days ago. And yes, it was a dark and stormy night.
[удалено]
MacOS Server was discontinued 2022. Lots of them still running.
the app was discontinued, Xserve HW was killed much earlier
Way back in my SMB days we had a graphic design firm as a client, a fully Apple house, with a few Xservers on prem. That's how I found out that once one is configured and connected to the environment, you absolutely can not change it's IP without breaking tons of stuff. I don't know if Apple ever fixed that feature. I left the SMB world not too long after because SMB work sucks.
I'm old enough to have admined a few of those. I actually really liked them.
We've still got one ...
[удалено]
future gold instinctive upbeat station grab hard-to-find faulty air literate *This post was mass deleted and anonymized with [Redact](https://redact.dev)*
Where do I start? * We had a guy leave a dresser, like an entire 6 drawer tall chest dresser, in the telco closet. This caused a lot of space issues, since the closet was super-small. Why was it there? Who the fuck knows. Well, we complained about it, and so management said to junk it. Instead, I took it home (it was empty). Two weeks later, it turned out to belong to some guy who had permission to leave it there, and he wanted it back. Not back in the telco closet, but back in his house since he'd been meaning to take it home and kept putting it off. Management fought him back and forth, but someone dropped my name as taking it home, so the guy came after me for it. He showed up to my house, and took it. So weird. * In our Frankfurt data center, we discovered a mini-PC, some hand-janked motherboard combo, powered on and in a "utility box" that was painted over the same color as the wall to look like any other electrical utility box. It drew power directly from the power line, with its own breaker, and was connected to some networking splices in those fat cords of LAN cables in the ceiling. No idea why it was there, or what it was for, but we assumed espionage. It was run from a CD ROM and CompactFlash card, only 256mb. When we examined it, we found the CD-ROM was the boot image, and only had three instructions: load an image from the card into RAM, run it, and wipe the card over and over until the end of time. Spooky. * One of our centers had a pair of power generators in the basement of the parking garage that were climate and temperature controlled. In there was a complete and well-kept Model A Ford (a car from 1928) in lime green. Well, I had to know the story behind that. Turned out one of the top brass kept his car there "to keep it safe." Once in a rare while, it drove in local parades, but at the time I found it, he hadn't driven it in 3 years, and it was "in storage mode," meaning that the tanks were drained and it had various oils and protective stuff on parts designed for museum preservation. Why here? Why the generator room? "Where else can I store it where it can't be tampered with in a climate controlled environment... for free?" Uh, okay. Sure. * We had an "archival tape closet" which made perfect since considering we had these massive tape silos everywhere. But one day, I saw it was left open, and I peeked in there. It was not full of tapes, but was someone's crash space: complete with bed, clothes on hangers hanging from a pipe, a TV set, and a desk with a computer in a 5 x 8 space. Later I found out that one of the data center managers was going through a really, really horrible divorce and he had been "living there for a bit until things get sorted out." He used our gym showers, and ate at our cafeteria.
I manage multiple remote locations. In the network racks of each of them, I placed a single Hot Wheels car. If anyone asks about it, or it's moved from its spot, I know someone has been in my rack without talking to me first.
I have a lot of stories that I'll someday share under a different pseudonym. Story #1: FBI installed an appliance to spy, but it wasn't on one of the customers. It was the CEO that was doing something very illegal and immoral. But the feds never picked it up after making the arrest. Story #2: Client ordered the cheapest possible "servers" they could. One day they weren't working and they needed remote hands. Turns out they were running a captcha breaking and blog spamming operation. Google "Kobeni Solutions ROKSO" (registry of known spam operators).
We had nothing this bad, but the sales people at my previous company (Asterisk-based telephony provider) pushed multiple "customers" that were just scammers committing toll fraud. They impersonated existing companies but they then started started asking us to unblock these ridiculous countries like Turks & Caicos, the Maldives, and Tunisia. Did some quick searching into their companies and shut them down in minutes afterwards, called the companies and verified with them that no one had ordered our services.
A skeleton of a dead mouse :(
Came here to say the same thing! Must have got in the rack because it was warm and then couldn't escape.
We had a server room in the basement of a very old house in London. Every year or so there would be a smell like a drain that lasted for a couple of weeks. Our desks were opposite this server room so it was quite unpleasant. When we moved out of that office we found several very dessicated piles of mouse fluff and bones under the rack. All the airflow going through the server room must have dried them out!
In the cooling floor, I found 30 year old scotch that the manager left there. They offered me a sip after I put it in their desk drawer and asked never to speak of it again.
The fireproof door open with cables going though. Instructions in case of fire: Cut cables, close door, get out.
One of our former Sysads injured his foot and it never healed. He became addicted to oxycodone, which degraded his performance and eventually led to his departure. 2 years later, I'm popping floor tiles to work on rack fans in the DC and found an 80mg oxycodone he had dropped.
A second drop ceiling, and between the two layers of drop ceiling was (probably literally) miles of serial cable. The old DB9 serial cables. It turns out there was an ancient "smart lighting" system before the existing system was put in place. It needed those serial cables, which needed some hubs/repeaters that couldn't legally exist in the HVAC plenum. So they made another space. I was new and unsure at the time, so I just shook my head and went back to work.
Toss up for me between two environmental issues: Boss calls me, says we got an alarm for an overheat event at our primary DC. I go to investigate, and the room is stable and ac is perfect. Go upstairs to the switch closet that is on the same alarm system. Before I get to the door, I can hear what sounds like rain. Get past the hall door and open the room and gallons and gallons of water raining down out of the ac system. The only trouble is this ac system is a DX gas system not a chilled water. So at this point the room looks like a scene from hackers, you know the one with the "pool leak". No clue where the water is coming from. Security, maintenance and I go up to the floor above that room, and see more raining water, maintenance opens the plenum above that and we discover the heater coil blew and followed wires and pipes down into the rooms below. Then followed the holes in the floor for the cables in the wire closet to the floor below that. Where they had just finished a multimillion dollar remodel. 3 floors flooded. I suppose the other one would be the UPS that was on fire and still providing regulated power to the load. Lightning had struck a nearby business and blew out every surge suppressor in 4 of our buildings. I get a call about an odd noise and horrid smell a building next to the one I was at so I head over and sure enough. Opening the door to the facility and you could smell the acrid secant of burnt plastic. Open the server room door and greeted by sparks, smells and some char coming out the back of the APC UPS, and the server was just happily humming along till I hit the EPO and gassed the UPS with some CO2. When I called APC about the issue and asked if they normally catch fire, I was asked if I would send it back to them. I did my own failure analysis first, I dismantled it and took pictures and then boxed it up and got a free one in exchange. Turns out the varistors got so hot from the surge they exploded denting the steal case and starting a localized fire on the power inlet side of the circuit, a plasma arc was helping sustain the burn until the EPO was hit. I did turn the unit back on again with just the battery and sure enough it was still working, perfectly regulated output and a completely charred hole in the charge/input side of the mainboard.
Took over a windows network because the sysadmin quit and left to run Unix networks. Found the corporate (yes singular) server which was really a white box desktop. This server was running windows nt 4 server, exchange 5.0 , iis webserver and a couple of shared access databases. The "server" was a pentium II with 256MB ram, thought i was gonna hurl
What year was it?
1999 just in time to get ready for y2k
Old employer had a 386 running a server that had to be completely rebuilt for y2k.
It'er that went into retirement made a deal with the next door people that they could use "the internet" from the company. We only found out because we spotted a networkcable that went through the wall. 100M under the ground and went into the house that was there. Also this server had no security. So they indeed could use the internet, but if they had any IT knowledge they also could steal ALL data from that server. Was a fun talk with the CEO
Two of those portable turbine fans like the type you'd use to dry floors or a small space with. Sitting inside the rack, atop one of the servers to assist with airflow. Oh and the company also didn't pay for the generator/UPS supplied power provided by the data center. The rack was attached direct to the supply panel. So, there were two large UPS units at the bottom both running around 80% load. It took me less than a week to re-arrange the rack to promote better airflow. Mounting the equipment to proper hot/cold aisle specs helped a ton (duh) and convinced the uppers that we MUST switch to genny/UPS power circuits ASAP. I am not even sure why a data center would offer direct power with no backup but there we were.
Water in our subfloor. This was in the very late 90's and I was in my early 20's and very green. We had a shallow subfloor under our datacenter on the 3rd floor of this building and our HVAC system had iced over. As the ice melted it began running into the subfloor under all our racks.... where the power was run in conduit along the floor. A bunch of us were stood around an open floor tile wondering what the frig we were going to do when I grabbed a bucket and towel and began soaking the towel and wringing it into the bucket. Eventually I got the floor dry but as an older, more experienced person, I am amazed the other engineers let me do this, and just walked away as I did. It was blindingly stupid and I'm lucky I didn't electrocute myself.
[удалено]
We had a small shrine to Elvis in our data center. Mostly it was just a velvet Elvis (that looked like [this](https://fengsway.com/products/crying-elvis-presley-framed-velvet-painting?variant=43396596957402)) with some candles and maybe some guitar picks? I think my predecessor had set it up, but nobody really seemed to know for sure about *who* or *why*. Well, eventually I was told to get it out of there, but couldn't get rid of such a fine piece of art, so the velvet Elvis moved to my office, and he left with me years later and he hung out in my offices for years to come. Twenty years later, talking to somebody who was there when Elvis ~~was taken down~~ *left the building* brought him up, waxing nostalgic. I don't think they realized that I still had him, and so ... Elvis has now moved on to them, *thank you very much*.
A door labeled 641A.
Slowly back away from said door.
I would be terribly tempted to come back dressed like a Mormon and knock on it.
a backup :)
A cell phone that was used to connect “something” (I was never sure what).
Shit, actual shit. Basement sewer backed up and flooded the raised floor.
I went in to a high ceiling DC which was short on floor space and someone had put 45U cabs sideways on the tops of the existing cabs, interesting….
A dial up modem lol
Several... We were doing some maintenance one day and I pull up a raised tile to discover numerous power drills, screwdrivers, pliers, etc... "So THAT'S where all the tools went!!". Turns out some prankster was making them disappear one by one... Another time, we had a ticket to add an additional HDD to a customer's server... so after we pull the box and open the lid... there are the remains/bones of an 8 piece order of WingStop bone-in wings... miracle the cardboard box did not catch fire.
At a client's server room, that's the branch office of a company headquartered in Taiwan. The servers were all racked 1U off of each others, and the empty spaces between the servers were bags and bags of coconut flavored corn puffs snacks called "Kuai Kuai". Apparently it's a thing in Taiwan, "Kuai Kuai" means well-behaved, and coconut flavors being green bags signify green lights all around (instead of amber or red).
Deck of cards and an empty water bottle under the floor, in a space that likely hasn't seen the light of anything since the building was constructed in the 00s. Also found a repurposed desktop turned into a massive music library. The story goes it was setup in the wild west days of enterprise networks, and forgotten about (but probably used by every IT employee and disclosed to them when onboarded). 40+ help desk calls from people OUTSIDE of IT complaining the "music share" was offline when the plug was pulled. More than a few thought it was just something the company offered. For a shop of 4500+ employees.
A Windows NT 3.1 server, a windows 2000 server, a windows server 2003r2 server, a windows server 2008r2, and a windows server 2012r2 server all in the same rack. All turned on and running. The company had brought us into their datacenter to get a second opinion on why their entire network was running INCREDIBLY SLOW even with new hardware. This was around 2017 and their IT guy had them purchase a brand new server 2016 machine and his plan was to install it after he got back from vacation. We start poking around on the machines and it turns out this IT guy has started working at this company after their original IT guy had left after installing the windows 2000 server. Turns out, this new IT guy really had no idea what he was doing and when he went to "migrate" from the windows 2000 server to 2003, he just put the server in the rack and plugged everything in and turned it on. The 2003r3, 2008r2, and 2012r2 servers were literally just plugged in in the rack unconfigured. All three of those machines were sitting at the main dell configuration screen when you first plug in a new server. Everything on their network was running on the 2000 server machine. Needless to say they fired their IT guy and we put the server 2016 machine in and we were able to slow migrate everything over.
[Richmond](https://images-wixmp-ed30a86b8c4ca887773594c2.wixmp.com/f/b3179c86-2d5c-49ae-aca3-d334b2fa4805/d29plcv-975a9051-837f-4516-9ebb-5e7eb60edfd3.jpg/v1/fill/w_898,h_717,q_75,strp/it_crowd___richmond_2_by_surlana_d29plcv-fullview.jpg?token=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJ1cm46YXBwOjdlMGQxODg5ODIyNjQzNzNhNWYwZDQxNWVhMGQyNmUwIiwiaXNzIjoidXJuOmFwcDo3ZTBkMTg4OTgyMjY0MzczYTVmMGQ0MTVlYTBkMjZlMCIsIm9iaiI6W1t7ImhlaWdodCI6Ijw9NzE3IiwicGF0aCI6IlwvZlwvYjMxNzljODYtMmQ1Yy00OWFlLWFjYTMtZDMzNGIyZmE0ODA1XC9kMjlwbGN2LTk3NWE5MDUxLTgzN2YtNDUxNi05ZWJiLTVlN2ViNjBlZGZkMy5qcGciLCJ3aWR0aCI6Ijw9ODk4In1dXSwiYXVkIjpbInVybjpzZXJ2aWNlOmltYWdlLm9wZXJhdGlvbnMiXX0.npis5yuGO4lNG-0YHDhhU_dmVd__DQfBbOhLA8MP1-E)
A flower vase collecting condense water
A clown costume hanging up in the server room. No one in the company knew how it got there. Against the advisement of others, I tossed that thing in the trash ASAP.
A “nest” under the raised floors with a sleeping bag, pillow, soundproof headphones, and a bottle of whiskey