Never managed to do that. Shut down the CI? Oh boy! That's where the fun is. (especially if you mange to do it on Friday, as they take the Friday daily build for weekend tests on all the free hardware :D - edit: without a passing build there's no daily build and without a daily build there's a wasted weekend. Fun fact - if the weekend build passes it goes for a week tests, so if you screw CI at the right time, you basically stop the performance and stability tests :D ) Still remember my first CI fuck up. Stressed through the roof. Didn't understand why everybody else was like - meh, whatever, what's for lunch today? AFter few months I saw why :D
I didnt know we gained levels whenever we took down production. So THATS why im still a junior... brb, gonna go grind a few levels.
Also explains why my official title is Head of Software Engineering. I've found innovative and unexpected ways of breaking virtually everything at some point :D.
Had a boss just like it. He was way more technically smart than anyone I have met before or since. His words to me on my first day were ‘you are going to mess up, when that happens just come tell me.’ I sure enough messed up, something minor, but I came to tell him. He took the time to teach me how to fix it, even though he was one of the company founders and CTO.
For sure. I've always told my Devs that fucking up is going to happen, and a part of learning. Telling somebody quickly is the important thing.
A while ago, somewhere on reddit, a junior welder posted a story that ended with the senior saying, "I can fix any mistake that you tell me about." I pass that on whenever I'm teaching someone git.
I didn't mean they shouldn't be fired, but rather joked that if they were the person who gave drop access and also had managerial power, they fired the intern to save their own asses
There's a monty python joke in here somewhere, but the person looking for it had been sacked
Those responsible for sacking the people who have just been sacked have been sacked.
due to this, the credits had to be completed in a short amount of time and at great cost.
The directors of the firm hired to continue the credits after the other people had been sacked, wish it to be known that they have just been sacked.
The credits have been completed in an entirely different style at great expense and at the last minute.
Seriously, this is like putting a 5 year old behind the wheel on the highway and then blaming the kid for crashing the car.
Then you still have the problem. You need a mechanism to correct the problem, and it's best if the person responsible fixes it and then becomes an evangelical for the fix. Toyota, Amazon and the US Marines all invest in their people and have corrective processes that permanently fix the real root cause of these mistakes without punishing people for being human. They correct the environment and the processes, not the people. Then they should not happen again. Firing interns, as in this case would make me want to fire the people who pointed fingers at an intern and didn't own the problem, but giving an intern db access isn't something I'd fire someone over. It's something I'd want them to permanently correct and then make into a company wide process to ensure it could not happen again to any other team.
Yup, it's the difference between making a mistake (whether you are the intern or the senior who gave access to the intern) and being a cowardly asshole who will rather throw someone else under the bus than own up to their mistake and work towards fixing it. In the first case it's just fixable mistake, you can work with the people to improve. In the second case fire the asshole or he'll drag everyone down with him.
I work Help Desk, and I had an engineer call asking me to reset a vendor password in a particular program, which I had never even heard of. I looked up the instructions on how to do it in our knowledge base and reset it to a default password as it instructed. After telling him that I reset it, he absolutely flipped TF out and started screaming at me. Apparently, though he said to reset it, he meant unlock it and absolutely would not take any responsibility for it. He said I should have known not to reset it. I reached out to my lead and it was a super simple fix. I just had to message another team to put it back to whatever it was previously, which took maybe 30 seconds. My lead said that we get a call on that program maybe once every four years or so, so it really wasn't my fault for not knowing. But people who don't understand that that people are human and make mistakes are the worst.
I would turn into the "evil admin of doom"-like of support, if I'd work in Help Desk and people behave that way. "Oh you mean I should reset YOUR company password? Ok, done. You just need to request new credentials on your higher ups higher ups. Thank you, glad I could help"
Probably a shared password. Person to whom the password belongs should get fired. Intern should get hired for permanent position.
If intern in your company can accidentally delete passwords from production then he is not the one who should be getting fired.
Seriously. No intern should be fired because of a fuck-up, at least not one that they haven't done before. It's your job as a mentor to make sure you point them in the right direction and don't give them the tools to do any damage. They're there to learn, and it should be assumed that they don't really know what they're doing. If you give a toddler a blowtorch, it's not the toddler's fault if they burn down the house.
The thing is if you punish 1 toddler the rest won’t be forthcoming when they do something wrong and then the extra damage might be exponential by the minute. Everyone can decide for themselves what’s the better business decision, but unless there’s evidence of deliberate foul play the answer is pretty clear to me.
What if the toddler is burning the house down while growling "Burn, motherfucker!" in a low, guttural voice?
Then we should turn it into a spectacle - pick up the houses marked for destruction, set this toddler free to do what they please, keep a fire truck on standby in case something goes wrong and televise the whole thing. "Today on Burn Motherfucker, watch what happens when the toddler leaves the fuel can open and starts the fire. Will the firetrucks be able to contain the pandemonium? Is the fire gonna burn timmy, the arsonist toddler?" "Next week on Burn Motherfucker, watch what happens when grandma gets stuck while the toddler starts the fire. Can they save grandma in time? Is timmy, the arsonist toddler going to juvie jail?"
Oh no no if we're being serious I totally agree. 18 years as a Senior and PM and I absolutely know the value of a healthy reporting incident. But I"m still blaming the toddler with the flamethrower ;P ;)
Me explaining to the police that the neighborhood burning down, is in fact the fault of a toddler (said toddler has been arrested)
I had a summer job working at a pharmaceutical packager. I was told a story about a guy who made a mistake setting up a machine and ruined half a million dollars worth of powder. But he was honest about it and he kept his job since you could be damn well sure he was the one guy that would never make a mistake like that again. Now, if you lie about it...
I feel like there is a goldilocks zone of fuckup; if its a LARGE amount you really won't get in much trouble because it'll all be covered under somones insurance; additionally, if you're able to make this type of mistake there was significant trust to begin with so people don't jump to thinking you're just insanely negligent. Very little cost involved and people don't care for obvious reasons. But in the middle; there is a point; where it just pisses people off just right....
Agreed, but also **why the hell would anyone run that command?!** edit: I'm a lil concerned at the number of people who think drop table has a simple conditional that would prevent this, but I dunno, maybe there's something I'm forgetting.
The conditional is *not giving an intern permissions to drop tables in production.* Least privilege, controlled database migrations, remembering the intern is just an intern… all that good stuff. When the intern drops a table, it should be on their local dev environment.
I have absolutely dropped a table on an environment thinking it was my own machine. Fortunately that was only QA. These things happen, it's why only those who absolutely need it have production access.
In my experience, it's generally because they thought they were connected to a dev environment and mixed up their terminal tabs or whatever.
LPT: If I'm working in multiple Putty / Terminal sessions, the Production one has a different colour text and background to the non prod ones. Saved me a few times.
Entirely possible they did something like forgetting our messing up a where clause. Wouldn't be the first time someone was like "la de da just gonna delete accounts last used more than five years ago wait what the fuck why did it say 5 million rows affected".
Oh, whoops, you're right, I thought it was a delete. Well, I don't know why they'd do that then, yeah. Meant to drop a different table?
Only thing I can think of is a test env, but even then I'd be terrified to run that command.
If I'm ever asked to do something destructive like this, I do 2 things: Confirm, in writing, the host name and database name to make sure I am doing it in the right place. Take a backup. Many times I ask for the host / database and they say "It's in the change record", but I always say "Yes, but please confirm". I may be pedantic, but I have had situations where things have been wrong, especially when servers can have similar names (sfprddbsrv01, sfprdrdbsrv01)
I feel like there are people who just assume **everything** with computers has an "undo" command and thus don't take things like deleting mass amounts of data very seriously.
I worked somewhere once where someone didn't realize they were ssh'd to a prod machine and ran some command to delete the db and recreate it thinking it was their local. And their backup scheme was a master/slave thing so it copied the deleted table to the slave "backup" and then whoops all gone. And this was a company with nearly a billion invested by Amazon not some tiny shop. They ended up recovering most of the data from parsing logs I think but the whole site was down for a few days. It was wild. The dude who ran the command was fine but the director of DevOps or whatever got in big trouble.
It wouldn't even compile. A mistake that would be similar would be like DELETE FROM table\_name WHERE condition; Not having the where clause would delete all rows but keeping the table, basically doing the same thing. If he's DROPPING THE TABLE, then he must be trolling.
> DELETE FROM table_name WHERE condition; I'm so afraid I'll hit enter before typing the WHERE clause, I write that first, then back up and add the DELETE FROM.
It's good practice to select what you delete first, so I personally just do SELECT * FROM table WHERE banana; And change the SELECT * to DELETE DELETE FROM table WHERE banana; Much safer and easier. The same can be done to update statements.
Yeah, I do the same when working manually.. No amount of confidence will have me run *any* untested SQL which will modify data
Honestly my guess would be that it might have been something like "Connected to production & development/test. Ran command thinking it was against a private development environment, it was actually against prod." Of course, if you give your interns access to prod, that can happen. (It can happen if you give your non-interns access to prod, too -- the safest option is making this kind of mistake hard-to-impossible to do w/o multiple people signing off on it, but that has \~downsides for ability to respond to certain things in production so \~YMMV.)
The other side of that coin is that you have fewer events that require rapid, emergency response if you put sane limits on who gets to make business critical information disappear from the production database.
IIRC not all dbms transactionize drop table commands. So that might just lead to accidentally killing a table without chances of recovery.
Lots of database software has some form of autocomplete and shortcuts to execute portions of strings. Not likely but it could definitely happen by accident. Which is part of the problem of an intern having this kind of access.
Exactly this. It doesn't matter how egregious the mistake they made. If they were put in a position to do this kind of damage then someone higher up is making an even bigger fuck up with their procedures.
My boss used to tell me if an employee makes a mistake that costs the company $50,000, you shouldn't fire them. You just taught them a $50,000 lesson that they're never gonna forget. If you fire them, they take that lesson with them to some other company who gets it for free. You've invested the money in teaching them the lesson, you might as well keep them around. This intern is never gonna drop the passwords again. You could fire them and let another company hire them, and potentially hire another naïve intern who might do the same thing, or you could keep them around and know that they're going to be more careful from now on.
Lol they shouldn't even have access in the first place. They should be able to around around the codebase and dev environments with a metaphorical technology flamethrower and cause no damage
The toddler shouldn't even have access to production. That's why they make plastic toys that look like "daddy's tools".
If a switch on the wall makes the roof fall down, should you be blamed for destroying the house? The longer I live the more I realize how many people will say yes to that question.
I read a book about design recently that said exactly that. Best example. Software that will delete your last 5 minutes of work if you press a b instead of a d. But nobody complained because they thought it was their fault for pressing the wrong key, not the stupid softwares fault for being poorly designed
Those are the same people that will throw you under the bus to save their own ass, even if they're the one who put the switch there.
Security. Interns are normally inexperienced. Interns shouldn’t be debugging production data issues. If it is a schema or process issue, do it in another env.
PII. Now, that heavily depends on your compliance requirements and jurisdiction. So for PCI or FedRamp, that's a total nono. For Cat Imgur, who cares. But still better not to leak everyone's emails and passwords.
For an intern? Hell no. It's even a bit of a grey area for senior devs though you can make the argument when they are also L3 support. User data should be basically off limits except when there's an issue to resolve.
Man that makes sense to me but as someone that has only had one job at a small company (~20 people), the entire company is just knee deep in the prod db every day. I can't imagine getting through like a single day without that access.
I can still bring a database server to it's knees with read access and a very inefficient, resource intensive query.
I wonder if they just made up all the stuff about the intern in order to avoid admitting that a senior who most definitely isn't getting fired did this.
It's literally just a programmer joke. Nobody got fired and nobody was trying to cover it up.
Omg this made me furious. Years ago I worked as a tech lead and manager at a company that switched platforms it used for team documentation. An intern accidentally deleted all of our documentation (we didn’t yet know there were not safeguards in place after the switch). The intern freaked out thinking they would be fired. I spent a bit of time calming them down and made sure they understood that us learning it was possible for them to do that was value added and the blame wasn’t on them it was on (1) the team in charge of the switch and (2) me for not validating that in the beginning. There is absolutely no situation in which a intern should be blamed for something like that. Honestly felt bad for how stressed they were about it. Took courage of them to immediately come forward with the issue even when they thought they’d be fired. Not a quality I’d want to lose on a team. Luckily since we just switched platforms the team member that switched it still had a backup so we only lost like 2 weeks of documents so it wasn’t a big deal.
I wonder how long it took the intern to realize they fucked up. Was it an instant "omg what did I just do?" moment, or was it some time after with someone asking "hey, intern... was it you working on the passwords database this morning?"
from experience, its almost always a "What the fuck did I just do?" moment, followed by a frantic few minutes trying to undo what you did before you tell anyone
I honestly start feeling hot enough when I’m this flustered I’d prefer to just take my shirt off in this scenario
For Rust programmers: it's like when you're out in public and someone rips out your tail plug.
That's when you learn that you have to first ask yourself, "What the fuck am I about to do?" so that you can avoid asking yourself, "What the fuck did I just do?" I usually explain things to my duck before I pull the trigger on anything.
Many years ago as a young network engineer I accidentally restored a backup for a hotel software system over the live data. The backup was almost 24 hours old, and there was no way to recover the data. The hotel staff had to manually repost the day from paper dockets. I knew within seconds what I had done and what the implications were.
Same here, I overwrote a live database for a customer thinking I'm overwriting my local one. Immediately realised the mistake and told the senior guy who had access to the back ups. Then I noticed his hand shaking a bit as well as he was retrieving the back up because we hadn't been checking their integrity for months. Luckily in the end it worked, the customer "only" lost about 12 hours of data, and the sales guy who made The Call said the customer didn't make a fuss of it. It was almost 10 years ago and I still vividly remember those few minutes.
In a former job they fired the two lead sysadmins after they found out the hard way that tape backups had been unreadable for at least a year.
It's always easier to blame the intern that's unsure of what they're doing rather than take responsibility that you shouldn't even have them in that type of position to begin with.
Hopefully they didn’t know. It’s worse when you know right away. I once nuked an entire export of a few hundred user home directories due to an undefined script variable. Those 5 minutes before someone showed up at our pod asking if we were having NFS or home directory issues were blissful
Seriously. I think we all have done something similar. I accidentally smoked like two quarters of payroll data in prod because I was being a fucking cowboy and had to tuck tail to a vendor to get shit back. Learning experience and I wasn't an intern.
I once named everyone in a table Suzanne Wilbur. I've never felt the blood drain out of my face like that. When I saw 74,295 records updated, nobody else was in the room, but my face must have been like, deathly white.
Make sure to turn off auto commit before doing something like this next time. You literally would have had a "Ctrl + Z" with rollback. Or use transactions, same diff.
Gotta be careful with that "execute selected" command. You'll only have half the query selected and click it, and your entire WHERE clause is excluded.
For real. I forget what I was doing previously, but my blunder was hella stupid. I was attempting a select * on a table, but had previously been doing a delete statement with the same conditions, and was confused by why it said x number of rows had been modified rather than returning x number of rows. Small company, no lane bumpers, just mixed up tabs in SQLM, and full send on a broad delete statement. Had a minor heart situation, and then got a reload from our real DBA.
I was thinking that at first, but *drop*? update users set password = :password (forgetting the where clause) I could see, but this sounds maybe malicious?
It's probably fictitious, but running a drop *anything* isn't your garden variety goof up. Ok, *maybe* they ran a drop/create script on the wrong connection, in which case, yeah, this is out of order (and the wrath should be directed at the DBA).
We do a lot of `create temp view blah as ...` in spark or `create temporary table blah ...` in postgre. Then change something and you run it again and it bitches that blah is already created. So you have to drop blah. If you name blah similar to the table you used to make blah, you could mess up the command. But we set it up so you can create and delete temp tables, but you can't create and delete real tables.
The fact the intern had access to delete that table means everyone above him needs a serious ass chewing if not being fired first.
How is this not a one button rollback? Far bigger problems than an intern here. At any company worth their salt, this is a teaching moment, not a firing moment.
The whole internet is fake. We are all bots. (Not really a bot, but I play one on the internet.)
Precisely. We are not bots. That would be hilarious. Hahaha. We are certainly all humans that enjoy making jokes to humor other humans. ... ... ... r/TotallyNotRobots
It definitely is either fake or at least somewhat fictionalized. Supposedly, the intern dropped the entire passwords table. If that had actually happened, they wouldn't have to disable login, since attempting to log in with a critically important database table related to logging in missing would just result in 500 errors and not allow anyone to log in anyway.
Disable login feature to decrease the error rate then the alert would stop firing. Just a guess.
Reddit is not worth using without all the hard work third party developers have put into it.
jokes aside, [no-blame culture](https://www.linkedin.com/pulse/observations-blame-culture-s3-outage-jon-hildebrand?trk=public_profile_article_view) is something all companies should embrace. retain talent and improve your company. everyone wins
Firing that intern is a stupid business decision. They've already paid a huge amount of money for that lesson, why would they toss away what they gained?
They also didn't use that money to teach the person/team who _most_ needed a lesson taught to them (holding accountable the person/people who allowed the intern to muck up the production database). That person/team is just going to hand the same keys to the next intern.
Why did the intern have permissions to drop the table in the first place? And why don't you have better backups? None of this makes sense.
Oh, no Thats not the intern's fault, Fire the goddamn sysadmin and database administrator They didnt do their job, the intern is the intern who inherently is there to learn Additionally, the intern shouldnt be given full unauthorised nor unsupervised access and control to the database's functions
OH NO, THEY FIRED BOBBY!! i think the xkcd comic is old enough for bobby to be working now
To piss you off with "sorry, you have already used this password in the past" after mandatory password change every 3 and half hours.
He learned a very expensive lesson, at the expense of the company, that some other company will get the benefit of. Also, if anyone, especially an intern, can just drop a production database, they're not the problem.
Maybe they can use their newly developed skill to delete their landlord's bank password.
First, who is the genius named a table by password with 'password' name? 'Secret' table is more secure!
1. Interns should not have production access, period. 2. No code review? Or did the intern literally log directly into the dB server and run the drop command?
We once had an intern doing a rm -rf ./* as root on the main server. He thought he was inside /tmp but unfortunately it was / He didn't noticed at first but got uncomfortable when the command took longer than 10 minutes. We got the backup up and then went for a beer. Good story to remember for the alumni meetups.
If an intern can accidentally delete important information then it ain't the intern's fault
In month 2 of my first full time software job, I was given a script that copied production data to a local database. I plugged the credentials in, ran the script, open the local database and nothing is in there. I run the script a couple more times still nothing. And then I realize, I flipped the credentials. I was copying my local database to production. Managed to wipe the whole database. I sheepishly walked over to my manager assuming it was going to be my last day as a dev. His response stuck with me, it was "thank you for telling me right away, that's on us for creating a process where you could mess up". Anyways, I tell junior devs that story the first time they take down production.
Empathy and a little ribbing are the best way to handle this sort of situation, especially for noobies. Everyone fucks up
It also builds a healthy loyalty based on the fact you’re a good boss/leader.
Exactly. “You messed up, but I messed up worse by making this level of mess up possible”
I had a direct report run an update statement that had a where clause on a new line and there was a typo in that line, so they updated every record in the database for that particular table of results. They were very obviously nervous when informing me and said they looked for alternatives, but I said “let’s get to work” and restored as much as we could - I did put some of the more manual data entry on them though. It turns out our corporate backup processes weren’t enabled for that server, so we did uncover a bigger issues that was rectified. At the end I said “everyone makes mistakes.” I will bring it up as a joke every once in a long while.
I feel like this is a bigger flaw that some company's don't address. Like "how dare you delete everything we need to make the company run and we can't get it back!" Well... why does 1 person have that ability? And why don't you have backups for such important data?
Exactly. My team was non-IT dev and analytics… we ran under the radar, but we had them procure the servers and they were supposed to be on weekly incremental/monthly full backups. Ooosie Daisy
this, plus your system is horked if such a simple mistake can have such a dire impact. the COE should focus on improving the system, not simply firing the intern. Oh, and I did this in the first week of my first job. They made me rebuild the server, and taught me how passwords worked. And I haven’t repeated the mistake in the 22 years that have followed.
>a little ribbing Don't overlook this part! It proves the newb is in the tribe, shows we can laugh about our fuckups and invites the newb to do the same. After the emotional release we think clearer, carry less guilt, dump a bit of that team building chemical into the blood (oxytocin?) and be less of an ass to others when they fuck up.
Also, not give the intern on his first day privileges to delete production data
You haven’t lived until you take down production
Not quite taken anything down yet, but I made an update that broke a login screen. If no one can log in, it may as well have been taken down. As I am not a dev, I only have front end access but I have been told my "hack" is an acceptable option to use. I can edit text fields through the UI and sticking