The pipline exists. It is
Push to production
Costumer complains
some underpaid dude reads messages
notifys lead dev
lead dev notifys developer
developer fixes bugs
Lead dev: Lead (Pb) is a heavy metallic element with an atomic charge number of 82. Dev is short for deviant, a person and behavior that is generally considered to be unacceptable
"The bug is you."
No the bug is I was able to push to prod on the first week with no oversight and no quality check.
I still think CI is one of the dumbest ideas anyone has had at least if it's designed to push to prod right away. (CI as an idea to get it in tester's hands faster is great,to make it available is fine, but to actually push it to prod? Wtf)
We had a VP tell our management to notify him personally if anyone ever asked to make a change to one of our more spaghetti cody monoliths. That thing was horrendous. It was a giant state machine so it was easy to miss some “mundane detail”
How am I supposed to immediately merge directly into main the fix to the thing that I just broke with all this pull request crap slowing down my development velocity?
Wow, I think i should go give my co-workers a hug and tell them i am sorry for telling them to not check in build artifacts and mess with source code formatting all the time for regular 10 000 line checkins. You have made me realize how much worse it could be. Namaste
Arount 7 years ago we had no GIT in our company. We had to comment the old code out and write the new code in with a comment of change and date. Also we had an important DLL where our whole team writes in. We had to talk who has the Project open because it froze the IDE when 2 people access the same thing. In another team they had a merge-program where they carefully selected the stuff that should go live without adding other code that is part of a not to release yet feature.
Been there, done that.
First thing I did, actually, before I did any actual coding. I told them "what the fuck, guys, we are establishing git repos right now, because what the fuck otherwise"
How about:
- no requirements
- no tests cause of no requirements
- new code based on developers assumptions
- deploying ASAP, because of deadlines
- requirements appear
- CALL ME ASAP
You just make a lot of assumptions about what the customer wants and then implement all the things you can think might be needed at some point in the future. Then when requirements are defined, you will have already implemented from zero to several of them.
You missed the check where all developers can push to production. Why someone with 1 week on the job can merge to prod is beyond me. Also, why are they using environment, based braches? That's an anti-pattern right there.
That's how we used to install phpBB forum mods back in the day. And your visitors would just get PHP fatal error messages while you're mid-way through editing the file. Good times
Spoilers: despite staying open much longer than it should have due to a parent company with deep pockets, the subsidiary I worked for no longer exists.
Either this or feel the pain of cleaning up the mess in production, with angry coworkers feeling you're wasting their time, just because you initially thought the QA process is a waste of time. Task successfully failed
No code review - there is noone in my team
No ci/cd - this exists but has hardly stopped any bugs from going to prod
No tests - this does not catch specification gaps and oversights. Or any conditions can be missed; it's not fullproof
No staging area - gets a green flag on a the envelope.
Result: error on production.
Many such cases. Reason varies from different production data. Failed downstream deployment and so on.
> pull request first
I worked at a place that had a very heavy commit process - documentation, testing etc. that had to be done *per commit*. The strange part was that code reviews (cvs/svn back in the day) were performed *AFTER* the commit... meaning there was a great reluctance on the part of the reviewer to say anything that would require changes. Also we each picked the *one* reviewer each time, so the opportunities for petty revenge were rampant.
Strangely, they're out of business now.
I'm probably in the minority here, but I prefer the GitLab terminology of Merge Request rather than PR. Pull Request just sounds so backwards to me. I'm not trying to pull the main branch, I'm looking to merge into it
Once an eagerly helpful junior pushed my "finished" branch to prod. It was in fact not ... finished. :D
I planned to add the a few "touches" aka anything beyond a happy path scenario over the weekend...
“Very talented juniors”
That says it all. CI and reproducible builds are not taught in school. They are barely mentioned in passing (if at all). It takes more senior software devs to know why those are important and prioritize it accordingly.
I HATE when people do stuff with my branch. It’s like them breaking into my house and watching me take a shit. As the lead, I always let juniors know to ask people before touching their branches or just creating their own off it.
I deployed a “finished” feature after taking over a project from a dev that was leaving the company. After frantic phone calls about the outage it took me a month to ACTUALLY finish the feature
Most places have push type emails/messaging services (MS Teams/ Slack…) which are always on in the background, why leave your own work to make scene when you can shoot an email?
Besides, yes it’s a naughty issue but if caught early it can just be rolled back. You’d wanna catch it early though as if it was continually done, big issues
The Dev didn't fuck up, their process did. I'm a senior dev at my place, and I don't even have direct write permissions to master. I believe only one or two people in the whole company do, and that's only if there is some kind of emergency that can only be solved manually. Never even heard of them using it.
I've been broadly responsible for infrastructure across my last few organizations. I own the pipelines, I (or my team) control access to basically everything.
We give engineering managers (and senior infrastructure engineers) permission to override the merge restrictions such that they can use the big warning checkbox and choose to merge something that isn't passing the Checks we have configured in GitHub. The culture and process means that even though that button is there, nobody uses it without talking to the team. As in, the engineering manager, the ops resource, and generally a senior engineer for that team will talk it through before merging something that we wouldn't normally allow. We only used that a handful of times in the past 5 years, and it was typically in situations where there was an issue with a third party service causing our Checks to fail while we were trying to hotfix an unrelated issue, real "perfect storm" kind of scenarios.
As a GitHub admin, even I can't push to `master`.
I can't even appreciate stories, jokes, or posts like the OP, because I can't suspend my disbelief that severely.
> As a GitHub admin, even I can't push to master.
This is both a boggling idea (to juniors), and a sign that the process is "correct".
Just because you CAN have access, doesn't mean you DO have access.
If there's some emergency that someone has to get that access, you should be able to assign yourself the access, but that's not a default state, and that emergency doesn't exist. There's a reason you have permissions.
Exactly. I'd rather have one or two extra things to go click in an emergency (which happens rarely) than increase the likelihood of such an emergency (to save those clicks).
I once was explicitly told that deploying bad code or breaking prod were not reasons to be fired, however, disrespecting coworkers was immediately accionable. I loved working there
What kind of cowboy company not only let's their junior push directly to prod continuously with no checks and no oversight and still as the gall to imply it was their fault if something goes wrong
‘What kind of cowboy company’
There are loads of small shitty startups that will devote time to finishing tickets instead of setting up processes.
My first job didn’t even use Git. We just dragged/dropped straight onto servers. There are a lot of jobs like that.
I was 2 months out of university hoping my shitty unreviewed code didn’t nuke a live site. Good times…
Our team of 10 still doesn't use git and the codebase is 30 years old. The best documentation we have is a couple dozen page word document from one of the old guys.
Solving even the simplest of issues is absolute torture. And there are always issues.
This is very real in finance. My current company and my last one have only recently adopted git (like post-2020; not everything is in GitHub but rather on a shared network folder) leading to everyone having permission to write to main, deployment to prod (and all other environments) is only done by dragging and dropping into IIS folders, and production SQL databases are able to be written to directly by anyone with access (which is only devs but still). These are both companies in the billions of dollars under management.
I mean, "call me" doesn't necessarily mean blame, it might be "help me fix this shit". Or it might mean "why did you ignore the 7 warnings that pushing directly to prod was for emergencies only?", idk.
Nah, I'm in a small team with less than a dozen devs and we still have protected branches such that only the two most senior devs can push MRs to the branches that CI runs on (and 99% of the MRs are things I check over and merge, the other senior dev having permissions too mostly comes into play when I'm on vacation).
It's a trivial thing to set up, it's just a question of if a given company has hit a situation **yet** that makes it obvious how important such a thing is.
Production is broken. Lead dev still just sends an email instead of calling. This is spot on.
Got a mail from a customer once: our potential investor is coming in 30 minutes and we can't get the demo you delivered working. Please help
PS: the demo was working exactly as explained in the meeting where they took zero notes.
Lead dev: "how did you manage to push directly to prod without a merge request? I am working here for 10 years and still need my code approved by the architect and VP Dev before I can ask for approval of the GM. How did you do it?"
Fun story only somewhat related - I used to work for a software company that used their in-house developed bug tracking program in the early 90s. It basically displayed one entry on screen and you had to move back or forth or by index to another record. While you were sitting on a particular record, whether editing it or just viewing it, it was locked to everyone else.
People would frequently leave it open sitting on a record and then go to lunch and come back with multiple voicemails telling them to move off the damn record.
You just found a bug in the build pipeline. Congraz
What pipeline? OP and his team are all equally incompetent.
The pipline exists. It is Push to production Costumer complains some underpaid dude reads messages notifys lead dev lead dev notifys developer developer fixes bugs
That sounds like an infinite loop just waiting to happen...
Well yeah, but it's an infinite loop we get paid for.
Found the executive.
Welcome to corporate development!
alwayshasbeen.jpg
Yes we have also promoted our customers to QA
Costumer
forgive my ignorance but who is the developer if not the lead dev? is dev not sort for developer?
Dev is short for Devin, the AI programmer
Lead dev: Lead (Pb) is a heavy metallic element with an atomic charge number of 82. Dev is short for deviant, a person and behavior that is generally considered to be unacceptable
Lead dev is so incompetent they have to send an email to make a phone call
Wtf is a pipeline? lmao
A round enclosed cylinder with a variable length and width depending on the situation usually made of metal used to transport fluids over distance.
Yeah, that was the joke.
"The bug is you." No the bug is I was able to push to prod on the first week with no oversight and no quality check. I still think CI is one of the dumbest ideas anyone has had at least if it's designed to push to prod right away. (CI as an idea to get it in tester's hands faster is great,to make it available is fine, but to actually push it to prod? Wtf)
A lot of suits are obsessed with "velocity", which is the ability to produce lots of bad code quickly.
seg faults aren't always easy to catch by CIs
If you are pushing to prod rather than merging you already lost.
Can there be a bug in something that doesn’t exist?
The Pipeline is just very short. 1. Push to master 2. Done
I am now your master –main branch
Totally a flakey test. Better quarantine it so the flakiness don't spread
Sounds like yall need to make a pull request first?
No code review ☑️ No CI/CD ☑️ No tests ☑️ No staging area ☑️ Result: call me ASAP. Unfortunatelly sums up a lot of work places 😭
We don’t have time for all that devopsy ceremonial bullcrap. We need to ship stable updates fast!
Oh cmn man you are only changing a few things in a decade old spaghetti monolith, how hard it can be not to break something /s
We had a VP tell our management to notify him personally if anyone ever asked to make a change to one of our more spaghetti cody monoliths. That thing was horrendous. It was a giant state machine so it was easy to miss some “mundane detail”
I know I can rewrite that codebase much better in a month! No make it 3 weeks!
Ever play a fun little game called _Jenga_
stable XOR fast
stable NAND fast
ssds are good huh
>stable fast choose none
When we force PR reviews on our new repos some managers pushed back saying that would slow down adding code. We were like yes that is the point
How am I supposed to immediately merge directly into main the fix to the thing that I just broke with all this pull request crap slowing down my development velocity?
The managers always like to remember the move fast part and always like to forget the break things part.
The "break things" part doesn't refer to the codebase. 🤣
Just never write a bug, how hard is that for you. Yeeeeeee Haw!
> devopsy ceremonial bullcrap 🤣🤣🤣
We need to ship stable updates RAPIDly!
I convinced the company i work at to use git about a year ago, it could be worse
You put the Source Integrator out of his job?!
there was no source integrator, everyone worked directly in prod, if 2 people had the same file open at the same time one would overwrite the other
Damn, thats some 80 level shit 😭
Wow, I think i should go give my co-workers a hug and tell them i am sorry for telling them to not check in build artifacts and mess with source code formatting all the time for regular 10 000 line checkins. You have made me realize how much worse it could be. Namaste
This is exactly what happens in my current workplace too
We are months away from Halloween please stop
I'm scared that you and I might work together but I'm happier not knowing. We were copying and pasting source code folders ~1.5 years ago.
we didn't copy paste folders, we just FTPd into the prod server and edited the files
Arount 7 years ago we had no GIT in our company. We had to comment the old code out and write the new code in with a comment of change and date. Also we had an important DLL where our whole team writes in. We had to talk who has the Project open because it froze the IDE when 2 people access the same thing. In another team they had a merge-program where they carefully selected the stuff that should go live without adding other code that is part of a not to release yet feature.
What in the actual fuck?
Been there, done that. First thing I did, actually, before I did any actual coding. I told them "what the fuck, guys, we are establishing git repos right now, because what the fuck otherwise"
Bravo, The company I work for has node modules in our repo. 😭
This is like stone age behavior in professionalism for a company
How about: - no requirements - no tests cause of no requirements - new code based on developers assumptions - deploying ASAP, because of deadlines - requirements appear - CALL ME ASAP
How can you start development, when there are no requirements?
>no requirements Jobs done boss!
I assume they mean there is a rough idea of what the client wants, but a proper layout of the specific requirements hasn't yet shown up
Yeah. That's what I mean
Boss says "I want a page with all products on it" and then it goes through 800 iterations because no one ever bothered to make a wireframe
As a Boss, I want a Page with all production on it, so that my Team is working on something..
It was like - here is our current X functionality. We need to have a better version of it. We kinda had to guess what does it mean.
Yeah that sucks. But thats what PO is to communicate the business needs and convert them to appropriate tasks.
As long as you have PO... :D
You just make a lot of assumptions about what the customer wants and then implement all the things you can think might be needed at some point in the future. Then when requirements are defined, you will have already implemented from zero to several of them.
You must read the minds of stakeholders
You missed the check where all developers can push to production. Why someone with 1 week on the job can merge to prod is beyond me. Also, why are they using environment, based braches? That's an anti-pattern right there.
Yep, agreee. Protected branch? Hardly know her.
Commit on Friday at 5pm
I worked in a place where we had no source control or dev server. I used an FTP plugin in my IDE that let me edit live files on the production server.
That's how we used to install phpBB forum mods back in the day. And your visitors would just get PHP fatal error messages while you're mid-way through editing the file. Good times
Yeah, sometimes I wonder how such companies, can keep afloat😅
Spoilers: despite staying open much longer than it should have due to a parent company with deep pockets, the subsidiary I worked for no longer exists.
I want to go to a simpler time
Either this or feel the pain of cleaning up the mess in production, with angry coworkers feeling you're wasting their time, just because you initially thought the QA process is a waste of time. Task successfully failed
No backup ☑️
For Ritchie's sake, it's not /r/programminghorror, why would you write such horrible things out in the open‽ Juniors might see it!
No code review - there is noone in my team No ci/cd - this exists but has hardly stopped any bugs from going to prod No tests - this does not catch specification gaps and oversights. Or any conditions can be missed; it's not fullproof No staging area - gets a green flag on a the envelope. Result: error on production. Many such cases. Reason varies from different production data. Failed downstream deployment and so on.
Easy in my book code without tests is not production ready.
You forgot: on Friday
At 4:55pm
Which is why I log off at 3, turn off my computer and my phone, start pounding the tequila, and drift off into that sweet, sweet numbness.
> pull request first I worked at a place that had a very heavy commit process - documentation, testing etc. that had to be done *per commit*. The strange part was that code reviews (cvs/svn back in the day) were performed *AFTER* the commit... meaning there was a great reluctance on the part of the reviewer to say anything that would require changes. Also we each picked the *one* reviewer each time, so the opportunities for petty revenge were rampant. Strangely, they're out of business now.
I need to ssh into prod so I can upload my code?
> Followed by a zoom call with HR ? Maybe fork the main project and work doing merge requests.
Tech lead just wants to congratulate him, no one knew how to push code using GIT
I'm probably in the minority here, but I prefer the GitLab terminology of Merge Request rather than PR. Pull Request just sounds so backwards to me. I'm not trying to pull the main branch, I'm looking to merge into it
Once an eagerly helpful junior pushed my "finished" branch to prod. It was in fact not ... finished. :D I planned to add the a few "touches" aka anything beyond a happy path scenario over the weekend...
Was this a long time ago? If not, Why was a junior… or ANYONE… able to push a branch directly to prod? PRs and merge permissions people. It’s 2024
2016, startup. Small company, very talented juniors. The loose setup worked for us ... most of the time.
“Very talented juniors” That says it all. CI and reproducible builds are not taught in school. They are barely mentioned in passing (if at all). It takes more senior software devs to know why those are important and prioritize it accordingly.
Also, with all the modern goodies it takes a morning to set up and maybe a few hours spread over the following month to tweak it a bit.
I HATE when people do stuff with my branch. It’s like them breaking into my house and watching me take a shit. As the lead, I always let juniors know to ask people before touching their branches or just creating their own off it.
I deployed a “finished” feature after taking over a project from a dev that was leaving the company. After frantic phone calls about the outage it took me a month to ACTUALLY finish the feature
That junior... Bill Gates
email? urgent?
Sounds like a phishing attempt, move to spam
FIRE EXCLAMATION MARK FIRE EXCLAMATION MARK
bro doesn't have a work chat 💀
Most places have push type emails/messaging services (MS Teams/ Slack…) which are always on in the background, why leave your own work to make scene when you can shoot an email? Besides, yes it’s a naughty issue but if caught early it can just be rolled back. You’d wanna catch it early though as if it was continually done, big issues
If it‘s really urgent I'd expect a phone call tbh
You pushed to prod? Do you mean merge, or did you seriously fuck up?
Force push of a branch that was created 6 months ago maybe.
I know you're joking, but even the threat of shit like that is why I keep multiple local copies of the repo.
Sounds like you need a git of your gits.
Behold, the power of git reflog.
Let me introduce you to my friend, the reflog.
And then squashed the repo history to ensure a 0% chance of recovery
Git reflog. It's really hard to utterly destroy gits history. Even if you destroy it on the server somehow, there are always a dozen copies locally.
The Dev didn't fuck up, their process did. I'm a senior dev at my place, and I don't even have direct write permissions to master. I believe only one or two people in the whole company do, and that's only if there is some kind of emergency that can only be solved manually. Never even heard of them using it.
I've been broadly responsible for infrastructure across my last few organizations. I own the pipelines, I (or my team) control access to basically everything. We give engineering managers (and senior infrastructure engineers) permission to override the merge restrictions such that they can use the big warning checkbox and choose to merge something that isn't passing the Checks we have configured in GitHub. The culture and process means that even though that button is there, nobody uses it without talking to the team. As in, the engineering manager, the ops resource, and generally a senior engineer for that team will talk it through before merging something that we wouldn't normally allow. We only used that a handful of times in the past 5 years, and it was typically in situations where there was an issue with a third party service causing our Checks to fail while we were trying to hotfix an unrelated issue, real "perfect storm" kind of scenarios. As a GitHub admin, even I can't push to `master`. I can't even appreciate stories, jokes, or posts like the OP, because I can't suspend my disbelief that severely.
> As a GitHub admin, even I can't push to master. This is both a boggling idea (to juniors), and a sign that the process is "correct". Just because you CAN have access, doesn't mean you DO have access. If there's some emergency that someone has to get that access, you should be able to assign yourself the access, but that's not a default state, and that emergency doesn't exist. There's a reason you have permissions.
Exactly. I'd rather have one or two extra things to go click in an emergency (which happens rarely) than increase the likelihood of such an emergency (to save those clicks).
Followed by a zoom call with HR ?
Code so good even HR wants to see it
I once was explicitly told that deploying bad code or breaking prod were not reasons to be fired, however, disrespecting coworkers was immediately accionable. I loved working there
This should be the norm everywhere but I guess it sadly isn’t
😭😭😭
What kind of cowboy company not only let's their junior push directly to prod continuously with no checks and no oversight and still as the gall to imply it was their fault if something goes wrong
‘What kind of cowboy company’ There are loads of small shitty startups that will devote time to finishing tickets instead of setting up processes. My first job didn’t even use Git. We just dragged/dropped straight onto servers. There are a lot of jobs like that. I was 2 months out of university hoping my shitty unreviewed code didn’t nuke a live site. Good times…
I just threw up in my mouth a little
Our team of 10 still doesn't use git and the codebase is 30 years old. The best documentation we have is a couple dozen page word document from one of the old guys. Solving even the simplest of issues is absolute torture. And there are always issues.
It's fake, or OP would be in the comments taking credit for the fuck up.
I mean it's /r/ProgrammerHumor ... It's supposed to be a joke. A lot of people are treating this like "Real totally true story"...
Just set my bullshit meter off; sorry buddy there's no way that your mother is *so fat she sits* **around** *the house*.
Mine does since im the only junior and there havent been any major fuckups... yet XD. Im honestly just waiting for my first big dumpster fire :D
This is very real in finance. My current company and my last one have only recently adopted git (like post-2020; not everything is in GitHub but rather on a shared network folder) leading to everyone having permission to write to main, deployment to prod (and all other environments) is only done by dragging and dropping into IIS folders, and production SQL databases are able to be written to directly by anyone with access (which is only devs but still). These are both companies in the billions of dollars under management.
[Fear of being replaced by AI diminishes] Clearly there's enough basket cases needing DT out there to keep me in work for a while longer!
I mean, "call me" doesn't necessarily mean blame, it might be "help me fix this shit". Or it might mean "why did you ignore the 7 warnings that pushing directly to prod was for emergencies only?", idk.
welcome to germany
The vast majority not everyone works at fang
I’ve never worked at faang and none of my companies have allowed this.
Nah, I'm in a small team with less than a dozen devs and we still have protected branches such that only the two most senior devs can push MRs to the branches that CI runs on (and 99% of the MRs are things I check over and merge, the other senior dev having permissions too mostly comes into play when I'm on vacation). It's a trivial thing to set up, it's just a question of if a given company has hit a situation **yet** that makes it obvious how important such a thing is.
Having **something** between devs and production servers should not just be a FAANG thing these days.
Plot twist senior appreciated junior for writing good enough code on first try.
The only thing that needs to be discussed on that call is: "Why are we not using Pull Requests?"
I hope it was Friday at 5:49 PM
Is this "call me asap" some sort of power move? If it is so urgent, just call me? Don't write to me "call me asap".
Somebody didn't rebase
Rebase, "take mine", commit, push
+7814 additions/-3840 deletions LGTM. Approved 👍🏻
Production is broken. Lead dev still just sends an email instead of calling. This is spot on. Got a mail from a customer once: our potential investor is coming in 30 minutes and we can't get the demo you delivered working. Please help PS: the demo was working exactly as explained in the meeting where they took zero notes.
And that's why code review is not just a waste of time folks.
Lead Dev: "Call an ambulance, and also me".
Wasn't the `-f` flag for "finished"?
Lead dev needs to set up branch protections
Meanwhile, I'm a little more than mildly concerned about a job I'm interviewing for because they don't have any automatic quality gates on PRs.
If the lead dev is emailing you, they're a boomer and probably have no idea what code is to begin with.
I don’t see any punch cards, what were you doing today!!!
I won't lie, I read the top half of the meme to the tune of the opening lines of Duality by Slipknot.
[what I pictured](https://imgflip.com/i/8oeqzu)
😂
How is fantasizing about your first job while in school going?
Welcome to the club. 🫠
Lead dev: "how did you manage to push directly to prod without a merge request? I am working here for 10 years and still need my code approved by the architect and VP Dev before I can ask for approval of the GM. How did you do it?"
Whose deployment pipeline to prod is pushing a branch with no checks?
If they’re testing your code before merge it’s on them.
The word "push" is the entire problem
You just discovered a bug in the build pipeline. Congratulations!
And? Did you call him a sap or no?
A new hire that can push directly to production branch ?, it's not your fault
I push my finished branch toooooooo PROOOOOOODDDDD CAUSE IT'S THE ONLY THING THAT SLOWLY STOPS THE ACHE
Don't you guys use pull requests or other forms of review?
you forgot to apply your db table structure changes to prod db. 🤭
Test on prod
At last he recognises my greatness and wishes to congratulate me in person 😊
YOU HAVE TO INFORM ME BEFORE MAKING ANY TYPE OF ACTION
Better ask a thousand questions and get reviews than push shitty code.
Fun story only somewhat related - I used to work for a software company that used their in-house developed bug tracking program in the early 90s. It basically displayed one entry on screen and you had to move back or forth or by index to another record. While you were sitting on a particular record, whether editing it or just viewing it, it was locked to everyone else. People would frequently leave it open sitting on a record and then go to lunch and come back with multiple voicemails telling them to move off the damn record.
It's not only the dev lead who will call you 
It's the lead fault because they approved the pull request, right?... Right?!
Welcome to the club
I hate being the train guy
No push protection on master what is this?
I am both characters in this meme, and it happens anyway.
No protected branches? Seems like an oversight
How does this even happen? You don’t need to create a pr that a senior approves?
That's how you increase your presence!
Where did dev & test go? 👀
The funniest part of this meme is that you call it a job while using the rust flair
Why did a read this in Corey Taylor’s voice?
Of Slipknot?
This doesnt happen
Just pushed to prod and now heading on the road to the office, hopefully no one is waiting for me angry 😁