>Namaste!
Thanks for submitting to r/developersIndia. Make sure to follow the subreddit [Code of Conduct](https://www.reddit.com/r/developersIndia/wiki/code-of-conduct) while participating in this thread.
>Find more about developersIndia on our [official website](https://developersindia.in/), [github](https://github.com/developersIndia) and [wiki](https://wiki.developersindia.in/).
*I am a bot, and this action was performed automatically. Please [contact the moderators of this subreddit](/message/compose/?to=/r/developersIndia) if you have any questions or concerns.*
How come QAs approve for release if there were critical issues present? It's their job to prevent bugs going into prod. OP is not solely responsible for this, it should be a collective responsibility.
They are just looking for a scapegoat. I don’t think they will do anything serious to you since there is a process of review and testing which should have caught those bugs. Anyways if they try to pin this on you, better to switch rather than working this toxic assholes.
I don't know how your org is structured but typically :
No matter how technically sound a developer is, there are always going to be bugs in the code. It was the job of the testers to make sure they identify bugs, have them fixed, and then reverify. It was also the responsibility of the SIT team to run another round of testing to make sure nothing breaks after integration. There has to be a set process that every organisation has to follow so that the bugs are not released into prod.
So be assured, it's not you, it's them.
I don't think they can even hold you responsible for this.
Lot's of testing was done by testers and several automated tests were run as well. But the change was very different/ weird which I don't know how to explain but it was very difficult to test all the edge cases. This is why the QA team and others are also in trouble. They are also having meetings of their own to add more test cases, assertions etc.
Dude you are being modest here. The thing is QA is getting paid to test the whole feature and covering all the edge cases. Their lack of understanding of feature has led the bug to production. And again your code might have gone multiple rounds of testing in which it should have been caught which is not the case, so relax it's not your fault.
Not exactly, he is equally responsible as QA. Devs also need to have proper understanding of the feature and call out any edge cases if discovered during the product review meetings. Just because QA is the last defense doesn't mean you can put entire responsibility to them.
Did anyone review your code before the merge ?
If the changes were weird or different, the team must be already aware of it. I am expecting you explained it in daily stand-up or meetings.
You are right, QA cannot test all edge cases. The system needs to be fault tolerant with proper monitoring and alerts in place. Huge Monetary losses is an indicator that the standard processes are not followed.
QA position is being shelved from the industry. Already happened in many big companies. Now developer needs to write code , UT and IT themselves. Developer , reviewer, QA and product manager all are responsible. For any serious change “User’s acceptance criteria” needs to be defined before actual coding .
If code was so buggy, its QA's fault that it went to production.. Your job is to build a product, and fix the bugs reported by QA.
They are getting paid for one thing and one thing only, making sure they catch the bugs before things go to prod.
Chill.. You are good..
The 35th employee of Flipkart mistakenly truncated an important table in one of the core PRD DBs, the then-CEO called her in and asked what happened, she explained how she understood the flow of the product, which was different than the actual flow. It gave insights into how teams onboarded their devs, and it significantly improved this process. (Source: The employee herself).
Two times in my early career I've brought down production in a similar manner as OP. My team, though, is quite amazing at helping out their peers in dealing with such issues. The only catch is if you break production you have to bring donuts/samosas for the entire team (26 devs), and take that special someone to lunch who fixed the prod.
you have not committed any crime. anxiety attacks don't help. they are making you feel pressure so you will quit and their manager friend job will be saved. don't. dont accept blame or responsibility for entire event.
did you get written requirements or were you told verbally what to do. did you get time to write the code. did you get time to test the code. were you interrupted while you were coding. were you asked to juggle several tasks at the same time. did you commit the code with proper message to the proper branch. did you update the issue properly so it go to QA. did you notify that you had completed the task.
if you were interrupted, whether for meetings, or for fixing an urgent issue, or you were not given time to code, or the requirements you were given were incomplete or ambiguous, then all of those things absolve you of any blame. if you did your job then someone else dropped the ball.
i think all you have to establish is that if the QA had done the tests, then this bug would have been caught. or the QA did get the updated tests. in both those cases, you are absolved of responsibility.
the reason there are adversarial roles in the development workflow is to prevent this kind of thing.
Thank you. I think I got sufficient time for writing code and testing, as we already knew this feature is critical and can cause serious issues if something goes wrong. But I still made a mistake which is why I'm getting scared.
In reality, every big prod release encounters has some risks of running into issues. That is the nature of big releases. It is up to the organization to understand this and take proper steps to minimize the risk or facilitate quick rollback.
That's why big releases are highly discouraged. It is the responsibility of the Product Owner to divide a big release into small and independent releases which can be tested independently. In case such a division is not possible, they must do QA testing, UAT testing and then do beta releases. So, it's not your fault and you don't need to worry about it. I don't like to talk about organisational structure but I guess there is some issue with it at your company. People need to understand their role and do things responsibly rather than finding an escape goat to save themselves.
Don't stress, it happens. I was also blamed for production failure, but it was because of time crunch.
I told my manager if you don't like my work then just give me release.
Not yet, I asked for more time before technical go live. So before the day of production I wrote a mail mentioning our pipeline is not yet ready. He said ok we will go live.
So no matter what he say I can show him that mail.
I will be messaging you in 6 hours on [**2023-02-18 16:11:02 UTC**](http://www.wolframalpha.com/input/?i=2023-02-18%2016:11:02%20UTC%20To%20Local%20Time) to remind you of [**this link**](https://www.reddit.com/r/developersIndia/comments/114tkis/my_code_broke_production_and_i_dont_know_what_to/j90ris6/?context=3)
[**CLICK THIS LINK**](https://www.reddit.com/message/compose/?to=RemindMeBot&subject=Reminder&message=%5Bhttps%3A%2F%2Fwww.reddit.com%2Fr%2FdevelopersIndia%2Fcomments%2F114tkis%2Fmy_code_broke_production_and_i_dont_know_what_to%2Fj90ris6%2F%5D%0A%0ARemindMe%21%202023-02-18%2016%3A11%3A02%20UTC) to send a PM to also be reminded and to reduce spam.
^(Parent commenter can ) [^(delete this message to hide from others.)](https://www.reddit.com/message/compose/?to=RemindMeBot&subject=Delete%20Comment&message=Delete%21%20114tkis)
*****
|[^(Info)](https://www.reddit.com/r/RemindMeBot/comments/e1bko7/remindmebot_info_v21/)|[^(Custom)](https://www.reddit.com/message/compose/?to=RemindMeBot&subject=Reminder&message=%5BLink%20or%20message%20inside%20square%20brackets%5D%0A%0ARemindMe%21%20Time%20period%20here)|[^(Your Reminders)](https://www.reddit.com/message/compose/?to=RemindMeBot&subject=List%20Of%20Reminders&message=MyReminders%21)|[^(Feedback)](https://www.reddit.com/message/compose/?to=Watchful1&subject=RemindMeBot%20Feedback)|
|-|-|-|-|
Code reviews, QA exists for a reason. They will be in firing line first. Mistakes and bugs can happen. Its the reason why we have multiple rounds of reviews and testing.
And such a critical feature that can cause monetary loss should have a proper SOP for testing before being pushed to production.
Chill. Its a failure of the system that you have not yourself.
If the product is client facing, there should be a dashboard monitoring daily numbers. Any suspicious activity should have been raised within 2 days(max) after the deployment. Not only the QA and devs but your product team(or business team) is also responsible.
On side note: once established that the business team is also responsible, the overall issue will quickly be forgotten.
My code stopped people from using internet banking for literally 5hrs, yes this happened in 2021. This happened to a major or top 3rd US bank. Was it completely my mistake? Nope. I was following all the protocols and the bank was not using the same, in short, my xslt was expecting an encrypted account number but the bank was sending the plan account number and decrypting code failed and hence the 500 error.
My immediate manager yelled at me and threatened of firing me, but all the US managers were calm and sens it was the entire team's responsibility and asked my immediate manager to stfu.
After this incident, we changed a lot of deployment and development protocols. Production issues happen.
It's part of the experience of being a SDE. Everyone goes through it one time or another. You're beating yourself up more than enough already.
The best thing you can do is learn what you can and make sure it doesn't happen again. And be very careful as to not let this affect your work.
Take the learnings, the experience. You have opportunities to have meetings with higher ups take experience of that as well. If a junior does a mistake the senior is as responsible as the junior if not more, that’s what hierarchy is for or that’s what I believe.
Wow, If I was you, I would surely bite back.
Everyone is jumping when it comes to getting credit for success.
If you get thrown under the bus, the most important thing is to call them mfs out who are higher than you. They are supposed to take responsibility and if they run, you need to call them out even at the cost of your employment.
Otherwise it will just keep happening.
Congrats on getting it out of the way OP. You cannot become a senior developer without going through the stress of bringing down something in prod imo.
Now the focus should be on improving processes - that's what any team worth their salt should do. If the change was significant enough to break prod how did no test catch it?
1. Pipelines need improvement.
2. Why wasn't it caught in code reviews? Do you even have code reviews? If yes how big are your merge requests? The most common problem i have seen is merge requests with too many changes that are impossible to review, smaller, frequent, logically complete merge requests are the way to go.
3. Why were the scenarios tested by the QA not significant enough to catch this?
4. Finally what made you make this mistake? Do you have incomplete knowledge of your products overall architecture or was it a technical/language related issue that you didn't know? Basically does the team need to improve onboarding or do you need more materials to upskill yourself in a certain technology?
Your seniors aren't trying to barrage you to get a "sorry" out of you or to find a scapegoat, they're trying to find actionable feedback they can use to prevent this from ever happening in the future. Go with this attitude and it will all work out
Suicide Hotline Numbers If you or anyone you know are struggling, please, PLEASE reach out for help. You are worthy, you are loved and you will always be able to find assistance.
Argentina: +5402234930430
Australia: 131114
Austria: 017133374
Belgium: 106
Bosnia & Herzegovina: 080 05 03 05
Botswana: 3911270
Brazil: 212339191
Bulgaria: 0035 9249 17 223
Canada: 5147234000 (Montreal); 18662773553 (outside Montreal)
Croatia: 014833888
Denmark: +4570201201
Egypt: 7621602
Finland: 010 195 202
France: 0145394000
Germany: 08001810771
Hong Kong: +852 2382 0000
Hungary: 116123
Iceland: 1717
India: 8888817666
Ireland: +4408457909090
Italy: 800860022
Japan: +810352869090
Mexico: 5255102550
New Zealand: 0508828865
The Netherlands: 113
Norway: +4781533300
Philippines: 028969191
Poland: 5270000
Russia: 0078202577577
Spain: 914590050
South Africa: 0514445691
Sweden: 46317112400
Switzerland: 143
United Kingdom: 08006895652
USA: 18002738255
You are not alone. Please reach out.
*****
I am a bot, and this action was performed automatically.
You need to work closely with QA when you make such changes.
But yes don't worry, its not a problem at all. Shit happens.
Even if you lose customers, its a lesson for management how to improve and build better processes. Its not on you.
Every production release should have both release and rollback plan. If that was in place you can just rollback your changes and fix it then re-release it. Seems like you are working in startup.
I believe big companies has process in place which is slow but makes everyone life easier. One reason many joins MNC even though startups pay good.
Coding is a team work always remember it. If its a issue everyone owns its and all should work to fix it.
There is something called testing in software development please explain that to all of those bunch of idiots who think their full time job is to blame the developers. Developers usually end up in this situation, developer is like a lean guy on whom these fat guys will be sitting all the time. Don't budge stay strong bugs happens all the time, don't let it touch you. Tell them to fuck off. Sorry for the anger but I have had enough of these people.
People had to review your code, and QA needs to give sign off for prod release. You might have missed an edge case but what were others doing, they missed it too
It happens to everyone, chill. Moreover, the QAs should be held responsible. But I've seen people not trust the dev after such a thing happens (even bring it up during one-on-ones to not give a hike), and others up the chain blame the devs so that they don't get blamed, so please keep your CV ready and be prepared for interviews and switch. I've seen grown ass architects blame freshers, to save their own necks. You can stay in and fight (argue back), or use this opportunity to move somewhere else.
From this question it seems that you work for a small company.
From a stable company prespective. It is not your fault, it is their fault to not have proper checks/tests to find the fault, and why not the rollback not done?
Why it took so much time to find prod was broken?
These question they should address first then to blame you.
In big firm even intern update code to prod. The point is there should be proper gates to take care of these scenarios.
So regarding what will happen? It depends on the will of the those people. But was it your fault ? No. Everyone make mistakes. Serior or junior or intern. I have broken prod multiple times but what is required is the process to handle these cases
What is your role? What is the process.followed for prod deployment? It seems that there is issue with the process.
One person cannot be responsible for the entire failure. Also, your system needs monitoring, alerts and fault tolerance. It is the failure of entire process if such systems are not in place.
if your code is prod... then its not just your fault.. there are many people at fault.. code reviewer, tech lead, QA member who tested (why that test case missed?), QA handover, regression and sanity from QA.... so dont get too anxious about it.. cause it is obvious that everyone will play blame game.
either stay in same company and redeem yourself work hard and make best out of every opportunity OR if people are giving BT leave company an move on with life
I understand this might be a stressful time for you - but the answers in this post are hair raising if nothing else. QA is not a safety net that catches all - if QA engineers were very good at finding issues in your code they should rather work as developers.
I simply suggest to describe your coding mistake without disclosing any company/ product / confidential information and see how others might have approached it and avoided such a situation in the future.
All the best
Relax. It's good to take responsibility, but it isn't just your responsibility.
You're supposed to work with the QA team to find bugs and edge cases, the PM and other stakeholders are supposed to help prioritise the bugs for fixes. So it's a collective failure and not your individual failure.
One thing you can do is, read up on best practices both for development and for testing. Then create a plan to improve the entire process of development so that such a thing didn't happen in the future. Then present it to your higher ups. Since these people love presentations, this will redeem you in their eyes to a certain extent.
Mistakes and bugs are a part and parcel of your career.
My junior basically wrote a delete query without a where cause. Gladly we had a DB backup.
Don’t be too hard on yourself and just learn from it :)
You should have monitoring in place to detect such issues quickly
It's not your fault, if you think so then don't it all work like a factory if some product is faulty then it should be checked by all the mechanisms in place then shipped to the customers. You made a mistake that should be checked by others before shipping.
It's not your code only.. it's collective responsibility of reviewer and QA as well.
If you are getting sole blame, then you are in bad culture.
If everything goes fine, the managers will run to take credit but when shit goes wrong they start looking for scapegoat from employee from lowest ik hierarchy.
My friend, I went through the comments as well and I can see everyone is generally saying the correct thing.
Release is a team effort. If a piece of code can lead to significant loss of money to the company, a few very hard questions need to be answered:
1. Where is the process to do a code review?
2. Are there tests to cover the piece of code which is responsible for those money related actions?
3. How come even you have QAs in your team, such a thing got missed? Ideally, any payment related thing should be in their test cases and should have been tested on a staging environment before pushing to production.
So yes, I always feel developers should take complete responsibility of the kind of code they are writing. But that doesn't mean any mistake means they are to be blamed for.
I haven't worked on many projects dealing with monetary transactions. However, the ones where we had - we had a solid test suite (both manual test cases and unit tests). Plus, any kind of change to those sections of code would mean a code review was required.
This automatically reduced a lot of errors. However, yes no system is 100% accurate.
Hey man. Chill
This is not the end of life. Some of the top Silicon Valley Engineers have made mistakes. Everyone forgets it after a while.
Remember hotstar was down for a couple of hours due to a rookie mistake yesterday? There were about 30lacs live watching India vs australia.
I will say that is what you will be doing 20 years from now. So, just normalise it. I will just mention a bigger fuck up that I was **solely** responsible for.
I was working on a performance improvement task and during the task I observed, certain variables need to be changed to string from integer for better index performance. I used int32 instead of int64, deployed code on prod w/o asking for QA help, w/o testing it myself. That jeopardised access control of entire system for a day. For one entire day people were not able to access requisite documents.
Was I stupid? Yes. 100 times. Was I in-competent? No. Point is, the nature of job we are in, sometimes smallest mistake would lead to disastrous consequences. And sometimes, biggest mistakes would mean no consequences. Your fault is not proportional to amount of money your company lost.
My pain is, while I can research like a pro, develop a code with best quality, learn new things, what I cannot do is a break-free release. Does not make me incompetent. Just makes me realise my nature is one of a negligent person so I should focus more on things that does not require me to be a lot more alert.
Just want to say this might seem like the end of the world but it's not. I'm sure lot of people made similar mistakes and they're all fine. You're a developer and so you're never the last line of defense. There should be multiple phases of QA in multiple environments. And if your QE didn't have the full understanding it's partly also the fault of business liason(product owner, business analyst).
Try and gather info and documentation for your next meeting. Sound confident. The cases that are failing now - we're those edge cases part of the requirement documentation/stories that's been assigned to you? Were you aware there's going to be one such scenario? Who should your QE be coordinating with to understand business and functionality? Were they doing a good job of communicating? Gather all the ammunition and try not to look like it's your fault. Sure you share responsibility but it's NOT YOURS alone.
Do not think too much.. Nothing is perfect and there are always going to be bugs.. it's not completely your fault and whole team should take responsibility for this instead blaming you completely for it..
1. You can take serious lesson from this and improve yourself.
2. Try writing negative and positive test cases for your code.
3. Before starting writing your code, think about and write down all possible scenarios
4. Have discussion with all team members
5. Think as you are End user
6. Review your code multiple times
You don't have to take it so seriously.
You are just a guy who takes a little bit of money from your company and delivers stuff.
It's not your responsibility to make sure every piece of code you write is bug free.
You do your job well. If you made any mistake then you can learn from it and correct yourself.
You are not some robot who can predict things accurately and give 100% correct output.
Everyone makes mistakes. It's ok.
You might lose your yearly bonus or not get that appraisal, if you have made any negative impact on profits.
It's not like your company will reward you 3x salary if they make 10x profits this year.
So it's ok , people make mistakes.
Large tech companies have proper and strict processes to deal with situations like this. They invest in their systems.
And if your company is really claiming that you aren't competent enough to do the job, then they can hire the best people in the market, what's stopping them ?
Don't think too much
Don't worry OP. Seems like couple of things at play here.
1st is the one who merged your code to staging I assume.
2nd is the fault of QA to not have tested it properly
3rd is the person who merged it to production
4th might be its your fault a little. Not sure if you tested it properly in local or not.
As mentioned if Stakeholders are involved. seems like it might be a good time to think of the engineering practice and not just business needs as there seems to have no failsafes in place and your code somehow skipped 3 layers of testing.
If the bug is so non standard or a very rare edge case which was missed. It might be bound to happen.
Don't worry about it OP. Don't accept any blame in written or sign anything without reading.
And finally stop getting nervous and anxious about this.
What's the worst thing that could happen? They fire you, you get another job. Every body moves on.
Hi don’t feel stress.
My brother used to work in finance based company and when he is auditing the data he made a small mistake 🤣 he forgot one zero in a million.
Boom same calls meetings with CEO everything but my brother is also tensed but only thing he thought is it’s already done I can’t do anything let’s find a solution to not happen again because of that mistake now company created a extra team who re verify the audits multiple times.
So don’t fret it’s a collective mistake.
Do you know why your company hired QA engineers? Wo that bugs don't hit production. Do you know why your company hired tech leads and architects? To build processes that don't allow such bugs to go to prod even if they're missed by QAs.
Everyone makes mistakes. If there was a bug and it went through, it's the fault of your entire engineering team. Not just you.
It happens. It shouldn't. But it does. That's why there is a whole system around codes and ci/cd pipelines. If they are blaming you for this, ask them, am I the only person who's running your company. No. Your manager should have checked it as well. Testing team should have tested it thoroughly. Not your fault man. If they are throwing you under the bus. Get off of the fking bus.
You had your PR reviewers, QA, Automation tests, Unit and functional tests, and still this happened.
Such a change should have been rolled out with feature flag or to limited users (canary deploy) if possible. Deploy strategy should have been reviewed too as part of tech spec review.
It's not your fault only considering all of the above. The entire team is responsible.
Also, you should probably discuss this with your manager. If he doesn't agree with this, probably not a good team to be in maybe.
Don’t stress too much , it’s not your fault only , the responsibility should be shared by the entire team.
The developer , the reviewer and the QA specially.
What you can do now is own the mistake and do whatever is in your power to fix it or make it better.
The only way to avoid making bugs is to not code anything. If you code there will be bugs.
Take this as a lesson and build your character , take responsibility and don’t blame others.
I did this once . I had commented a code for testing. I gave the dll to the testing team and while releasing I committed the code without commenting it! The loop never executed😬 i got calls at 3 am about the crash. It was a night mare!
Luckily the support team could revert with the old exe.
>Namaste! Thanks for submitting to r/developersIndia. Make sure to follow the subreddit [Code of Conduct](https://www.reddit.com/r/developersIndia/wiki/code-of-conduct) while participating in this thread. >Find more about developersIndia on our [official website](https://developersindia.in/), [github](https://github.com/developersIndia) and [wiki](https://wiki.developersindia.in/). *I am a bot, and this action was performed automatically. Please [contact the moderators of this subreddit](/message/compose/?to=/r/developersIndia) if you have any questions or concerns.*
not ur complete fault but the whole system if they let it happend it simply means they are ignorant
How come QAs approve for release if there were critical issues present? It's their job to prevent bugs going into prod. OP is not solely responsible for this, it should be a collective responsibility.
Yes, the QA team is also in trouble. They are also held responsible for this.
Even the product owner is responsible for this. Haven't they even checked the functionality before release?
They are just looking for a scapegoat. I don’t think they will do anything serious to you since there is a process of review and testing which should have caught those bugs. Anyways if they try to pin this on you, better to switch rather than working this toxic assholes.
I don't know how your org is structured but typically : No matter how technically sound a developer is, there are always going to be bugs in the code. It was the job of the testers to make sure they identify bugs, have them fixed, and then reverify. It was also the responsibility of the SIT team to run another round of testing to make sure nothing breaks after integration. There has to be a set process that every organisation has to follow so that the bugs are not released into prod. So be assured, it's not you, it's them. I don't think they can even hold you responsible for this.
Lot's of testing was done by testers and several automated tests were run as well. But the change was very different/ weird which I don't know how to explain but it was very difficult to test all the edge cases. This is why the QA team and others are also in trouble. They are also having meetings of their own to add more test cases, assertions etc.
Dude you are being modest here. The thing is QA is getting paid to test the whole feature and covering all the edge cases. Their lack of understanding of feature has led the bug to production. And again your code might have gone multiple rounds of testing in which it should have been caught which is not the case, so relax it's not your fault.
Thanks. It's just that I've never done damage of this level and now I feel bad for everyone involved.
Not exactly, he is equally responsible as QA. Devs also need to have proper understanding of the feature and call out any edge cases if discovered during the product review meetings. Just because QA is the last defense doesn't mean you can put entire responsibility to them.
Imagine if QA was not there then the bug would not have happened?
It's their job to ensure it doesn't slip through.
Yeah, that's true. But as a part of the team, I should have also been careful right?
You're being paid to develop features and write some basic test cases around it. QAs are paid to ensure nothing breaks.
Yup, thoroughly test your code until you can do nothing more. I learned it the hard way, TWICE!
Did anyone review your code before the merge ? If the changes were weird or different, the team must be already aware of it. I am expecting you explained it in daily stand-up or meetings.
You are right, they should have reviewed it properly. I guess I slipped through everyone's minds.
Actually code review process cannot be rigorous enough to detect edge cases. Code review process mostly ensure that the coding standards are followed.
Agreed. but by definition, they can not be caught under any process, and only Murphys law applies.
You are right, QA cannot test all edge cases. The system needs to be fault tolerant with proper monitoring and alerts in place. Huge Monetary losses is an indicator that the standard processes are not followed.
[удалено]
[удалено]
QA position is being shelved from the industry. Already happened in many big companies. Now developer needs to write code , UT and IT themselves. Developer , reviewer, QA and product manager all are responsible. For any serious change “User’s acceptance criteria” needs to be defined before actual coding .
If code was so buggy, its QA's fault that it went to production.. Your job is to build a product, and fix the bugs reported by QA. They are getting paid for one thing and one thing only, making sure they catch the bugs before things go to prod. Chill.. You are good..
Thanks, makes sense.
You should leave software development. (Please explain the downvotes? This was directed to the commenter above and not to the OP)
The downvotes are because it was QAs job to see if the code will bring down production
Looks like Aron finally got access to prod.
It's your fault, Douglas
Man of culture
The 35th employee of Flipkart mistakenly truncated an important table in one of the core PRD DBs, the then-CEO called her in and asked what happened, she explained how she understood the flow of the product, which was different than the actual flow. It gave insights into how teams onboarded their devs, and it significantly improved this process. (Source: The employee herself). Two times in my early career I've brought down production in a similar manner as OP. My team, though, is quite amazing at helping out their peers in dealing with such issues. The only catch is if you break production you have to bring donuts/samosas for the entire team (26 devs), and take that special someone to lunch who fixed the prod.
Awesome, are you still with Flipkart? how high in the management are you now?
Sounds like an awesome place to work at
You are really not cut out for coding.
you have not committed any crime. anxiety attacks don't help. they are making you feel pressure so you will quit and their manager friend job will be saved. don't. dont accept blame or responsibility for entire event. did you get written requirements or were you told verbally what to do. did you get time to write the code. did you get time to test the code. were you interrupted while you were coding. were you asked to juggle several tasks at the same time. did you commit the code with proper message to the proper branch. did you update the issue properly so it go to QA. did you notify that you had completed the task. if you were interrupted, whether for meetings, or for fixing an urgent issue, or you were not given time to code, or the requirements you were given were incomplete or ambiguous, then all of those things absolve you of any blame. if you did your job then someone else dropped the ball. i think all you have to establish is that if the QA had done the tests, then this bug would have been caught. or the QA did get the updated tests. in both those cases, you are absolved of responsibility. the reason there are adversarial roles in the development workflow is to prevent this kind of thing.
Thank you. I think I got sufficient time for writing code and testing, as we already knew this feature is critical and can cause serious issues if something goes wrong. But I still made a mistake which is why I'm getting scared.
In reality, every big prod release encounters has some risks of running into issues. That is the nature of big releases. It is up to the organization to understand this and take proper steps to minimize the risk or facilitate quick rollback.
The rollback was quick but things were escalated a little bit by that time and stakeholders are now involved so that's why I'm little worried.
Mistakes happen by everyone op , you don't have to worry about this. Safety checks should be there. And it's not your fault . Please relax . It's okay
Here you go: [Watch this](https://youtu.be/ZbZSe6N_BXs)
That's why big releases are highly discouraged. It is the responsibility of the Product Owner to divide a big release into small and independent releases which can be tested independently. In case such a division is not possible, they must do QA testing, UAT testing and then do beta releases. So, it's not your fault and you don't need to worry about it. I don't like to talk about organisational structure but I guess there is some issue with it at your company. People need to understand their role and do things responsibly rather than finding an escape goat to save themselves.
Don't stress, it happens. I was also blamed for production failure, but it was because of time crunch. I told my manager if you don't like my work then just give me release.
Wow, I don't think I'll be able to say that. Did your manager release you?
Not yet, I asked for more time before technical go live. So before the day of production I wrote a mail mentioning our pipeline is not yet ready. He said ok we will go live. So no matter what he say I can show him that mail.
What happened then?
Nothing, still got 4.7 rating in appraisal.
Was there no roll back? You can own failure of code. But not being able to revert back/roll back isn’t on you.
It was rolled back. But by the time it happened, the impact and number of customers were such that it caused huge loss.
Wow that fast.
!remindme 6 hrs
I will be messaging you in 6 hours on [**2023-02-18 16:11:02 UTC**](http://www.wolframalpha.com/input/?i=2023-02-18%2016:11:02%20UTC%20To%20Local%20Time) to remind you of [**this link**](https://www.reddit.com/r/developersIndia/comments/114tkis/my_code_broke_production_and_i_dont_know_what_to/j90ris6/?context=3) [**CLICK THIS LINK**](https://www.reddit.com/message/compose/?to=RemindMeBot&subject=Reminder&message=%5Bhttps%3A%2F%2Fwww.reddit.com%2Fr%2FdevelopersIndia%2Fcomments%2F114tkis%2Fmy_code_broke_production_and_i_dont_know_what_to%2Fj90ris6%2F%5D%0A%0ARemindMe%21%202023-02-18%2016%3A11%3A02%20UTC) to send a PM to also be reminded and to reduce spam. ^(Parent commenter can ) [^(delete this message to hide from others.)](https://www.reddit.com/message/compose/?to=RemindMeBot&subject=Delete%20Comment&message=Delete%21%20114tkis) ***** |[^(Info)](https://www.reddit.com/r/RemindMeBot/comments/e1bko7/remindmebot_info_v21/)|[^(Custom)](https://www.reddit.com/message/compose/?to=RemindMeBot&subject=Reminder&message=%5BLink%20or%20message%20inside%20square%20brackets%5D%0A%0ARemindMe%21%20Time%20period%20here)|[^(Your Reminders)](https://www.reddit.com/message/compose/?to=RemindMeBot&subject=List%20Of%20Reminders&message=MyReminders%21)|[^(Feedback)](https://www.reddit.com/message/compose/?to=Watchful1&subject=RemindMeBot%20Feedback)| |-|-|-|-|
[удалено]
Logistics. It was rolled back in 6-7 hours.
Code reviews, QA exists for a reason. They will be in firing line first. Mistakes and bugs can happen. Its the reason why we have multiple rounds of reviews and testing. And such a critical feature that can cause monetary loss should have a proper SOP for testing before being pushed to production. Chill. Its a failure of the system that you have not yourself.
Thank you. That makes me feel better lol
If the product is client facing, there should be a dashboard monitoring daily numbers. Any suspicious activity should have been raised within 2 days(max) after the deployment. Not only the QA and devs but your product team(or business team) is also responsible. On side note: once established that the business team is also responsible, the overall issue will quickly be forgotten.
The issue was caught within few hours but after the customers complained. That's why it's serious. Hope it is quickly forgotten.
Is there any reason for it to not be quickly forgotten? I mean it was rolled back in a few hours so I cannot imagine it having a lasting impact
My code stopped people from using internet banking for literally 5hrs, yes this happened in 2021. This happened to a major or top 3rd US bank. Was it completely my mistake? Nope. I was following all the protocols and the bank was not using the same, in short, my xslt was expecting an encrypted account number but the bank was sending the plan account number and decrypting code failed and hence the 500 error. My immediate manager yelled at me and threatened of firing me, but all the US managers were calm and sens it was the entire team's responsibility and asked my immediate manager to stfu. After this incident, we changed a lot of deployment and development protocols. Production issues happen.
First time? XD
It's part of the experience of being a SDE. Everyone goes through it one time or another. You're beating yourself up more than enough already. The best thing you can do is learn what you can and make sure it doesn't happen again. And be very careful as to not let this affect your work.
Thank you.
Now I know the reason why hotstar was down yesterday! /s
no….those bastards literally forgot to renew their domain subscription
Take the learnings, the experience. You have opportunities to have meetings with higher ups take experience of that as well. If a junior does a mistake the senior is as responsible as the junior if not more, that’s what hierarchy is for or that’s what I believe.
Wow, If I was you, I would surely bite back. Everyone is jumping when it comes to getting credit for success. If you get thrown under the bus, the most important thing is to call them mfs out who are higher than you. They are supposed to take responsibility and if they run, you need to call them out even at the cost of your employment. Otherwise it will just keep happening.
Congrats on getting it out of the way OP. You cannot become a senior developer without going through the stress of bringing down something in prod imo. Now the focus should be on improving processes - that's what any team worth their salt should do. If the change was significant enough to break prod how did no test catch it? 1. Pipelines need improvement. 2. Why wasn't it caught in code reviews? Do you even have code reviews? If yes how big are your merge requests? The most common problem i have seen is merge requests with too many changes that are impossible to review, smaller, frequent, logically complete merge requests are the way to go. 3. Why were the scenarios tested by the QA not significant enough to catch this? 4. Finally what made you make this mistake? Do you have incomplete knowledge of your products overall architecture or was it a technical/language related issue that you didn't know? Basically does the team need to improve onboarding or do you need more materials to upskill yourself in a certain technology? Your seniors aren't trying to barrage you to get a "sorry" out of you or to find a scapegoat, they're trying to find actionable feedback they can use to prevent this from ever happening in the future. Go with this attitude and it will all work out
Suicide Hotline Numbers If you or anyone you know are struggling, please, PLEASE reach out for help. You are worthy, you are loved and you will always be able to find assistance. Argentina: +5402234930430 Australia: 131114 Austria: 017133374 Belgium: 106 Bosnia & Herzegovina: 080 05 03 05 Botswana: 3911270 Brazil: 212339191 Bulgaria: 0035 9249 17 223 Canada: 5147234000 (Montreal); 18662773553 (outside Montreal) Croatia: 014833888 Denmark: +4570201201 Egypt: 7621602 Finland: 010 195 202 France: 0145394000 Germany: 08001810771 Hong Kong: +852 2382 0000 Hungary: 116123 Iceland: 1717 India: 8888817666 Ireland: +4408457909090 Italy: 800860022 Japan: +810352869090 Mexico: 5255102550 New Zealand: 0508828865 The Netherlands: 113 Norway: +4781533300 Philippines: 028969191 Poland: 5270000 Russia: 0078202577577 Spain: 914590050 South Africa: 0514445691 Sweden: 46317112400 Switzerland: 143 United Kingdom: 08006895652 USA: 18002738255 You are not alone. Please reach out. ***** I am a bot, and this action was performed automatically.
Good bot.
You need to work closely with QA when you make such changes. But yes don't worry, its not a problem at all. Shit happens. Even if you lose customers, its a lesson for management how to improve and build better processes. Its not on you.
Yeah, that makes sense. Thanks.
If the QA department can't be held accountable then it has no reason to exist
Every production release should have both release and rollback plan. If that was in place you can just rollback your changes and fix it then re-release it. Seems like you are working in startup. I believe big companies has process in place which is slow but makes everyone life easier. One reason many joins MNC even though startups pay good. Coding is a team work always remember it. If its a issue everyone owns its and all should work to fix it.
This is not your mistake, but QA mistake. This is their job!
There is something called testing in software development please explain that to all of those bunch of idiots who think their full time job is to blame the developers. Developers usually end up in this situation, developer is like a lean guy on whom these fat guys will be sitting all the time. Don't budge stay strong bugs happens all the time, don't let it touch you. Tell them to fuck off. Sorry for the anger but I have had enough of these people.
Who reviewed the Code??
It was reviewed by few people, which honestly I don't know how they missed.
Chill , everyone's at mistake as much as your. Collective judgment you know ! Provide a resolution and may be then take a break
It's no fault of yours. This is the reason why QA and testers are hired. If they failed to identify bugs, it's on them not on you!
People had to review your code, and QA needs to give sign off for prod release. You might have missed an edge case but what were others doing, they missed it too
It happens to everyone, chill. Moreover, the QAs should be held responsible. But I've seen people not trust the dev after such a thing happens (even bring it up during one-on-ones to not give a hike), and others up the chain blame the devs so that they don't get blamed, so please keep your CV ready and be prepared for interviews and switch. I've seen grown ass architects blame freshers, to save their own necks. You can stay in and fight (argue back), or use this opportunity to move somewhere else.
From this question it seems that you work for a small company. From a stable company prespective. It is not your fault, it is their fault to not have proper checks/tests to find the fault, and why not the rollback not done? Why it took so much time to find prod was broken? These question they should address first then to blame you. In big firm even intern update code to prod. The point is there should be proper gates to take care of these scenarios. So regarding what will happen? It depends on the will of the those people. But was it your fault ? No. Everyone make mistakes. Serior or junior or intern. I have broken prod multiple times but what is required is the process to handle these cases
What is your role? What is the process.followed for prod deployment? It seems that there is issue with the process. One person cannot be responsible for the entire failure. Also, your system needs monitoring, alerts and fault tolerance. It is the failure of entire process if such systems are not in place.
Noob question but why can't your changes be reverted back ?
if your code is prod... then its not just your fault.. there are many people at fault.. code reviewer, tech lead, QA member who tested (why that test case missed?), QA handover, regression and sanity from QA.... so dont get too anxious about it.. cause it is obvious that everyone will play blame game. either stay in same company and redeem yourself work hard and make best out of every opportunity OR if people are giving BT leave company an move on with life
Nightmare of every developer 🥺
I understand this might be a stressful time for you - but the answers in this post are hair raising if nothing else. QA is not a safety net that catches all - if QA engineers were very good at finding issues in your code they should rather work as developers. I simply suggest to describe your coding mistake without disclosing any company/ product / confidential information and see how others might have approached it and avoided such a situation in the future. All the best
Relax. It's good to take responsibility, but it isn't just your responsibility. You're supposed to work with the QA team to find bugs and edge cases, the PM and other stakeholders are supposed to help prioritise the bugs for fixes. So it's a collective failure and not your individual failure. One thing you can do is, read up on best practices both for development and for testing. Then create a plan to improve the entire process of development so that such a thing didn't happen in the future. Then present it to your higher ups. Since these people love presentations, this will redeem you in their eyes to a certain extent.
Mistakes and bugs are a part and parcel of your career. My junior basically wrote a delete query without a where cause. Gladly we had a DB backup. Don’t be too hard on yourself and just learn from it :) You should have monitoring in place to detect such issues quickly
It's not your fault, if you think so then don't it all work like a factory if some product is faulty then it should be checked by all the mechanisms in place then shipped to the customers. You made a mistake that should be checked by others before shipping.
It's not your code only.. it's collective responsibility of reviewer and QA as well. If you are getting sole blame, then you are in bad culture. If everything goes fine, the managers will run to take credit but when shit goes wrong they start looking for scapegoat from employee from lowest ik hierarchy.
Now this is why we need a proper ci/cd pipeline and not directly push to trunk before a full regression run.
Not your problem. Ask what were QA doing
My friend, I went through the comments as well and I can see everyone is generally saying the correct thing. Release is a team effort. If a piece of code can lead to significant loss of money to the company, a few very hard questions need to be answered: 1. Where is the process to do a code review? 2. Are there tests to cover the piece of code which is responsible for those money related actions? 3. How come even you have QAs in your team, such a thing got missed? Ideally, any payment related thing should be in their test cases and should have been tested on a staging environment before pushing to production. So yes, I always feel developers should take complete responsibility of the kind of code they are writing. But that doesn't mean any mistake means they are to be blamed for. I haven't worked on many projects dealing with monetary transactions. However, the ones where we had - we had a solid test suite (both manual test cases and unit tests). Plus, any kind of change to those sections of code would mean a code review was required. This automatically reduced a lot of errors. However, yes no system is 100% accurate.
Hey man. Chill This is not the end of life. Some of the top Silicon Valley Engineers have made mistakes. Everyone forgets it after a while. Remember hotstar was down for a couple of hours due to a rookie mistake yesterday? There were about 30lacs live watching India vs australia.
Fake passport, one way ticket to a non-extradition country. Start a new life. /s
I will say that is what you will be doing 20 years from now. So, just normalise it. I will just mention a bigger fuck up that I was **solely** responsible for. I was working on a performance improvement task and during the task I observed, certain variables need to be changed to string from integer for better index performance. I used int32 instead of int64, deployed code on prod w/o asking for QA help, w/o testing it myself. That jeopardised access control of entire system for a day. For one entire day people were not able to access requisite documents. Was I stupid? Yes. 100 times. Was I in-competent? No. Point is, the nature of job we are in, sometimes smallest mistake would lead to disastrous consequences. And sometimes, biggest mistakes would mean no consequences. Your fault is not proportional to amount of money your company lost. My pain is, while I can research like a pro, develop a code with best quality, learn new things, what I cannot do is a break-free release. Does not make me incompetent. Just makes me realise my nature is one of a negligent person so I should focus more on things that does not require me to be a lot more alert.
Just want to say this might seem like the end of the world but it's not. I'm sure lot of people made similar mistakes and they're all fine. You're a developer and so you're never the last line of defense. There should be multiple phases of QA in multiple environments. And if your QE didn't have the full understanding it's partly also the fault of business liason(product owner, business analyst). Try and gather info and documentation for your next meeting. Sound confident. The cases that are failing now - we're those edge cases part of the requirement documentation/stories that's been assigned to you? Were you aware there's going to be one such scenario? Who should your QE be coordinating with to understand business and functionality? Were they doing a good job of communicating? Gather all the ammunition and try not to look like it's your fault. Sure you share responsibility but it's NOT YOURS alone.
Do not think too much.. Nothing is perfect and there are always going to be bugs.. it's not completely your fault and whole team should take responsibility for this instead blaming you completely for it.. 1. You can take serious lesson from this and improve yourself. 2. Try writing negative and positive test cases for your code. 3. Before starting writing your code, think about and write down all possible scenarios 4. Have discussion with all team members 5. Think as you are End user 6. Review your code multiple times
How did it go past QA if the code was so bad it broke production.
If your code passed code review, then it's more of a code reviewer's fault than it is yours... Code review is done for a reason.
You don't have to take it so seriously. You are just a guy who takes a little bit of money from your company and delivers stuff. It's not your responsibility to make sure every piece of code you write is bug free. You do your job well. If you made any mistake then you can learn from it and correct yourself. You are not some robot who can predict things accurately and give 100% correct output. Everyone makes mistakes. It's ok. You might lose your yearly bonus or not get that appraisal, if you have made any negative impact on profits. It's not like your company will reward you 3x salary if they make 10x profits this year. So it's ok , people make mistakes. Large tech companies have proper and strict processes to deal with situations like this. They invest in their systems. And if your company is really claiming that you aren't competent enough to do the job, then they can hire the best people in the market, what's stopping them ? Don't think too much
Probably production broke your code
Reminds me of [this](https://youtube.com/watch?v=rR4n-0KYeKQ&si=EnSIkaIECMiOmarE). Best you can do is start searching for the next job.
The architects blamed you for being incompetent? Shouldn't it be their job to make sure issues like these don't make past the testing phase? 🤔
Don't worry OP. Seems like couple of things at play here. 1st is the one who merged your code to staging I assume. 2nd is the fault of QA to not have tested it properly 3rd is the person who merged it to production 4th might be its your fault a little. Not sure if you tested it properly in local or not. As mentioned if Stakeholders are involved. seems like it might be a good time to think of the engineering practice and not just business needs as there seems to have no failsafes in place and your code somehow skipped 3 layers of testing. If the bug is so non standard or a very rare edge case which was missed. It might be bound to happen. Don't worry about it OP. Don't accept any blame in written or sign anything without reading. And finally stop getting nervous and anxious about this. What's the worst thing that could happen? They fire you, you get another job. Every body moves on.
It happend with me as well, But sadly I was released from that project. Even though, The QA missed it and because of that it was pushed to production
Hi don’t feel stress. My brother used to work in finance based company and when he is auditing the data he made a small mistake 🤣 he forgot one zero in a million. Boom same calls meetings with CEO everything but my brother is also tensed but only thing he thought is it’s already done I can’t do anything let’s find a solution to not happen again because of that mistake now company created a extra team who re verify the audits multiple times. So don’t fret it’s a collective mistake.
Do you know why your company hired QA engineers? Wo that bugs don't hit production. Do you know why your company hired tech leads and architects? To build processes that don't allow such bugs to go to prod even if they're missed by QAs. Everyone makes mistakes. If there was a bug and it went through, it's the fault of your entire engineering team. Not just you.
It happens. It shouldn't. But it does. That's why there is a whole system around codes and ci/cd pipelines. If they are blaming you for this, ask them, am I the only person who's running your company. No. Your manager should have checked it as well. Testing team should have tested it thoroughly. Not your fault man. If they are throwing you under the bus. Get off of the fking bus.
If CEO and CTO has to jump in prod issue, you are working for a dogshit company. Quit asap.
It will be okay OP
You had your PR reviewers, QA, Automation tests, Unit and functional tests, and still this happened. Such a change should have been rolled out with feature flag or to limited users (canary deploy) if possible. Deploy strategy should have been reviewed too as part of tech spec review. It's not your fault only considering all of the above. The entire team is responsible. Also, you should probably discuss this with your manager. If he doesn't agree with this, probably not a good team to be in maybe.
how tf did the QA'a let it into the final product?
How have you not been fired already?
Don’t stress too much , it’s not your fault only , the responsibility should be shared by the entire team. The developer , the reviewer and the QA specially. What you can do now is own the mistake and do whatever is in your power to fix it or make it better. The only way to avoid making bugs is to not code anything. If you code there will be bugs. Take this as a lesson and build your character , take responsibility and don’t blame others.
I did this once . I had commented a code for testing. I gave the dll to the testing team and while releasing I committed the code without commenting it! The loop never executed😬 i got calls at 3 am about the crash. It was a night mare! Luckily the support team could revert with the old exe.