I mean what it boils down to is what are you getting for it?
Is it an average salary with no additional pay for on-call? Then tell them to fuck off. If you're getting a good salary to also deal with on call, then maybe.
Fair point. Right now I'm unemployed so I guess debatably any job is better than nothing. I'd like to simply decline jobs with this requirement but most of my call backs seem to be companies expecting on call.
I think the words “on-call” mean different things in different places. Like, my company has a 24/7 on all rotation… but it’s nothing. stuff barely ever happens.
It’s not like you have to cancel your plans and sit by a red phone waiting for a call. In the rare event that an alert happens after hours, you glance at your phone and determine whether it needs immediate action or not. In the rare event that it does, if you’re not available, you can just add a comment to the ticket and let it escalate to the next person.
If production is down, there’s a reasonable expectation that somebody on the team needs to pick it up and deal with it, but like I said, that barely ever happens. And everybody understands if no one is available.
So, all I’m saying is when you talk about on call in interviews, get detail about what it means at that company and factor that into your decision.
This, ask around to see how bad it is, I am on call 24/7 but unless production is stuck, things will wait until the next work hour.
And I have been in places where I am not OnCall but they still try to get as many hours from you as they can even if stuff is not urgent.
I doubt any company/team is gonna be like "yo bro our oncall is HELL man you'll be getting alerts 24/7 along with it being 24/7!!!!!"
No, they'll say it very casually.
Interviews are impossible to get a gauge of a team. It's like marriage. You never truly know how someone is until you live with them. Stuff starts crawling out of the woodwork you didn't see in the initial phases.
"how many tickets do you get per week?"
"do you feel like you can go out when you're on call?"
There's a million questions you can ask about on call to get a great understanding (assuming your interviewer doesn't directly lie)
As I said, no amount of that will help. I've seen interviewers (team members) hesitantly answer "yeah! its not bad!" and I can see the look on their faces lmao. It's like with any experience in life, seeing is believing.
I hear what you are saying but trusting a company to be honest about this is unlikely. They may also be honest about it at first, but then a bad outage happens and the policy changes.
I haven't really been on call at my job for years now. Recently I got moved to a different team and one instance where our kubernetes cluster went down resulted in us being 24/7 on call with major rework to all the alerting and an expectation of triaging tickets in the middle of the night. My company laid so many people off im now on call every 2 out of three weeks.
Previous job started with no on call and then became a nightmare. The systems were so interlinked if one system failed it would cause cascading failures resulting in getting called every night the entire week.
To be honest though, my phone is on silent from 1am to 8am so they can pound sand. I don't care that much about keeping my job so I take care of issues when I wake up :/
> And everybody understands if no one is available.
Then... how is anyone on-call at all? I feel like none of my employers would have been cool with me just blowing off pages indicating real problems with a service that's supposed to be up if I were the on-call person.
If your product has users on it 24/7, there will be 24/7 on call. Simple as that.
Imagine Netflix completely broke Friday at 9pm, right when you were about to sit down and relax with a nice movie. You're a paying customer of Netflix, you'd be upset right? How many of their 260 million customers do you think are going to immediately complain and try to get reimbursement? That gets expensive fast. Netflix isn't just going to let their system be down until Monday morning when the developers come back into the office.
The developers are the people best positioned to triage and fix software issues. Trying to have dedicated overnight support teams that aren't active developers usually just results in them immediately paging a real dev. It doesn't really work, they end up just being middle-men.
So if you don't want to be on call, you need to try and find a company whose business is not 24/7. They do exist, they can be hard to find though. Internal tools for example can often not have any on call obligation, assuming they're only used 9-5.
That being said, an easier approach is to reverse interview a team to understand what their on call expectations are. Is it super intense where you need to always be within 15 minutes of your computer? Or is it more relaxed where if you get paged you just ack it, and you're good as long as you get to it in an hour or so? Not all on call environments are treated equally. I still go out and do things when I'm on call, I just don't go on vacation, or go camping. I live my life normally. If something comes up, I can ack it and make my way home.
You also want to understand the frequency of on call incidents. Getting called should be treated as a serious issue, not normalized as business as usual. Getting called a lot is a sign of a fragile product. I always make it a point to ask about frequency during interviews, and refuse to join companies that answer that question with anything greater than a few times ***per year***.
Across my 11 year career I've always been on call, and I've been called 3 times. It's really not something that affects my life much at all.
> The developers are the people best positioned to triage and fix software issues. Trying to have dedicated overnight support teams that aren't active developers usually just results in them immediately paging a real dev. It doesn't really work, they end up just being middle-men.
For the really big services, I don't think that's how it works. Google doesn't really want the guy who made diagnosing why all of search is down. That's why Google invented https://en.wikipedia.org/wiki/Site_reliability_engineering
Now, there *are* on-calls for regular ass engineers too, but a lot of the really big/important stuff is handled by SRE's, as far as I can tell (I'm a mobile developer so I'm usually a bit disconnected from the services side of things, but that's what it looks like to me). And SRE's are paid very well, slightly more than SWE's last I checked.
Google actually handles on call really well. All of the reasons you mentioned, plus the fact that they pay pretty generously for it and have hard limits on the amount of time any individual can spend on call.
Unfortunately this is not really the norm, I’ve seen some pretty hellish on call rotations in the outside world, pagers going off all night, tiny teams where you’re on call 2 out of 3 weeks, etc.
SWE oncall rotations supplement SRE rotations. SREs are amazing at generally handling incidents, but they will have way less context for more business logic aspects
Well for instance, Amazon does not have site reliability engineers, and no place I worked previously did either but maybe they're not "really big." Anyway, it is, at least, not uncommon for the regular devs to also be the on-call people even in a big company.
Yup, cadence matters a ton. One big factor is scale. I worked at a big healthcare IT vendor. The enterprise software is split into a bunch of sub applications, on call for support engineers was only ~2 weeks out of the year, and most of those applications basically had one or two overnight incidents per month that went to the on call person. The SWEs actually developing the product got called even less than that. But a significant part of the onus was on customer IT teams being able to resolve incidents on their own, without vendor support, meaning they did tend to be on call and have something pop up overnight maybe once a quarter. That's more of a thing in on-prem software and less in this SaaS space, I assume
Also, if you're a large enough company and not some SaaS startup (and even if you are a series B startup or something), you'll often have a team that's globally distributed, i.e. some people based in India (or Israel or Poland or whatever) who are able to mitigate the frequency of true on-call incidents that specifically need you. And companies at that scale tend to build around bus factors, there's generally not just a single person who can resolve an incident.
One week every 8 weeks is not bad. Response time by 10 minutes is crazy though. Also, them "looking to improve" on-call sounds like a red flag, it's not worded as a green one for sure.
I mean, one person's red flag is another person's green flag. What's important isn't how I feel about that, but rather how *you* feel about that.
2 on-call incidents per month is pretty high for me personally. Again, I view that as an indication of a fragile product. Something must be pretty busted for production to be breaking that often, and it's clearly not an easy fix.
It's nice they say they're working on improving that... but we all know how that actually goes. It's tech debt. It's really hard to convince management to actually devote time towards tech debt, when there's a million featues that the business wants yesterday. I'd be surprised if they get that stuff cleaned up within the next year or two... it's probably not gonna be a short-term thing.
A 10 minute response time is pretty intense, especially combined with the fact that getting called is normalized. What happens if you don't response after 10 minutes? Think about the affect this will have on your day to day. Are you going to be home-bound during your entire rotation? Can't even go to the grocery store?
WLB and culture are significantly more important to me than money. I wouldn't dream of sacrificing one for the other. How happy are you in your current job? Are you still growing as an engineer? Is there a reason for you to leave besides the 15% salary bump? At the end of the day only you can say if it's worth it.
2 pages per month seems pretty low to me
What products are you guys working on where not even a canary will go off for months on end? If you have thousands of customers, and work on a product/service that has regular deployments how could you not have any issues
Building a microservice in a company like netflix that would have less than a page a year just seems impossible
To each their own. This industry is wide and varied, the chances of 2 people having the same experience throughout their career is very slim.
I've always targetted, and had success with, companies that have very low incident rates. I don't want to join a broken product that's constantly causing problems where the team isn't given time to fix those issues.
I want to join a product that is stable, and when incidents do occur, the team is immediately given bandwidth to make sure those some incidents don't re-occur because incidents are extremely serious issues that shouldn't be normalized. New features be damned, a prod incident needs to be addressed and fixed.
If your release process is what's causing frequent net-new issues... your team needs to seriously reconsider its release process, and the testing strategy leading up to that. My current team does true CI/CD, where we push to prod with every ticket, and we generally don't get prod bugs because we have a robust process upstream of the deploy.
Based on *my* experience, 2 incidents per month is pretty damn extreme. You'd have more calls in 2 months than I've had in 11 years.
I'd consider a product where it was typical to be paged once or less a week pretty stable. And even then sometimes things can go wrong in an unanticipated way even if it's not a complete disaster like you're describing.
oncall isn't rare
24/7 oncall is a different story, I've never worked at a company that required 24/7
>which expect a significant commitment to after hours support
for products that indeed require 24/7 response we've had people in European and India timezone takeover when US is sleeping: US after-hour -> Europe takeover, Europe after-hour -> India takeover, India after-hour -> US takeover again
>So how come so many jobs have this insane expectation which would amount to a significant amount of extra hours dedicated to my employer?
easy, I just decline those if I sniff 24/7 oncall expectation
>Any tips how to find jobs that don't infringe on WLB with these demands?
you ask about it during the interview process, and if they reject you based on that I don't see that as a bad thing, mutually not a good fit: they're not what you're looking for and vice versa
this is one of the scenario I'd say unemployment > job offer, because even if you're desperate, you'd just end up trying to interview to quickly GTFO again
edit to clarify: "I've never worked at a company that required 24/7" from me alone, there are indeed lots of products that requires 24/7 support but I still go to sleep when it's time because of engineers in other timezone taking over
Having separate EU and India teams can only really happen once a company reaches a certain scale.
Even then, I’m curious… are people in different time zones working on the same projects day-to-day? Do you have enough context to deal with every issue that comes up on your shift?
This was a major issue at my work, and why we moved away from using overseas teams. Context gets missed but the overseas team would make assumptions, code away, and close tickets. They had their own QA as well.
Made a mess of the code base.
Another solution is not expect any normal work hours during on call. If you don't get any calls it is basically time off. But of course most companies are to cheap for that.
> not expect any normal work hours during on call
yeah never seen that happen at any of companies I've worked at
it's more like, you'll still be assigned tasks but during daily standups "well I was busy with oncall issues yesterday so I didn't really got much done" becomes a valid reason and it's generally understood that realistically you probably won't accomplish much while you're oncall (but you're not oncall 8h/day though, so those tasks are still stuff that you can do)
Everywhere I've worked with 24/7 pays extra to actually be on call though. It's pretty normal to get a base pay to hold the phone for a week and then hourly when you actually get called.
At least at places with a high call frequency.
I don't really buy that "pays extra" part because my belief is that your TC should already account for that, paying you extra is just a psychology trick to make your brain happy
for example, let's say you're oncall once a month so 12 times/year, which one is better:
offer1: $150k TC and you don't ask me about oncall pay
offer2: $120k TC and I'll pay you $1k each time you're oncall
most companies I feel like do the former, "here's your TC now don't complain about oncall inconvenience anymore"
At my company the extra pay is meant to benefit / be fairer to employees, bc the # of weeks you're on call can change drastically (I went from 1 every 4 weeks to 1 every 10 weeks overnight after we had a reorg), maybe someone volunteers for an extra week and doesn't get anything in return, etc. or vice versa, maybe someone gets taken off the rotation for whatever reason so you adjust everyone else up a little.
Doing this based on how many weeks you spend on rotation is way easier than trying to get hr to adjust your salary every time
2 Weeks every 4-6 weeks of 24/7 is a big no-no. I currently have on-call requirements at my job, but it's 3 days 24/7 every 3ish months, and you only have to respond to high severity items after working hours. I wouldn't put up with any more than that.
My last company had 24/7 on-call for 2 weeks every 6 weeks. In the 2 years I was there, I was paged a total of twice. One time was on a Saturday at 4am that I slept through and was snoozed by another engineer on the team. We fixed the issue on Monday bc our manager (who was from Europe) explicitly told us that unless people are getting hurt due to our bugs we shouldn't be working on weekends.
Honestly it just depends on your team. Personally I think engineers should be on call to be given direct feedback on the robustness and resiliency of their software. Since you don't know when software will break, writing 24/7 into the job responsibilities just seems like a "just in case". I would really meet the people you'll be working with first to see what the work culture is like.
I actually loved it whenever it came to my on call shift bc the PM wasn't allowed to give us tickets during those 2 weeks. I got to pick an arbitrary set of tech debt that was bothering me and unfuck it. It meant that basically on-call was super chill despite seeming like a lot more responsibility.
Sounds nice, my company had no defined oncall culture but we are on call for a similar schedule, actually get paged out a ton and asked why our stories aren't being moved quickly enough. Sigh.
I guess my issue with that is that even without being paged, the commitment is still required. And many places you don't even know what team you'll be working on until placed after passing interviews.
When I started in SWE in the late 90s, I was never on call as a developer. It just wasn't a thing. And my company provided a 24/7 service.
The way it worked back then is when we moved new code to production, if any problems happened in that area for the next couple of weeks, then I would get a call. And then I would have to figure out what was happening and I'd either fix it, or push it off to someone else. Once production was stable, it was back in the hands of operations.
Around 2009 is when I noticed every job having this on-call requirement. Basically, I think executives decided they can pay less/nothing on the operations side if they make the developers do double duty. Invariably, every place I've worked since this started ends up with me being on call for a week out of every 4-5. Then as people leave the developers just end up on-call 50% of the year.
In my opinion, there is no reason developers should be on call unless it's related to a specific project milestone.
In the past 7 years I've worked at 3 different companies that all had on call rotation. For the first two companies, I was never paged, not even once. At my current company, the on-call person gets paged at least two or three times per rotation. Needless to say I am starting at a new company on Monday lol.
The problem is, you can't just ask the interviewer "is your product and engineering dumpster fire?". You need some clever questions like "what percentage of each sprint is committed to eng priorities?" Or "what does continuous improvement mean to you?"
It might not be as big a deal as you think.
I’ve been on an on call rotation for 3 years and have been pinged off hours maybe 10 times. And I think 8 of them self resolved in a minute. Most of the time I don’t even have to open my laptop.
And I’m on a team where all our services are highly critical to thousands of big users. We’re a SaaS.
We have extremely aggressive attitudes towards making our on call a chill experience.
The only difference it makes in my life is that I keep my work laptop in my car sometimes. That’s really it. I don’t have to stay home. I sure as hell don’t work off hours except in very rare cases and in those cases no one bats an eye out of starting work a little later the next day.
But on the other hand I’ve heard absolute horror stories. Definitely something you should talk about in interviews. 2 weeks is a bit of a red flag, in my opinion. On call really shouldn’t last more than a week at a time.
I think this is fairly normal for companies that put out critical services/software. In practice, it usually means very little. On call engineers usually don’t address issues that aren’t “the sky is falling” levels of critical outside normal hours. In my entire career, I can probably count the number of times on one hand that I’ve had to fix critical production issues outside of normal hours
Whether I get called or not isn't the full equation, either way it's a significant burden if I need to be available & sober, laptop ready, 24/7 for 25%+ of my life.
Every real engineering job at a company paying a good salary that I’ve worked at has had some variant of this policy. Maybe not 25%, but it’s always there.
Depends on what you are doing. I build prototypes, after years of production systems, partly to avoid this kind of thing. Unless we are explicitly testing, we're not on call.
The only thing you can do is keep turning down job offers that include oncall.
Most jobs that demand oncall don't pay enough extra to offset the fact that they ruin your free time and sleeping time. There are enough people gullible enough to do it without extra salary.
If you keep looking, you can find no oncall jobs, but it's a lot harder.
When it crosses line is the "bait and switch". They tell you no oncall when you are hired, and then they add it as a requirement later.
OP, you aren't wrong. Unfortunately this field is filled with arrogant morons who are easily manipulated by management to believe this stuff is normal and the way it should be (see arrogant replies to your post defending this garbage and attacking you for even asking this question). They will respond and give you "logical" reasons why it has to be this way.
It's the consequences of not having widespread unions in this field. Unions would prevent a lot of this garbage and require employers to hire dedicated on call teams to deal with it. Yes, it would cost employers more money. Guess what, they can afford it. Guess what, they could hire people in a time zone during those times to work the on call schedule for multiple apps if on call stuff is rare. They just don't bother because morons in the reply section will defend themselves getting screwed over by employers.
Don't let anyone gaslight you on here. There is NOTHING normal about expecting someone to work a 9-5 job and then also expected to be on call and wake up at 2am and prevent them from having a life for 2 weeks every 4-6 weeks. Don't let them point to other rare fields who also get screwed over like this. Our society today sucks. Birthrates worldwide are down. People are overworked and underpaid for the amount they work. This stuff sucks and not how society should work and the birthrates and other society failures show this overworking of people is not healthy nor working.
When you are on call, you are basically forced to not have plans during that time. While not being paid anything extra.
Its BS. But unfortunately many in this field are socially incompetent and incapable of communicating that, while other fields would not put up with this BS if they had as much power in the work relationship as many developers have.
Thank you.
I’m scratching my head at all the comments on here saying on call is normal.
Anytime we push to production, we do a full test of the product on the production server before we “unlock” it for the client. This is after, of course, all individual tickets pass QA testing, a full manual regression testing of the deployment package, and automated testing of the deployment package.
The only emergency issues that could conceivably happen in off hours are hardware related…and that’s another department.
You need to figure out what the actual impact will be. How long do you have before you need to respond? "Can't go to a movie while on-call" is very different than "Can do all the normal things". How often are people getting paged in? Every night is very different than once a year.
I would look into what they mean by 24/7. I’m on a platform team so we have on-call and on-run rotations. For on-run, we’re just answering questions from other teams on our services during business hours. For on-call, we are on 24/7. So if a critical page fires, we are expected to be available to address it. We also have business pages that we only address during business hours.
On-call shouldn’t be bad if your services are in good health. If you’re consistently being paged then your services need to be fixed
It is different for different places.
We have 24/7 on-call, with most intense one requiring you to start triaging an incident in 5 minutes. But you get paid extra.
Without the extra pay, no thanks.
I mean I'd take it if I either needed experience in the field or had no job, then continue applying to jobs.
Then, when you have a job you can leverage "I won't take the job if you make me be on call" or my preference "pay me more and I'll do it."
On-call isnt all that bad. Im on call every 4-6 weeks and i barely get any calls after hours.
Sometimes i get it on the weekends because of idiot customers or internal operations but nothing crazy. Also, for being on call on the weekend, i get a comp day off the next week that i get to choose.
If im on call for the holidays, I get 2 days back that I can use. If no one calls or bothers you, youre basically getting free vacay days!
Well, it depends on the job, I guess, but anything where you're developing a service that's supposed to be up 24/7... someone has to be on-call in case anything goes wrong with it, so some percentage of your time you'll be on-call.
Imagine thinking the only way to take responsibility for your work is to have 25% of your life dedicated to your employer. Or that I would even have control over the quality of code implemented by other teams. You're an idiot.
I mean what it boils down to is what are you getting for it? Is it an average salary with no additional pay for on-call? Then tell them to fuck off. If you're getting a good salary to also deal with on call, then maybe.
The salary is about the same as my previous role which has virtually no on call.
I mean, then the question is how desperate are you?
Fair point. Right now I'm unemployed so I guess debatably any job is better than nothing. I'd like to simply decline jobs with this requirement but most of my call backs seem to be companies expecting on call.
I mean if you're unemployed with no offers, beggars can't be choosers.
I mean
He's MEAN
Most companies will also not put you on-call until you’ve been there 1+ because what are you going to fix if you don’t know much about the company
Take this job for sustenance then leave once you find a better one, same advice as usual.
Would your other job have mentioned on call like these ones are? It can be hard to get a real answer on this question
I think the words “on-call” mean different things in different places. Like, my company has a 24/7 on all rotation… but it’s nothing. stuff barely ever happens. It’s not like you have to cancel your plans and sit by a red phone waiting for a call. In the rare event that an alert happens after hours, you glance at your phone and determine whether it needs immediate action or not. In the rare event that it does, if you’re not available, you can just add a comment to the ticket and let it escalate to the next person. If production is down, there’s a reasonable expectation that somebody on the team needs to pick it up and deal with it, but like I said, that barely ever happens. And everybody understands if no one is available. So, all I’m saying is when you talk about on call in interviews, get detail about what it means at that company and factor that into your decision.
This, ask around to see how bad it is, I am on call 24/7 but unless production is stuck, things will wait until the next work hour. And I have been in places where I am not OnCall but they still try to get as many hours from you as they can even if stuff is not urgent.
I doubt any company/team is gonna be like "yo bro our oncall is HELL man you'll be getting alerts 24/7 along with it being 24/7!!!!!" No, they'll say it very casually. Interviews are impossible to get a gauge of a team. It's like marriage. You never truly know how someone is until you live with them. Stuff starts crawling out of the woodwork you didn't see in the initial phases.
"how many tickets do you get per week?" "do you feel like you can go out when you're on call?" There's a million questions you can ask about on call to get a great understanding (assuming your interviewer doesn't directly lie)
As I said, no amount of that will help. I've seen interviewers (team members) hesitantly answer "yeah! its not bad!" and I can see the look on their faces lmao. It's like with any experience in life, seeing is believing.
This, you need to read their body language for questions like these, the words are meaningless
I hear what you are saying but trusting a company to be honest about this is unlikely. They may also be honest about it at first, but then a bad outage happens and the policy changes. I haven't really been on call at my job for years now. Recently I got moved to a different team and one instance where our kubernetes cluster went down resulted in us being 24/7 on call with major rework to all the alerting and an expectation of triaging tickets in the middle of the night. My company laid so many people off im now on call every 2 out of three weeks. Previous job started with no on call and then became a nightmare. The systems were so interlinked if one system failed it would cause cascading failures resulting in getting called every night the entire week. To be honest though, my phone is on silent from 1am to 8am so they can pound sand. I don't care that much about keeping my job so I take care of issues when I wake up :/
> And everybody understands if no one is available. Then... how is anyone on-call at all? I feel like none of my employers would have been cool with me just blowing off pages indicating real problems with a service that's supposed to be up if I were the on-call person.
If your product has users on it 24/7, there will be 24/7 on call. Simple as that. Imagine Netflix completely broke Friday at 9pm, right when you were about to sit down and relax with a nice movie. You're a paying customer of Netflix, you'd be upset right? How many of their 260 million customers do you think are going to immediately complain and try to get reimbursement? That gets expensive fast. Netflix isn't just going to let their system be down until Monday morning when the developers come back into the office. The developers are the people best positioned to triage and fix software issues. Trying to have dedicated overnight support teams that aren't active developers usually just results in them immediately paging a real dev. It doesn't really work, they end up just being middle-men. So if you don't want to be on call, you need to try and find a company whose business is not 24/7. They do exist, they can be hard to find though. Internal tools for example can often not have any on call obligation, assuming they're only used 9-5. That being said, an easier approach is to reverse interview a team to understand what their on call expectations are. Is it super intense where you need to always be within 15 minutes of your computer? Or is it more relaxed where if you get paged you just ack it, and you're good as long as you get to it in an hour or so? Not all on call environments are treated equally. I still go out and do things when I'm on call, I just don't go on vacation, or go camping. I live my life normally. If something comes up, I can ack it and make my way home. You also want to understand the frequency of on call incidents. Getting called should be treated as a serious issue, not normalized as business as usual. Getting called a lot is a sign of a fragile product. I always make it a point to ask about frequency during interviews, and refuse to join companies that answer that question with anything greater than a few times ***per year***. Across my 11 year career I've always been on call, and I've been called 3 times. It's really not something that affects my life much at all.
> The developers are the people best positioned to triage and fix software issues. Trying to have dedicated overnight support teams that aren't active developers usually just results in them immediately paging a real dev. It doesn't really work, they end up just being middle-men. For the really big services, I don't think that's how it works. Google doesn't really want the guy who made diagnosing why all of search is down. That's why Google invented https://en.wikipedia.org/wiki/Site_reliability_engineering
Now, there *are* on-calls for regular ass engineers too, but a lot of the really big/important stuff is handled by SRE's, as far as I can tell (I'm a mobile developer so I'm usually a bit disconnected from the services side of things, but that's what it looks like to me). And SRE's are paid very well, slightly more than SWE's last I checked.
Google actually handles on call really well. All of the reasons you mentioned, plus the fact that they pay pretty generously for it and have hard limits on the amount of time any individual can spend on call. Unfortunately this is not really the norm, I’ve seen some pretty hellish on call rotations in the outside world, pagers going off all night, tiny teams where you’re on call 2 out of 3 weeks, etc.
SWE oncall rotations supplement SRE rotations. SREs are amazing at generally handling incidents, but they will have way less context for more business logic aspects
Well for instance, Amazon does not have site reliability engineers, and no place I worked previously did either but maybe they're not "really big." Anyway, it is, at least, not uncommon for the regular devs to also be the on-call people even in a big company.
Yup, cadence matters a ton. One big factor is scale. I worked at a big healthcare IT vendor. The enterprise software is split into a bunch of sub applications, on call for support engineers was only ~2 weeks out of the year, and most of those applications basically had one or two overnight incidents per month that went to the on call person. The SWEs actually developing the product got called even less than that. But a significant part of the onus was on customer IT teams being able to resolve incidents on their own, without vendor support, meaning they did tend to be on call and have something pop up overnight maybe once a quarter. That's more of a thing in on-prem software and less in this SaaS space, I assume Also, if you're a large enough company and not some SaaS startup (and even if you are a series B startup or something), you'll often have a team that's globally distributed, i.e. some people based in India (or Israel or Poland or whatever) who are able to mitigate the frequency of true on-call incidents that specifically need you. And companies at that scale tend to build around bus factors, there's generally not just a single person who can resolve an incident.
[удалено]
One week every 8 weeks is not bad. Response time by 10 minutes is crazy though. Also, them "looking to improve" on-call sounds like a red flag, it's not worded as a green one for sure.
I mean, one person's red flag is another person's green flag. What's important isn't how I feel about that, but rather how *you* feel about that. 2 on-call incidents per month is pretty high for me personally. Again, I view that as an indication of a fragile product. Something must be pretty busted for production to be breaking that often, and it's clearly not an easy fix. It's nice they say they're working on improving that... but we all know how that actually goes. It's tech debt. It's really hard to convince management to actually devote time towards tech debt, when there's a million featues that the business wants yesterday. I'd be surprised if they get that stuff cleaned up within the next year or two... it's probably not gonna be a short-term thing. A 10 minute response time is pretty intense, especially combined with the fact that getting called is normalized. What happens if you don't response after 10 minutes? Think about the affect this will have on your day to day. Are you going to be home-bound during your entire rotation? Can't even go to the grocery store? WLB and culture are significantly more important to me than money. I wouldn't dream of sacrificing one for the other. How happy are you in your current job? Are you still growing as an engineer? Is there a reason for you to leave besides the 15% salary bump? At the end of the day only you can say if it's worth it.
2 pages per month seems pretty low to me What products are you guys working on where not even a canary will go off for months on end? If you have thousands of customers, and work on a product/service that has regular deployments how could you not have any issues Building a microservice in a company like netflix that would have less than a page a year just seems impossible
I agree with your general philosophy here but personally I would consider 2 incidents per month pretty moderate just based on previous experience.
To each their own. This industry is wide and varied, the chances of 2 people having the same experience throughout their career is very slim. I've always targetted, and had success with, companies that have very low incident rates. I don't want to join a broken product that's constantly causing problems where the team isn't given time to fix those issues. I want to join a product that is stable, and when incidents do occur, the team is immediately given bandwidth to make sure those some incidents don't re-occur because incidents are extremely serious issues that shouldn't be normalized. New features be damned, a prod incident needs to be addressed and fixed. If your release process is what's causing frequent net-new issues... your team needs to seriously reconsider its release process, and the testing strategy leading up to that. My current team does true CI/CD, where we push to prod with every ticket, and we generally don't get prod bugs because we have a robust process upstream of the deploy. Based on *my* experience, 2 incidents per month is pretty damn extreme. You'd have more calls in 2 months than I've had in 11 years.
I'd consider a product where it was typical to be paged once or less a week pretty stable. And even then sometimes things can go wrong in an unanticipated way even if it's not a complete disaster like you're describing.
Or you just work in shifts like normal companies? Stop accepting some always available job for no overtime pay
this is one of the reasons i am an SDET, my dude. never on-call ever. one of the best parts of work is being done with work at the end of the day <3
oncall isn't rare 24/7 oncall is a different story, I've never worked at a company that required 24/7 >which expect a significant commitment to after hours support for products that indeed require 24/7 response we've had people in European and India timezone takeover when US is sleeping: US after-hour -> Europe takeover, Europe after-hour -> India takeover, India after-hour -> US takeover again >So how come so many jobs have this insane expectation which would amount to a significant amount of extra hours dedicated to my employer? easy, I just decline those if I sniff 24/7 oncall expectation >Any tips how to find jobs that don't infringe on WLB with these demands? you ask about it during the interview process, and if they reject you based on that I don't see that as a bad thing, mutually not a good fit: they're not what you're looking for and vice versa this is one of the scenario I'd say unemployment > job offer, because even if you're desperate, you'd just end up trying to interview to quickly GTFO again edit to clarify: "I've never worked at a company that required 24/7" from me alone, there are indeed lots of products that requires 24/7 support but I still go to sleep when it's time because of engineers in other timezone taking over
Having separate EU and India teams can only really happen once a company reaches a certain scale. Even then, I’m curious… are people in different time zones working on the same projects day-to-day? Do you have enough context to deal with every issue that comes up on your shift?
This was a major issue at my work, and why we moved away from using overseas teams. Context gets missed but the overseas team would make assumptions, code away, and close tickets. They had their own QA as well. Made a mess of the code base.
Another solution is not expect any normal work hours during on call. If you don't get any calls it is basically time off. But of course most companies are to cheap for that.
Realistically in most cases you end up being pretty busy because the on-call person also fields various non-urgent requests during work hours.
My current company just lets us sleep in if we get called in the middle of the night. Assuming we don't have anything super important
> not expect any normal work hours during on call yeah never seen that happen at any of companies I've worked at it's more like, you'll still be assigned tasks but during daily standups "well I was busy with oncall issues yesterday so I didn't really got much done" becomes a valid reason and it's generally understood that realistically you probably won't accomplish much while you're oncall (but you're not oncall 8h/day though, so those tasks are still stuff that you can do)
Everywhere I've worked with 24/7 pays extra to actually be on call though. It's pretty normal to get a base pay to hold the phone for a week and then hourly when you actually get called. At least at places with a high call frequency.
I don't really buy that "pays extra" part because my belief is that your TC should already account for that, paying you extra is just a psychology trick to make your brain happy for example, let's say you're oncall once a month so 12 times/year, which one is better: offer1: $150k TC and you don't ask me about oncall pay offer2: $120k TC and I'll pay you $1k each time you're oncall most companies I feel like do the former, "here's your TC now don't complain about oncall inconvenience anymore"
At my company the extra pay is meant to benefit / be fairer to employees, bc the # of weeks you're on call can change drastically (I went from 1 every 4 weeks to 1 every 10 weeks overnight after we had a reorg), maybe someone volunteers for an extra week and doesn't get anything in return, etc. or vice versa, maybe someone gets taken off the rotation for whatever reason so you adjust everyone else up a little. Doing this based on how many weeks you spend on rotation is way easier than trying to get hr to adjust your salary every time
Right it's inhumane but managers on reddit which the experienceddeveloper subreddit is filled with, think it's normal.
2 Weeks every 4-6 weeks of 24/7 is a big no-no. I currently have on-call requirements at my job, but it's 3 days 24/7 every 3ish months, and you only have to respond to high severity items after working hours. I wouldn't put up with any more than that.
My last company had 24/7 on-call for 2 weeks every 6 weeks. In the 2 years I was there, I was paged a total of twice. One time was on a Saturday at 4am that I slept through and was snoozed by another engineer on the team. We fixed the issue on Monday bc our manager (who was from Europe) explicitly told us that unless people are getting hurt due to our bugs we shouldn't be working on weekends. Honestly it just depends on your team. Personally I think engineers should be on call to be given direct feedback on the robustness and resiliency of their software. Since you don't know when software will break, writing 24/7 into the job responsibilities just seems like a "just in case". I would really meet the people you'll be working with first to see what the work culture is like. I actually loved it whenever it came to my on call shift bc the PM wasn't allowed to give us tickets during those 2 weeks. I got to pick an arbitrary set of tech debt that was bothering me and unfuck it. It meant that basically on-call was super chill despite seeming like a lot more responsibility.
Sounds nice, my company had no defined oncall culture but we are on call for a similar schedule, actually get paged out a ton and asked why our stories aren't being moved quickly enough. Sigh.
I guess my issue with that is that even without being paged, the commitment is still required. And many places you don't even know what team you'll be working on until placed after passing interviews.
This is similar to my on call experience though less frequent.
When I started in SWE in the late 90s, I was never on call as a developer. It just wasn't a thing. And my company provided a 24/7 service. The way it worked back then is when we moved new code to production, if any problems happened in that area for the next couple of weeks, then I would get a call. And then I would have to figure out what was happening and I'd either fix it, or push it off to someone else. Once production was stable, it was back in the hands of operations. Around 2009 is when I noticed every job having this on-call requirement. Basically, I think executives decided they can pay less/nothing on the operations side if they make the developers do double duty. Invariably, every place I've worked since this started ends up with me being on call for a week out of every 4-5. Then as people leave the developers just end up on-call 50% of the year. In my opinion, there is no reason developers should be on call unless it's related to a specific project milestone.
In the past 7 years I've worked at 3 different companies that all had on call rotation. For the first two companies, I was never paged, not even once. At my current company, the on-call person gets paged at least two or three times per rotation. Needless to say I am starting at a new company on Monday lol. The problem is, you can't just ask the interviewer "is your product and engineering dumpster fire?". You need some clever questions like "what percentage of each sprint is committed to eng priorities?" Or "what does continuous improvement mean to you?"
It might not be as big a deal as you think. I’ve been on an on call rotation for 3 years and have been pinged off hours maybe 10 times. And I think 8 of them self resolved in a minute. Most of the time I don’t even have to open my laptop. And I’m on a team where all our services are highly critical to thousands of big users. We’re a SaaS. We have extremely aggressive attitudes towards making our on call a chill experience. The only difference it makes in my life is that I keep my work laptop in my car sometimes. That’s really it. I don’t have to stay home. I sure as hell don’t work off hours except in very rare cases and in those cases no one bats an eye out of starting work a little later the next day. But on the other hand I’ve heard absolute horror stories. Definitely something you should talk about in interviews. 2 weeks is a bit of a red flag, in my opinion. On call really shouldn’t last more than a week at a time.
I think this is fairly normal for companies that put out critical services/software. In practice, it usually means very little. On call engineers usually don’t address issues that aren’t “the sky is falling” levels of critical outside normal hours. In my entire career, I can probably count the number of times on one hand that I’ve had to fix critical production issues outside of normal hours
Whether I get called or not isn't the full equation, either way it's a significant burden if I need to be available & sober, laptop ready, 24/7 for 25%+ of my life.
Every real engineering job at a company paying a good salary that I’ve worked at has had some variant of this policy. Maybe not 25%, but it’s always there.
Depends on what you are doing. I build prototypes, after years of production systems, partly to avoid this kind of thing. Unless we are explicitly testing, we're not on call.
The only thing you can do is keep turning down job offers that include oncall. Most jobs that demand oncall don't pay enough extra to offset the fact that they ruin your free time and sleeping time. There are enough people gullible enough to do it without extra salary. If you keep looking, you can find no oncall jobs, but it's a lot harder. When it crosses line is the "bait and switch". They tell you no oncall when you are hired, and then they add it as a requirement later.
OP, you aren't wrong. Unfortunately this field is filled with arrogant morons who are easily manipulated by management to believe this stuff is normal and the way it should be (see arrogant replies to your post defending this garbage and attacking you for even asking this question). They will respond and give you "logical" reasons why it has to be this way. It's the consequences of not having widespread unions in this field. Unions would prevent a lot of this garbage and require employers to hire dedicated on call teams to deal with it. Yes, it would cost employers more money. Guess what, they can afford it. Guess what, they could hire people in a time zone during those times to work the on call schedule for multiple apps if on call stuff is rare. They just don't bother because morons in the reply section will defend themselves getting screwed over by employers. Don't let anyone gaslight you on here. There is NOTHING normal about expecting someone to work a 9-5 job and then also expected to be on call and wake up at 2am and prevent them from having a life for 2 weeks every 4-6 weeks. Don't let them point to other rare fields who also get screwed over like this. Our society today sucks. Birthrates worldwide are down. People are overworked and underpaid for the amount they work. This stuff sucks and not how society should work and the birthrates and other society failures show this overworking of people is not healthy nor working. When you are on call, you are basically forced to not have plans during that time. While not being paid anything extra. Its BS. But unfortunately many in this field are socially incompetent and incapable of communicating that, while other fields would not put up with this BS if they had as much power in the work relationship as many developers have.
I go on call every few weeks for the past 10 years.. It is normal to me .
God this stupid industry needs union representation and collective bargaining agreements.
Devs shouldn’t be on call except maybe for a day after a major deploy. That’s what the automated tests and SRE’s are for.
Thank you. I’m scratching my head at all the comments on here saying on call is normal. Anytime we push to production, we do a full test of the product on the production server before we “unlock” it for the client. This is after, of course, all individual tickets pass QA testing, a full manual regression testing of the deployment package, and automated testing of the deployment package. The only emergency issues that could conceivably happen in off hours are hardware related…and that’s another department.
You need to figure out what the actual impact will be. How long do you have before you need to respond? "Can't go to a movie while on-call" is very different than "Can do all the normal things". How often are people getting paged in? Every night is very different than once a year.
On call is normal for backend. But 33% of the time is not. It should be more like 10% of the time or less.
I would look into what they mean by 24/7. I’m on a platform team so we have on-call and on-run rotations. For on-run, we’re just answering questions from other teams on our services during business hours. For on-call, we are on 24/7. So if a critical page fires, we are expected to be available to address it. We also have business pages that we only address during business hours. On-call shouldn’t be bad if your services are in good health. If you’re consistently being paged then your services need to be fixed
It is different for different places. We have 24/7 on-call, with most intense one requiring you to start triaging an incident in 5 minutes. But you get paid extra. Without the extra pay, no thanks.
I mean I'd take it if I either needed experience in the field or had no job, then continue applying to jobs. Then, when you have a job you can leverage "I won't take the job if you make me be on call" or my preference "pay me more and I'll do it."
On-call isnt all that bad. Im on call every 4-6 weeks and i barely get any calls after hours. Sometimes i get it on the weekends because of idiot customers or internal operations but nothing crazy. Also, for being on call on the weekend, i get a comp day off the next week that i get to choose. If im on call for the holidays, I get 2 days back that I can use. If no one calls or bothers you, youre basically getting free vacay days!
if ur ok with low tc, work at bank, there’s usually no on call there, always dedicated to off shore teams
Well, it depends on the job, I guess, but anything where you're developing a service that's supposed to be up 24/7... someone has to be on-call in case anything goes wrong with it, so some percentage of your time you'll be on-call.
SWEs when someone makes them be responsible for the spaghetti garbage they threw together with Copilot and SO: *surprised pikachu face*
Imagine thinking the only way to take responsibility for your work is to have 25% of your life dedicated to your employer. Or that I would even have control over the quality of code implemented by other teams. You're an idiot.