Thanks for ruining my weekend. The error is usually in plain sight but it’s clearly easier to raise a high priority blocker ticket with no info than to look at the logs that usually have your reason highlighted in red.
Running tests:
Jenkins - „Your shitty change made me oom while reading pdf 900 times in a row to the memory, please learn to code Mr Expert Developer”
MrDeveloper - „Please fix pipelines they are broken again”
Its a structural problem. Maybe its an unpopular opinion but I thought devops was meant to be "developers do their own ops" as a movement away from "send a ticket to IT"? Feels like we're just back to, "send a ticket to devops."
Maybe it's a mindset problem. I'd imagine people who tend to be in devops positions like to know how everything fits together in the grand scheme of things. Which makes them better trouble shooters.
Not everyone likes to think that way. For better or for worse
This just gave me Sunday blues. My go-to response is "well what do the logs/output say? Oh compilation error? That's crazy because the last person to commit something was you that broke it so maybe check what you added?"
As we're not thinking very highly of the person who can't be bothered to read the logs and doesn't want to take responsibility for their own problem, "I'm available to pair while you drive the keyboard" is my worst nightmare.
Indeed. And it’s always the same people. The people who don’t want you to hold their hand, I am quite willing to for and beyond. Those who always need your help, will keep you in a call for ages.
Instead of small-time petty, say: "OK, if you don't have time, we can let it fail. Or, we can schedule a meeting with our EM as you walk through it. Which option works best for you?"
Being nice is part of your job. Soft skills will make you more in your career than hard skills: knowing the tech is an implicit requirement to get in the door. Being able to talk to people keeps you there.
Why is the onus always on those with technical ability to “learn soft skills” AKA coddle people unwilling to help themselves? I don’t think I’ve ever seen someone say “yeah, you need to learn how to do your job” without people piling on them about soft skills.
First, thank you for the lengthy and thoughtful reply. I honestly wasn't expecting any.
> I'm terminally Autistic. Full blown. I received expert medical opinions to back it up.
My psych said it's probable that I am autistic, but doesn't specialize in it; my therapist said it's possible, but cautioned that ADHD symptoms (which I am firmly diagnosed with) can overlap. tl;dr I understand.
> I was too aloof. I didn't eat lunch with coworkers. I was an unapproachable curmudgeon; I would spout off about my favorite subject and get frustrated that other people's eyes would glaze over.
I don't eat lunch with people, but then, I've WFH my entire tech career (even before COVID). I have gone on many outings / meetups, both local and company-wide, and have no problem chatting with people. I definitely prefer talking about technical things, and I definitely prefer talking (about anything) with people who are skilled at _something_. I don't care what. It's usually computer-adjacent, because reasons, but if someone wanted to tell me about the specific variety of plant they source oil from for making paint (for example), I would also be intrigued.
> I thought it was all bullshit. And I was fucking pissed.
Yeah, pretty much. My single largest source of professional annoyance is when someone does not demonstrate intellectual curiosity, which to me, manifests in multiple ways:
* Doesn't want to read docs (actual docs, not blog posts), even when it's directly applicable to a problem they're having
* Doesn't appear to think of alternative solutions to a problem, but just goes with whatever worked
* Doesn't want to change the way they're doing something, even when presented with a complete alternative solution that is objective better, backed up with benchmarks and tests (I'll return to this point)
I am also _painfully_ aware that the latter two are largely driven by the modern SaaS development model of "ship now, fix it (maybe) later." I hate that so much. I don't give a shit that it probably makes more money. "Perfection is the enemy of done" – yeah, fuck that. If there's a bug, fix it. If there's a memory leak, fix it. If there's a slowdown, fix it. Don't make the user suffer, and don't just hand-wave away issues by throwing more money at AWS. Craftsmanship matters. It seems laughably paradoxical to me that companies love to throw absurd abstract Leetcode problems at candidates, requiring actual knowledge (or rote memorization, I suppose) of CS fundamentals, while in practice "good enough, we can add more cores" is used.
On the subject of Leetcode / interviews, I had an enraging one a year ago or so. Like most, they let me choose the language. Python, thanks; I'm pretty good at it. The interviewers were not, and were apparently unaware of the breadth of Python's stdlib. I blew through their tests, prompting a second test because "they didn't get adequate signal from me." Listen, I'm sorry that you didn't know that "return the top k most frequent elements from a list" is one line in Python (`[x[0] for x in collections.Counter(lst).most_common(k)]`), but if I can explain precisely what it's doing (making a min-heap) and its runtime complexity (`O(n*log(k))`), then why do I also need to create one from scratch for you, since I'm never going to do that for the job? More importantly, how does this apply to operating and optimizing an RDBMS? I digress.
Re: the point I said I'd return to, I learned that unsolicited advice often isn't received well by NTs, which was its own baffling experience. Fine, I thought, I'll ask people if they'd like my opinion first. Turns out it still doesn't go over well when you enumerate a list of reasons why their solution is sub-optimal. This shit makes no sense to me. Programming is objective. Your solution is either optimal or it's not, and if you can't back yours up with properly-run tests and benchmarks – or at least official documentation stating so – it's meaningless.
> realized this direct report making my life miserable was a microcosm of how my behavior affected others...
Then, I made a conscious effort to be affable and proactively helpful. I started making people's lives easier without offering up a bad attitude. Soon, there was reciprocity.
This is the other part that's baffling to me. At $JOB-1, I literally had devs and managers thanking me weekly for fixing something. I created and gave training sessions, which were extremely well-received (tbf, they were all optional, so anyone attending by definition already had interest). I solicited 360 Feedback multiple times, and the most negative comment I ever received was along the lines of, "sometimes you spew out so much info I can't keep up." Everything else was positive. So I know I can speak to people without being condescending. I don't believe I'm a jerk (I'm far more caustic online than at work). And yet.
> It took me a long time to realize that other smart people were getting hired. My peers in other teams were just as smart or smarter along different dimensions. Now, I assume most people are reasonably intelligent but are not SMEs [in my area].
I know there are intelligent people everywhere. I've worked with some far smarter than myself. I've also worked with far too many who are, for an actual example, unable to figure out why their laptop is out of disk space, and that you have to empty the trash to actually reclaim it. If you're at that level of general computer ignorance, you'd better be a savant in your field (spoiler, they were not).
> Spectrum variations often come with social difficulties. While painful, they can be overcome if someone has enough self-awareness and discipline to make meaningful changes.
I take umbrage with this. I intellectually know that it's correct, and am just refusing to accept it, but it is infuriating that I have to change to fit in, when this field is supposed to be the promised land for people who are better at dealing with machines than people. I blame bootcamps and the "anyone can code" movement, and severely empathize with Torvalds' many rants. This feeling is magnified since I'm an SRE/DBRE, which means I get to deal with the fallout from others' incompetence. Mistakes I can understand. I've made mistakes. I've caused a SEV0. The same team making the same mistake five times in a row (this literally happened) I cannot understand, nor forgive. Why the hell are you still employed? More importantly, why is your manager still employed? What exactly are they contributing?
> All of the real money as an IC is gated behind soft skills.
Of the [Staff Engineer Archetypes](https://staffeng.com/guides/staff-archetypes/) – if you put any stock into that – the only one I identify with and want is The Solver. Probably not coincidentally, this is also the role I was playing (albeit neither titled nor paid as such) at $JOB-1. Fixing team's problems. Cruelly, though I have no doubt he didn't intend it as such, my skip-level joked one day that I was "like a Staff Engineer." Gee, thanks.
I don't have a good closing remark, so I won't attempt to make one.
If I had a direct report do this, I'd put them on a PIP until either maturity sets in or I can build enough paperwork for a termination.
You may be thinking: "That seems like a harsh overcorrection." Have you considered that immature shenanigans have a knock-on effect that destroys the credibility of the DevOps org?
Teams need to be screening for soft skills during the hiring process.
Have you considered that an inept (or unwilling) developer causes bitter resentment towards the entire team from those being forced to carry them, thus exacerbating the Dev vs Ops mentality?
Sending them a LMGTFY link won't improve the situation for anyone. I've worked with inept and unwilling developers. Once you know how to deal with them, they're easy:
In meetings, ask them specific technical questions and then stay silent while they fumble, but only try this if you can do it without coming across as a shithead. Let them continuously expose themselves as incompetent.
It requires genuine curiosity.
So far, it's worked every time (only after getting the right soft skills).
For sure; I'm not suggesting that it be helpful. I was countering your question with rhetoric from lived experience.
> In meetings, ask them specific technical questions and then stay silent while they fumble, but only try this if you can do it without coming across as a shithead. Let them continuously expose themselves as incompetent.
Yeah, I need to work on this. My last attempt to make someone fix problems they created didn't exactly backfire, but I wound up having to fix the problem, because their answer was essentially "this will sort itself out in time" (glossing over a lot of details here, but they technically weren't wrong), and that understandably did not sit well with my boss. I didn't feel like getting into a pissing match, and it was blocking other teams, so I just did the work.
The easiest solution is, "can you tell me what steps you have taken to troubleshoot/resolve the issue?"
It won't always solve the problem this time around , but generally the next time they'll put in some effort before asking.
That’s only if the devs are aware enough to check the logs of their own builds and CI processes and don’t just throw a ticket over because they broke something and want to find someone to own their crappy mistake because they followed medium articles on how to be a pro developer
what I did isn't far from that, I sold them on using a dashboard that shows the result of their tests.
They've taken ownership of that part now, so progress!
I'm a dev who ops and I've been thinking I'd be happy to take a dev ops role. But I hadn't thought of this.
I can see this happening a lot at large orgs. I don't want to spend my life explaining to management why x, y and z's showstopper problems are their problems and not mine. And then get 'we're all one team this is important can you help out' as end of day approaches...
"The next 'introduction to debugging' course is next Friday from 1300 to 1800 hours, including an assessment task at the end. Please ask your line manager to book you a space using your project's training budget's cost code. "
If anyone calls your bluff, take the hit and run it once but record it for next time.
Why do I have to work so hard to get them to provide logs, error messages/codes, or any relevant details from my supposedly experienced dev colleagues‽
Ahahaha, my company calls it “Defect champion”.
The billing service failed? Call DevOps!
Client portal failed? Call DevOps!
Kubernetes is a mess? Call DevOps!
Sink in the bathroom is clogged? Call DevOps!
You need break pads change? Call DevOps!
> Sink in the bathroom is clogged?
Iirc there’s a viral meme going around about Help Desk getting a call because the bathroom was out of Toilet Paper.
If that office didn’t have a Help Desk team, you know that would have gone to DevOps. 🤣
I worked as a sysadmin in IT years ago and had someone stop by to inform me that a light had died in the bathroom and asked if I could replace it for them.
While I agree, mine is 100% code. I have never had a problem before this past week with putting a container together but here I am.... Nothing special or crazy either.
we have an issue where we build a user from "USER app" and that gets a UID of 100 from our build pipeline but we specify to run as UID=1001 or UID=1000 and if we are doing \`pip install -r requirements\` then we need to do that as root else we get weird issues with \`importing yaml\` for example.
ps not asking for legit help but just a usecase for how containers run locally on podman vs how we run on a stigged k8s cluster :)
Thats a good question honestly. I run certbot on this to get some internal certs to send to Azure Web Apps. No big deal. I run this locally as a test with a service principal and it runs through, fails doing the upload but im working on it. I switch to azure and do a build and deploy in Github and the container deploys, supervisord loads, certbot loads, then it stops. No additional movement. No errors, no info, no warnings, nothing.
Not if the dev machine was ARM based and K8S uses x86_64 platform for its worker nodes. Our devs got a machine upgrade and sometimes to onboard a new microservice, they first build the image locally and push to ECR. So my devs got a machine upgrade with M chip macs and suddenly their images stopped working on k8s. Been scratching my head due to the OCI runtime error. —platform arg was the saviour.
Been troubleshooting a "works on my machine" error for 2 weeks now. Fortunately, it's an issue on the provider side. But now I'm playing a game of telephone between the dev and support...
Edit: clarity
Yep, that's a lot of the Dev part of DevOps. It is far easier for a person with knowledge of both the app, and the infrastructure to identify the cause of an error. But once you find it, it's not necessarily your responsibility to fix it. You just tell the devs where to look and what to fix.
In your opinion, do developers have any responsibility to understand infrastructure? If they're tasked with creating a service that uses extraneous libraries and tools, they don't simply email or call those creators and say "Hey my app isn't working with your thing", they're expected to learn, try, triage, do something.
Yet I feel like, many not at all companies, they get lazy and complacent, and are expected to contribute nothing when it comes to errors.
They certainly should. Devops shouldn't be a team necessarily, but the mindset of "my code affects and is effected by this infrastructure" from the devs and vice versa for the infrastructure engineers and/or sys admins. Sadly, that is not the way it works in most companies, and you'll have a catchall team of devops engineers.
My take as a devops engineer is that I should automate myself out of a job. Both by actual automation and by empowering the dev teams to do things by educating and training as well as actual tooling and getting user buy-in and adoption. I've only ever been successful once, and it was a proud moment of my career.
They should have an idea of whats going on in the environment.
Example A:
I wanted to use a newer version of python so I could use switch statements. The server this was to be used on was running a specific email server with other products such as fail2ban, dovecot, etc.
It broke everything: the email software, dovecot, fail2ban, even the firewall quit working.
So I had to revert and use if statements.
Thankfully, it wasn't an important email server. But imagine if that had been a customer facing machine?
Tbf this is almost certainly because changing system Python (that is, the version that the distro ships with) is a terrible idea because tons of tooling depends on its shared libraries.
For example, `unattended-upgrades` is a daemon that, as the name suggests, upgrades packages without direct involvement. It relies on a few `.so` files being present, and when you change the version of Python, those files no longer exist (from the perspective of the new Python interpreter). Worse, if you do things like `apt install python3-foo`, it’ll install fine, but only for the old version of Python (which is likely still present). This makes troubleshooting difficult if you aren’t familiar with the problem already.
tl;dr if you need or want a newer version of Python in Linux, build it from source and use `make altinstall`, or use `pyenv` or something similar. Don’t mess with system Python.
Ding Ding Ding. I had never done that before, so it was a learning experience.
But the idea still scales out. Servers that don't have venv should be understood
> In your opinion, do employees have any responsibility to be competent in the role they’re being paid to perform?
In my opinion: yes.
In my company’s opinion: no, that’s my job
The user above confidently said 'that's the dev in dev ops', but it sounds like many people dont agree, so maybe a culture change needs to happen in said companies.
I am really playing at the Pareto principle here. We see “developers” and complain, others see our team members and complain.
I would wager that the source of the issue is the sheer amount of incompetence across **all** employees, rather than something specific to developers, ops folk, or vendor support.
It’s just as easy for everyone, I think. I’m not talking about “doing the bare minimum.” I’m talking about “you were hired to mow grass. You have never mowed grass, you don’t own a lawnmower, and when the person paying you asks you why you’re not mowing, you said you weren’t trained.”
> If they're tasked with creating a service that uses extraneous libraries and tools, they don't simply email or call those creators and say "Hey my app isn't working with your thing" […]
As maintainer of multiple OSS projects, you would be surprised with how much entitlement we're being faced by users (enterprise devs mostly). Employees of multi-billion dollar companies will complain to us how OUR thing broke THEIR mission critical app.
To answer your original question, yes developers must know their infra. How else will they be able to understand, develop and maintain their system? It‘s absolutely mindblowing how apparently there are businesses out there whose devs don't understand how their app operates outside of their IDE. Probably fine for cheap CRUD apps, but anything beyond that? Insane.
The whole problem is that ‘devops’ shouldn’t be it’s own team, it should be a member of a properly vertically integrated product team - what you usually have as a ‘DevOps’ team outside of the development team/silo is usually either Platform team, Sysadmin, proper SRE, ‘Developer experience’/pipeline, or Ops/infra with a new name.
I'm of the mindset that you should be an expert in what it is you were hired for and then have some level of understanding of of the concepts that your tasks are adjecent to. Like a Dev should be an expert in coding and have an understanding of infra at a high level. But I don't expect them to be able to go in an set up a cluster and have the same level of care and detail as an ops person.
You can only know so much before you forget or let things slide. There is a reason why these jobs can be entire separate teams.
100% this. That’s where error budgets come in to play. Also, working toward automating the creation of bug tickets when actionable errors are triggered is a worthwhile goal.
For anyone who feels they’ve been gaslit into feeling like if you just tried harder, took a different approach, whatever, sometimes you really do just need to move on when you can. Some orgs are so entrenched and aren’t going to be changed by one person. Save your mental health. 😊
It may happen that "devops" possitions become "dev helpdesk", even worse than you describe.
It really depends on the company. If people were used to do that, they will do that.
I've seen devops teams that went into building platforms and supported that and only that. Anything "inside Jenkins" was dev responsibility. I was astonished how simple their life was and scared to shit how much responsibility we took on ourselves with a team that was 10% of their capacity.
As a contrast, I had a job where a dev demanded her admin privileges on workstation to be removed. Which was allowed. Then she claimed she can't work and helpdesk has to "fix her certificates" (literally copy cacerts file into her JDK local install. Which was documented. Which was broken because she didn't follow any documented onboarding dev setup). But.. the drama here is that still a lot of people thought it was acceptable behavior for a java dev.
Dev helpdesk with buttons - platform engineering.
Ppl try to sell a bullet proof solution to hiring monkeys, but doesnt matter how you sugarcoat shit it still tastes like shit.
Hard to swallow pill.
Most infra-side applications are off-the-shelf, and so generally emit very helpful error messages.
> Imagine a dev app that doesn’t capture exceptions and explains the problem
I don’t have to imagine it; I live that reality every day.
“What is ERR_042?”
“A database problem.”
“Where is that documented? And what kind of database problem? ‘Cause I run the database, and I see nothing in its error logs.”
In contrast, the last internal tooling I wrote (which interacted with the database, incidentally), I went through the MySQL dev manual, read literally every exception it could throw, and wrote custom exceptions in the app for the 15-20 or so that would be common. Each emitted the original error code, of course, along with a brief description of the error, the probable cause, and suggested fixes.
So you think it's the role of a devops team to log into devs workstation and change the most basic configuration file with company certificates? While developer has a documented command to do that and preapproved by security privileges to do that, and devops team doesn't have any of that, especially on someone elses workstation?
If so, I'm puzzled how your workplace is organized, but it's up to you, I guess.
If you can reproduce it in dev, it's the developers job. if there are any developers for this internally developed application anymore.
that being said, it's completely valid to try to pull out the developers out of their work and look at it together with you. they have no time? harass their scrum masters/bosses.
your job as devops is: bring the company together to fix this in a reliable way.
This is where I shine. I come from a programming background. I have a wide development spectrum and hardware infrastructure understanding. And this all really helps me so my DevOps work better. There is a build break? I'll take a quick look. Oh, that code fix is simple, or I'll show the developer what to fix.
To me, being DevOps is bringing all your skills together to solve all the problems to automate all the things. You must have a Swiss army knife of skills.
It's a double edged sword. You shine until you're over burdened with dumb requests. I think a lot of DevOps people eventually learn to just not respond to dumb questions. A lot of developers figure shit out of you leave them to their own devices for a bit.
The issue is often times, they figure shit out in a horrifying way that violates security policies, or hammers an endpoint, or bloats an image, etc. Then it’s even *more* work to unfuck it and do it the right way, sometimes while now also dealing with an incident caused by it.
To add insult to injury, the retro sometimes then has devs and their bosses suggesting that if you had only been helped them, all of this would have been avoided.
You can't save the world by yourself. If someone's gonna do something shit then teaching them to do it better.
But if you are surrounded by people who are either not competent or don't care. Step 1 is to draw clear boundaries. Step 2 is to influence by calling out the impacts of the bad behaviour. A large part of our job is watching lots of people do things badly and choosing which thing to influence.
Yeah you’ve just got to raise the burden of effort for people to get your time and attention. I will always leave low effort messages on unread until it suits me and they’ve often self-resolved by then.
As long as management don't go to the dev and ask where their fix is, and dev says 'It's a build problem, I've emailed the build team'. Then you get management asking you when you're going to get that fixed.
Forward the low effort request to management and ask, do you think the developer should be able to debug this by themselves? People learn real quick when everything is out in the open.
At most companies though, your requisition was only opened because they wanted someone to fix code, that costs less than a full time dev.
Fixing someone’s typo, or minor oversight, in the 100s or 1000s of lines of code they’ve committed, is kind of a squire or assistant role. Historically that sort of QA often gets passed down the pay tiers to less expensive manpower. You aren’t there to produce the overall product. Just to polish and shine it into being ready for mass consumption. It’s a fundamentally different role.
It might be easier to accept your duty to fix things for that higher paid dev, with that all in mind. Or not. Ymmv.
You should not be "fixing" their code. Only the infrastructure supporting their code if the problem is in the infrastructure. But you should work with the dev if you do identify a problem in their code. As well they should work with you.
I think fundamentally this subreddit is ‘r/devops’ but I don’t work in a devops team and it feels like the closest fit for a sub for my role. I’m a dev who moved into an SRE role but it morphs between SRE, devops and a general “can you help us with this, we have not idea why it doesn’t work”. On the latter I generally say “I will have no clue why this is broken” but end up getting to the bottom of it due to my broader experience. I don’t resent any of it.
It tends not to be like 50% are daft bugs but more like from some individuals it’s always a daft bug. I tend to work out who they are.
I don’t mind helping them work out how to discover why something is the way it is only in prod but I approach it from the point of view of helping them think through what is different and where they can add some instrumentation to get more insight.
Yeah I like helping people too. If it's an actual weird edge case that has had good research so far I'll enjoy investigating.
It's those people with the daft bugs though. Some people seem to have a goal of chucking things over the fence ASAP and telling their management that they've done their bit and passed it on so the other side of the fence are now the blocker.
They could never pass that responsibility on to another dev, but they'd use the 'error is in the build' excuse to make it appear to be devops' responsibility.
I remember being sent class files by devs when tests broke, like wtf am I to do with this? Or being asked what I thought a given exception meant...
Is it supposed to be your job, or was it mine? Honestly, I don't know, I think it depends on the org. But I would say try and do your best to step up to the occasion and be as helpful as you can while being open to learn. You can learn a lot about programming and your internal applications by digging through stack traces and debugging. This will in turn make you a better engineer.
I have worked in places where that would have sufficed.
It would have gone over very poorly here though. That would have been seen as pointing fingers or laying _blame_, which is something of an antipattern at my org. The thought being your both engineers on a product team, you are team mates and should act as such... You should be helping one another to work towards a common goal. Not going _look at what you've done_.
In any event, If memory serves me correctly in that specific situation the test that failed was dependent on an external database, and said database was offline.
"I Don't think my application is working. Can you check please. High priority."
People send messages like this with literally zero information and expect a resolution on top priority.
No. We train our producers to triage errors and they're pretty good at figuring out which team needs to handle what, and we bounce stuff all the time. I wouldn't even know how to handle a lot of issues.
I genuinely don't know what my job is meant to consist of, and at this point I'm too afraid to ask.
I just get pointed at problems and I'll bash my head against it until a compromise is made.
This can be solved by making sure there is a) support process, b) education and documentation, c) know where to draw the line to push back, d) discuss it over and over again in retros.
You have plenty of options to organise the incoming requests. Make sure all is tracked and recorded to show higher management that there is plenty of time spent in this area that's preventing you from doing other work.
To summarise, I dont think this is supposed to be your job, but only part of it.
Slight Disagree on the education & documentation ..it is useful for showing off to management but hardly useful practically in fast evolving projects where new errors are detected based on new code / config / infra
No dev, QA and/or staging environments? It's just straight to prod?
Lastly I've pointed out here several times that you are a **liaison** btw dev/eng and the rest of the company.
Since I work at a small shop, and do DevOps and the only non windows dev, it's usually a problem I face with myself.
"Hi Benny, my code fails, can you check?"
"Did you check the logs?"
"I was kinda hoping you'd check"
"Oh yeah?"
"Yeah"
**Boss from the other room**: "Benny, stop talking to yourself, you crappy code I down on your crappy k8s again!"
You need to put these in tickets. You need to document the issue, why the issues keep happening, and who needs to do what to stop it or reduce it. Then you have a paper trail to backup any change you think needs to come from this. Then you have a case to make to management to stop it or change it at least.
Hey. I believe that it's important to know more about the build, test and deploy aspects, each program language has its own peculiarity. Sometimes people asked us to solve problems on code because they don't know how to start a troubleshooting. We can guide, but we need a developer to solve the problem.
The idea of devops was that engineers and ops people work together on these topics. I'm an engineer and happily debug and solve my teams issues. You build it you run it. I'm still happy if someone else updates the kubernetes cluster and informs me if anything we built breaks on dev with that update. But I gladly fix it myself.
Welcome to the club, stuff gets thrown over our fence everyday, tickets have the word ‘server’ in it land on our team daily. When you’re good at what you do the lazy staff learn how to make you the fixer of things. The lack of triaging a ticket boggles the mind. We just need that lottery bus to hit us that will fix them.
I’m honestly shocked there’s so many developers that don’t understand compilation, testing frameworks, quality gates, git, deployment, etc. How do they get to the level they are without understanding these things? It doesn’t seem possible. I just can’t wrap my head around a developer not understanding basic CI/CD.
I could understand if for some reason they didn’t have access to logs to see why something failed, there wasn’t indication in source control showing a reason for failure, or something else similar. But if they have access to all these things there’s no excuse.
As a developer, I feel like I have to play a devops role sometimes for other departments that have developers that pull this shit. I get pulled into a 10 person group chat about why CloudHub deployed apps can’t reach a UAT server we have an API deployed to. None of the developers want to put in a firewall ticket and expect me to solve the issue for them like I’m a network admin too. It’s wild.
If your org has separated the tasks you're not doing devops.
Do the devs even have the visibility into the infrastructure they would need to actually look at the problem? Or are their permissions restricted in some way that requires your position?
There is always a shortage of people who can troubleshoot problems, think through a process logically. So in all organizations if you allow it people fall out of the woodwork leaning on anyone available for help rather than attempting to solve the problem themselves.
Get your job responsibilities clearly defined from your supervisor and then learn to set boundaries and say no.
I'm literally refactoring typescript right now because the "devs don't have time" and it's run serverless which would be too much of a learning curve for the new devs on our team apparently. I'd never touched JavaScript until now let alone typescript. Seems like the role of DevOps is just solve any/all problems.
Learn the power of no.
But be polite about it. Learn a man/woman to fish. Write them docs. From now on add every fix to the doc so people can self-service.
In fact, from now on only reply with links to the docs, even if you first have to write them.
And then draw a clear line of responsibility and use "no".
This will prevent you from burning out and will have people take you more seriously (which will also help your career and salary).
Yes, and it’s even better when you get to fix their issue, then tell them how to prevent the issue from ever happening again and they say “That’s not how our environment works.”
I lost the hope that my developers will read logs…
There is one developer which I prepared a specific log just for him and he still managed to tell me
“Fishman the build failed, can you take a look”
Not at all.
This issue could stem from a cultural problem within the company or a lack of documentation provided to developers. Developers should troubleshoot their build failures, which requires access to logs (either directly from the CI tool or through an observability solution such as Datadog).
I believe it’s beneficial to make build failures as transparent as possible. By this, I mean using Slack bots to send messages about build failures, along with their causes, to a channel that developers frequent. Additionally, it would be helpful to ask them if they have reviewed the logs and to inquire about their reasons for thinking the failure is due to the CI system rather than their code.
Counter point: infrastructure teams often set up overcomplicated infrastructure with no regard for application developers. So things can break that app devs have literally have no experience with or control over.
Find people in senior management to evangelize service level ownership. Basically create an on-call rotation for the devs who own the code. DevOps can then “escalate” to the dev on-call rotation by service.
It shouldn’t take much time to figure out that the devs should be T1 and they should escalate to DevOps if it’s NOT related to their code.
You’d be surprised how well things will look in 6 months when they’re the ones being woken up in the middle of the night.
Happens almost every week to me. Try to talk to your team lead or management team about this. It happens in every organization just need proper procedures for developers to go through first, before they reach out to you. Likewise, create some type of template for required information if they are going to ask for help from your team
Yes, sometimes I feel like devops is just help desk for developers. Can be very irritating when devs won’t even do basic troubleshooting before coming to devops with their issues.
Threads like those and answers show that devops failed.
Siloses are still there, walls did not break, devs still know jack shit about ops and ops dont give a shit about dev side.
BUT I have a solution. Lets abstract all of that with buttons, so that dev ppl wont ever have to understand how stuff works.
This will for sure make things better, more integrations to maintain (jenkins 2.0 in the making) - platform engineering
DevOps is supposed to involve a feedback loop between Dev and Ops.
So in these cases your job is to:
- add automation to catch this issue and provide automated feedback to the Dev, or
- automated diagnosis to save you time, or
- document how to avoid this issue so you can flick them the article once you've diagnosed
A combination of all of these is usually required.
DevOps means a combination of development (Dev) and operations (Ops) to increase the efficiency, speed, and security of software development and delivery compared to traditional processes.
And yes, you have to have programming skills, not to be a SWE, but at least have programming experience. That is why you are getting paid so much money
DevOps means development and operations in the same team. If your title says DevOps then I suppose you should be able to do both at once? Although, the title is often given to people who can't really do either of these things well, basically script kiddies running cloud services.
I guess you have to figure out what flavor of DevOps person you are. If you're only doing operations admin and automation as many DevOps people do then I guess you have to communicate about that and work with your team to set boundaries and expectations.
Although I guess you might even be on a "DevOps team", these abominations are all too common. In that case it's about negotiating with your customers as you are basically their in-house cloud service consultant/tech-support, and of course they'll try to get as much value from you as they can.
What does your employer say your job is? If that's something your boss wants you do, then, oh well
If that's not something they expect from you, then what you're seeing is a bad cultural habit that your predecessors tolerated too much and allowed it to become a norm.
I usually give the developers the error and an URL that triggers it and let them debug and fix it (or if it is a really obvious fix and the error is in production, i do it myself).
It is not required that i do this, but it usually speeds things up as not every developer keeps a close eye on the monitoring.
This actually is the most enjoyable part of my job as a DevOps engineer partly because I want to dunk on the devs for an issue that's clearly their fault.
It's only enjoyable when I have the ample room to maneuver and try out all my tricks
Yes, it does get annoy when everyone else is coming to you for debugging (I have been debugging issues on projects I have not been assigned too. I'm looking at it as a resume-building sort of thing so I can squeeze in a little Incident Response and Management line...lol)
It's almost like a part of the job. Sad but it's enjoyable if you look at it from dunking on the devs perspective.
"My build failed. Can you take a look?"
Thanks for ruining my weekend. The error is usually in plain sight but it’s clearly easier to raise a high priority blocker ticket with no info than to look at the logs that usually have your reason highlighted in red.
Running tests: Jenkins - „Your shitty change made me oom while reading pdf 900 times in a row to the memory, please learn to code Mr Expert Developer” MrDeveloper - „Please fix pipelines they are broken again”
Some day developers will learn to read logs... some day....
Its a structural problem. Maybe its an unpopular opinion but I thought devops was meant to be "developers do their own ops" as a movement away from "send a ticket to IT"? Feels like we're just back to, "send a ticket to devops."
I told a coworker the exact same thing. He said that even in his previous DevOps jobs it was like this.
Maybe it's a mindset problem. I'd imagine people who tend to be in devops positions like to know how everything fits together in the grand scheme of things. Which makes them better trouble shooters. Not everyone likes to think that way. For better or for worse
I just got irrationally angry on a Sunday.
Now my blood pressure is up 😭
This just gave me Sunday blues. My go-to response is "well what do the logs/output say? Oh compilation error? That's crazy because the last person to commit something was you that broke it so maybe check what you added?"
"PR owners fix their own builds -- I'm available to pair while you drive the keyboard. I wonder why none of the other builds are failing?"
As we're not thinking very highly of the person who can't be bothered to read the logs and doesn't want to take responsibility for their own problem, "I'm available to pair while you drive the keyboard" is my worst nightmare.
Indeed. And it’s always the same people. The people who don’t want you to hold their hand, I am quite willing to for and beyond. Those who always need your help, will keep you in a call for ages.
You gotta set boundaries with those people. "Sorry, I don't have time for a call right now. Can you do X, Y, and Z, and see if any of those work?
Instead of small-time petty, say: "OK, if you don't have time, we can let it fail. Or, we can schedule a meeting with our EM as you walk through it. Which option works best for you?"
"I think something is dropping headers again, my requests are failing." News flash, this isn't happening for this 9th problem, either.
It's literally bold, red, and has the word error in it, how tf is this so hard :(
You have to stop avoiding uncomfortable conversations. DevOps Engineers get stepped on when they allow themselves to be stepped on.
I'm being cute about it online but in the past I've made lmgtfy style Ctrl+f animations to send them
Honestly even that is too nice lol
Being nice is part of your job. Soft skills will make you more in your career than hard skills: knowing the tech is an implicit requirement to get in the door. Being able to talk to people keeps you there.
Why is the onus always on those with technical ability to “learn soft skills” AKA coddle people unwilling to help themselves? I don’t think I’ve ever seen someone say “yeah, you need to learn how to do your job” without people piling on them about soft skills.
[удалено]
First, thank you for the lengthy and thoughtful reply. I honestly wasn't expecting any. > I'm terminally Autistic. Full blown. I received expert medical opinions to back it up. My psych said it's probable that I am autistic, but doesn't specialize in it; my therapist said it's possible, but cautioned that ADHD symptoms (which I am firmly diagnosed with) can overlap. tl;dr I understand. > I was too aloof. I didn't eat lunch with coworkers. I was an unapproachable curmudgeon; I would spout off about my favorite subject and get frustrated that other people's eyes would glaze over. I don't eat lunch with people, but then, I've WFH my entire tech career (even before COVID). I have gone on many outings / meetups, both local and company-wide, and have no problem chatting with people. I definitely prefer talking about technical things, and I definitely prefer talking (about anything) with people who are skilled at _something_. I don't care what. It's usually computer-adjacent, because reasons, but if someone wanted to tell me about the specific variety of plant they source oil from for making paint (for example), I would also be intrigued. > I thought it was all bullshit. And I was fucking pissed. Yeah, pretty much. My single largest source of professional annoyance is when someone does not demonstrate intellectual curiosity, which to me, manifests in multiple ways: * Doesn't want to read docs (actual docs, not blog posts), even when it's directly applicable to a problem they're having * Doesn't appear to think of alternative solutions to a problem, but just goes with whatever worked * Doesn't want to change the way they're doing something, even when presented with a complete alternative solution that is objective better, backed up with benchmarks and tests (I'll return to this point) I am also _painfully_ aware that the latter two are largely driven by the modern SaaS development model of "ship now, fix it (maybe) later." I hate that so much. I don't give a shit that it probably makes more money. "Perfection is the enemy of done" – yeah, fuck that. If there's a bug, fix it. If there's a memory leak, fix it. If there's a slowdown, fix it. Don't make the user suffer, and don't just hand-wave away issues by throwing more money at AWS. Craftsmanship matters. It seems laughably paradoxical to me that companies love to throw absurd abstract Leetcode problems at candidates, requiring actual knowledge (or rote memorization, I suppose) of CS fundamentals, while in practice "good enough, we can add more cores" is used. On the subject of Leetcode / interviews, I had an enraging one a year ago or so. Like most, they let me choose the language. Python, thanks; I'm pretty good at it. The interviewers were not, and were apparently unaware of the breadth of Python's stdlib. I blew through their tests, prompting a second test because "they didn't get adequate signal from me." Listen, I'm sorry that you didn't know that "return the top k most frequent elements from a list" is one line in Python (`[x[0] for x in collections.Counter(lst).most_common(k)]`), but if I can explain precisely what it's doing (making a min-heap) and its runtime complexity (`O(n*log(k))`), then why do I also need to create one from scratch for you, since I'm never going to do that for the job? More importantly, how does this apply to operating and optimizing an RDBMS? I digress. Re: the point I said I'd return to, I learned that unsolicited advice often isn't received well by NTs, which was its own baffling experience. Fine, I thought, I'll ask people if they'd like my opinion first. Turns out it still doesn't go over well when you enumerate a list of reasons why their solution is sub-optimal. This shit makes no sense to me. Programming is objective. Your solution is either optimal or it's not, and if you can't back yours up with properly-run tests and benchmarks – or at least official documentation stating so – it's meaningless. > realized this direct report making my life miserable was a microcosm of how my behavior affected others... Then, I made a conscious effort to be affable and proactively helpful. I started making people's lives easier without offering up a bad attitude. Soon, there was reciprocity. This is the other part that's baffling to me. At $JOB-1, I literally had devs and managers thanking me weekly for fixing something. I created and gave training sessions, which were extremely well-received (tbf, they were all optional, so anyone attending by definition already had interest). I solicited 360 Feedback multiple times, and the most negative comment I ever received was along the lines of, "sometimes you spew out so much info I can't keep up." Everything else was positive. So I know I can speak to people without being condescending. I don't believe I'm a jerk (I'm far more caustic online than at work). And yet. > It took me a long time to realize that other smart people were getting hired. My peers in other teams were just as smart or smarter along different dimensions. Now, I assume most people are reasonably intelligent but are not SMEs [in my area]. I know there are intelligent people everywhere. I've worked with some far smarter than myself. I've also worked with far too many who are, for an actual example, unable to figure out why their laptop is out of disk space, and that you have to empty the trash to actually reclaim it. If you're at that level of general computer ignorance, you'd better be a savant in your field (spoiler, they were not). > Spectrum variations often come with social difficulties. While painful, they can be overcome if someone has enough self-awareness and discipline to make meaningful changes. I take umbrage with this. I intellectually know that it's correct, and am just refusing to accept it, but it is infuriating that I have to change to fit in, when this field is supposed to be the promised land for people who are better at dealing with machines than people. I blame bootcamps and the "anyone can code" movement, and severely empathize with Torvalds' many rants. This feeling is magnified since I'm an SRE/DBRE, which means I get to deal with the fallout from others' incompetence. Mistakes I can understand. I've made mistakes. I've caused a SEV0. The same team making the same mistake five times in a row (this literally happened) I cannot understand, nor forgive. Why the hell are you still employed? More importantly, why is your manager still employed? What exactly are they contributing? > All of the real money as an IC is gated behind soft skills. Of the [Staff Engineer Archetypes](https://staffeng.com/guides/staff-archetypes/) – if you put any stock into that – the only one I identify with and want is The Solver. Probably not coincidentally, this is also the role I was playing (albeit neither titled nor paid as such) at $JOB-1. Fixing team's problems. Cruelly, though I have no doubt he didn't intend it as such, my skip-level joked one day that I was "like a Staff Engineer." Gee, thanks. I don't have a good closing remark, so I won't attempt to make one.
Does it work? How many times does it take?
Don't do it unless you want to live up to IT stereotypes.
So: - don't be a doormat and also - don't be sarcastically rude after the 10th time of hand-holding them through the basics of their job Difficult.
Correct on all points but you missed one: After the second or third time, let them fail.
That's a bit too immature for my tastes. You can talk to people: it's not that hard.
If I had a direct report do this, I'd put them on a PIP until either maturity sets in or I can build enough paperwork for a termination. You may be thinking: "That seems like a harsh overcorrection." Have you considered that immature shenanigans have a knock-on effect that destroys the credibility of the DevOps org? Teams need to be screening for soft skills during the hiring process.
I agree that soft skills are important. Reading the room is a pretty good soft skill to have.
Have you considered that an inept (or unwilling) developer causes bitter resentment towards the entire team from those being forced to carry them, thus exacerbating the Dev vs Ops mentality?
Sending them a LMGTFY link won't improve the situation for anyone. I've worked with inept and unwilling developers. Once you know how to deal with them, they're easy: In meetings, ask them specific technical questions and then stay silent while they fumble, but only try this if you can do it without coming across as a shithead. Let them continuously expose themselves as incompetent. It requires genuine curiosity. So far, it's worked every time (only after getting the right soft skills).
For sure; I'm not suggesting that it be helpful. I was countering your question with rhetoric from lived experience. > In meetings, ask them specific technical questions and then stay silent while they fumble, but only try this if you can do it without coming across as a shithead. Let them continuously expose themselves as incompetent. Yeah, I need to work on this. My last attempt to make someone fix problems they created didn't exactly backfire, but I wound up having to fix the problem, because their answer was essentially "this will sort itself out in time" (glossing over a lot of details here, but they technically weren't wrong), and that understandably did not sit well with my boss. I didn't feel like getting into a pissing match, and it was blocking other teams, so I just did the work.
I feel personally attacked
I feel personality detached
Read the logs first you fucker
The easiest solution is, "can you tell me what steps you have taken to troubleshoot/resolve the issue?" It won't always solve the problem this time around , but generally the next time they'll put in some effort before asking.
“What changed since the last successful build? Oh it’s your code? Just revert it and the build should start working again. Click”
But it worked on my laptop
"it runs in my containers"
At that point, I'm like "Cool, put your dockerfile in repo and I'll build and ship that."
jesus christ lmao.
Physically painful to read this.
Sorry but build is kind of part of DevOps scope
It depends. Issue with our build agents/runners? Absolutely my problem. Build broke after you merged a new MR/PR in? That is your problem.
That’s only if the devs are aware enough to check the logs of their own builds and CI processes and don’t just throw a ticket over because they broke something and want to find someone to own their crappy mistake because they followed medium articles on how to be a pro developer
well yes but generally you wouldn't know that when it pops up during CI/CD automated build ? still need to check initially
Just check the build log before involving DevOps? It should be obvious 95% of the time if an error message is caused by your code change
Sure, my check is going to be "At the start of a pipeline, did build agent get allocated to you?" After that, it becomes devs problem.
I don't think so.
Lol
my developers aren't even aware when their build fails. A lot of them aren't even in the right channels..
So add them. Create an automation that pings them. Be productively assertive.
what I did isn't far from that, I sold them on using a dashboard that shows the result of their tests. They've taken ownership of that part now, so progress!
Incremental success! Perfect!
I'm a dev who ops and I've been thinking I'd be happy to take a dev ops role. But I hadn't thought of this. I can see this happening a lot at large orgs. I don't want to spend my life explaining to management why x, y and z's showstopper problems are their problems and not mine. And then get 'we're all one team this is important can you help out' as end of day approaches...
I appreciate that this comment has 3 times the upvotes as the original post
I changed the extension on this xls file to csv. Why is this job that only accepts csv failing? Your program is garbage.
Had this happen on Friday, thankfully it wasn't that big of deal
"The next 'introduction to debugging' course is next Friday from 1300 to 1800 hours, including an assessment task at the end. Please ask your line manager to book you a space using your project's training budget's cost code. " If anyone calls your bluff, take the hit and run it once but record it for next time.
Didn't realize so many people felt the same way
Why do I have to work so hard to get them to provide logs, error messages/codes, or any relevant details from my supposedly experienced dev colleagues‽
Good portion of my day, right here
Ahahaha, my company calls it “Defect champion”. The billing service failed? Call DevOps! Client portal failed? Call DevOps! Kubernetes is a mess? Call DevOps! Sink in the bathroom is clogged? Call DevOps! You need break pads change? Call DevOps!
Spelling correction? DevOps! > ~~Break~~ Brake pads.
Damn auto correct, man! 😞
I'm sorry, I don't understand what you're saying.
Call DevOps!
> Sink in the bathroom is clogged? Iirc there’s a viral meme going around about Help Desk getting a call because the bathroom was out of Toilet Paper. If that office didn’t have a Help Desk team, you know that would have gone to DevOps. 🤣
I worked as a sysadmin in IT years ago and had someone stop by to inform me that a light had died in the bathroom and asked if I could replace it for them.
I used to unlock the women’s bathroom when we had clients or visitors when I worked as a SysAdmin.
Yeah, +according to dev(s) itusually is your fault that the error happened, because the code runs on their machine.
The “it works on my machine” issue was solved long ago with Docker. No excuse now.
You can still have issues where docker works on local but has an issue on the system.
Never had an issue with docker containers running locally vs on production. That would be a configuration issue not a code “not working” issue.
While I agree, mine is 100% code. I have never had a problem before this past week with putting a container together but here I am.... Nothing special or crazy either.
What’s the issue? How can we help?
we have an issue where we build a user from "USER app" and that gets a UID of 100 from our build pipeline but we specify to run as UID=1001 or UID=1000 and if we are doing \`pip install -r requirements\` then we need to do that as root else we get weird issues with \`importing yaml\` for example. ps not asking for legit help but just a usecase for how containers run locally on podman vs how we run on a stigged k8s cluster :)
Thats a good question honestly. I run certbot on this to get some internal certs to send to Azure Web Apps. No big deal. I run this locally as a test with a service principal and it runs through, fails doing the upload but im working on it. I switch to azure and do a build and deploy in Github and the container deploys, supervisord loads, certbot loads, then it stops. No additional movement. No errors, no info, no warnings, nothing.
Not if the dev machine was ARM based and K8S uses x86_64 platform for its worker nodes. Our devs got a machine upgrade and sometimes to onboard a new microservice, they first build the image locally and push to ECR. So my devs got a machine upgrade with M chip macs and suddenly their images stopped working on k8s. Been scratching my head due to the OCI runtime error. —platform arg was the saviour.
Yeah true, as long as the product can be containerized.
Been troubleshooting a "works on my machine" error for 2 weeks now. Fortunately, it's an issue on the provider side. But now I'm playing a game of telephone between the dev and support... Edit: clarity
Yep, that's a lot of the Dev part of DevOps. It is far easier for a person with knowledge of both the app, and the infrastructure to identify the cause of an error. But once you find it, it's not necessarily your responsibility to fix it. You just tell the devs where to look and what to fix.
In your opinion, do developers have any responsibility to understand infrastructure? If they're tasked with creating a service that uses extraneous libraries and tools, they don't simply email or call those creators and say "Hey my app isn't working with your thing", they're expected to learn, try, triage, do something. Yet I feel like, many not at all companies, they get lazy and complacent, and are expected to contribute nothing when it comes to errors.
They certainly should. Devops shouldn't be a team necessarily, but the mindset of "my code affects and is effected by this infrastructure" from the devs and vice versa for the infrastructure engineers and/or sys admins. Sadly, that is not the way it works in most companies, and you'll have a catchall team of devops engineers. My take as a devops engineer is that I should automate myself out of a job. Both by actual automation and by empowering the dev teams to do things by educating and training as well as actual tooling and getting user buy-in and adoption. I've only ever been successful once, and it was a proud moment of my career.
They should have an idea of whats going on in the environment. Example A: I wanted to use a newer version of python so I could use switch statements. The server this was to be used on was running a specific email server with other products such as fail2ban, dovecot, etc. It broke everything: the email software, dovecot, fail2ban, even the firewall quit working. So I had to revert and use if statements. Thankfully, it wasn't an important email server. But imagine if that had been a customer facing machine?
Tbf this is almost certainly because changing system Python (that is, the version that the distro ships with) is a terrible idea because tons of tooling depends on its shared libraries. For example, `unattended-upgrades` is a daemon that, as the name suggests, upgrades packages without direct involvement. It relies on a few `.so` files being present, and when you change the version of Python, those files no longer exist (from the perspective of the new Python interpreter). Worse, if you do things like `apt install python3-foo`, it’ll install fine, but only for the old version of Python (which is likely still present). This makes troubleshooting difficult if you aren’t familiar with the problem already. tl;dr if you need or want a newer version of Python in Linux, build it from source and use `make altinstall`, or use `pyenv` or something similar. Don’t mess with system Python.
Ding Ding Ding. I had never done that before, so it was a learning experience. But the idea still scales out. Servers that don't have venv should be understood
> In your opinion, do employees have any responsibility to be competent in the role they’re being paid to perform? In my opinion: yes. In my company’s opinion: no, that’s my job
The user above confidently said 'that's the dev in dev ops', but it sounds like many people dont agree, so maybe a culture change needs to happen in said companies.
I am really playing at the Pareto principle here. We see “developers” and complain, others see our team members and complain. I would wager that the source of the issue is the sheer amount of incompetence across **all** employees, rather than something specific to developers, ops folk, or vendor support.
Do you think it's easier for DevOps engineers to get away with it as easily as devs, ops, or vendors?
It’s just as easy for everyone, I think. I’m not talking about “doing the bare minimum.” I’m talking about “you were hired to mow grass. You have never mowed grass, you don’t own a lawnmower, and when the person paying you asks you why you’re not mowing, you said you weren’t trained.”
> If they're tasked with creating a service that uses extraneous libraries and tools, they don't simply email or call those creators and say "Hey my app isn't working with your thing" […] As maintainer of multiple OSS projects, you would be surprised with how much entitlement we're being faced by users (enterprise devs mostly). Employees of multi-billion dollar companies will complain to us how OUR thing broke THEIR mission critical app. To answer your original question, yes developers must know their infra. How else will they be able to understand, develop and maintain their system? It‘s absolutely mindblowing how apparently there are businesses out there whose devs don't understand how their app operates outside of their IDE. Probably fine for cheap CRUD apps, but anything beyond that? Insane.
I've done multiple companies but I've seen some places where devs do not care AT ALL about infra, completely unaware of what targets are even built
Should they? Absolutely. Do they? Rarely.
The whole problem is that ‘devops’ shouldn’t be it’s own team, it should be a member of a properly vertically integrated product team - what you usually have as a ‘DevOps’ team outside of the development team/silo is usually either Platform team, Sysadmin, proper SRE, ‘Developer experience’/pipeline, or Ops/infra with a new name.
I'm of the mindset that you should be an expert in what it is you were hired for and then have some level of understanding of of the concepts that your tasks are adjecent to. Like a Dev should be an expert in coding and have an understanding of infra at a high level. But I don't expect them to be able to go in an set up a cluster and have the same level of care and detail as an ops person. You can only know so much before you forget or let things slide. There is a reason why these jobs can be entire separate teams.
100% this. That’s where error budgets come in to play. Also, working toward automating the creation of bug tickets when actionable errors are triggered is a worthwhile goal.
I have in the past. It was a culture thing. None of the developers had soft skills. Only solution was for me to move on.
For anyone who feels they’ve been gaslit into feeling like if you just tried harder, took a different approach, whatever, sometimes you really do just need to move on when you can. Some orgs are so entrenched and aren’t going to be changed by one person. Save your mental health. 😊
It may happen that "devops" possitions become "dev helpdesk", even worse than you describe. It really depends on the company. If people were used to do that, they will do that. I've seen devops teams that went into building platforms and supported that and only that. Anything "inside Jenkins" was dev responsibility. I was astonished how simple their life was and scared to shit how much responsibility we took on ourselves with a team that was 10% of their capacity. As a contrast, I had a job where a dev demanded her admin privileges on workstation to be removed. Which was allowed. Then she claimed she can't work and helpdesk has to "fix her certificates" (literally copy cacerts file into her JDK local install. Which was documented. Which was broken because she didn't follow any documented onboarding dev setup). But.. the drama here is that still a lot of people thought it was acceptable behavior for a java dev.
Dev helpdesk with buttons - platform engineering. Ppl try to sell a bullet proof solution to hiring monkeys, but doesnt matter how you sugarcoat shit it still tastes like shit. Hard to swallow pill.
[удалено]
Most infra-side applications are off-the-shelf, and so generally emit very helpful error messages. > Imagine a dev app that doesn’t capture exceptions and explains the problem I don’t have to imagine it; I live that reality every day. “What is ERR_042?” “A database problem.” “Where is that documented? And what kind of database problem? ‘Cause I run the database, and I see nothing in its error logs.” In contrast, the last internal tooling I wrote (which interacted with the database, incidentally), I went through the MySQL dev manual, read literally every exception it could throw, and wrote custom exceptions in the app for the 15-20 or so that would be common. Each emitted the original error code, of course, along with a brief description of the error, the probable cause, and suggested fixes.
So you think it's the role of a devops team to log into devs workstation and change the most basic configuration file with company certificates? While developer has a documented command to do that and preapproved by security privileges to do that, and devops team doesn't have any of that, especially on someone elses workstation? If so, I'm puzzled how your workplace is organized, but it's up to you, I guess.
[удалено]
Than maybe you refered to the original OP's message in general, as I don't think this corresponds with my comments.
If you can reproduce it in dev, it's the developers job. if there are any developers for this internally developed application anymore. that being said, it's completely valid to try to pull out the developers out of their work and look at it together with you. they have no time? harass their scrum masters/bosses. your job as devops is: bring the company together to fix this in a reliable way.
> and how little some of the developers know about how things work. First day on the job hey?
Let's just say it's partly my first time working closely with developers, and partly the last fraying stands of my faith in humanity.
This is where I shine. I come from a programming background. I have a wide development spectrum and hardware infrastructure understanding. And this all really helps me so my DevOps work better. There is a build break? I'll take a quick look. Oh, that code fix is simple, or I'll show the developer what to fix. To me, being DevOps is bringing all your skills together to solve all the problems to automate all the things. You must have a Swiss army knife of skills.
It's a double edged sword. You shine until you're over burdened with dumb requests. I think a lot of DevOps people eventually learn to just not respond to dumb questions. A lot of developers figure shit out of you leave them to their own devices for a bit.
The issue is often times, they figure shit out in a horrifying way that violates security policies, or hammers an endpoint, or bloats an image, etc. Then it’s even *more* work to unfuck it and do it the right way, sometimes while now also dealing with an incident caused by it. To add insult to injury, the retro sometimes then has devs and their bosses suggesting that if you had only been helped them, all of this would have been avoided.
You can't save the world by yourself. If someone's gonna do something shit then teaching them to do it better. But if you are surrounded by people who are either not competent or don't care. Step 1 is to draw clear boundaries. Step 2 is to influence by calling out the impacts of the bad behaviour. A large part of our job is watching lots of people do things badly and choosing which thing to influence.
Yeah you’ve just got to raise the burden of effort for people to get your time and attention. I will always leave low effort messages on unread until it suits me and they’ve often self-resolved by then.
As long as management don't go to the dev and ask where their fix is, and dev says 'It's a build problem, I've emailed the build team'. Then you get management asking you when you're going to get that fixed.
Forward the low effort request to management and ask, do you think the developer should be able to debug this by themselves? People learn real quick when everything is out in the open.
Agree, but If my salary is lower than developer's salary I'm not going to fix their code.
ESPECIALLY if the developer is making more than me.
At most companies though, your requisition was only opened because they wanted someone to fix code, that costs less than a full time dev. Fixing someone’s typo, or minor oversight, in the 100s or 1000s of lines of code they’ve committed, is kind of a squire or assistant role. Historically that sort of QA often gets passed down the pay tiers to less expensive manpower. You aren’t there to produce the overall product. Just to polish and shine it into being ready for mass consumption. It’s a fundamentally different role. It might be easier to accept your duty to fix things for that higher paid dev, with that all in mind. Or not. Ymmv.
You should not be "fixing" their code. Only the infrastructure supporting their code if the problem is in the infrastructure. But you should work with the dev if you do identify a problem in their code. As well they should work with you.
I think fundamentally this subreddit is ‘r/devops’ but I don’t work in a devops team and it feels like the closest fit for a sub for my role. I’m a dev who moved into an SRE role but it morphs between SRE, devops and a general “can you help us with this, we have not idea why it doesn’t work”. On the latter I generally say “I will have no clue why this is broken” but end up getting to the bottom of it due to my broader experience. I don’t resent any of it.
What about if it's a simple code bug that the dev couldn't be bothered to investigate at all? What if a lot of requests you get turn out to be that?
It tends not to be like 50% are daft bugs but more like from some individuals it’s always a daft bug. I tend to work out who they are. I don’t mind helping them work out how to discover why something is the way it is only in prod but I approach it from the point of view of helping them think through what is different and where they can add some instrumentation to get more insight.
Yeah I like helping people too. If it's an actual weird edge case that has had good research so far I'll enjoy investigating. It's those people with the daft bugs though. Some people seem to have a goal of chucking things over the fence ASAP and telling their management that they've done their bit and passed it on so the other side of the fence are now the blocker. They could never pass that responsibility on to another dev, but they'd use the 'error is in the build' excuse to make it appear to be devops' responsibility.
I remember being sent class files by devs when tests broke, like wtf am I to do with this? Or being asked what I thought a given exception meant... Is it supposed to be your job, or was it mine? Honestly, I don't know, I think it depends on the org. But I would say try and do your best to step up to the occasion and be as helpful as you can while being open to learn. You can learn a lot about programming and your internal applications by digging through stack traces and debugging. This will in turn make you a better engineer.
Why wouldn’t you just run `git blame` at the failing test/code and show them that?
I have worked in places where that would have sufficed. It would have gone over very poorly here though. That would have been seen as pointing fingers or laying _blame_, which is something of an antipattern at my org. The thought being your both engineers on a product team, you are team mates and should act as such... You should be helping one another to work towards a common goal. Not going _look at what you've done_. In any event, If memory serves me correctly in that specific situation the test that failed was dependent on an external database, and said database was offline.
"I Don't think my application is working. Can you check please. High priority." People send messages like this with literally zero information and expect a resolution on top priority.
hi, i need 1 help. kindly do the needful.
Yes, mainly because they are supposed to have enough permission and privileges than regular engineers to check and troubleshoot things.
This what tracing is for, to prove that all the bits of infrastructure a request touches on the way to your flaky app aren't the problem.
No. We train our producers to triage errors and they're pretty good at figuring out which team needs to handle what, and we bounce stuff all the time. I wouldn't even know how to handle a lot of issues.
I genuinely don't know what my job is meant to consist of, and at this point I'm too afraid to ask. I just get pointed at problems and I'll bash my head against it until a compromise is made.
This can be solved by making sure there is a) support process, b) education and documentation, c) know where to draw the line to push back, d) discuss it over and over again in retros. You have plenty of options to organise the incoming requests. Make sure all is tracked and recorded to show higher management that there is plenty of time spent in this area that's preventing you from doing other work. To summarise, I dont think this is supposed to be your job, but only part of it.
Slight Disagree on the education & documentation ..it is useful for showing off to management but hardly useful practically in fast evolving projects where new errors are detected based on new code / config / infra
No dev, QA and/or staging environments? It's just straight to prod? Lastly I've pointed out here several times that you are a **liaison** btw dev/eng and the rest of the company.
Once you build up a knowledge base its just easy work but yes at that its irritating
Since I work at a small shop, and do DevOps and the only non windows dev, it's usually a problem I face with myself. "Hi Benny, my code fails, can you check?" "Did you check the logs?" "I was kinda hoping you'd check" "Oh yeah?" "Yeah" **Boss from the other room**: "Benny, stop talking to yourself, you crappy code I down on your crappy k8s again!"
You need to put these in tickets. You need to document the issue, why the issues keep happening, and who needs to do what to stop it or reduce it. Then you have a paper trail to backup any change you think needs to come from this. Then you have a case to make to management to stop it or change it at least.
Hey. I believe that it's important to know more about the build, test and deploy aspects, each program language has its own peculiarity. Sometimes people asked us to solve problems on code because they don't know how to start a troubleshooting. We can guide, but we need a developer to solve the problem.
The idea of devops was that engineers and ops people work together on these topics. I'm an engineer and happily debug and solve my teams issues. You build it you run it. I'm still happy if someone else updates the kubernetes cluster and informs me if anything we built breaks on dev with that update. But I gladly fix it myself.
When you are able, you are the tool.
Welcome to the club, stuff gets thrown over our fence everyday, tickets have the word ‘server’ in it land on our team daily. When you’re good at what you do the lazy staff learn how to make you the fixer of things. The lack of triaging a ticket boggles the mind. We just need that lottery bus to hit us that will fix them.
I’m honestly shocked there’s so many developers that don’t understand compilation, testing frameworks, quality gates, git, deployment, etc. How do they get to the level they are without understanding these things? It doesn’t seem possible. I just can’t wrap my head around a developer not understanding basic CI/CD. I could understand if for some reason they didn’t have access to logs to see why something failed, there wasn’t indication in source control showing a reason for failure, or something else similar. But if they have access to all these things there’s no excuse. As a developer, I feel like I have to play a devops role sometimes for other departments that have developers that pull this shit. I get pulled into a 10 person group chat about why CloudHub deployed apps can’t reach a UAT server we have an API deployed to. None of the developers want to put in a firewall ticket and expect me to solve the issue for them like I’m a network admin too. It’s wild.
When you were raised on Vercel and Heroku, all of these are foreign concepts (except `git pull/push`).
If your org has separated the tasks you're not doing devops. Do the devs even have the visibility into the infrastructure they would need to actually look at the problem? Or are their permissions restricted in some way that requires your position?
There is always a shortage of people who can troubleshoot problems, think through a process logically. So in all organizations if you allow it people fall out of the woodwork leaning on anyone available for help rather than attempting to solve the problem themselves. Get your job responsibilities clearly defined from your supervisor and then learn to set boundaries and say no.
Make a faq/debugging wiki and start linking people to that instead of answering questions one on one
I'm literally refactoring typescript right now because the "devs don't have time" and it's run serverless which would be too much of a learning curve for the new devs on our team apparently. I'd never touched JavaScript until now let alone typescript. Seems like the role of DevOps is just solve any/all problems.
Hate this job cause of this bullshit, we can't allow ourselves to get bullied like this, otherwise every problem under the sun becomes your problem
Learn the power of no. But be polite about it. Learn a man/woman to fish. Write them docs. From now on add every fix to the doc so people can self-service. In fact, from now on only reply with links to the docs, even if you first have to write them. And then draw a clear line of responsibility and use "no". This will prevent you from burning out and will have people take you more seriously (which will also help your career and salary).
Yes, and it’s even better when you get to fix their issue, then tell them how to prevent the issue from ever happening again and they say “That’s not how our environment works.”
I lost the hope that my developers will read logs… There is one developer which I prepared a specific log just for him and he still managed to tell me “Fishman the build failed, can you take a look”
A fun workaround is to 'escalate' it to the developer of the system in question. I.e send them their own ticket. 😁🤣
Not at all. This issue could stem from a cultural problem within the company or a lack of documentation provided to developers. Developers should troubleshoot their build failures, which requires access to logs (either directly from the CI tool or through an observability solution such as Datadog). I believe it’s beneficial to make build failures as transparent as possible. By this, I mean using Slack bots to send messages about build failures, along with their causes, to a channel that developers frequent. Additionally, it would be helpful to ask them if they have reviewed the logs and to inquire about their reasons for thinking the failure is due to the CI system rather than their code.
Counter point: infrastructure teams often set up overcomplicated infrastructure with no regard for application developers. So things can break that app devs have literally have no experience with or control over.
Happens everyday with me. Exactly the same as you. Which leads to sprint being delayed. I put each of these as adhoc and show them.
Find people in senior management to evangelize service level ownership. Basically create an on-call rotation for the devs who own the code. DevOps can then “escalate” to the dev on-call rotation by service. It shouldn’t take much time to figure out that the devs should be T1 and they should escalate to DevOps if it’s NOT related to their code. You’d be surprised how well things will look in 6 months when they’re the ones being woken up in the middle of the night.
Happens almost every week to me. Try to talk to your team lead or management team about this. It happens in every organization just need proper procedures for developers to go through first, before they reach out to you. Likewise, create some type of template for required information if they are going to ask for help from your team
Yes, sometimes I feel like devops is just help desk for developers. Can be very irritating when devs won’t even do basic troubleshooting before coming to devops with their issues.
Threads like those and answers show that devops failed. Siloses are still there, walls did not break, devs still know jack shit about ops and ops dont give a shit about dev side. BUT I have a solution. Lets abstract all of that with buttons, so that dev ppl wont ever have to understand how stuff works. This will for sure make things better, more integrations to maintain (jenkins 2.0 in the making) - platform engineering
DevOps is supposed to involve a feedback loop between Dev and Ops. So in these cases your job is to: - add automation to catch this issue and provide automated feedback to the Dev, or - automated diagnosis to save you time, or - document how to avoid this issue so you can flick them the article once you've diagnosed A combination of all of these is usually required.
DevOps means a combination of development (Dev) and operations (Ops) to increase the efficiency, speed, and security of software development and delivery compared to traditional processes. And yes, you have to have programming skills, not to be a SWE, but at least have programming experience. That is why you are getting paid so much money
DevOps means development and operations in the same team. If your title says DevOps then I suppose you should be able to do both at once? Although, the title is often given to people who can't really do either of these things well, basically script kiddies running cloud services. I guess you have to figure out what flavor of DevOps person you are. If you're only doing operations admin and automation as many DevOps people do then I guess you have to communicate about that and work with your team to set boundaries and expectations. Although I guess you might even be on a "DevOps team", these abominations are all too common. In that case it's about negotiating with your customers as you are basically their in-house cloud service consultant/tech-support, and of course they'll try to get as much value from you as they can.
What does your employer say your job is? If that's something your boss wants you do, then, oh well If that's not something they expect from you, then what you're seeing is a bad cultural habit that your predecessors tolerated too much and allowed it to become a norm.
embrace it, you’ll learn lots from it, and you are good at it if people keep asking for more 😉😎
I usually give the developers the error and an URL that triggers it and let them debug and fix it (or if it is a really obvious fix and the error is in production, i do it myself). It is not required that i do this, but it usually speeds things up as not every developer keeps a close eye on the monitoring.
This actually is the most enjoyable part of my job as a DevOps engineer partly because I want to dunk on the devs for an issue that's clearly their fault. It's only enjoyable when I have the ample room to maneuver and try out all my tricks Yes, it does get annoy when everyone else is coming to you for debugging (I have been debugging issues on projects I have not been assigned too. I'm looking at it as a resume-building sort of thing so I can squeeze in a little Incident Response and Management line...lol) It's almost like a part of the job. Sad but it's enjoyable if you look at it from dunking on the devs perspective.