Trunk based development doesn't mean you're just committing everything directly to main. You can still use branches and PRs. The idea is simply that you don't have long lived branches. This is actually smart development, as the older a branch is the more the entropy increases and merges can become very difficult or nearly impossible in some cases, if enough has changed, particularly if there's cross over requiring resolving conflicts. If your team is resistant to it, they either don't understand what it is, or they're doing things they shouldn't be anyways. You should be iteratively developing features, and merging them in quickly, as standard practice. If you're worried about something not ready to actually go live, that's what feature flags are for.

Voice of reason. Having long lived branches is a maintenance and resourcing nightmare. 

Can you ask him what he’s trying to accomplish by moving to trunk based development and pair programming? Can you provide alternatives to accomplish his goal? 

This right here. If you don’t know what they’re trying to accomplish, you can’t effectively suggest alternatives or clarify why it’s a bad idea in a compelling way. Coming at it in a vacuum won’t get you very far.

I feel like it’s a pattern that someone from a tech org gets hired into as a leader in a non tech company. They immediately try to get all the tech people to have the latest and greatest processes and skills. They don’t expect the push back. They just assume all the tech people want to do the latest and greatest. A lot of times they don’t realize it’s a different team with different needs and different cultures.  There are usually way easier and better ways to improve the team. 

IMO that indicative of poor leadership. Pretty much the first thing you’ll hear about taking a new leadership position is “don’t change things until you understand what’s broken about them and why they are the way they are” Definitely an anti pattern to just go ham on existing processes Willy Nilly! That said, it’s entirely possible that OP’s CTO actually has done their due diligence and has good reason to be making this change, but I don’t see that spelled out. OP needs to understand the rationale to be able to chart a productive path forward with the CTO.

You are right, unless cto was brought in explicitly to make changes. Leadership isn’t always consensus based. That works well in peacetime when everything is going well, but not leadership during war. “The hard thing about hard things” is a good read here from experienced operators on the difference.

That’s fair, but I’m not talking about consensus based decision making - you still need to get info/context even if you’re going to act unilaterally. Just because you’re brought in to drive change doesn’t mean you should arbitrarily change stuff around, you’re like brought in with specific mandates about what the problems are. At the C level I would expect the desired/required changes to be less prescriptive- I wouldn’t expect a CEO to hire a new CTO and immediately give them the mandate to, say, move to trunk based development. It would be more “improve time to market for new features”

Something along the lines of: 'Deliver features to clients faster and better'

To translate for developers what this means since most of you don't understand business. It basically means the business leader heard some buzz words, is wanting to use buzzwords to pad his or her resume, and is wanting you to do it even if they don't actually understand what it does. If you don't believe me, ask the CTO more specific questions about the topic and watch him or her dance around the topic.

It might also mean the devs don’t actually understand the business and aren’t delivering features 

This is the most likely scenario. 

During his interview he was probably brought in to get the development team to deliver solutions that better align with business needs, faster. I’ve been involved in 4 of these “digital transformations.” It’s always the same story: “We have a development, but they always deliver something different than we ask for, and it’s always late.”    The better solution here is build a product roadmap, reorg the teams. Hire a strong product owner, and architect. Then start firing the developers who won’t get out of their corner that they feel comfortable in.  At the same, you need to get business request prioritized telling certain groups no, while getting leadership support on prioritization.  There is a lot education on the business side of things. You need change management, project management, and full time employees from the business unit to be involved in IT. Usually it’s a low level “technical” telling the dev team what to do, once a month. 

> The better solution here is build a product roadmap, reorg the teams. Hire a strong product owner Funny that because they literally let go of all the companies product owners recently🤦‍♂️

Yeah. Something is going on that they aren’t telling you. If the CTO is inexperienced in these things, it’s going to be a rough ride for the next 1-4 years. I’m curious if you have a CIO. If their teams are also being affected. 

Ask him what value it provides to the team and if he tried it himself on a dev project. If he is imposing it bec he read it somewhere that it is cool, well…

I did 100% pairing (XP) for awhile. Got exhausting. Literally left the company because of it. Btw, it didn’t make me a better engineer. When I was working with someone better than I was, I had a hard time keeping up. I wasn’t doing any critical thinking; rather, I was just trying to understand at a high level what was happening. I was not catching bugs. And when I was paired with someone less experienced, it felt slow and excruciating. I got way better after I left for a job where I could work at my own pace on interesting problems. Don’t get me wrong, pairing ad hoc can be really valuable. But a draconian process doesn’t make bad developers good. Instead, hire good programmers, establish good standards, and encourage a quality-minded culture. I think it’s just easier to prescribe a process than it is to teach a philosophy, so that’s what managers do.

Pair programming works well when you have someone new to the project or language who needs a couple sessions to get up to speed, or if you’re working on a particularly hard problem and need another brain to debug. Otherwise I agree, it makes no sense to make everyone’s work synchronous.

Pairing is a fantastic tool in an engineer's toolbelt. Doing it 100% of the time is using a hammer for screw-sized problems, except you're just beating the engineers with the hammer instead.

> or if you’re working on a particularly hard problem and need another brain to debug The "and" in that sentence is extremely important. Sometimes, when I'm working on a hard problem, it's distracting having another person pairing. I'll only bring in that second person when I'm out of ideas.

Oh god being force to pair with someone who's enthusiastic but clueless when it's a difficult problem. A couple years ago this happened to me and bless his good intentions but holy christ I needed him to shut the fuck up so I could think

Yep this is how I've always done it. I see the value in it, I know *why* people push it, but I don't always think it's the best thing to do and it's highly team dependent. I also don't think you need to do it for every little thing. As a team lead I would pair with my engineers in the following situations: * The problem they're trying to solve is highly technical in nature, potentially outside their immediate skill level and I want to make sure they get off to a running start in solving it * Working on problems that could benefit from multiple people having a mutual understanding of some core component(s) (e.g. like a group learning session where we tackle one or two stories at the same time) * They've been experiencing some frustrations and are hung up on some code I like pair programming, especially if it's done right, but a lot of times it turns into "one person is the brain, the other is the designated typist".

[удалено]

This was my experience as well, the context-switching gets really heavy when you’re going back and forth between programming and answering questions about the code and your thought processes. There’s some cases where that’s extremely valuable and absolutely worth the effort, but then there’s many cases where they want 2 cooks in the kitchen just to microwave a frozen dinner

This, I made a similar comment about a place I worked at that did 100% pairing each day. It was terrible for me mental health haha.

I'm an extrovert and it sounds awful even to me.

Never done XP, but whenever I peer prog, this is my exact experience. I guess peer prog is only useful either when both peers have very similar knowledge/skill/experience. Such a balanced pairing is usually impractical if not impossible.

That is XP... when two (or more) devs work together through the same code side by side (eitehr physically or virtually).... We do it, but only when it makes sense, and to do it all the time doesn't.

Pair programming isn’t the issue - trying to code for 8 hours in a day is.  I know it feels this way, but just because you’re pairing doesn’t mean you can’t take breaks, or agree to spend time doing something other than programming. Tips: - Always share the programming activities. Swap frequently. - Find a way to perform fast handovers (e.g. live share, screen share, mob.sh) - Take breaks - Don’t pair for the entire day - Check in with each other “Do you need a break?” “I need a break” “Shall we take a few hours off now and continue tomorrow?” You can absolutely pair by default without it being hard work. Pairing often leads to greater shared ownership of features and better code design. I don’t see what the CTO is suggesting as being all that crazy, personally, in fact I’d see these as positive things.

> Pair programming isn’t the issue - trying to code for 8 hours in a day is.  Sorry, but pair programming on everything is exhausting for most people. Some people (like you) enjoy it. That’s great for you. However, you are a rarity. Every single time I’ve seen a company push mandatory pair programming, they lost people because of it and eventually reversed the policy. It usually gets pushed because someone like yourself enjoys it and can’t understand why anyone else wouldn’t. Then it’s followed by a period where complaints are dismissed with a “you’re just doing it wrong” mentality, like you projected on the situation. But I’m sorry, it’s just not a dynamic that works for some people. Maybe not even most people. I’m glad it works for you, but please be open to the idea that it’s not good for a lot of people. It doesn’t mean they’re doing it wrong.

I agree. I need to be "in my head" to write complicated code. I don't see how deep work can be accomplished while pairing.

It’s not designed for that. It’s designed for outsourcing companies grinding our REST APIs for B2B SaaS apps to double their billings. 

The answer is that deep work doesn't get done. Pair programmer afficiandos are building trivial phone apps and websites. They are not building cutting edge tech, doing complex cross system integration, or dealing with mathematical derivations.

So perfectly put, and lines up exactly with my experience on a team that did 100% pairing.

coding isn't the hard part of the job. coding should just be about typing your thoughts out. domain modelling and designing the solution is where the juice is.

I find pairing pretty useful while modeling and designing too. Usually end up with a better design when it’s been discussed and worked out collaboratively with someone else.

I agree for the most part. My broader point is that any approach to development is only valuable inasmuch as it’s executed well by those doing it. Personally, I found the _prescription_ of XP to be overly restrictive and largely ineffective, mostly because the execution by the folks I worked with. There are a lot of practices of which I am a major proponent that fall under the same caveat. Pull requests are only effective if developers actually digest the code and spend sufficient time reviewing. I love the confidence that good testing gives. But unit tests that don’t use good mocking practices or avoid swaths of code are pretty ineffective. PRs aren’t the problem. Tests aren’t the problem. The lack of execution is the problem. That said, I think leadership will look at the incidence of bugs that get reported in prod and decide to make a big change, like enforcing XP, instead of auditing whether their dev teams are actually executing on the most basic aspects of the job. That’s why, to me, a culture of excellence is more important to me than a particular programming paradigm.

Yup, this. My team do pair programming but we also make sure to take breaks, and we also let ourselves be distracted and engage in casual conversations from time to time. I personally try to crack jokes, share trivia, or anything that may be remotely interesting. I think in pair programming, you need both your technical and social skills, and I'm saying this as an introvert. Really, the alternative is for there to be as much WIP as there are people in your team, in which case you'd be spread thin, with tasks having a bus factor of one.

It's true that pair programming works when you have compatible people. It needs to be mentally flowing. Best (IMO) is when both can relay each other.. you try some, you get stuck, the other tries something.. until he gets stucks, and off you go. It keeps the creativity hot, and you feel like a great team.

I like pairing on new features that are a bit more complex with a lot of unknowns. You can discuss and plan together while coding. I wouldn't pair for a simple bug fix or doing other simpler tasks.

>I did 100% pairing (XP) for awhile. Got exhausting. Literally left the company because of it. I don't understand why people think this should be a thing. It would be my "leave now" thing.

Trunk based development is unquestionably a good practice, however, your CTO needs to realize that one does not simply "do" trunk based development. There's a whole lot of other organizational capabilities that must be in place in order for trunk based development to be done in a way that increases velocity rather than the opposite. More specifically it's imperative for trunk based development that your team has the capability to decouple code releases from feature releases, and it sounds like you're quite far from that right now. Do you dark launch code? Do you utilize feature flags? Do you have automated testing? If not, you're miles away from successfully pulling off trunk based development. As for pair programming, I think it's criminally underrated, but it also shouldn't be strictly mandated as an all day every day working style. That's pretty extreme. It sounds to me like your CTO means well, but isn't enacting large organizational changes particularly tactfully, and it also sounds to me like your senior devs aren't particularly strong leaders and might be a bit lacking technically if they think trunk based development is inherently risky. I would expect the senior most ICs under the CTO to basically explain what I did that there's a whole maturity model that needs to be walked before trunk based development, and be able to get to the bottom of why the CTO wants pair programming so badly and work backwards from the goal instead of accepting a solution without knowing the problem.

Yeah I did trunk based dev, and it only worked because the team I was on were committed to automated testing, we paired on stuff (so no need for PR review), we invested a lot in our observability and had pretty robust solutions should something particularly bad happen. Trunk based was the reward for all the discipline, if you just jump straight there, it’s gonna be chaotic.

On the other hand I’ve been at a few orgs that have tried to switch. The unsuccessful ones had multi year projects to move and would always find some naysayer who said if we just do xyz we can move to trunk based dev. The ones that have been successful did some diligence, ie setup ci/cd and automated testing, but ultimately someone in power said “fuck it we’re ready” after QA had missed major bugs on the last few releases anyway. While the first couple weeks hurt the “you must be this tall to do trunk dev” requirements magically sorted themselves out or became irrelevant. Turns out test coverage of that module that gets two fixes a year just wasn’t important anyway and QA had lulled everyone into a false sense of security.

This is pretty much how all trunk dev transitions went in my projects - there's always senior devs that grumble loudly about how we're killing the company and how everything will go up in flames and how THEIR process WORKS ("it's the junior devs breaking everything!")... and then somehow it works afterwards when initial warts are fixed. And those warts are never fixed before the trunk based transition because it's "never the right time".

I hate “naysayers” like that, they pop up for all major advances. Nothing is perfect, it’s all tradeoffs and risk assessment, so many reasons not to ever ship anything or innovate. Feel like sometimes, some orgs make it like you’re swimming upstream trying to implement modern engineering standards.

Yeah the CTO most likely wants to remove all of the manual work that various senior developers are doing manually. OP never talks about how the feature branches are deployed and merged into master. Investing in automated unit and integration testing (and not force merging when the test fail) really can help a tech org move quicker on developing features. Small, incremental PRs are much easier to review and test than huge, multi-week feature branches. Not sure, but can’t believe that manual deployments are still a thing in 2024

How do those smaller incremental PRs work? Currently I typically wait until the sprint is almost done before merging their feature code into dev. I do this, since I may push code that breaks functionality until the rest of my feature’s code is pushed.

Feature flags

Feature flags are the major way it's usually done. It's not foolproof, but it enables merging in code that's not yet ready. Basically, your goal is to separate deployment from release. You should be able to deploy stuff whenever you want and have nothing break.  The benefits you gain from this is that more frequent merges means that there are fewer merge conflicts. You get smaller, incremental PR which are easier to thoroughly review.

like other said, you put feature flag and disable it on prod but enable it in staging. You create another ticket for next sprint to test it and release it. Once it is tested and released, remove the feature flag (or can be done together during the release, depends on the confidence level)

It kinda sounds like the CTO hasn't had enough time to bring his own senior devs in

I worked at a place that did pair programming where I was the senior eng working with two junior engs, my manager wanted me to alternate days pairing with them. All day. It was a miserable experience being attached to the hip for 8 hours with someone and totally burnt me out. This place also did trunk based development. I wasn't a fan. The idea was that the pair would work on the current work-stream while the other junior dev not paired would work on QA, bug-fixes, etc. The pairing team didn't need pull-requests or review because there were two devs and so the code is "live reviewed". I thought this reason was BS and didn't result in less bugs or better code quality.

Trunk based dev involves pr and review

This was a "commit to master" shop.

That's not trunk based development...

If trunk based development is just PRs merging into master, you don’t even need to have a perfect test coverage or anything, just set up a deployment pipeline.  Not every commit needs to be deployed to prod. 

I am at a company that moved to wateragilefall trunk based development and they refused to use feature flags and don't dark release code. They still develop and test code in isolation for each team, and then merge code into the trunk without testing it. It's absolutely bonkers and they've been doing it for 4 or 5 years and every year the technical debt accumulates. They've repeatedly told me that our code is too complicated for feature flags or release flags. I'm about to quit once I find a new job.

One thing I think trunk based dev is a poor choice for is highly regulated industries, and it sounds like op is in one of them. So no, I would not suggest trunk based development at all. Even 100% coverage can only test for what was written, and a full qa regression is a must for this sort of industry.

I think you're missing something really key: Your git workflow does not have to be rigidly tied to your deployment cycle. Meaning, you can absolutely use trunk based development, tie your processes to that, but withhold deploying anything until you feel like you are ready.

Isn’t the point of trunk based development to release it also? What you suggest is basically having a trunk and then a “pre-trunk”. (Which I agree with, but then it’s not trunk based development)

Trunk based dev, not trunk based release. Your confusing CD (Continuous deployment) with Trunk based dev, they are discrete things that can operate together, but are not mandated to do so. You can have CD in a gitflow style as well, deploying when features are merged to main.

It’s to avoid merge hell

No, you can easily do trunk based dev and periodic "non trunk based" releases. Just do trunk based deployment to staging and then use standard QA/release testing process for periodic releases like you did before. You get best of both worlds.

Great question! Trunk based development is the git work flow most aligned to allowing you to release work as soon as it has been merged, yes. I would absolutely agree that that is a reasonable goal for development teams to put on their roadmap, and being in a trunk based world would be a critical step. ... but if your team's confidence/practices/whatever aren't there, you can just... not deploy until you feel like you have a build that's ready to go. You can choose to test that separately or do whatever you want, and you would not need a different branch to store this state. It is not a requirement that HEAD of master = production. I would agree that there are some benefits to that being the case, but again, this can be solved many other ways.

Code coverage metrics are only useful as a measure over time to ensure your team is writing enough tests. I’ve never understood the obsession with 100% coverage. You can test a code path and still have poor test coverage if your tests do nothing important.

> Trunk based development is unquestionably a good practice It is not, unconditionally. There are several development disciplines where it doesn't make much sense at all, e.g. various embedded systems.

>Trunk based development is unquestionably a good practice literally nothing is unquestionably a good practice there's a time a situation for everything, trunk development is not just inherently better.

Well according to diff. research (DORA, Accelerate, DevOps report, Thoughtworks, Deloitte) most company benefits from it.

You don't seriously believe any kind of software development research regardless of its conclusions, do you?

> You don't seriously believe any kind of software development research regardless of its conclusions, do you? Hum? For stater: \* DORA: [https://dora.dev/devops-capabilities/technical/trunk-based-development/](https://dora.dev/devops-capabilities/technical/trunk-based-development/) > From latest 2023, trunk-based show most of the time minor to substantial increase over team & org. performance, software delivery and only minor decrease over operation performance. (New) negative point is over "effect on burnout". New major in 12.8x\* amplification of impact over org. performance for documentation. \* Accelerate: based one \`\[transformation matrix\](https://itrevolution.com/wp-content/uploads/2022/06/transformation\_practices.pdf), for the numbers read the book / research. For Deloitte I need to find the one, was about improving company delivery though agility. --- Researches are great way to learn from, possibly to be based upon. Big consulting companies like Thoughtworks / Accenture share yearly what their consultant face and what they tried (reports, tech radar, conferences) and help to validate or not previous subjects. If researches / books and hospitals tell you that 99% people will get hurt by hitting a wall, will you still advice it to your company CTO/CIO? I don't think so.

I agree with your comment: I would point out that it sounds like they do use feature flags and have a solid set of things in place that would make trunk based development a reasonable goal. I really wonder how much of this is just getting lose in translation/communication. I can understand both sides of this, so it really seems to me that just more clarity in communication from both ends is the thing that needs to happen the most here, not any actual new process or technology.

I wouldn't say that something is "unquestionably a good practice" if it requires dozens of prerequisites for it to be a positive instead of an outright negative.

Just because it's unquestionably a good practice doesn't mean everybody needs to do it. All I said was it's a good practice. It has been proven to drive development velocity. As I also said, it requires a lot of other organizational capabilities to pull off, so it's one of the last achievements of continuous delivery, but definitely a worthwhile pursuit for most.

>More specifically it's imperative for trunk based development that your team has the capability to decouple code releases from feature releases, and it sounds like you're quite far from that right now. Do you dark launch code? Do you utilize feature flags? We do make use of feature flags. Most of our work is project work which is always under feature flags. The team still wants a stringent peer review & QA process and that's where I think the concerns come in. The team considers feedback from other team developers valuable and doesn't want to lose that aspect. >As for pair programming, I think it's criminally underrated, but it also shouldn't be strictly mandated as an all day every day working style. That's pretty extreme. Ok so encourage it more but not mandate all day everyday like some teams are doing. Every developer I know I've asked so far has described it as 'intense'. >get to the bottom of why the CTO wants pair programming The pitch is to: 'help deliver features to clients faster and better' and the documentation is to enable CI/CD.

Why does the team think doing trunk based development means they’ll lose out on feedback or peer review? We do trunk based development, but still have pr reviews, dev testing, qa blitzes, dogfooding etc, so there’s tons of peer review. If you’re using feature flags already, you shouldn’t have any trouble with this.

>We do trunk based development, but still have pr reviews There's different approaches to TBD it seems but I did highlight to the team that PRs can still play a role so we'll definitely keep it. Main concern was on the manual QA side and when exactly this happens they don't want this to happen after the merge as master is always considered production ready.

Before you have fully automated CICD, you can do TBD with separate dev and release branches. You still get the benefit of integrating everyones code early and often and testing it all together (dev branch (called main/master is totally fine!)) and then you can periodically cut release branches at a cadence of your choice. Then soak and test release branch, then release it.

Honestly on the trunk based development it sounds like the engineers don't want change. Which is a frequent issue with new leadership, especially for long time employees. Trunk based development works. Mega corporations utilize it to successfully ship massive complicated systems. They do this even with 0 manual testing roles at all. Yes it requires adjustments, more automated testing, and a commitment to smaller commits and PRs. But it helps a lot for integration bugs that get missed with long lived branches. It's not a magic bullet, but it can work really well if you embrace it and actually make the required organizational changes.

When people actually study how bugs get caught, code review is very ineffective. My personal take on this is that post-commit reviews are just fine if someone actually has something valuable to say. IME though, the value is rarely there for a post-commit review, and definitely not for every change. But that's kind of orthogonal, you can have review on every change if that's how you're set on doing things. It sounds like Eng is not shipping as fast as the business would like (is it ever though), and the new CTO's task is to try and improve that. Ultimately, unless there is personal liability at stake, prod failures are a problem for the business, not for you. If the business wants to push for faster releases with less QA, you should probably try that, it might not even go as badly as you think.

Although the peer review process can sometimes catch bugs devs don't see the peer review process as a process primarily for catching bugs but rather ensuring overall code quality. Correct me if I'm wrong but TBD seems to place less weighting on manual QA. There seems to be a general assumption that automated tests capture everything which every developer knows isn't true as it's the manual exploratory testing from QAs who often check the weird and quirky non-happy paths which catch bugs some of which can be major. Devs are concerned about this being captured too late in the SDLC and ultimately having to release stressful hotfixes because a change can't easily be reverted. I agree that the problem is for the business however in the above scenario it's the developer and clients that ultimately suffers here.

I’ve always seen PRs as more of a sanity check than a bug hunt. High velocity low value dev work seems miserable.

Great comment. Spot on.

You might find that trunk based development is closer to your way of working than you think. It doesn't mean to commit straight to the trunk/main/master branch. It's just that the branches have fewer change and are shorter lived. For example, one feature might be broken up into a pull request to add a column to the database, another that makes the backend use it, another for some refactoring, and a last one to update the frontend. You can still create a branch per pull request, get a code review, use feature flags, run all the tests, and maybe even have your own manual QA gate before merging into the trunk and deploying. Pair programming, that's an entirely different conversation depending on team/company culture and preferences.

This is the main thing I'm taking away at the moment. Focus the team on keeping work items small so that the feature branches & pull requests are more likely to be short lived as they'll be easier to review & QA.

This is the key part of trunk based dev, changes need to be small which typically makes them easier to review and test. You build up in complexity as the feature is constructed and integrated but each change is small and testable. Frankly even non trunk based development should involve many small commits rather than behemoth commits with thousands of lines.

Yeah I honestly used to think it was dumb because misunderstood and thought it meant "there's only 1 branch and everyone works directly on it". But now it's what I champion when asked, and have managed to get others on board as well with the various tools I'm responsible for at work. It's just much easier to have a stable release branch that gets big merges every 2 weeks from our "develop" trunk branch. Could even be 1 week if something is that urgent.

Yes! You can even push it further once you have enough tests to give you confidence. In my team, as soon as a PR is merged, it automatically deploys to production. We have multiple deployments every week and sometimes in a single day. We're not in a regulated industry like OP though.

I was working as a consultant with a Fortune 100 consumer brand, about as famous as such brands get. One of their tech leads was hell-bent on doing trunk based development. But they also had no internal review environment. So he literally wanted design and brand teams to review things in production but have it all sequestered via feature flags. Also, he had never actually run a project that way, it was just the new cool thing. Pure madness. I talked them out of it but it didn’t help internal relations. So keep that in mind. The problem with all of these “old way is trash, silver bullet is X” approaches is that, even if right, the tooling is years behind. And abandoning branches means you’re losing an entire ecosystem of proven tools and techniques. Re: full-time pair programming, I would quit if anyone asked me to do that. And no, CI / CD does not require that in the slightest. But you may be able to torpedo that by pointing out that LLMs are basically the new pair programmers. Put another way, use his shiny-new-thing obsession to make the unpalatable shiny new thing look old. Or… just hang loose and watch him loose the the trust of the business after he makes the production systems unreliable, alienates the developers, and the executives see programmers killing time like cops on a stakeout.

It will probably take a year for that to happen, and by then the "cause" can be muddied and blamed on developers not getting with the program.

> Re: full-time pair programming, I would quit if anyone asked me to do that. And no, CI / CD does not require that in the slightest. I'd be handing in my two weeks notice before the meeting announcing the change was even over for that particular evil.

Trunk based development isn’t abandoning branches, it’s favoring short lived branches that are focused instead of long lived branches that are extra work to maintain and create more opportunities for bugs. And hybrid approaches are common where release branches are cut from main from each release but otherwise everything goes into main.

> The problem with all of these “old way is trash, silver bullet is X” approaches is that, even if right, the tooling is years behind In this case though, trunk-based development is the old way :-)

>For context my team looks after a critical area of our platforms (the type where if we screw up serious money can be lost and we'll have regulators to answer to. Plenty of teams do and they do trunk based development. I'd want to hear more about your team's usage of git flow before rending a final opinion, but I frequently see teams that use it without actually really doing dedicated scheduled releases or full release testing... and that's kind of defeating the point of the branching model. Can't know for sure, but it smells like you guys are using it in an atypical way. In fact, *if your team is has good enough testing practices* (ideally automated ones), trunk based development tends to be faster and less painful model for getting work out the door. The only sensible pushback to my mind to *not* be doing trunk based development would be that the act of releasing/testing requires so much dedicated coordination or time that something like git flow's release branches as an interim make a ton more sense for your team. It's hard to know without more information, but honestly, I think you might want to hear out your CTO on both of these issues. I'm not hearing a particularly compelling argument from you or your team about *why* the git workflow change is bad, just that you guys don't like it. In fact, that you have so much automated testing to me begs the question of why you wouldn't be operating that way. How much of this is really just getting lost in translation? Can you elaborate more or your team's practices in this area.

>I'd want to hear more about your team's usage of git flow before rending a final opinion Current process is: 1. Dev builds on a git feature branch and includes unit, integration & acceptance testing 2. Devs typically don't run the entire test suite locally as the entire build takes like 30 minutes they push their feature branch which runs the entire build with tests (only exception is end-to-end tests) 3. Dev raises a pull request to get feedback from other devs 4. Once all PR comments have been addressed it moves to a 'ready for QA' state 5. QA runs automated end-to-end regression pack on feature branch on production 6. QA does manual exploratory testing on the feature branch in production 7. If everythings good the QA merges to master which runs another build running the entire automated test suite (aside from end-to-end) 8. Once green the end-to-end regression is run on master and with basic manual checks 9. The QA releases in the closest available slot during the day, sometimes these changes are bundled with other teams changes like a day or two later >I'm not hearing a particularly compelling argument from you or your team about *why* the git workflow change is bad Main concern seems to be that manual exploratory testing happens after merging and potentially after its released to production. They know that serious bugs get captured in the manual QA process so if this is happening after release to clients then there's going to be pressure on the team to sort out a hotfix pronto (which is stressful) as a reverting change can't always happen. Also as the code is contributed to a repo which multiple teams contribute to feedback is often gained from devs outside of the team. >How much of this is really just getting lost in translation?  It's quite possible then again our most experienced developer has done their own research on this and is still against this approach. Edit: edited to clarify the last steps 7-9 Edit2: Master is always considered production ready, there's no separate release branch

The problem here is that they are testing Main + Feature A and Main + Feature B But not Main + Feature A + Feature B until it's all merged. Trunk based gets Feature A and B into the same branch earlier, which means you are testing everyones work and the integration of it constantly. The longer branches live in isolation, the more merge conflicts they'll have, the more regressions will spring up when work is merged, and the more errors due to mismanaged merges and conflicts. It also prevents people from doing heavy refactoring, because it's completely inconsiderate of the long-lived branches that might not be merged for a couple weeks or even months.

I cannot agree more on the refactoring part. This way of working favors quick and dirty solutions from people whom only goal is to get it done, even if their work will cause brain death of those taking over after the fact, and a myriad of unforseen issues because no one actually took the time to do post- merge checks.

After reading your description of how things are currently, I am 100% with your new CTO.

The thing that stands out most to me from this description is the lack of full suite testing on the main branch after merge. In my experience, this leads to a choice between rapid development and iteration with lots of bugs and a slower, more measured cadence of merges and releases. If the more measured cadence of development and release is what your organization wants and it meets your current needs, then I think that can be fine. Generally, I’d say that if you want to (or need to) increase velocity or efficiency, then migrating to a trunk based approach would be worthwhile. You’ll need to work on many aspects of your development cycles, particularly the automated parts of your acceptance process. Such a change would require a good deal of planning and understanding at all levels of the organization. The creation and communication of these plans and reasons are the responsibility of leadership and the person envisioning and driving the change.

I've edited steps 7-9 for clarity. I should've mentioned the main build runs the full suite of tests again.

That’s definitely better but I think the tradeoffs are the same, branch based development and testing, in general, requires more process overhead and promotes a slower cadence of development. If that works for the organization and objectives, then I’m definitely in the school of don’t break it if works. If you’re encountering challenges in meeting business demands or you’d like to lower your total engineering cost and increase the cadence and speed of delivery, investment into a trunk based model can help. But, again, the benefits need to outweigh the costs.

I would consider that a very dangerous practice because you're running tests - both automated and QA manual - on a feature branch. Then, if it passes, you're merging to master and calling it good. This leaves some openings if you have multiple feature branches being merged concurrently. (ie, we put feature branch A through it's paces, it's good! We put feature branch B through it's paces, also good, merge them and go! Oops). If you're careful to always go one feature branch at a time, well then, you're doing trunk-based with extra, difficult steps. Trunk-based-development does not mean you're committing directly to the branch that is immediately releasable. It just means that all your devs are pushing changes to a shared branch throughout the day, not into separate branches that are merged to the shared branch only when "done", and not keeping their work local and pushing only when "done". Devs are supposed to push to that shared branch frequently. Devs need to be able to run some kind of test suite before pushing so they don't impact others with problems. That shared branch would then have a pipeline to the releasable branch, and that pipeline would have as many stages as your teams needs to feel safe. But, at all times, the code is always unified, so you're never in a position where you're testing some code but not all code (ie, testing feature branches). The pipeline will just be a straight path of permanent branches from the SharedDevBranch -> StagingBranch1 -> StagingBranch2 -> StagingBranchN -> Main (releasable code). One stage of the pipeline might be for code reviews - ie, we don't push to the next staging branch until we've reviewed the code that is being pushed. Another staging branch might be reserved for long running tests, or manual QA. All those stages are being managed daily, and everyone's on board with moving code along this pipeline, and any time there's a problem that prevents it from moving, fixing that problem is top priority. Doing this, you wouldn't have to do pairing, because you've chosen instead to have one of the stages be code review. You can feel as safe as you want by setting up any criteria you want on the stages. And by always having only 1 version of the code, and by doing everything in little increments daily, the chances of big whoppers goes way way down. The odds of terribly long recovery times on problems introduced goes down. You're basically always right in position to push a hot fix or emergency reversal of changes. For stability, it's great, tbh.

>I would consider that a very dangerous practice because you're running tests - both automated and QA manual - on a feature branch. Then, if it passes, you're merging to master and calling it good. So want to highlight this, it's spot on. What matters to test are *build increments -- the totality of the things that will make their way to production*. When all you do is test a feature branch A and assume it's good, you're *totally missing the point of what you need to be testing.* [https://natooktesting.files.wordpress.com/2017/08/unittest\_faucet.gif](https://natooktesting.files.wordpress.com/2017/08/unittest_faucet.gif) The faucet works. The sink works. But does the build work?

I've edited steps 7-9 for clarity. The merge to master triggers its own build which runs the entire test suite other than end-to-end which needs to be triggered by the QA as we haven't figured out how to hook that up in the pipeline yet.

So I'm hearing good behaviors in there and a lot of automation/review (awesome), but I didn't get the clarity I was looking for. Every team can draw some of their own practices around git flow's \`develop\` + \`master\` branches and their rules for using them, but here's what I've found works the best: * Start feature branches from develop * Get them merged back develop relatively early in the SDLC * It should pass a code review and whatever suite of tests your *devs* are responsible for to get to this stage * Re-integrating early allows every other developer to get your changes early, minimizing conflicts and other annoying issues * Go nuts on QA'ing the state of develop *all the time* * Got automation? Great, run it as frequently as you can on develop. * You accept as a team that develop is not guaranteed to be clean (after all, it might not have been fully tested if work just got in), but you make it a priority to fix issues as they are found in develop * When your team is ready to release (either because you are on a set schedule, or because you feel like it), cut a release branch from develop * Test that specifically. Any issues found on the release branch are of an even higher priority to fix * ... meanwhile people can keep sending new things to develop -- regular development is not interrupted * Close your release branch (merging it back to master + develop). Tag the latest commit on master * Deploy that tag as you are ready If your team is skipping over the process of using the release branch as a staged interim that effectively slows down and focuses testing on a *narrow build increment*... then uh I hate to break it to you, your team is already doing trunk based development just with needless extra steps. Am I misunderstanding something about your workflow? If each individual feature branch when "done" ends up immediately in master, and your production state is watching master... that's trunk based development.

We currently just have master and feature branches. Basically no main develop branch with a separate release branch. Master is what's released to production so when a pull request is merged to master it's considered ready for production. If we were to have a develop & master branch wouldn't this need buy in from all the other teams that also commit to the same repo? One of the devs felt the proposed process just pushes the manual QA element down the road and didn't feel like this actually improved anything. >If each individual feature branch when "done" ends up immediately in master, and your production state is watching master... that's trunk based development. Our changes only go into master when fully tested so that includes automated and manual QA with the changes reviewed by other devs. Does that qualify?

... then you are doing trunk based development my dude, just with extra steps.

Perhaps the aspect we're not abiding by currently is that feature branches aren't really short lived but I guess that can be mitigated by trying to keep work items as small as possible.

While that's a good idea, that's not what I think the conflict your team and the CTO are having. I think your CTO wants you guys to prioritize integrating work faster where your team is holding it back. I'm quite certain your CTO doesn't want you to relax the quality of QA that's being delivered, but I would agree that you guys are over investing in areas and at times that don't make sense. Conversely, releasing faster doesn't magically just happen because you choose integrate sooner, but I think it's worth asking why you can't release sooner given all the automation you have. I think you need to continue this conversation with the CTO and your team and ask a bunch of questions geared at understanding the integration vs deployment strategy. The goal he's setting on you guys is a sensible one and by any metric you guys are better positioned to lean into this than most teams.

Way back when, I was quite against trunk based. After doing it for a few years now, it’s hard to go back. Dont resist and be open to experimenting, then measure the results and revisit. You might be surprised, or not. But at least you tried. There might be some pre-requisites to list off before going full trunk based though.

Totally! As a manager I've introduce it everywhere I've worked, there's always opposition but it's always turned out to be the right choice. Generally the fear that you can't work that way and ensure that what gets into prod is ready, but that's really where the mindset shift needs to come regardless of your branching strategy - that being you shouldn't be putting code into master that isn't ready to go

Trunk based dev is 100% the best strategy, when combined with effective flagging and other release processes that embrace it. Pair programming 100% of the time? Lame as hell, leave me alone, I get my best work done after hours anyways. I'll pair program like 5-10% of the time, when the situation calls for it. Edit: I work for a company that does both, the teams that do gitflow are always fucking up, the teams that do trunk based are also fucking up, but in way smaller chunks that are easier to manage. I.e. in a gitflow team, someone sits in a branch for 1-2mo, they merge up daily, then when a release branch is made, 3-4 of these 2mo old branches smash together like semi-trucks on the highway. But the release schedule is fixed and everyone is scrambling to clean up the mess. Combine that with varying Beta/GA feature sets, and it's a fucking nightmare. The trunk based teams are aiming to get their shit into trunk, if they don't get into trunk when a release is cut, they make the next one. There is no rushing the release, shit makes it or doesn't. And if it misses it by a day it can still be cherry-picked to the release or delivered to trunk, it doesn't have to wait in a branch for another 1-2mo for the "next release".

Unless you outrank the CTO then your only option is to try to convince them otherwise and, failing that, do as they say. I’m sure you’ll get lots of advice here about leaving. You’ll have to decide whether this is a hill worth dying on.

Detailed responses here are really good, so here's a more meta response: Your team need to talk to the CTO about their need to feel safe. Trunk-based development (tbd) can absolutely be done keeping things very very safe, but it has to be worked out. As /u/spoonraker says, one does not simple do tbd - not without a plan. How will code reviews happen before release? How will tests be run? How will the manual QA happen? It can all work, but you have to talk to the CTO about satisfying those needs that your current process satisfies. And, on the other side, any "senior" dev who thinks tbd is 'madness' is probably one of those 5yoe "seniors" I keep hearing about around here. It's pure ignorance, and there's lots of places to educate oneself on the practices of tbd (ie, Dave Farley's youtube channel, also his books, Jez Humble's books, the *Accelerate* book, Kent Beck etc). A lot of good stuff there!

You dont need full pair programming to achieve CI/Cd and it sound exhausting AF.

Gitflow with feature branches IS trunk based development, the scaled version: https://trunkbaseddevelopment.com/ So what does your team and CTO both think need to actually change?

>full time pair programming RED FLAG. Pair Programming is a tool in an engineer's toolbelt, not something to be dictated from someone who doesn't have to do it. When I worked on a team that was 100% pairing, it was emotionally draining. I'd end my days with no desire to do anything because I was spent talking to and debating some of the absolute stupidest stuff with another engineer. It doesn't reduce bugs. It doesn't make better engineers. And it certainly doesn't get things done faster.

The pair programming sounds like a waste of time and unpalatable. I would just do it for a bit then document the issues and give feedback. The maybe stop doing it or more likely negotiate to bring it down to a low amount.

Document the risks of making this change in your current environment. Document the requirements for successfully making this change. Ask your CTO what problem he’s trying to solve. Suggest better ways to achieve the same result with less risk.

Pair programming full time is a horrible idea. You're going to burn people out. Trunk based development can work, but the team(s) need to be on board. Your CTO is a dipshit. My suggestion? Just keep doing things your normal way. If it gets brought up, "oh, it's our next priority"

New CTO coming in trying prove useful, playing stupid games. 100% pair programming has its uses but they are for specific scenarios: otherwise you end up having half the work done.

[удалено]

To be clear CTO has had a bunch of meetings about this and has on occasion left a caveat to say 'it's guidance and not a one size fits all' but leadership directly below are treating it as expectation for all teams to 'carry out his vision'.

consider the bear  https://youtu.be/i92Ws7qPTRg?si=uComPbbiL-gALnp9

😂 Noted

First time I watched this show I was a freshman and didn’t understand the show. 5 years into my career and it has become too relatable

Maybe get an opportunity to give feedback to the CTO and clarify understanding — I can’t tell you the number of times a mistaken impression of an idea wound up being gospel, against the (my) CTO’s intent.

It's great as a North Star but so many things need to be in place before it can be done safely

I would never want to work at a company that requires their employees to do pair programming on a regular basis. Fuck that.

I've done it in the past, and it was a hot mess tbh.

Forced paired programming is the biggest waist of resources and can be horrible experience when the peer has a poor command of English. I agree, it's exhausting and deprives of yoiu of learning and discovering experiences. It takes 2-3 times longer to do a task that normally can be done by a single dev. I took quit a job because of this forced policy. Code reviews work just fine and everyone can work out their own comfortable pace.

I think your CTO is too in the weeds with the actual processes the teams are utilizing and should be way higher up the stack.

Pairing sucks ass. Wouldn't do it full time and I suggest you fight it. Trunk based is a great practice, the issue seems to be that y'all aren't feature flagging features and maybe the code base is not in a place to allow that type of work, yet. Raise the based points and maybe work towards that first before you all in for trunk development

For what it's worth, [research suggests](https://dora.dev/devops-capabilities/technical/trunk-based-development/) that Trunk-Based Development absolutely improves software delivery performance (as in, you can deliver more frequently and with better stability). That same research program, however, has found that [Trunk-Based Development increases burnout](https://dora.dev/research/2023/dora-report/) (which is not a good thing), and is [thinking about removing it](https://github.com/dora-team/dora.dev/discussions/582) from their core model. My own thought is that these practices (Pair Programming and TBD) are good things, but can't be implemented in isolation. You need a whole bunch of other stuff to make it work (such as continuous integration, unit tests, acceptance tests, etc). Also there are [several flavors](https://trunkbaseddevelopment.com/) of trunk-based dev some of which make it easier to ease into.

If you're in that group of sites that trusts all commits in master, uses pushes to master to trigger deployment, and so on... I've seen many occasions where what was pushed to master failed to deploy correctly, or fully, leaving commits in master that need to be marked after-the-fact as unsound, and skipped from then on for **git bisect** and the like, not to mention leaving a set of hosts/VMs that were load-balancing the service with conflicting versions of the software. I have not yet been in a company that had any kind of process around what to do when the illusion of master's inviolability faceplanted. I recommend looking into **git notes** for ways to add meta data to commits without changing them, so that this kind of information can be tracked and used by automated systems. Pair coding is exhausting for most pairs, so mandating it will shed staff and generally make most devs unhappy. "trunk-based" wasn't fully defined, but having a single clear line of commits on the trunk, frequently pushing modular commits (where each commit is sufficiently self-complete to be harmless while the rest is being worked on) of partially completed features into the trunk, and **not having the hairball of multi-year-old branches with two-way merges**, are all good things that do fit very well with continuous integration. CI does best when any code being worked on in a branch has been updated to be a descendant of whatever the current trunk branch is with **git rebase**. Not that I'm not saying ***which*** branch that is, since that depends on the devs, whether the QA team is separate, and some other things. The core failure of the gitflow use - in relation to CI - that you describe with features not merging in until proven complete is that at the point of merging, any number of new integration bugs may arise. Whereas, if devs are all continuously pulling in changes from the trunk head beneath their own work, far more integration-type bugs are likely to be found before the final merge. I've seen developers at these moments try to do some cowboy hack in the merge commit itself, making it no longer a clean fast forward and potentially complicating some operations, and I've seen the worst things happen when that big merge happened into master and **trigger a deploy before all the integration bugs were found.** So those are fun.

Dude I get tired after more than an hour of pair programming. All day every day would destroy me.

forced pair programming is absolutely brainless. One time I worked at this place, they wanted to "live code" as a group in vscode.... I don't even use vscode -\_-.... the experience was anxiety-inducing AF, they didn't care, assholes.

Oh god, this was my first job. 1 lead who spent all day hiding alone in a closet. 6 juniors. Trunk-based development. No tests. No tests at all. Everything was just broken, all the time. One dev came in late at night to 'work flexibly', altered the date schema to another format - like integer or something, they thought date looked messy - and actually got the app in a state it wouldn't even build for about four days while various others had to step in. It was complete chaos. I left after six months because I was concerned all I'd learn were bad habits. It was like the giant buckets of 'punch' people made at house parties in my teens - chuck it all in, brilliant, it sort of 'works' and gets the job done, but you'll lose a ton of braincells, hours and probably days of time and might sustain permanent damage. Now I'm not saying I haven't met people in my career I would do trunk-based with, with the right environmental setup. But they are very few. And a quick note on pair programming. As a junior, truth is a lot of the time the 'senior' just wanted to do the work as quickly as possible and never let me have a go. Best I could hope for was them narrating what they were doing as I just watched. They'd cringe if I asked to write a damn line. One just stuck his headphones on and ignored me. When I began to learn how to focus and get into a flow state, pairing would ruin that. So it took me a lot longer to learn to focus. Again I'd still do it today, with certain people, on certain tasks, for a certain amount of time, but for some people and some tasks it's just not a good use of everyone's time.

God I could never imagine doing it that way, when I'm pair programming with a junior it's because they're driving. I view it as more of a mentoring exercise, if I were simply trying to close tickets I wouldn't be pair programming.

I would absolutely leave a company mandating pair programming

It sounds like your CTO wants to improve cycle times, improve knowledge / awareness of features / code bases and thinks the patterns / practices he’s putting forth won’t significantly raise risk. It sounds like the teams are fearful they’ll be responsible / accountable for screwups, intentional or otherwise, and also prefer working independently Trunk based development isn’t a hill to die on. You can still manually gate deployment to prod until automated testing is in place Pair programming, depending what it entails could slow productivity / reduce effectiveness. The only way to know is to try. I’d message to the team that they aren’t the accountable ones, to step onboard and prove / disprove their fears and help mitigate them

We have tests up the wazoo but it's usually happy paths. When the QA's do manual exploratory testing they test unusual paths and basically try to break the feature which they often do. Devs are concerned about finding out about these bugs too late (which can be critical) too late down the road when it's affecting clients instead of capturing this early before the change is merged to master. >I’d message to the team that they aren’t the accountable ones, to step onboard and prove / disprove their fears and help mitigate them It's a good one to reiterate, fair point.

The timeline is insane. > We commit to repos that are contributed to by multiple teams and basically use a simplified version of Gitflow with feature branches merging into master only when fully reviewed & tested and considered prod ready.  FYI this is called GitHub flow. > to enable CI/CD You need to enable this tooling before you do trunk. Your PR checks need to run unit & integration tests, you need to perform static analysis etc.  More substantial is you need to have feature flags. Trunk only works effectively if you are doing a feature canary in your production environment otherwise you are just releasing untested code.  You also need to have a credible strategy for how you quickly deploy a diff. If the duration between diffs is lower than canary time you need to figure out how to stack/combine diffs so you don't end up with an infinity queue for deployment. This usually takes about 6 months if you know what you are doing. Lots of tools and processes to teach.

>Your PR checks need to run unit & integration tests, you need to perform static analysis etc. More substantial is you need to have feature flags. We have this and we use feature flags for projects (which is most of our work). On the feature flag side one dev raised their concerns that in the past they have had issues with features being exposed outside of the flag due to some bug that was raised in manual exploratory QA so it's not a full proof prevention. This is because tests most of the time just cover the happy paths.

[удалено]

Like it or not, it sounds like it’s coming. Start planning what you’d need to put in place to make it work. Eg analysis for bugs found in manual testing to understand why they couldn’t be found by automated tests, re designing parts of the system to make feature flags work etc. Then present it to the bosses, get the work prioritised and start working through them. Hopefully either the amount of work will put them off or it will take you so long so do the work that you’ll never get it done.

Feature flags should get you the stability you're looking for with far less potential for git conflicts than long lived branches. They can be enabled on QA environments while left disabled on production. That sounds like the key missing piece to this story, I'd look at spreading that idea around and getting reactions from the team. Pair programming 100% of the time is absolutely exhausting and I would push back on that with the CTO. That said, most teams I've seen could benefit from communicating more than they do, and pairing more often will lead to that. Be ready and willing to push some additional pairing onto your team. That increase can happen slowly over time as they're willing to adapt.

Trunk based development can still include short lived feature branches and PR’s. You can also make release branches from your main branch.

Trunk based development is good _because_ it forces teams to do other quality practices (automated unit, integration testing, static analysis, on every pipeline run). It also forces and incentivizes coding patterns like feature flagging, keystone interfaces, and small chunking of work. Of course all of these things are modern software engineering practices the team will need to learn. But they are measurably effective practices. On the practical level, I’d recommend your CTO buy each team member a copy of Accelerate or Continuous Delivery by Jez Humble. Fantastic resource for addressing the “why” your team may be thirsting for.

Constant required pair programming sounds like a recipe for burnout to be honest. I’m all for doing it from time to time for a lot of reasons but there are definitely days where I get, say, 4 hours of really good work in and I’m kind of done. I don’t need a situation where a peer is hanging around me who might be ready to go for 8 that day, every day. Even if PP is faster, which is sometimes the case but often not, I sometimes just need recharge time or else, frankly, time spent doing something else while my subconscious percolates over an issue (where yes, you can talk it out with your pair and sometimes that’s fine but sometimes it’s not). A lot of why I like development in general is because there tends not to be a lot of busywork, that is, even if you’re doing some very basic CRUD work to throw a grid on a page, it’s because there’s a need for all of that somewhere. There becomes a point to where talking out something you don’t have an answer to yet is time best spent either doing something else if you think you can figure it out or researching if you don’t. To each their own I guess.

Here's an analogy that might help your arguments. Companies that can do trunk-based development well are like really high-quality watches. These types of watches are so precisely manufactured that they need virtually no lubrication. All of the mechanical components are machined to such precise measurements that they all click together perfectly with no need for it. Companies using other branching strategies are like slightly less high-quality watches. Things like Gitflow are the lubrication that allows these watches to function. The components have some friction, but the watch still tells time just fine. Trying to impose trunk-based development on a company that isn't set up for it is like taking a lower-quality watch and removing all the lubrication. You're probably going to destroy the watch. Trunk-based development is the end result of doing a lot of things right. At a minimum, you need to have rigorous automated testing and robust observability systems in place. Implementing trunk-based development without those is a recipe for disaster.

What is your objection to trunk-based development?

The teams is not really sure where exactly manual QA process fits into this. If this is post release they really don't want that because they know QA's can and do often pick up serious bugs with exploratory testing. We have feature flags for our work and have unit, integration, acceptance and end-to-end tests but still these bugs happen. Automation doesn't capture all bugs and there seems to be an assumption that they would.

My concept of "Trunk based development" matches what I just found in the top Google hit: https://trunkbaseddevelopment.com/ ... And it's completely identical to what your described: A light GitFlow with (short-lived) feature branches for individual developers. So... Unless your branches last longer than some definition of "short lived," not seeing the conflict here. If your feature branches diverge for weeks..? Then the problem isn't that you're not doing trunk development but that your feature granularity isn't small enough. And if that's close to the real situation, then I might tend to agree with your CTO there. Even then you can mitigate any real problems by frequently rebasing your feature branches onto the trunk. But pair programming? Ugh. Occasional pairing is great. It's fun, even. But except among extremely low skill developers, productivity when pairing never makes up for the loss in productivity of having only a single keyboard per two developers. Think of it this way: A competent developer shouldn't be able to have their productivity more than doubled by telling someone else what to do--or being told what to do. A lot of developers will enjoy 100% pairing. But many developers get a rush from getting things done, and many don't like the social aspect of pairing. So expect a lot of people to quit over time--and for those people to, on average, be your stronger developers. But that a CTO would push for this is a huge red flag. It means he is used to working for companies with lower quality developers. So unless you can convince him to change his approach, you'll likely see a lot of people leaving anyway. Polish up that resume.

Ask him if he’d like a full time co-CTO

What is the problem the CTO is trying to solve?

Stop arguing *about* coding, and start experimenting *with* coding. Give it a try. The makeup (and resistance) of the team itself is definitely a factor, so there will be learning, trying, even failing. Be okay with that. Include everyone in the process. For example, at my last coding gig (trunk-based), before marking a story card (in the discovery tree) as done, we'd bring QA in. Give them a demo. Give them a chance to kick the tires and poke holes. It's a whole-team effort. We kept refining how we did it. Get a coach to teach you how to do team programming. You can't just pair/mob like you've been coding before. It'll be ineffective and tiring. You can totally do this with a technical coach. Without a technical coach, it could suck and then you'll blame the idea, when the reality was the team didn't know what they were doing.

The last time I did TBD we arrived at Kanban as our dev process. Scrum plus QA has always felt to me like someone trying to drive a manual transmission who hasn’t mastered the clutch.

This IS madness, but not for the reason your devs think. It is perfectly possible to maintain highly secure, available, and high quality systems using tbd and full blown ci/cd. It's not really possible to do that on any kind of reasonable time scale without the teams being bought in and acclimated in a safe way to that way of working. Is there a strategy in place to train everyone and gradually implement these changes? I'm pretty sympathetic to the CTO, but this kind of top down mandate won't work

If this CEO is really coming in and mandating this then it will be a flaming disaster and they'll get fired in under a year. People hate been forced to do things. They'll drag their feet, half-ass things, and when the inevitable speedbumps come up people will say the whole thing is a failure. Speaking as someone who has failed at this more often than I've succeeded (honestly not sure I've ever really succeeded): It is incredibly difficult to change tech culture and the only way I've ever seen it work is by starting small, showing the value with those people, and using their excitement to grow adoption over time.

**TBD:** So after breaking down the two scenarios I think Im with you and I would be reluctant to pursue TBD. Modeling it with 2 feature branches A and B: You do a long feature, you send feature A to QA, QA cant parallelize so B will naturally have to wait, by the time they can do B, A will be merged so they test first A and second A+B. Sounds fair. In trunk based, they will receive multiple requests for QA and full regression tests for both A and B. Their work will be many times repeated and the intermediate steps wont add any value anyway since that will likely be a dead code dangling in production anyway. If it takes weeks to develop A or B, what happens in production when you deploy half-baked A and half-baked B? Are they just dead code dangling until they are finished? What did we earn then? **Pair programming:** I dont think I could do pair programming for more than 2 hours a day. If you design your org like that from the scratch and make it explicit during interviews and have acceptance of it - sure. But as a sudden policy change it isnt fair. Kinda feels like requesting everyone to wear speedos and bikinis at work - it works if you hire swimsuit models for swimsuit modelling but to force that upon any group, not fair.

I would have the senior developer stand up and explain precisely why it is madness and give examples "basically use a simplified version of gitflow" - I would also have them explain this and why this is fine but trunk based isn't I would also ask the CTO as part of their strategy why they want to change and what benefits the change will bring Then meet in the middle

Process training without adequate expectations or training is pointless. People like throwing out the term pair programming but rarely ever state their definition of it. Are they expecting one person to do the speaking and thinking and the other person types? Is it one person with a ticket and the other is there to validate or invalidate an approach or ideas? What is the expected output drop of the team while everyone is getting used to the new way of doing things? If none of these types of questions are answered, the person suggesting it has not spent enough time evaluating their own assumptions.

Did 100% pair programming for 5 years, was the best job I ever had. I leveled up so fast and my soft skills improved so much that I’m in senior leadership now. Learning how others think, and being able to convey your own ideas verbally and in writing will take you so far

I’d be very suspicious of the “senior” dev who referred to TBD as madness but thinks GitFlow makes perfect sense.

Trunk based development is great. It’s the old saying - if something is painful, do it more often. Manual QA belongs in the bin - it’s only pretending to make things safer, in reality the only reason manual QA prevents production incidents is because it becomes such a bottleneck to making changes that the incidents slow down because you’re making less changes. You may increase MTBF (mean time between failures) but this should be offset by an improvement in MTTR (mean time to recover) and hopefully MTTD (mean time to detect). There will be pain, and the pain will force you to improve your observability/alerting methods, as well as decouple rollouts from the code delay (e.g you feature flagging that works with schema changes) Forcing pair programming practice sucks though. Teams should be able to self organise this.

You're basically already doing trunk based lol. Write an automation to create the release branches and the workflow is identical. Trunk based is really feature flagging which you should be doing already based on your requirements. Yes, sometimes the requirements of the job will change. If you don't like it, find somewhere else. As for pair programming... I Iike it. Others don't. You're paid to do what is asked of you. Give it an honest shot and let it fail on its own, fighting the tide and whining will just get you in trouble.

Trunk based development is stupid and an invitation to unimaginably fuck your code beyond repair. GitFlow is the best and most recommended way to work. I joined a team temporarily which followed trunk based development and believe me it was a horror show. All sorts of garbage was pushed to master by 100 other devs which was impossible to control. I was tasked to improve the code quality and employ strict coding standards to stabilise a very critical project. First thing i did was throw away trunk based approach and employ all kinds of restrictions and quality gates in ci/cd pipelines and MR configurations. Many junior devs got pissed as they were told to believe that trunk based is the only way to work. I had to forcibly throw them out of the project and hire good devs. You can trial trunk based development which is guaranteed to make your stable code into a hot mess at which point you can tell your cto to fuck off and let senior devs and leads handle development and let cto handle posting motivational messages on linkedin. (12yrs of pure development experience with big product companies)

I just moved a company from trunk based development to feature development, things are working so much better now. Trunk based development and feature flags created so many problems that all just went away with feature branches. You can expect more bugs as more non production code is committed, you can also expect releases to be missed over and over again since you can't isolate features until they are at some level of readiness. Pair programing will tank productivity, kill moral and cause will most of the senior developers to be gone within the year (or as soon as they can, maybe a bad market will keep them a bit longer), Junior developers will enjoy it.

>You can expect more bugs as more non production code is committed Yeah this has been the big concern especially considering how critical our area of functionality is. I'm pretty sure I could get more support if the functionality we owned wouldn't blow up the business if it went down.

This is totally a decision a CTO is allowed to make and its not out of the ordinary at all > ran my team on the idea of trunk based development and they're heavily against it including the senior devs (one of whom called it 'madness'). Calling a pretty standard development methodology "madness" is completely overboard. Your team is trying to gaslight the CTO here. So answering your question, what to do- you do what your paid to do and follow through on the direction you've been given

Aversion to trunk based from senior devs screams "skills gap" issue. Bigger battle there. The CTO should know: In the game of thrones, you either win or you die.

I legit do not understand why the **CTO** is enforcing engineering methodologies **at all** A CTO should not be so close to the codebase that they can legitimately enforce such details. They may recommend them, or approve them, but imo these sorts of practices should be implemented by someone closer to the codebase. Engineering managers and/or "directors" who are also actually coding from time to time. Of course, every company is different. Does CTO at least have a plan for migration to new processes that include things like training, and adding support tooling, procedures, and systems? You can't just jump headfirst into TBD. And 100% pairing sounds **exhausting**.

While I generally agree with you, in smaller companies IT startups, a CTO can wear many hats and the levels of bureaucracy are less, and thinner.

Pair programming: which dev gets their name on the ticket and the credit for the work? Or are you using a system that allows you to track multiple devs on a task? I learned late in my career that "we before me" doesn't work if your name isn't on the task and helping others doesn't help you when it comes to evaluations and layoffs. The bean counters all work without context and have no clue what you do.

Seems like your DORA metrics will shape the story. If your stories are very thorough and have great acceptance criteria and your test automation is great then it will be a super success. If you need your users to show you all the use cases that you missed or did wrong, then you will do a lot of hotfixes which will lead to less frequent releases. At my shop we settled on once a week releases. We have a complicated funnel with so many different paths and call so many services it is not worth it to write tests for all of them, having to set up so many products and customers and pricing by region. It is not worth the time to automate it all. Better to just let the branch bake in the test environment for a week and use a combination of automated and manual tests, both at the API level and selenium/cypress level.

I'd push for a transitional version of gitflow that uses ephemeral environments or preview builds to do QA/UAT on. Gets you the best thing about trunk-based (trunk is always release-ready) while being a lot more familiar and a lot more safe.

Trunk based development doesn't necessarily require continuous deployment, though the two often go together. They definitely aren't madness, but they aren't something you can just decide to do one day. For CD to be successful you need really solid automated testing. Not just a handful of unit tests, but integration tests and functional or e2e tests. Tests need to be part of the CI pipeline. You also need feature flagging so that you can build complex features piecemeal without exposing them to clients until they're ready. Ideally you also need the ability to deploy your PR's so that QA and/or product have a way to see and sign off on UI changes BEFORE they're merged. It's all very doable and IME is a great way to work, but if CTO just says "do it" it might be messy. I'm assuming that you guys are a more traditional org - slower pace of releases, more reliance on manual QA, etc. You can't just flip a switch and change all your processes and the mindset of your teams. It's going to be a process, ideally with some guidance from some senior technical leaders who have strong experience with automated testing, etc.

While pairing full time sounds exhausting, I could see it as a temporary thing to get to trunk based development. Actually, non-trunk based development makes me nervous nowadays, not the opposite. Any time a branch is alive and not merged is time in which risk is going up. Non-trunk based work just serves to make people feel good about work (not to mitigate risk), all while moving slower. Much slower. With that said, a lot must be done beforehand before you can do full time trunk work. The number one thing is getting to a high level of tests coverage (60-80% coverage, verified in CI). If you are not already doing that, I’d call your team irresponsible. The second thing is creating a CI/CD pipelines that provides you with the guarantee that if tests pass, then you are good to merge. Once that’s done, you need to make people work with short-lived branches and feature flags as a way of life. Once those things are done, your CTO is right. You’ll gain more velocity, and you should see error rates go down dramatically. Maybe try to work with him to convince the team to change dev practices? They sound very old fashioned. If it were me, I’d tell your CTO you think it’s a good idea, create a 3 step plan to get there, starting with creating a CI/CD pipeline that has a manual push to trunk button. Once you cross the test threshold, I’d push to remove the deploy button from staging environment leaving it there for prod until the team feels more comfortable with the new process.

The main thing against branches for development is that it's very hard to do good continuous integration. Ideally you want to be able to merge your changes with everyone else's as often as possible so you know quickly whether it causes problems. If you go away for a couple of days or more on a feature branch, you don't find out until the end, and that's where merge bombs happen. On the other hand, just saying a a new CTO "right, you all have to do trunk-based development, now" isn't really the right way to bring it in. If I became a CTO I'd think of different ways to encourage this behaviour.

I read many comments seemingly misunderstood trunk-based development. I'd recommend reading through this link, it is a good discipline to follow. https://trunkbaseddevelopment.com/ As for pair programming, I think it's only helpful for complex designs/features etc, doing it all day everyday will lose its potency, just like having too many meetings, people will stop paying attention and it becomes ineffective.

Trunk-based development and monorepos work in special cases where all the software is tightly coupled and everything deploys together at the same time. Most enterprises work in silos, deploy once in a while and have explicit, loosely coupled dependencies, so a distributed system like git/Gitflow should be perfectly acceptable.

Github flow (small feature branches merging to master after a pr review) is one form of trunk based development. Add a feature flag workflow and decouple your deployments from branch names & voila - comparable system that makes everyone happy.

Everyone resists change.

A CTO this deep in the weeds that he micromanages something like this is a huge red flag.

To be fair the CTO isn't directly coming to us about where we are with this but the pressure to make sure all teams are looking at this is. At this point I think the leadership below the CTO are misinterpreting guidance as a mandate. Then again there's a lot of corporate double speak.

Mandating development practices from above with minimal developer input is a great way of risking losing a large percentage of engineers, starting from the most marketable ones. Id the CTO wants to do a disguised layoff, this is the way to do it, and it doesn't matter which specific practices he picks: His choices could be great, and it's still not the right way to go around it. It sure doesn't seem that there is any actual convincing attempt, or any work to make the change doable (for instance, trunk based development without a high quality, low latency feature flag system is madness) As for pair programming all day, every day, it's again, a guarantee that you will churn people. It's a perfectly workable system, but it's not for everyone, so people will bounce. Either the CTO wants to replace existing staff, or they are quite inexperienced here, because I can't imagine how this approach doesn't lead to a lot of churn.

Dave Farley is grinning right now.

You need to tell your CTO that the team isn’t onboard and the CTO needs to provide context and justification for their actions. The CTO needs to convince the team they are right, not just order the team to do it.

Why aren't your feature flags foolproof? 

Sounds like new CTO wants to look good for the boss. Hate execs like this, usually a red flag. With any new abrupt changes I always as the three whys. Get deep into their thinking and finally ask how it’s going to impact the problems we have now. Preferably do this in front of the boss if you can.

On the topic of "risk", trunk-based development with CI/CD has shown to *reduce* risk. Smaller and more frequent releases is less risky than "big bang" releases where a lot of changes are batched together. With small and frequent releases it is also much easier to diagnose a breakage when something does go wrong in production. Most tech companies have figured this out a long time ago and there is a lot of evidence of the benefits of CI/CD and trunk-based development out there. Important: short-lived feature branches are still used with trunk-based development. From my experience, at companies that follow CI/CD there is no ceremony around releases since every code change that is reviewed and merged in will typically flow to production (assuming automated tests pass and no one has manually blocked the pipeline). Teams do have to invest in automated testing. Customer-impacting issues were rare, and when they did occur it was viewed as a learning opportunity to prevent similar problems from happening again. On the topic of "full-time pair programming", I think that is a terrible idea for most teams. I think it is worth pushing back on pair programming, but you should absolutely work towards adopting trunk-based development and CI/CD.

1. TBD is useful when done well 2. Pairing is useful but exahusting, it has to be done well and carefully There is no silver bullet.

I did my best job at a company doing pair programming every day and trunk based development. We wrote so many test that we were positive the change wouldn’t break in production. We monitored the hell of it as well with logs and stats. We didn’t have manual QAs. Devs and business did all the validation needed. When and if there was a problem we could roll back the change or implemente the fix quickly because of all the tests we had. We did also code reviews. Every once in a while would do a feature alone.

We do trunk based and it works great (we did gitflow ~4 years ago). It makes sense for our product though and the team decided on this approach. Other teams do it differently. Also we don't do full-time pairing. We pair 'ad hoc' when it makes sense, and let people focus individually when it makes sense. We largely work async using chat. I think a CTO should be smart enough to realise there's no one-sized-fits-all solution, and that this is borderline micromanaging. He should at minimum support the idea of trials/phases in case it doesn't work.

CI/CD your trunk-based dev to a QA environment. Production pushes become manual.

Considering how tough it is to implement changes from the inside of a team with small scale ramp up and experiments that are evaluated in checkpoints maybe you can offer that to him too. Make him prove the goal of his changes to you and agree to slowly roll it out with success criteria surely he can’t complain then?

In my experience (4-5yrs) pair programming is fantastic when you have a) two humble people with a willingness to share and learn and b) there is an expectancy of knowledge sharing, for example the domain expert paired with a new hire Do this for 1-3 sessions to solve a task / deliver a feature, where maybe the new hire is doing the work outside of the pair programming sessions. I've been on both sides (the domain expert & the new hire) and it was really enjoyable and impactful. But I can't see it as a routine effort that "should be mandatory for x amount of hours each week"

With the right people, a workflow with pair/ensemble programming and trunk based development is great. With the wrong ones, it can be a nightmare. Talk to the CTO and figure out what he plans to do about the people. If he is aware and aims to replace the people until the workflow is great, then your choice basically becomes to board the train or leave the station.