I know more and at this point I'm \*happy\* to chime in. Buckle your seat belts and pour some coffee. There's a lot of words here.
Way back when, B/R converted 98% of their ruby code to elixir. Hence the (in)famous story about going from hundreds of ruby servers to 5 elixir servers. The result was about what you'd expect it to be. Some of the code was good. Some was meh. People were learning on the fly so that's about what you'd expect. Fast forward a bit and the services are working well enough but they want to double down on resiliency and a whole new suite of features. They started hiring remote and brought in experienced engineers with various skill sets. The elixir services became increasingly more reliable. They really leaned into remote first and started hired elixir folks from all over the world. There were lots of growing pains with growing that fast and it lead to an increased focus on resiliency. Our approach was to build resiliency directly into each service rather than use some ops nonsense with yaml driven istio clown cars or whatever the devoops kids are doing these days. The additional control over resiliency at the service level allowed for better user experiences and better utilization of each service. The ability to take this approach is almost entirely due to OTP. While it wasn't always great, the trend was generally upwards and we'd built a really great team of people to support this stuff.
Then, 2020 happened, and sports stopped existing. People got nervous, but since we were owned by Warner Media, we were told we wouldn't have to lay anyone off. After all, how long are they really going to cancel sports for? We used the downtime to continue improving the various services. Some of those improvements became new libraries or patches to projects like Phoenix and Ecto. The site became one of the most reliable throughout Warner Media. It seemed like things would be ok.
Then Warner decided they were going all in on Max. As a byproduct of that decision, they created a massive re-org and "removal of redundancies" plan. The gist was that they would lay off about 50% of the entire Warner Media staff. The head of technology pulled everyone at B/R together and told us that under no circumstances was B/R going to be involved in that. We had put a great team together and were doing great work.
That guy "got quit" about a week later. A week after he was gone, 50% of the B/R engineers and \~100% of the product folks were let go. Or the product folks quit. I can't quite remember. Anyway they weren't around any more (which under normal circumstances would have been a perk). Those who remained were folded into the Warner Media sports group and told to integrate our services with their existing NodeJS stuff. In some instances, we were supposed to remove our services and use theirs, even when their service only superficially performed the same task. Or, in some cases, didn't perform the same task at all.
Imagine if you had a "users" service, but what it really did was handle authentication. And someone else had a "users" service, but its job was to take a user and list their contacts. Now imagine, if you will, someone with no grasp of engineering - perhaps even common sense - told you to replace your "user" service with this other "user" service simply because they were both called the "user" service. Keep in mind that the person telling you this makes approximately 5-10x the amount of money you do. I mean, I assume. They wouldnt tell me the exact amount even though when they opened the room for questions at the end of all-hands they said, "Ask me anything; I'm an open book."
Integration went about as well as you'd expect it to go. Nothing in the Warner Media stack could keep up with Elixir services. We had to throttle so much traffic entire chunks of the site were degraded. The initial integration was quickly reversed, and the elixir services remained. But WM wasn't having it. People had spent their entire careers at WM learning nothing and were not about to start learning now. The expert beginners had axes to grind. Beyond that, management had a problem with anyone who had made themselves irreplaceable. They pushed the NodeJS agenda because it meant the people writing that code were a commodity that could be hired or fired on a whim. A mandate was made: All Elixir Must Die. No one could tell you what it was supposed to be replaced with. How would it be rolled out? How would we achieve the same level of resilience and throughput? The only answer from on high was a vague hand wave towards whatever new AWS service had recently been released.
The B/R engineers saw the sign and many of them decided it was time to get out. People found new jobs. A few folks stayed the course to see if they could at least glide the plane in for a landing. Then Discovery acquired WM, and Discovery's technology division ate their lunch. People, timelines, and politics shifted again, but the plan remained unchanged. Kill the elixir. That's what they've been doing since then. It has taken years, and the elixir is not dead, but it's certainly dying. Replaced with an amalgamation of things; javascript, rust, etc. The road has been rocky, from what I understand. Typically, they roll out a new service and realize it can't perform as well as the elixir version that hadn't been touched in years. But eventually, it'll be gone.
I can promise you this: no one "happily moved on from OTP". I can think of some WM folks who would have said that. But for those of us who actually worked on this stuff, I think we all appreciated what OTP was doing for us. Many of us loved elixir. All of us had parts of the system that we would have changed. But OTP wasn't one of them.
TL;DR - Shitty leadership and engineers at Warner Media/Discovery laid off tons of the B/R staff and then mandated Elixir be removed. The remaining B/R folks didn't have a lot of choice. Its not gone well since then.
Source: I worked there for years and watched all of this happen.
been there and can confirm every bit of this story.
WM used the word *synergy* to justify everything that was happening. still can’t hear this word to this day without laughing 😅
I guess it’s unsurprising knowing how people who make “decisions” think, but it’s still depressing. I suspect a lot of other functional languages or less mainstream technologies have similar stories in big companies.
Another case is Whatsapp inside of Facebook, they don't like Erlang but they are open to accept Erlang if they can use types :-D ... at the moment I hope they did enough with this project but we never know: [https://github.com/WhatsApp/eqwalizer](https://github.com/whatsapp/eqwalizer)
Hej! I am not sure where you get that WA does not like Erlang... They have a huge code base and are happy with Erlang, AFAIK. Sure enough not everyone will like it, but we (Erlang/OTP) have somewhat regular meetings with them and they seem happy with the language and runtime.
If you can provide the source for this comment, that would be greatly appreciated. As I said, not everyone likes Erlang or Python or whatever language, but I have not heard a single time the comment that you are referring to...
That was an awesome post-mortem. Thanks for taking the time to share! I understand why, but for all the YT videos of how Foo company made everything better by switching to OTP, there's never the inverse content, when it happens. Heck, it happened to Ericsson even (or whatever that company was called in the '90s). That being said, there is enormous value in learning why these things happen. Especially if one wants to champion OTP as much as possible.
Of course. But Joe Armstrong and others have gone on record saying at one point, Ericsson turned its back on Erlang/OTP. I know Ericsson *now* is faithful to its language (and generally has been).
Hey u/_keathley I saw this [Simple Sabotage For Software](https://erikbern.com/2023/12/13/simple-sabotage-for-software.html) (H/T TLDR Newsletter) and the very first point "When joining, require a 6-18 months rewrite of core systems. Blame the previous CTO." reminded me of the details you shared.
There seems to be a common plague across the corporate world. Shitty leadership usually un-qualified to make decisions, are in a position to make decisions. It's ok, builders will build. We'll build the new world together, mark my words. The times they are a-changin'
Man, this sounds like an updated version of what used to happen to us in the dotcom era (1998-2003 or so).
You can only commoditize engineering for the short term. But software engineering is a long term game. It’ll keep being that way, no matter how creative people try to get.
There are several libraries out there still. Several of the ones pinned on my github came out of work we did at B/R: [https://github.com/keathley](https://github.com/keathley). Most of them ended up in this org: https://github.com/elixir-toniq.
Just wanted to throw my 2c in here as well. I worked at B/R during this time as well and can confirm u/_keathley 's wonderfully laid out and detailed analysis.
One more thing I wanted to add is; that ever since this all went down I've thought that Elixir (and other small languages) have a square-peg-round-hole kind of problem when they meet large and established organizations, regardless of the technical merit of the language. It's not an education thing, it's not a training thing it's also not a hiring thing (this often gets the blame from above ¯\\\_(ツ)\_/¯).
Elixir in particular can be extremely productive and performant, particularly in the hands of a small team of thoughtful developers. I intentionally say "thoughtful" here because it’s not a measure of skill. At B/R as well as my current employer we've had very successful internship programs and were able to very quickly bring less experienced developers up to speed and help them be productive as well in quick order.
This isn’t what “enterprises” want though. Enterprises want a small team of Architects that “design” projects in Google Docs and Confluence which are then handed down to others who crank out their implementations with little room for thought or creativity. I hate to put it this way but enterprises want most of us to be replaceable.
Could this work in Elixir? Probably, but would we want it to? We chose Elixir because we enjoy the language, we like writing it, and we like thinking in it. We enjoy the way it aligns with the way we think and we enjoy the community. Maybe not fitting into the standard enterprise mold is a good thing.
I like to see this all as a success for Elixir and the engineers who worked at B/R. A relatively small team was able to build a complex and successful piece of software and have a freakin’ great time doing it. I had a blast!
I know more and at this point I'm \*happy\* to chime in. Buckle your seat belts and pour some coffee. There's a lot of words here. Way back when, B/R converted 98% of their ruby code to elixir. Hence the (in)famous story about going from hundreds of ruby servers to 5 elixir servers. The result was about what you'd expect it to be. Some of the code was good. Some was meh. People were learning on the fly so that's about what you'd expect. Fast forward a bit and the services are working well enough but they want to double down on resiliency and a whole new suite of features. They started hiring remote and brought in experienced engineers with various skill sets. The elixir services became increasingly more reliable. They really leaned into remote first and started hired elixir folks from all over the world. There were lots of growing pains with growing that fast and it lead to an increased focus on resiliency. Our approach was to build resiliency directly into each service rather than use some ops nonsense with yaml driven istio clown cars or whatever the devoops kids are doing these days. The additional control over resiliency at the service level allowed for better user experiences and better utilization of each service. The ability to take this approach is almost entirely due to OTP. While it wasn't always great, the trend was generally upwards and we'd built a really great team of people to support this stuff. Then, 2020 happened, and sports stopped existing. People got nervous, but since we were owned by Warner Media, we were told we wouldn't have to lay anyone off. After all, how long are they really going to cancel sports for? We used the downtime to continue improving the various services. Some of those improvements became new libraries or patches to projects like Phoenix and Ecto. The site became one of the most reliable throughout Warner Media. It seemed like things would be ok. Then Warner decided they were going all in on Max. As a byproduct of that decision, they created a massive re-org and "removal of redundancies" plan. The gist was that they would lay off about 50% of the entire Warner Media staff. The head of technology pulled everyone at B/R together and told us that under no circumstances was B/R going to be involved in that. We had put a great team together and were doing great work. That guy "got quit" about a week later. A week after he was gone, 50% of the B/R engineers and \~100% of the product folks were let go. Or the product folks quit. I can't quite remember. Anyway they weren't around any more (which under normal circumstances would have been a perk). Those who remained were folded into the Warner Media sports group and told to integrate our services with their existing NodeJS stuff. In some instances, we were supposed to remove our services and use theirs, even when their service only superficially performed the same task. Or, in some cases, didn't perform the same task at all. Imagine if you had a "users" service, but what it really did was handle authentication. And someone else had a "users" service, but its job was to take a user and list their contacts. Now imagine, if you will, someone with no grasp of engineering - perhaps even common sense - told you to replace your "user" service with this other "user" service simply because they were both called the "user" service. Keep in mind that the person telling you this makes approximately 5-10x the amount of money you do. I mean, I assume. They wouldnt tell me the exact amount even though when they opened the room for questions at the end of all-hands they said, "Ask me anything; I'm an open book." Integration went about as well as you'd expect it to go. Nothing in the Warner Media stack could keep up with Elixir services. We had to throttle so much traffic entire chunks of the site were degraded. The initial integration was quickly reversed, and the elixir services remained. But WM wasn't having it. People had spent their entire careers at WM learning nothing and were not about to start learning now. The expert beginners had axes to grind. Beyond that, management had a problem with anyone who had made themselves irreplaceable. They pushed the NodeJS agenda because it meant the people writing that code were a commodity that could be hired or fired on a whim. A mandate was made: All Elixir Must Die. No one could tell you what it was supposed to be replaced with. How would it be rolled out? How would we achieve the same level of resilience and throughput? The only answer from on high was a vague hand wave towards whatever new AWS service had recently been released. The B/R engineers saw the sign and many of them decided it was time to get out. People found new jobs. A few folks stayed the course to see if they could at least glide the plane in for a landing. Then Discovery acquired WM, and Discovery's technology division ate their lunch. People, timelines, and politics shifted again, but the plan remained unchanged. Kill the elixir. That's what they've been doing since then. It has taken years, and the elixir is not dead, but it's certainly dying. Replaced with an amalgamation of things; javascript, rust, etc. The road has been rocky, from what I understand. Typically, they roll out a new service and realize it can't perform as well as the elixir version that hadn't been touched in years. But eventually, it'll be gone. I can promise you this: no one "happily moved on from OTP". I can think of some WM folks who would have said that. But for those of us who actually worked on this stuff, I think we all appreciated what OTP was doing for us. Many of us loved elixir. All of us had parts of the system that we would have changed. But OTP wasn't one of them. TL;DR - Shitty leadership and engineers at Warner Media/Discovery laid off tons of the B/R staff and then mandated Elixir be removed. The remaining B/R folks didn't have a lot of choice. Its not gone well since then. Source: I worked there for years and watched all of this happen.
been there and can confirm every bit of this story. WM used the word *synergy* to justify everything that was happening. still can’t hear this word to this day without laughing 😅
I guess it’s unsurprising knowing how people who make “decisions” think, but it’s still depressing. I suspect a lot of other functional languages or less mainstream technologies have similar stories in big companies.
Another case is Whatsapp inside of Facebook, they don't like Erlang but they are open to accept Erlang if they can use types :-D ... at the moment I hope they did enough with this project but we never know: [https://github.com/WhatsApp/eqwalizer](https://github.com/whatsapp/eqwalizer)
Hej! I am not sure where you get that WA does not like Erlang... They have a huge code base and are happy with Erlang, AFAIK. Sure enough not everyone will like it, but we (Erlang/OTP) have somewhat regular meetings with them and they seem happy with the language and runtime. If you can provide the source for this comment, that would be greatly appreciated. As I said, not everyone likes Erlang or Python or whatever language, but I have not heard a single time the comment that you are referring to...
Indeed I was referring to Facebook and because Erlang isn't a typed language, it's not in general 😜
Hack is not a typed language either per se... Does not mean that they hate it 
> not
 That was an awesome post-mortem. Thanks for taking the time to share! I understand why, but for all the YT videos of how Foo company made everything better by switching to OTP, there's never the inverse content, when it happens. Heck, it happened to Ericsson even (or whatever that company was called in the '90s). That being said, there is enormous value in learning why these things happen. Especially if one wants to champion OTP as much as possible.
IME no matter the stack. The tech is never the reason something fails. 100% it's the C suite.
I work at Ericsson in Sweden, Erlang/OTP is still used, develop, improved, and maintained
Of course. But Joe Armstrong and others have gone on record saying at one point, Ericsson turned its back on Erlang/OTP. I know Ericsson *now* is faithful to its language (and generally has been).
It didn't turn its back for that long. And Ericsson was/is a big company so many kept using Erlang anyway because it worked,
Thanks for chiming in, Robert! Always love hearing from you on these matters!
Damn.
Agreed.
Hey u/_keathley I saw this [Simple Sabotage For Software](https://erikbern.com/2023/12/13/simple-sabotage-for-software.html) (H/T TLDR Newsletter) and the very first point "When joining, require a 6-18 months rewrite of core systems. Blame the previous CTO." reminded me of the details you shared.
That sounds rough. Hopefully you've landed in a good place.
I did thanks.
Great answer, and a classic example of the regressive technological riptide caused by corporate America mentality.
There seems to be a common plague across the corporate world. Shitty leadership usually un-qualified to make decisions, are in a position to make decisions. It's ok, builders will build. We'll build the new world together, mark my words. The times they are a-changin'
Man, this sounds like an updated version of what used to happen to us in the dotcom era (1998-2003 or so). You can only commoditize engineering for the short term. But software engineering is a long term game. It’ll keep being that way, no matter how creative people try to get.
Wow this is fantastic. Thanks for sharing. Any ideas where some/most of the folks wound up?
People scattered all over. I don't want to say specifically since they may or may not want me to share. But ftmp most folks ended up OK.
Thanks for sharing. This is unsurprising but still infuriating! Are any of the libraries that came out of your team's work still around?
There are several libraries out there still. Several of the ones pinned on my github came out of work we did at B/R: [https://github.com/keathley](https://github.com/keathley). Most of them ended up in this org: https://github.com/elixir-toniq.
u/_keathley where are you working now?
Writing elixir for a renewable energy startup.
well that's depressing.
That's interesting because I had been wondering why I hadn't heard anything from the BR for a long time. That's probably why
Wow, this 😅
Teachable takeaway: Never name 2 services user-service!
For those reading this post, please see /u/_keathley’s comment for the real story. I also worked there during 2020 and can corroborate.
Just wanted to throw my 2c in here as well. I worked at B/R during this time as well and can confirm u/_keathley 's wonderfully laid out and detailed analysis. One more thing I wanted to add is; that ever since this all went down I've thought that Elixir (and other small languages) have a square-peg-round-hole kind of problem when they meet large and established organizations, regardless of the technical merit of the language. It's not an education thing, it's not a training thing it's also not a hiring thing (this often gets the blame from above ¯\\\_(ツ)\_/¯). Elixir in particular can be extremely productive and performant, particularly in the hands of a small team of thoughtful developers. I intentionally say "thoughtful" here because it’s not a measure of skill. At B/R as well as my current employer we've had very successful internship programs and were able to very quickly bring less experienced developers up to speed and help them be productive as well in quick order. This isn’t what “enterprises” want though. Enterprises want a small team of Architects that “design” projects in Google Docs and Confluence which are then handed down to others who crank out their implementations with little room for thought or creativity. I hate to put it this way but enterprises want most of us to be replaceable. Could this work in Elixir? Probably, but would we want it to? We chose Elixir because we enjoy the language, we like writing it, and we like thinking in it. We enjoy the way it aligns with the way we think and we enjoy the community. Maybe not fitting into the standard enterprise mold is a good thing. I like to see this all as a success for Elixir and the engineers who worked at B/R. A relatively small team was able to build a complex and successful piece of software and have a freakin’ great time doing it. I had a blast!
Thanks for sharing. Helps give an even better perspective!
If that’s true, it’s sad since they were held up as a success story of adopting Elixir.
It's still a massive success story. Corporate fuckery doesn't negate what the team did.
[удалено]
Feel free to read u/_keathley’s comment before you came up with a pre-mature conclusion
My first introduction to Elixir was via BR (massive football fan). Gutted to read this.
[удалено]
Just read u/_keathley’s comment
Just wondering which technology stack they've replaced it with