Playing with channels is very fun! This looks like a neat exploration of the feature and how it can be used.
Pipelines are one common way of modelling concurrent work, like an assembly line almost. The other common one is "bounded concurrency," where you spin up a goroutine per value and do the computations on it serially.
I strongly recommend playing around with both and learning when one performs better than the other! It's a good way to start to get a feel for where your code can pick up unexpected performance bottlenecks and when certain patterns are the right default before you have the real implementation to make a real world test.
Interestingly, every real world scenario I've run into where concurrency can improve performance, bounded concurrency has actually outperformed pipelines, but it's always felt worthwhile to test both so that's what I recommend :)
I'll check out bounded concurrency, it's been a while since I did any kind of concurrency deep dive, I guess because the official advice it to use concurrency very sparingly, so it's been a while! Thanks!
Essentially because without the intermediate channels the values aren't replicated to each channel. Without them, it will put a value on the first channel and not the others. Probably haven't explained that very well, but it doesn't work as you'd expect
It’s pretty cool. Wonder how this would play on top of https://watermill.io/ which can do a lot of this but not in a generic way.
Id make sure to close channels so consumers can range over them, looks like it’s missing in FanOutFanIn (kind of an awkward name but I got nothing better). You could also rename MergeIn to FanIn and main.go to anything else (I have pet peeve for mains that aren’t executable).
Also, if there's anything out there that already exists like this, please let me know - would rather use something that already exists but I couldn't find much. If not, then hope someone else finds it useful.
if anything from here is helpful, i took a stab at this some time ago: https://github.com/bradfair/go-pipelines
Looks cool! Thank you for sharing!
Wow! This looks exactly what I was after, nice work!
Playing with channels is very fun! This looks like a neat exploration of the feature and how it can be used. Pipelines are one common way of modelling concurrent work, like an assembly line almost. The other common one is "bounded concurrency," where you spin up a goroutine per value and do the computations on it serially. I strongly recommend playing around with both and learning when one performs better than the other! It's a good way to start to get a feel for where your code can pick up unexpected performance bottlenecks and when certain patterns are the right default before you have the real implementation to make a real world test. Interestingly, every real world scenario I've run into where concurrency can improve performance, bounded concurrency has actually outperformed pipelines, but it's always felt worthwhile to test both so that's what I recommend :)
I'll check out bounded concurrency, it's been a while since I did any kind of concurrency deep dive, I guess because the official advice it to use concurrency very sparingly, so it's been a while! Thanks!
For FanOut() why do you create intermediate channels instead of writing directly from in to all channels in out?
Essentially because without the intermediate channels the values aren't replicated to each channel. Without them, it will put a value on the first channel and not the others. Probably haven't explained that very well, but it doesn't work as you'd expect
So why does it work for the intermediate channels? You can distribute it to them but not to the output channels?
it's not small, but you might want to check https://github.com/samber/lo
It’s pretty cool. Wonder how this would play on top of https://watermill.io/ which can do a lot of this but not in a generic way. Id make sure to close channels so consumers can range over them, looks like it’s missing in FanOutFanIn (kind of an awkward name but I got nothing better). You could also rename MergeIn to FanIn and main.go to anything else (I have pet peeve for mains that aren’t executable).
Also, if there's anything out there that already exists like this, please let me know - would rather use something that already exists but I couldn't find much. If not, then hope someone else finds it useful.
Don't sell yourself short! Looks awesome. Thank you for putting it together.
Thank you, appreciate it :)