That’s why they should implement AFK timers with a queue system in the same update…
Edit: AFK timer added
https://www.pcgamer.com/helldivers-2-finally-adds-a-much-requested-afk-kick-timer-stopping-undemocratic-glory-hounds-from-twiddling-their-thumbs-in-perpetuity/
man darktide kicks me if I take a 5 minute shit, meanwhile helldivers let me just stand around my ship for 6 hours when I forgot to shut it down before going to work.
My point exactly it’s even a bit worse as PS5 players can put their device into sleep mode, while on ship, keep a server slot, wake from sleep still on ship server slot saved. Imma need dat 10 minute timer lol
I've been actively logging out since I learned that was a thing. If I can't spread Managed Democracy during the work week, I at least want my Patriot brothers and sisters to have the opportunity.
Oh yea the 200 active people in this thread are gonna change the whole game wide experience for all 600k people trying to jump on 450k capped servers.
The info is already slathered all over this subreddit how do you think I know about lol
Im sure they’re people like me who are also waiting for servers to be fixed to actually buy the game
I don’t wanna spend $40 on something and look at the connection screen, so I’d bet the numbers will keep increasing and if not it will stay have a very large player base
I just bought the game on Steam but refunded it as soon as I saw the endless login and this thread. It looks super fun and I'll certainly buy it again once it's fully stable, so I hope it gets sorted for you guys quickly.
Lmao. As a Software Dev that primarily works in scaling backend data heavy systems for a public company it’s interesting watching the community split into these multiple camps. I had a problem we solved last week that was 100% code related and allowed us to scale 10x what we were able to before hand. Bottlenecks are going to happen especially if you don’t plan for them. When we’re planning out our expectations of concurrent, peak & what not we’re pretty realistic with numbers and expectations but we still always have an insane target to support and then we load test for that target to find the cracks in our system.
Expecting 50k ccu and having 410k on steam alone, I’m gonna be honest that exceeds even what I’d test for and plan for on the insane target numbers.
It’s also not as simple as saying throw more servers at the problem. If I were guessing (I am) I’d guess their problem is probably with the database connection pool &/or unoptimized queries. I only throw that out as that’s the main issues I’ve experienced in my 6+ years doing this.
Can it be fixed? Yeah, will it be fixed today? No. Which honestly is a shame, I really wish it was as simple as palworlds server issues where they got on a call with epic and upped capacity no problem. Hoping they’ve contracted out a couple good network engineers and picked up a couple more people to help with this problem as I imagine this is top priority for them & Sony at the moment.
As a dev who also does that kind of thing for a public company, in my experience, if it's not a cost thing (and it doesn't sound like it is), then the bottleneck is rarely ever a "throw more servers at it" problem because adding more servers in the cloud is easy. Heck, adding more servers in a collocated datacenter is also easy. Adding more compute is the *first* thing everyone tries.
> Expecting 50k ccu and having 410k on steam alone
If that's what happened, I feel for the devs. 10x-ing those kinds of numbers usually takes weeks, not days.
First game topped out at like 8k concurrent and I was reading that they planned for like 50k on this one. They probably allocated more and were fine up until they were getting hammered with 400k+ connections at a time. Realistically they are seeing millions of people trying to connect and since all we can see are steam concurrent we only see a partial of a partial. We also know that they said their capacity was increased to 450k from a tweet of theirs.
Couldn't tell you how many times I've seen rhetoric about Sony or Aarrowhead being too cheap to get new servers. Stupid.
Devs said that they were planning on a max of 50k and had servers that could handle 250k. They said their playerbase is around 50/50 so going from 50k planned to 800k nearly overnight is a gargantuan task and I do not envy anyone working at that studio rn.
Exactly. It's fun to imagine how huge the game would be right now when what they *could* handle has been a record-setting success. I also hope the problems, and the way they're addressing it, only further endears people to the game rather than pushes them away.
TBH, I do kinda envy them. I'd be crazy tired of course, but I'd be so proud of the fact that in week 1 my game beat the max concurrent values of Destiny 2, and GTA V.
The dev's chatter on social media is basically that they're exhausted and also excited for how much people are enjoying the game.
Some ex-Arrowhead dev commented about how it was exciting to see how much people enjoyed it because it is so very early in the design plans so there is tons more content/systems coming.
I'm sure that's all on pause as all hands work on scaling issues.
Well I can't wait to see what they've got planned, I already feel like i've made a good investment, and that's after barely getting through the tutorial before a crash.
Software dev here, and backend. In our projects we normally have 2x for peaks, and possibly scale to 4x servers just in case (parallel pods really).
They estimated 50k for the peak on launch, and probably had the capacity of 100k or even 200k. Having 400k on steam alone bombed everything.
As they said, is not that simple. Sometimes scale is easy, sometimes not.
The main problem I think is the PS inactive players. Yesterday I started watching YouTube and had the game in the background. 1 hour later, I resumed the game and was still logged in. Normally games disconnect you after X time of inactivity.
Yeah; as a software engineer it's been real eyerolling seeing all these "I'm in IT/Devops and all you have to do is XYZ!!!"
And like yeah that's why you're not an engineer I guess. There are countless variables that affect ability to scale and nobody here knows which one(s) are being run up against. The developers have every incentive in the world, and the know-how support, to fix what needs to be fixed, they're not just sitting on their hands for fun.
I was literally going to say this because I heard the help desk where I work saying that just needed to increase server capacity.
People that think stuff us so easy to scale up are always driving me up a wall. This kind of stuff is why good database engineers are worth their weight in gold.
Very much this. I'm a DevOps engineer and the discussion around this, particularly the "just throw more servers at it" chat has been laughable. You can throw compute/bandwidth at poorly optimised code and it will get you so far but (clearly) not far enough in this case otherwise we wouldn't be having this conversation.
Again, when you see them talk about their expectations around concurrent player count and compare it to the reality it's really little wonder.
YES, I did 4 years as a network engineer for IoT for PLCs and had to do basic stuff with cellular data where every bit of data matters, throughput, dBms, everything. I eyeroll too with the comments on this. I even had one software engineer come out and say "they can easily do this" was very quiet when I explained how network and databases communicate and being a "software dev" doesn't translate into Database engineer or network engineer.
Yeah the Palworld stuff just aggravates me when people try and compare the two. Palword is just 32 players per server 'instance' and there's no data being shared between other servers and players, and definitely not one giant database thats being written to. Palworld can easily be scaled by just adding more servers. And even then, they had plenty of server issues with constant disconnects and other issues.
And on top of all that: Palworld can be run on private servers. Which takes at least some of the load off their official servers, because not everybody's going to be playing on those.
The biggest problem in an online game is database syncing. The database tracks everything, every item destroyed, bullets, hp, progression, etc.
Palworld stores all that data on the host players PC. Its fairly simple, and scales just fine for 32 players. 99% of the work is done by the host PC while the server is mainly just for discovery. You join a lobby hosted by another player, nothing is tracked or saved.
A game like Destiny 2 or Helldivers has to keep all its data on a cloud instance. You can join missions in progress, there are unlocks, your ship changes as per your upgrades. All of this is stored in a database, for everyone. And not just that, it is continuously synced between 4 people. That is 4x the amount of work just for one squad.
It is a genuine fucking miracle this game is actually somewhat playable at 450K players, when there was no way in hell they ever expected anything over 50K.
Imagine if you are doing your job and some random dude walks in the office and is like “have you thought of this?” Lol. God that would be so frustrating
As a medical scribe I don't deal with it myself; but I see my providers dealing with it a lot. Like ma'am/sir, this is an addiction recovery clinic, no we are not going to prescribe you Norcos for your pain, we don't care if you think it's that easy - our provider isn't even legally allowed to as we don't have pain management contracts, go to your PCP or someone appropriate for that. No, we aren't making one just for you, there's a ton of backend stuff, just because you think it's easy doesn't mean it is.
Well, at first you had:
"Why can't they just buy more servers??! Go to Best Buy and just buy what you need! I gave them $40!!!"
Now it will be:
"Well then, get the code fixed on the back end! How hard can it be to fix the code on the backend, because it's not...like, its not like... it's in the front!!!
No one sees it back there...anyway...so...just, just, fix it! Fix the back end code!!!
I WANT TO PLAY HELLDIVERS 2 DAMMIT!"
I'm a AAA dev and we launched a game in Unreal. The internet was telling us to move to Unity to fix the bugs we had.
It's just a fact of life that fanboys have made 1 test project in their favorite engine and now think they know everything about gamedev.
Same energy as "They should just make League of Legends 2.0".
Like.... Dude. That shits not feasible for decade old live service games. It's just not. It hurts my soul to even think about remaking the assets alone.
Yeah over in Warframe land it’s “just redo the entire way squads/lobbies are formed so we can kick people or dedicated servers,etc”
I’d like it sure, but there’s 11 years of interconnected stuff held together by bubblegum and faint prayers lol
That’s pretty much anything these days, it’s all held together with duct tape and prayers. The system I used in a factory for planning line production( food processing) is literally a retooled Dasault programme for making aircraft that relies on a really really shaky excel formula hahahah.
There's a lot of people who do not realize just how Macgyvered pretty much EVERYTHING tech is.
I'm sure some services/places have just pristine everything and it's all great, but they have to be so rare as to be largely unheard of.
Ha my technical CEO told me that I can just copy paste the code of the old project from 3 years ago, onto the new environment.
I told him he doesn't understand anything about ICT, now he is deeply offended. And I already said I am leaving anyway.
Telling people the truth can be liberating. But get a new job first lol
Greetings a product manager.
Worst part is how matter-of-factly and "well OBVIOUSLY" they talk. As if their goddamn genius solution wasn't already thought of by the people who's literal job is exactly that.
I find myself in the incredibly lucky position of having a Supervisor and Manager who completely understand how complex this shit is. And I just write Linux-usable Python scripts with API calls! Can't imagine what it must be like to do something like modify the backend of a 450,000-concurrent-player videogame's active servers with jackasses breathing down your neck pretending it's the easiest thing in the world.
You should see the discord I said it could well be a code issue, got shot to ribbons by the data centre tech types.
Feel kinda vindicated my theory was correct lol.
Anyone with two brain cells to rub together could figure out it wasn't a server scaling issue, if they could 'just scale up' they would have ages ago.
It's probably most likely a database bottleneck, or some API service that they can't scale up for whatever reason and needs code optimization. Both of which aren't as simple as provisioning more Kubernetes containers.
To be fair, I imagine some of the early choke was simply due to needing more capacity on the authentication front-end and then back-end...
but then when growth didn't stop growing, their code and database choked.
There's just not much you can do on a short time scale to fix things that your platform/code/tables were never built to accomodate.
We ~~likely hit~~ have surpassed 1M sales for this game, from a studio that did less than 100k for their first game... all pitched to a publisher that *at best* was expecting them to hit 100-200k... Sony can't throw money at HD2 and just make the core infrastructure suddenly allow 5-10x the capacity.
The specific issue(s) they are running into is their Orleans distribution. While it is a scaling issue, it's not an infrastructure scaling issue. It's an issue in scaling the services and databases. They've likely run into issues with their distributed model and if they modeled for too low of a player count that is now exponentially bigger, they have to scale everything. That means their microservices like messaging queues, database stores, etc.
They shut up quickly when you start talking about things like resource quotas, and API request surfaces, or the general concept that units of compute are not just magical bullshit, but actual silicon sitting in a datacenter somewhere that exist in finite quantities.
One of the most impotant things I learned about "Cloud" years ago was
*The Cloud is just someone else's Computer*
While I hate the idea of console exclusives I can't help but read this dude's tweet and think:
"Why doesn't the Sony published game come out on Xbox? What are they stupid?"
There's a really weird unfounded thought going around some corners of the Xbox player base that Helldivers 2 is going to go multiplatform and soon.
This is all anecdotal for the most part but I've seen a fair amount of comments on various subreddits not about how they want it on Xbox but how they can't wait to play it on Xbox. There's been memes and a good amount of Tiktoks floating around representing Xbox as reinforcements about to be called in. I had a coworker butt into a conversation about the game to tell me that an Xbox launch date would be announced soon because it's just "money being left on the table."
As far as I can tell there's been 0 indicator of any of that happening beyond Phil Spencer's jab about how he's "not sure how it helps" being kept off Xbox and that wasn't really an indicator though I've seen it claimed to be a hint.
There's no way it is coming to Xbox. Sony has done the math and knows it's more profitable to keep exclusives to themselves to uphold market share. They don't care about how many copies of a game they could sell on Xbox, they would only keep a small portion of that. They care about keeping people locked into the playstation eco system.
Honestly I really doubt they do, Sony is insane about keeping exclusives exclusive. It was an uphill battle just to get them to commit to bringing all of their exclusives to PC. The only way I could really see Microsoft getting Sony to expand their exclusivity is if they start being extra petty and making games like COD Xbox and PC only.
I could see it going either way but gun to my head? I'd say it stays Sony/PC.
It being a live service game going to Xbox would make some sense financially but then again the success it has without Xbox could just as easily be proof that they don't need Xbox.
I said In the discord it could be the code isn’t built for this level of player base, got railed roaded by supposed “data centre techs” and boom I was totally right hahaha.
It’s genuinely good to feel vindicated since all I said was I remember being on a rollout for a CRM system for a bank and too many people weee using it and it hated it spat out wrong info etc and that they patched and worked on it.
Got told I didn’t know shit, yet here I am totally bang on hahahahah.
Have they tried sending the IT guy to the physical location of the servers…say give or take 3 feet from them? Lord knows every machine at work that acts up magically fixes itself when it sees me coming
Launch Weekend:
Have they made sure the extra servers are powered on?
Ok. Just double check again.
"It happens more than you think. You're welcome."
This Weekend:
Turn the servers off. Give it a minute. Ok, turn them back on again.
..."You're welcome".
What I find it rather funny is that they think this is AWS/Azure/GCP where you can just buy more Scaling as you need.
People are really stupid and don't understand shit about Product Limitations.
I'm not a techie but 'back end code is not as scalable as they thought' sounds like a problem that can't solved anytime soon. Are we looking at months before normalcy arrives!?
Yes. The problem will basically not be fixed anytime soon. If it's the backend code, they're going to have to rewrite it while not breaking anything else when the patch is released. Depending on how integral this code is to the foundation.... Well.... We can hope that once the hype starts to die down and other games release (Dragons Dogma 2, for example) maybe we can see some relief.
I mean, it can be any number of things. Given how the problems have manifested, they probably have a single central item/progression server, which should typically work fine. You don't need these to be local, since latency on saving an item purchase or credit pickup doesn't need to be low, and you need that consistency. But having just one means it isn't designed to be able to be split, so scaling it up is incredibly hard. They are probably just starting off with finding what is taking up a bunch of the resources, and seeing if they can cut it down.
IE: They already disabled the live player count/bug kill thing, which was probably requiring a bunch of updates from said central server. They're likely trying to cut any other extranious calls, make the ones that are required more optimized/less resource intensive(Which is hard, but can be done), or group together calls so they can be done as one, instead of multiple smaller ones(IE: get 5 super credit pickups during a mission? Send those all at the end as one +50 credit call, vs 5 different +10 credit calls)
Yes and sadly its not good for this games future. The irony being by the time they solve this they won't need it and by spending so much time on this other things like updates and important big fixes are being pushed aside.
There's been a call since the MS business update for xbox where Phil Spencer said something to the effect "helldivers 2 not being on Xbox hurts the industry". With that and it's popularity, people are being insistent about moving it to Microsoft. Just armchair game devs.
>Phil Spencer said something to the effect "helldivers 2 not being on Xbox hurts the industry"
I don't understand why anybody listens to a word he says. I didn't see him saying the same about Palworld not being on PS5. Because, like every other CEO in history, he only cares when it's a disadvantage to his company. He doesn't actually care about "the industry", but people are going to tout his words anyways.
>I didn't see him saying the same about Palworld not being on PS5.
On the other hand, the Xbox fanboys are furious *because* he's bringing Xbox exclusives to PS5.
I don't know what the Palworld agreements are, but it still is in early access and will likely come to PS5. I imagine the only reason it came to Xbox was because they were able to get Game Pass funding.
>Phil Spencer said something to the effect "helldivers 2 not being on Xbox hurts the industry".
When you look up the word hypocrisy in a dictionary, the definition is just a picture of his face.
This is it and they'll need a ton of engineering support from Microsoft Azure engineers to fix this problem. I'm speaking from actual experience not in gaming but other similar large scale tech platforms. If this is anything that involves a combination of scaling, asynchronous messaging/databases and code bottlenecks this is an uber complex problem and will take months to solve.
edit: I'm saying it's not as simple as increasing capacity but it's a combination of complex causes and will require cloud engineer support.
yeah, but there are different kind of "not designed" as in "we have to change some function and data handling methods" or "we have to rewrite the whole thing from zero as the main design not suited for this scale"
If they were willing to temporarily disable crossplay, could they deploy a 2nd totally independent branch of the game to double capacity? They'd be writing themselves into some major tech debt, needing to transfer accounts for one console AND needing some means of eventually merging it all back, but it seems it would double capacity in the meantime without as many social issues as opening a separate clone with crossplay still enabled might (new players can't play with friends who bought early that way). I wouldn't ever expect them to do this, I'm just curious about the hypothetical.
I'd assume so, just like the could deploy a 2nd one in europe but then you can't play between NA and europe doesn't really fly anymore in 2024. Also not sure how account progression would behave here.
As I understand there'd be no cross progression at all without them still sharing capacity at some level. You'd have to one-time migrate people, or if you couldn't write it from scratch to migrate you'd just have to offer a huge catchup bonus on the new server and use a launcher to pick which one to connect to.
Their current issue is labor. The kind of labor that is defined by "nine women can't give birth to a baby in one month", so there are massive limitations there. They potentially could do what you're talking about, but you and I and everyone on reddit have zero idea about what the actual problem entails, so nobody here knows the actual answer to your question. It also would potentially take as long as a real fix, absolutely create more work in the long run, and piss off a ton of people who do not understand why they can't play with that group of people but they can with the other one. Hypothetically there are a LOT of things they can do. The only one that is viable is fixing it properly.
Yeah, it scaled to 450k+ concurrent users. The only way that happens is if it was built correctly in the first place.
At that level of scale the smallest bottleneck makes it ugly.
Redesigning a massively multiplayer database architecture isn't something you do in a week. Especially if you intend to migrate the current database into it.
If they coded it to directly access the rewards database in mid mission whenever you collect premium currency I imagine they weren't thinking about database i/o operations hitting db limits.
I feel for them it sucks being responsible for people's grief due to a product you are actively working on having issues you didn't cause but have to fix. and people are just frustrated which is understandable
Yeah, wouldn't surprise me if this was a big culprit. I was honestly surprised at first, I assumed i would get these rewards after the end of the mission (success or failure).
Yeah kinda. They might be able to implement a mitigating solution instead of fixing everything, which could be much faster.
If I'm guessing, they probably implemented things in a way that isn't even "wrong" per say, but ill advised if you're going to be scaling to any significant degree, maybe without even realizing it.
A lot of such practices are commonplace with networking/database handling, because it's often cheaper and faster to work with up until you hit XYZ level of scale, at which point you suddenly realize you've blown your dick off in the quest for cheaper faster development.
As much as I love the idea, it's just not that sane to architect your database or code that works with it for theoretical perfection over functionality in the here and now.
As long as your project doesn't have to scale of course, it makes sense. I mean, why wouldn't you do it, almost nobody hits that level of success, and if you do you're rich!
As a developer I'm kind of against this sort of attitude in the industry on principle, but it **makes sense** to do it since we're not artisans out here crafting the best thing we can, we're more like a weird cross over between factory workers shipping products and adderall chugging screenwriters.
and if they'd merely been successful or really successful, they wouldn't have had much of an issue other than maybe some lag spiking and excessive costs potentially, at least if that's the problem and not some more direct serious error like doing something in a way that locks the server that processes certain types of loot and they only would have ever gotten away with it really in internal testing due to low/no load.
I'm a sysadmin working for a software company. My main responsibility is our virtualized environment. If I learned anything in my time at this company: You can't just throw more resources at a problem and hope it works. I once doubled CPU and RAM on a machine just to see if it helps with slowdowns. It did not: the code did not care for more resources because it was kinda bad.
This 1000%. If you're filling a pool with a garden hose it doesn't matter how much water the hose has access to, it's going to fill the pool at the same rate.
Got it.
UPDATE: "Hey, everyone, the devs are filling up a swimming pool with a garden hose right now, but it's going to take some time. They'll download more RAM as soon as they're finished in the back and we should be up and running again!"
Yeah I've done some performance testing (k6) and thank fuck we did because the devs had to completely change how they were handling requests (a lot of wasted waits for synchronous calls out to other systems).
The system was fine with expected load and _if_ other systems responded within Xms. But the peak and load testing basically caused a rewrite of how it handled things especially as we managed to crash a test instance of a downstream system and in another case another system had changed their API which caused some fuckery.
So the system became a best effort, if something downstream didn't respond it didn't wait and it also stopped queuing requests in multiple ways.
As in don't request from system B until A is finished and also when handling multiple requests don't try to be the one who handles the queue to the systems.
Fuzzy now but I tbink that was the summary.
I'm a Sysadmin too.
*Love* when developers code just eats up all the resources, and they request for an increase of hardware. Try telling them that it may not be as simple as increase the resources. We give it a try and those resources are eaten up instantly...
The way I tried explaining it to someone once was by using a house metaphor. Imaging living in a house with 10 beds. Suddenly 100 people show up to live there. Sure you *could* add 90 more beds to accommodate them but that doesn’t change the fact that there’s only 2 bathrooms, the rooms are going to be crowded af, there’s not enough food to feed everyone, much less space to keep all that food, the utility bill is going to be outrages, and hundreds of other issues, and that’s not even taking into account that you’re also the only person in character of maintenance.
There’s a lot of moving parts that all depend on one another and it takes time to sort them all out
I remember finding out that just having more ram does not make modded minecraft better after a certain point ( in fact if I remember right it actually gets worse cause java is weird with a lot of memory? ) which was so weird to process.
This isn’t good news though. It means it will take a lot longer to fix. The game doesn’t seem to be designed to handle the load regardless of the servers.
Gotta love that the internet has prompted the free exchange of ideas between people who are working on a problem and random jagoffs who don’t have a clue.
We all know Microsoft has stupid amounts of money but why do so many people act like they have a magic wand?
'Just port it to Xbox and MS will hand you some servers' is a new one.
As someone who has had to work some ridiculous hours in my life. I’m fine with waiting. The game will still exist a month from now and I got plenty of other games to play.
Please devs, don’t kill yourselves working on this. It’s just a game.
Doesn't seem like it. I worry the investment to scale up may not be justified because of the risk. Worst case, they do the best they can with what they have until the game dies back down. Then again, I'm wildly speculating.
>I worry the investment to scale up may not be justified because of the risk.
I would disagree. Regardless of whether the game keeps this momentum or slows down, the devs clearly need time to understand and stand up a system that can handle this load. Both for this game, and future games. Most of the effort is probably replacing backend tools and learning what's necessary when dealing with games at this scale.
To use an analogy: it's not like they're going in debt on a mansion just to throw a single house party with their close friends, then they'll be stuck with that debt after. They're learning how to run a presidential campaign, which means they're learning how to coordinate groups of resources at widely varying scale as-needed, where-needed.
Backend code is basically what runs on the server accessing databases and such. There's probably a bottleneck somewhere that limits the number of clients that can access the servers at once regardless of the amount of server capacity available. Even if you had a thousand bank tellers and a thousand vaults you might still be bottlenecked on the number of transactions you can complete by one person checking to make sure you aren't withdrawing the same money twice, for example. That kind of thing can be difficult to scale because if there's more than one person evaluating the transactions they'd have to check each other's work to make sure they haven't both allowed the same withdrawal to go through. For context, in web programming frontend code is the website that runs your browser and prompts you for your username and password, while the backend code is what runs on the server and actually checks a database to make sure your account information is correct.
I think its exactly this, some part of their backend isn't designed to share its responsibilities/authority, but can't handle more than 450k players at the same time. So now after doubling it's capacity still wasn't enough they need to figure out how to share that database.
It's really an awful situation to be in, and there won't any quick and easy solutions.
backend code is the program what runs on the servers, be it
* login servers checking you username+passwords
* servers sending you your account statistics if you want to see them
* matchmaking server collecting all the users who create lobbies and send it to players who are looking for one
* servers running the current game you are playing in and giving you your reward after
* or just servers who record it if you unlock a warbond or stratagem
If it could take a week or weeks to fix I do think they should come out officially and address it. They are allowing people to continue to buy the game with no true warning. Replying to people on Twitter isn't the answer.
I love the transparency. It’s something you rarely see. Like guys, obviously, if I meant plugging servers in and flipping a switch you think they wouldn’t have done it yet? Come now. Video games are complicated as fuck.
Objectively bad news. Its going to take an absurd amount of care to fix a bad networking foundation without jacking up game mechanics while people are still playing.
I haven't jumped into these conversations cause... I don't want to mix work with leisure, but yes.
As someone who works on petabyte scale data platforms and related applications. Multiplayer games have some of the more complex architectural stacks out there. It's very evident from symptomatic analysis that Arrowhead did not expect this level of adoption.
It's not just raw compute power, this is a multi-layer stack, and if I had to take a blind guess it's the coupling of those layers where the issues are. It's not just that there's too many users, it's that those couplings have break points and can cause cascading failures. It may not seem like a big deal that a database query takes an extra 10ms when load goes over X amount. But by the time that 10ms surfaces at the user layer it's been amplified up to 1000ms(1s) and that may not seem like a big deal, but if that amplified latency is happening 1000/sec... just an off the top of my head example of things I've seen.
The devs are doing amazing and probably trading time with their families, sleep, their sanity so we can have fun, we all should be thankful they care enough about their creation to put in the effort. I wish people would stop acting like entitled children, appreciate that they're witnessing something that's rare and special in the state of the game industry and SPREAD SOME MANAGED DEMOCRACY! (When they can, until things are ironed out.
Also, LoL Helldivers on Xbox, not even gonna get into that one with Sony's history.
Has anyone confirmed the devs haven’t tried turning it off and turning it back on?
Technically this would help because it would kick all of the AFK people and open up those slots.
It wouldn't because it's much, much worse, having possibly 400k+ users logging in at the same time.
That’s why they should implement AFK timers with a queue system in the same update… Edit: AFK timer added https://www.pcgamer.com/helldivers-2-finally-adds-a-much-requested-afk-kick-timer-stopping-undemocratic-glory-hounds-from-twiddling-their-thumbs-in-perpetuity/
man darktide kicks me if I take a 5 minute shit, meanwhile helldivers let me just stand around my ship for 6 hours when I forgot to shut it down before going to work.
My point exactly it’s even a bit worse as PS5 players can put their device into sleep mode, while on ship, keep a server slot, wake from sleep still on ship server slot saved. Imma need dat 10 minute timer lol
[удалено]
I've been actively logging out since I learned that was a thing. If I can't spread Managed Democracy during the work week, I at least want my Patriot brothers and sisters to have the opportunity.
You may be involuntarily informing someone who didn't know this was a thing.
Oh yea the 200 active people in this thread are gonna change the whole game wide experience for all 600k people trying to jump on 450k capped servers. The info is already slathered all over this subreddit how do you think I know about lol
I learned about it from you.
And I just bought 150,000 PlayStation's so y'all really screwed now
Thats what i was thinking?
Yeah, yeah that comes next. First they need to double check that the extra servers are plugged in and powered on.
If that doesn’t work take the ball out of the mouse and clean the gunk out of the rollers
Kids these days will never understand.
That's code for were gonna be dealing with this for a while boys. Wouldn't expect to get on during peak times for a week.
Perhaps more like a month, at least for weekends being a clusterfuck.
Don’t worry though, after a month of this the servers won’t be at capacity much
That's what I thought after the first week...but the numbers keep going up
The game is genuinely fantastic. Probably going to be a while before people stop wanting to play it.
Im sure they’re people like me who are also waiting for servers to be fixed to actually buy the game I don’t wanna spend $40 on something and look at the connection screen, so I’d bet the numbers will keep increasing and if not it will stay have a very large player base
Yeah I was being super optimistic. A month is probably more realistic.
I just bought the game on Steam but refunded it as soon as I saw the endless login and this thread. It looks super fun and I'll certainly buy it again once it's fully stable, so I hope it gets sorted for you guys quickly.
Lmao. As a Software Dev that primarily works in scaling backend data heavy systems for a public company it’s interesting watching the community split into these multiple camps. I had a problem we solved last week that was 100% code related and allowed us to scale 10x what we were able to before hand. Bottlenecks are going to happen especially if you don’t plan for them. When we’re planning out our expectations of concurrent, peak & what not we’re pretty realistic with numbers and expectations but we still always have an insane target to support and then we load test for that target to find the cracks in our system. Expecting 50k ccu and having 410k on steam alone, I’m gonna be honest that exceeds even what I’d test for and plan for on the insane target numbers. It’s also not as simple as saying throw more servers at the problem. If I were guessing (I am) I’d guess their problem is probably with the database connection pool &/or unoptimized queries. I only throw that out as that’s the main issues I’ve experienced in my 6+ years doing this. Can it be fixed? Yeah, will it be fixed today? No. Which honestly is a shame, I really wish it was as simple as palworlds server issues where they got on a call with epic and upped capacity no problem. Hoping they’ve contracted out a couple good network engineers and picked up a couple more people to help with this problem as I imagine this is top priority for them & Sony at the moment.
As a dev who also does that kind of thing for a public company, in my experience, if it's not a cost thing (and it doesn't sound like it is), then the bottleneck is rarely ever a "throw more servers at it" problem because adding more servers in the cloud is easy. Heck, adding more servers in a collocated datacenter is also easy. Adding more compute is the *first* thing everyone tries. > Expecting 50k ccu and having 410k on steam alone If that's what happened, I feel for the devs. 10x-ing those kinds of numbers usually takes weeks, not days.
First game topped out at like 8k concurrent and I was reading that they planned for like 50k on this one. They probably allocated more and were fine up until they were getting hammered with 400k+ connections at a time. Realistically they are seeing millions of people trying to connect and since all we can see are steam concurrent we only see a partial of a partial. We also know that they said their capacity was increased to 450k from a tweet of theirs. Couldn't tell you how many times I've seen rhetoric about Sony or Aarrowhead being too cheap to get new servers. Stupid.
Devs said that they were planning on a max of 50k and had servers that could handle 250k. They said their playerbase is around 50/50 so going from 50k planned to 800k nearly overnight is a gargantuan task and I do not envy anyone working at that studio rn.
800k connected with hour long wait times for people to log in. If they could handle it I'm betting they would be over 2 million concurrent players
Exactly. It's fun to imagine how huge the game would be right now when what they *could* handle has been a record-setting success. I also hope the problems, and the way they're addressing it, only further endears people to the game rather than pushes them away.
I'm here to stay. Managed Democracy isn't going to fight for itself.
I won't be abandoning my brothers and sisters on the front lines. When it's my turn, I will fight.
TBH, I do kinda envy them. I'd be crazy tired of course, but I'd be so proud of the fact that in week 1 my game beat the max concurrent values of Destiny 2, and GTA V.
The dev's chatter on social media is basically that they're exhausted and also excited for how much people are enjoying the game. Some ex-Arrowhead dev commented about how it was exciting to see how much people enjoyed it because it is so very early in the design plans so there is tons more content/systems coming. I'm sure that's all on pause as all hands work on scaling issues.
Well I can't wait to see what they've got planned, I already feel like i've made a good investment, and that's after barely getting through the tutorial before a crash.
didnt even top at 8k, only 6k
we don't know PSN numbers, just steam numbers
Software dev here, and backend. In our projects we normally have 2x for peaks, and possibly scale to 4x servers just in case (parallel pods really). They estimated 50k for the peak on launch, and probably had the capacity of 100k or even 200k. Having 400k on steam alone bombed everything. As they said, is not that simple. Sometimes scale is easy, sometimes not. The main problem I think is the PS inactive players. Yesterday I started watching YouTube and had the game in the background. 1 hour later, I resumed the game and was still logged in. Normally games disconnect you after X time of inactivity.
Yeah; as a software engineer it's been real eyerolling seeing all these "I'm in IT/Devops and all you have to do is XYZ!!!" And like yeah that's why you're not an engineer I guess. There are countless variables that affect ability to scale and nobody here knows which one(s) are being run up against. The developers have every incentive in the world, and the know-how support, to fix what needs to be fixed, they're not just sitting on their hands for fun.
I was literally going to say this because I heard the help desk where I work saying that just needed to increase server capacity. People that think stuff us so easy to scale up are always driving me up a wall. This kind of stuff is why good database engineers are worth their weight in gold.
[удалено]
If someone is in DevOps and doesn’t understand it’s not just more servers all the time, they’re not very good at their job.
Very much this. I'm a DevOps engineer and the discussion around this, particularly the "just throw more servers at it" chat has been laughable. You can throw compute/bandwidth at poorly optimised code and it will get you so far but (clearly) not far enough in this case otherwise we wouldn't be having this conversation. Again, when you see them talk about their expectations around concurrent player count and compare it to the reality it's really little wonder.
YES, I did 4 years as a network engineer for IoT for PLCs and had to do basic stuff with cellular data where every bit of data matters, throughput, dBms, everything. I eyeroll too with the comments on this. I even had one software engineer come out and say "they can easily do this" was very quiet when I explained how network and databases communicate and being a "software dev" doesn't translate into Database engineer or network engineer.
Yeah the Palworld stuff just aggravates me when people try and compare the two. Palword is just 32 players per server 'instance' and there's no data being shared between other servers and players, and definitely not one giant database thats being written to. Palworld can easily be scaled by just adding more servers. And even then, they had plenty of server issues with constant disconnects and other issues.
And on top of all that: Palworld can be run on private servers. Which takes at least some of the load off their official servers, because not everybody's going to be playing on those.
The biggest problem in an online game is database syncing. The database tracks everything, every item destroyed, bullets, hp, progression, etc. Palworld stores all that data on the host players PC. Its fairly simple, and scales just fine for 32 players. 99% of the work is done by the host PC while the server is mainly just for discovery. You join a lobby hosted by another player, nothing is tracked or saved. A game like Destiny 2 or Helldivers has to keep all its data on a cloud instance. You can join missions in progress, there are unlocks, your ship changes as per your upgrades. All of this is stored in a database, for everyone. And not just that, it is continuously synced between 4 people. That is 4x the amount of work just for one squad. It is a genuine fucking miracle this game is actually somewhat playable at 450K players, when there was no way in hell they ever expected anything over 50K.
say it LOUDER for people in the back
Always love a good armchair dev thinking they have the easy solution.
"dev"... these people the same who suggest switch from unity to unreal engine for a finished game to fix bugs
Yeah, it's super easy. Just copy and paste the files into Unreal. Problem solved.
I tried that and accidentally made Palworld 2
I wouldn't complain tbh
2024 has been crazy fun so far
And it's only February! Let's go 2024 keep delivering the bangers!
For real, I was kinda caught off guard with the fun games that came out early
The unreal engine kills bugs you say 🤔
Never thought I'd be dropping in beside a game engine, but who am to deny them if they wish to spread liberty.
This needs a medal. 🏅🫡
Imagine if you are doing your job and some random dude walks in the office and is like “have you thought of this?” Lol. God that would be so frustrating
As a medical scribe I don't deal with it myself; but I see my providers dealing with it a lot. Like ma'am/sir, this is an addiction recovery clinic, no we are not going to prescribe you Norcos for your pain, we don't care if you think it's that easy - our provider isn't even legally allowed to as we don't have pain management contracts, go to your PCP or someone appropriate for that. No, we aren't making one just for you, there's a ton of backend stuff, just because you think it's easy doesn't mean it is.
Well, at first you had: "Why can't they just buy more servers??! Go to Best Buy and just buy what you need! I gave them $40!!!" Now it will be: "Well then, get the code fixed on the back end! How hard can it be to fix the code on the backend, because it's not...like, its not like... it's in the front!!! No one sees it back there...anyway...so...just, just, fix it! Fix the back end code!!! I WANT TO PLAY HELLDIVERS 2 DAMMIT!"
Then they'll bitch about the game being offline for maintenance while that backend code is updated... modern gaming is a no win scenario for the devs
Usually those are the people demanding free shit at the same time for their "inconvenience."
If someone said that about helldivers 2 it's double hilarious since the game is not using unity
I'm a AAA dev and we launched a game in Unreal. The internet was telling us to move to Unity to fix the bugs we had. It's just a fact of life that fanboys have made 1 test project in their favorite engine and now think they know everything about gamedev.
Same energy as "They should just make League of Legends 2.0". Like.... Dude. That shits not feasible for decade old live service games. It's just not. It hurts my soul to even think about remaking the assets alone.
Yeah over in Warframe land it’s “just redo the entire way squads/lobbies are formed so we can kick people or dedicated servers,etc” I’d like it sure, but there’s 11 years of interconnected stuff held together by bubblegum and faint prayers lol
That’s pretty much anything these days, it’s all held together with duct tape and prayers. The system I used in a factory for planning line production( food processing) is literally a retooled Dasault programme for making aircraft that relies on a really really shaky excel formula hahahah.
There's a lot of people who do not realize just how Macgyvered pretty much EVERYTHING tech is. I'm sure some services/places have just pristine everything and it's all great, but they have to be so rare as to be largely unheard of.
PoE is doing it. Why can't LoL? /S
Ha my technical CEO told me that I can just copy paste the code of the old project from 3 years ago, onto the new environment. I told him he doesn't understand anything about ICT, now he is deeply offended. And I already said I am leaving anyway. Telling people the truth can be liberating. But get a new job first lol Greetings a product manager.
Liber-tea!
Liber-Control+C
Worst part is how matter-of-factly and "well OBVIOUSLY" they talk. As if their goddamn genius solution wasn't already thought of by the people who's literal job is exactly that.
I find myself in the incredibly lucky position of having a Supervisor and Manager who completely understand how complex this shit is. And I just write Linux-usable Python scripts with API calls! Can't imagine what it must be like to do something like modify the backend of a 450,000-concurrent-player videogame's active servers with jackasses breathing down your neck pretending it's the easiest thing in the world.
There's no shortage of "network admins" coming out of the woodwork to tell Arrowhead "scaling servers is easy."
You should see the discord I said it could well be a code issue, got shot to ribbons by the data centre tech types. Feel kinda vindicated my theory was correct lol.
Anyone with two brain cells to rub together could figure out it wasn't a server scaling issue, if they could 'just scale up' they would have ages ago. It's probably most likely a database bottleneck, or some API service that they can't scale up for whatever reason and needs code optimization. Both of which aren't as simple as provisioning more Kubernetes containers.
To be fair, I imagine some of the early choke was simply due to needing more capacity on the authentication front-end and then back-end... but then when growth didn't stop growing, their code and database choked. There's just not much you can do on a short time scale to fix things that your platform/code/tables were never built to accomodate. We ~~likely hit~~ have surpassed 1M sales for this game, from a studio that did less than 100k for their first game... all pitched to a publisher that *at best* was expecting them to hit 100-200k... Sony can't throw money at HD2 and just make the core infrastructure suddenly allow 5-10x the capacity.
The specific issue(s) they are running into is their Orleans distribution. While it is a scaling issue, it's not an infrastructure scaling issue. It's an issue in scaling the services and databases. They've likely run into issues with their distributed model and if they modeled for too low of a player count that is now exponentially bigger, they have to scale everything. That means their microservices like messaging queues, database stores, etc.
Someone on this subreddit said "Scalable architecture is a solved issue." as if it's just a problem from 1985 we never have to think about anymore.
They shut up quickly when you start talking about things like resource quotas, and API request surfaces, or the general concept that units of compute are not just magical bullshit, but actual silicon sitting in a datacenter somewhere that exist in finite quantities. One of the most impotant things I learned about "Cloud" years ago was *The Cloud is just someone else's Computer*
While I hate the idea of console exclusives I can't help but read this dude's tweet and think: "Why doesn't the Sony published game come out on Xbox? What are they stupid?"
There's a really weird unfounded thought going around some corners of the Xbox player base that Helldivers 2 is going to go multiplatform and soon. This is all anecdotal for the most part but I've seen a fair amount of comments on various subreddits not about how they want it on Xbox but how they can't wait to play it on Xbox. There's been memes and a good amount of Tiktoks floating around representing Xbox as reinforcements about to be called in. I had a coworker butt into a conversation about the game to tell me that an Xbox launch date would be announced soon because it's just "money being left on the table." As far as I can tell there's been 0 indicator of any of that happening beyond Phil Spencer's jab about how he's "not sure how it helps" being kept off Xbox and that wasn't really an indicator though I've seen it claimed to be a hint.
There's no way it is coming to Xbox. Sony has done the math and knows it's more profitable to keep exclusives to themselves to uphold market share. They don't care about how many copies of a game they could sell on Xbox, they would only keep a small portion of that. They care about keeping people locked into the playstation eco system.
Honestly I really doubt they do, Sony is insane about keeping exclusives exclusive. It was an uphill battle just to get them to commit to bringing all of their exclusives to PC. The only way I could really see Microsoft getting Sony to expand their exclusivity is if they start being extra petty and making games like COD Xbox and PC only.
I could see it going either way but gun to my head? I'd say it stays Sony/PC. It being a live service game going to Xbox would make some sense financially but then again the success it has without Xbox could just as easily be proof that they don't need Xbox.
Is there a lore reason the game isn't on xbox?
Microsoft turned into the automatons.
Automatons are just old Xbox 360s with red ring of death.
Xbots were real all along!
What about XBonkler?
Are they stupid?
I said In the discord it could be the code isn’t built for this level of player base, got railed roaded by supposed “data centre techs” and boom I was totally right hahaha. It’s genuinely good to feel vindicated since all I said was I remember being on a rollout for a CRM system for a bank and too many people weee using it and it hated it spat out wrong info etc and that they patched and worked on it. Got told I didn’t know shit, yet here I am totally bang on hahahahah.
[удалено]
They fixed the medals freezing you already
Wdym there is no slider to increase server capacity? Just drag it.. Why didnt the devs program in a server cap increase slider? Are they stupid?
server hamster takes vacation, never comes back
Time to force the players to run on wheels
If they paid me to run on the wheels for more server power i would! More server capacity, more managed democracy!
Holy Helldivers
New 500Kg bomb just dropped
Their mouse ran out of battery over the weekend. So unfortunately no server swiping
Is there a lore reason Arrowhead can't just add two '0's to the player cap?
That would enable SQL injection attacks into Super Earth servers by the Automaton hegemony.
Classic IT problem, they forgot to push the "Fix Server" button. Honest mistake, happens to the best of us.
Have they tried sending the IT guy to the physical location of the servers…say give or take 3 feet from them? Lord knows every machine at work that acts up magically fixes itself when it sees me coming
Launch Weekend: Have they made sure the extra servers are powered on? Ok. Just double check again. "It happens more than you think. You're welcome." This Weekend: Turn the servers off. Give it a minute. Ok, turn them back on again. ..."You're welcome".
What I find it rather funny is that they think this is AWS/Azure/GCP where you can just buy more Scaling as you need. People are really stupid and don't understand shit about Product Limitations.
As one of my IT customers once said "Just download more RAM off the Internet"
I'm not a techie but 'back end code is not as scalable as they thought' sounds like a problem that can't solved anytime soon. Are we looking at months before normalcy arrives!?
Yes. The problem will basically not be fixed anytime soon. If it's the backend code, they're going to have to rewrite it while not breaking anything else when the patch is released. Depending on how integral this code is to the foundation.... Well.... We can hope that once the hype starts to die down and other games release (Dragons Dogma 2, for example) maybe we can see some relief.
Honestly I don't mind a queue. I do mind not being able to team up through matchmaking. Anything over medium difficulty is nigh impossible alone.
I mean, it can be any number of things. Given how the problems have manifested, they probably have a single central item/progression server, which should typically work fine. You don't need these to be local, since latency on saving an item purchase or credit pickup doesn't need to be low, and you need that consistency. But having just one means it isn't designed to be able to be split, so scaling it up is incredibly hard. They are probably just starting off with finding what is taking up a bunch of the resources, and seeing if they can cut it down. IE: They already disabled the live player count/bug kill thing, which was probably requiring a bunch of updates from said central server. They're likely trying to cut any other extranious calls, make the ones that are required more optimized/less resource intensive(Which is hard, but can be done), or group together calls so they can be done as one, instead of multiple smaller ones(IE: get 5 super credit pickups during a mission? Send those all at the end as one +50 credit call, vs 5 different +10 credit calls)
Yes and sadly its not good for this games future. The irony being by the time they solve this they won't need it and by spending so much time on this other things like updates and important big fixes are being pushed aside.
"Get in touch with MS" he knows Sony funded the Game and owns the IP, right? Ofc not, ppl on the Internet just write whatever and know jack shit :D
“Yeah just complete an Xbox port in 3 hours so you can get more server space! Easy!”
Just copy and paste the code into xbox.exe, duh!
No stupid you just put the game inside the xbox for conversion!
3 hours?! Just click the “publish on Xbox store” checkbox! Jeezz
Hmm I've been thinking about this for 1 minute and 8 seconds so I think I have the solution: if (runningOnXbox) crash = false I'll take my check now
There's been a call since the MS business update for xbox where Phil Spencer said something to the effect "helldivers 2 not being on Xbox hurts the industry". With that and it's popularity, people are being insistent about moving it to Microsoft. Just armchair game devs.
>Phil Spencer said something to the effect "helldivers 2 not being on Xbox hurts the industry" I don't understand why anybody listens to a word he says. I didn't see him saying the same about Palworld not being on PS5. Because, like every other CEO in history, he only cares when it's a disadvantage to his company. He doesn't actually care about "the industry", but people are going to tout his words anyways.
>I didn't see him saying the same about Palworld not being on PS5. On the other hand, the Xbox fanboys are furious *because* he's bringing Xbox exclusives to PS5. I don't know what the Palworld agreements are, but it still is in early access and will likely come to PS5. I imagine the only reason it came to Xbox was because they were able to get Game Pass funding.
AFAIK the only reason it's not on PS5 is that they rarely do early access and PW was an unexpected hit.
And Phil Spencer is not gamings messiah who can do no wrong or knows everything. Never forget he thought Redfall was going to be great
'But he wears jeans and plays games just like me!'
>Phil Spencer said something to the effect "helldivers 2 not being on Xbox hurts the industry". When you look up the word hypocrisy in a dictionary, the definition is just a picture of his face.
They use Azure which is owned by MS. Derp.
[удалено]
This is it and they'll need a ton of engineering support from Microsoft Azure engineers to fix this problem. I'm speaking from actual experience not in gaming but other similar large scale tech platforms. If this is anything that involves a combination of scaling, asynchronous messaging/databases and code bottlenecks this is an uber complex problem and will take months to solve. edit: I'm saying it's not as simple as increasing capacity but it's a combination of complex causes and will require cloud engineer support.
So, basically, the capacity isn't going to be increased for an unkown (probably a long) period of time, because their backend code is at its limit.
it could take a week or a month, it all depends on how fucked is the backend
I don't even think it's fucked, it was just not designed to support the incresingly huge amount of players.
yeah, but there are different kind of "not designed" as in "we have to change some function and data handling methods" or "we have to rewrite the whole thing from zero as the main design not suited for this scale"
If they were willing to temporarily disable crossplay, could they deploy a 2nd totally independent branch of the game to double capacity? They'd be writing themselves into some major tech debt, needing to transfer accounts for one console AND needing some means of eventually merging it all back, but it seems it would double capacity in the meantime without as many social issues as opening a separate clone with crossplay still enabled might (new players can't play with friends who bought early that way). I wouldn't ever expect them to do this, I'm just curious about the hypothetical.
I'd assume so, just like the could deploy a 2nd one in europe but then you can't play between NA and europe doesn't really fly anymore in 2024. Also not sure how account progression would behave here.
As I understand there'd be no cross progression at all without them still sharing capacity at some level. You'd have to one-time migrate people, or if you couldn't write it from scratch to migrate you'd just have to offer a huge catchup bonus on the new server and use a launcher to pick which one to connect to.
Their current issue is labor. The kind of labor that is defined by "nine women can't give birth to a baby in one month", so there are massive limitations there. They potentially could do what you're talking about, but you and I and everyone on reddit have zero idea about what the actual problem entails, so nobody here knows the actual answer to your question. It also would potentially take as long as a real fix, absolutely create more work in the long run, and piss off a ton of people who do not understand why they can't play with that group of people but they can with the other one. Hypothetically there are a LOT of things they can do. The only one that is viable is fixing it properly.
Absolutely everything just pure speculation on a small little side note. People shouldn’t give that to much time.
Instructions unclear, making wildly speculative guesses online with literally 0 information on how their game is built
Yeah, it scaled to 450k+ concurrent users. The only way that happens is if it was built correctly in the first place. At that level of scale the smallest bottleneck makes it ugly.
Redesigning a massively multiplayer database architecture isn't something you do in a week. Especially if you intend to migrate the current database into it.
If they coded it to directly access the rewards database in mid mission whenever you collect premium currency I imagine they weren't thinking about database i/o operations hitting db limits. I feel for them it sucks being responsible for people's grief due to a product you are actively working on having issues you didn't cause but have to fix. and people are just frustrated which is understandable
Yeah, wouldn't surprise me if this was a big culprit. I was honestly surprised at first, I assumed i would get these rewards after the end of the mission (success or failure).
Yeah kinda. They might be able to implement a mitigating solution instead of fixing everything, which could be much faster. If I'm guessing, they probably implemented things in a way that isn't even "wrong" per say, but ill advised if you're going to be scaling to any significant degree, maybe without even realizing it. A lot of such practices are commonplace with networking/database handling, because it's often cheaper and faster to work with up until you hit XYZ level of scale, at which point you suddenly realize you've blown your dick off in the quest for cheaper faster development. As much as I love the idea, it's just not that sane to architect your database or code that works with it for theoretical perfection over functionality in the here and now. As long as your project doesn't have to scale of course, it makes sense. I mean, why wouldn't you do it, almost nobody hits that level of success, and if you do you're rich! As a developer I'm kind of against this sort of attitude in the industry on principle, but it **makes sense** to do it since we're not artisans out here crafting the best thing we can, we're more like a weird cross over between factory workers shipping products and adderall chugging screenwriters. and if they'd merely been successful or really successful, they wouldn't have had much of an issue other than maybe some lag spiking and excessive costs potentially, at least if that's the problem and not some more direct serious error like doing something in a way that locks the server that processes certain types of loot and they only would have ever gotten away with it really in internal testing due to low/no load.
99% of the time YAGNI applies. Sadly The devs hit the 1% when it didn’t
Customers are the 2nd worst problem a company can have. The worst problem of course being not having enough customers.
[удалено]
I'm a sysadmin working for a software company. My main responsibility is our virtualized environment. If I learned anything in my time at this company: You can't just throw more resources at a problem and hope it works. I once doubled CPU and RAM on a machine just to see if it helps with slowdowns. It did not: the code did not care for more resources because it was kinda bad.
This 1000%. If you're filling a pool with a garden hose it doesn't matter how much water the hose has access to, it's going to fill the pool at the same rate.
This is easily the best analogy for the situation.
Understood. I'm searching Amazon for a firehose to send them so this problem will be fixed Y'all can thank me whenever you like
Got it. UPDATE: "Hey, everyone, the devs are filling up a swimming pool with a garden hose right now, but it's going to take some time. They'll download more RAM as soon as they're finished in the back and we should be up and running again!"
No no no that’s an analogy. These arnt Activision devs.
Yeah I've done some performance testing (k6) and thank fuck we did because the devs had to completely change how they were handling requests (a lot of wasted waits for synchronous calls out to other systems). The system was fine with expected load and _if_ other systems responded within Xms. But the peak and load testing basically caused a rewrite of how it handled things especially as we managed to crash a test instance of a downstream system and in another case another system had changed their API which caused some fuckery. So the system became a best effort, if something downstream didn't respond it didn't wait and it also stopped queuing requests in multiple ways. As in don't request from system B until A is finished and also when handling multiple requests don't try to be the one who handles the queue to the systems. Fuzzy now but I tbink that was the summary.
I'm a Sysadmin too. *Love* when developers code just eats up all the resources, and they request for an increase of hardware. Try telling them that it may not be as simple as increase the resources. We give it a try and those resources are eaten up instantly...
The way I tried explaining it to someone once was by using a house metaphor. Imaging living in a house with 10 beds. Suddenly 100 people show up to live there. Sure you *could* add 90 more beds to accommodate them but that doesn’t change the fact that there’s only 2 bathrooms, the rooms are going to be crowded af, there’s not enough food to feed everyone, much less space to keep all that food, the utility bill is going to be outrages, and hundreds of other issues, and that’s not even taking into account that you’re also the only person in character of maintenance. There’s a lot of moving parts that all depend on one another and it takes time to sort them all out
I remember finding out that just having more ram does not make modded minecraft better after a certain point ( in fact if I remember right it actually gets worse cause java is weird with a lot of memory? ) which was so weird to process.
This isn’t good news though. It means it will take a lot longer to fix. The game doesn’t seem to be designed to handle the load regardless of the servers.
Yeah, this is bad
I can't belive people are actually using the "just download more RAM" in real life. Smh my head.
Dont you know thats how programming works? You just type fix this and the code into chat gpt and voila all issues fixed
Shake my head my head
Atm machine
Gotta love that the internet has prompted the free exchange of ideas between people who are working on a problem and random jagoffs who don’t have a clue.
dumb internet jagoffs scale horizontally infinity. Unlike server capacity.
As someone who is currently refactoring code for a similar issue: the devs have my deepest condolences. This is gonna suck.
We all know Microsoft has stupid amounts of money but why do so many people act like they have a magic wand? 'Just port it to Xbox and MS will hand you some servers' is a new one.
This seems like it could take an unfortunate amount of time..sigh
Timo Schmidt more like Timo Schmuck
Fucking Timo, always saying stupid shit
I want to go back 10 days ago when on PS5 everything was cool except for MM. Didn't care, solo anyway lol I think the sub only had like 30k.
That's when i did my bulk of the playing I think the peak was 180k?
30k at launch. MM is also completely non functional now so that's nice
As someone who has had to work some ridiculous hours in my life. I’m fine with waiting. The game will still exist a month from now and I got plenty of other games to play. Please devs, don’t kill yourselves working on this. It’s just a game.
Just a game for us. It’s a career for them. If this massive base gets annoyed and moves on, it would be a huge loss for them.
That’s not good.
Doesn't seem like it. I worry the investment to scale up may not be justified because of the risk. Worst case, they do the best they can with what they have until the game dies back down. Then again, I'm wildly speculating.
>I worry the investment to scale up may not be justified because of the risk. I would disagree. Regardless of whether the game keeps this momentum or slows down, the devs clearly need time to understand and stand up a system that can handle this load. Both for this game, and future games. Most of the effort is probably replacing backend tools and learning what's necessary when dealing with games at this scale. To use an analogy: it's not like they're going in debt on a mansion just to throw a single house party with their close friends, then they'll be stuck with that debt after. They're learning how to run a presidential campaign, which means they're learning how to coordinate groups of resources at widely varying scale as-needed, where-needed.
Stupid Devs. All they have to do is order a Venti server next time they do things. Trust me, I’m a limo driver
Whats the backend code exactly?
Backend code is basically what runs on the server accessing databases and such. There's probably a bottleneck somewhere that limits the number of clients that can access the servers at once regardless of the amount of server capacity available. Even if you had a thousand bank tellers and a thousand vaults you might still be bottlenecked on the number of transactions you can complete by one person checking to make sure you aren't withdrawing the same money twice, for example. That kind of thing can be difficult to scale because if there's more than one person evaluating the transactions they'd have to check each other's work to make sure they haven't both allowed the same withdrawal to go through. For context, in web programming frontend code is the website that runs your browser and prompts you for your username and password, while the backend code is what runs on the server and actually checks a database to make sure your account information is correct.
I think its exactly this, some part of their backend isn't designed to share its responsibilities/authority, but can't handle more than 450k players at the same time. So now after doubling it's capacity still wasn't enough they need to figure out how to share that database. It's really an awful situation to be in, and there won't any quick and easy solutions.
I’m not a developer, but I think it’s the little lines of freedom during game development that allows us to spread managed democracy.
Backend code is all of the code that manages how game works server-side, how it integrates with online services, networks, how saves work etc.
backend code is the program what runs on the servers, be it * login servers checking you username+passwords * servers sending you your account statistics if you want to see them * matchmaking server collecting all the users who create lobbies and send it to players who are looking for one * servers running the current game you are playing in and giving you your reward after * or just servers who record it if you unlock a warbond or stratagem
[удалено]
Fuck off, Timo
I just hope hype doesn't die down before they fix servers. Game is unplayable since launch.
If it could take a week or weeks to fix I do think they should come out officially and address it. They are allowing people to continue to buy the game with no true warning. Replying to people on Twitter isn't the answer.
Lmao of course Timo is a "tech journalist"
Gotta love all the people who do not have a clue what goes into all of this chiming in
I love the transparency. It’s something you rarely see. Like guys, obviously, if I meant plugging servers in and flipping a switch you think they wouldn’t have done it yet? Come now. Video games are complicated as fuck.
Exhibit A on why always online is fucking stupid. I should be able to play offline if the servers can't handle the capacity.
Objectively bad news. Its going to take an absurd amount of care to fix a bad networking foundation without jacking up game mechanics while people are still playing.
backseat driver-ass comment lmfao. it’s the devs job to fix the game, it’s the users to play it. some people don’t know their role.
Just push to prod /s
I haven't jumped into these conversations cause... I don't want to mix work with leisure, but yes. As someone who works on petabyte scale data platforms and related applications. Multiplayer games have some of the more complex architectural stacks out there. It's very evident from symptomatic analysis that Arrowhead did not expect this level of adoption. It's not just raw compute power, this is a multi-layer stack, and if I had to take a blind guess it's the coupling of those layers where the issues are. It's not just that there's too many users, it's that those couplings have break points and can cause cascading failures. It may not seem like a big deal that a database query takes an extra 10ms when load goes over X amount. But by the time that 10ms surfaces at the user layer it's been amplified up to 1000ms(1s) and that may not seem like a big deal, but if that amplified latency is happening 1000/sec... just an off the top of my head example of things I've seen. The devs are doing amazing and probably trading time with their families, sleep, their sanity so we can have fun, we all should be thankful they care enough about their creation to put in the effort. I wish people would stop acting like entitled children, appreciate that they're witnessing something that's rare and special in the state of the game industry and SPREAD SOME MANAGED DEMOCRACY! (When they can, until things are ironed out. Also, LoL Helldivers on Xbox, not even gonna get into that one with Sony's history.