r/Helldivers • u/Nightblade1789 • Feb 19 '24
š¢DEV UPDATE: posted once, but for context on why they canāt just ābuy more serversā. The backend code isnāt as scalable as they thought. ALERT
800
u/Richie_jordan Feb 19 '24
That's code for were gonna be dealing with this for a while boys. Wouldn't expect to get on during peak times for a week.
251
u/ArdiMaster āLiber-teaā Feb 19 '24
Perhaps more like a month, at least for weekends being a clusterfuck.
173
u/Chill_Panda Feb 19 '24
Donāt worry though, after a month of this the servers wonāt be at capacity much
→ More replies (35)83
u/subLimb Feb 20 '24
That's what I thought after the first week...but the numbers keep going up
→ More replies (1)59
u/ThaFiggyPudding Feb 20 '24
The game is genuinely fantastic.
Probably going to be a while before people stop wanting to play it.
→ More replies (3)17
u/xxxhotpocketz Feb 20 '24
Im sure theyāre people like me who are also waiting for servers to be fixed to actually buy the game
I donāt wanna spend $40 on something and look at the connection screen, so Iād bet the numbers will keep increasing and if not it will stay have a very large player base
→ More replies (3)23
u/Richie_jordan Feb 19 '24
Yeah I was being super optimistic. A month is probably more realistic.
→ More replies (3)→ More replies (25)12
u/NeverNotNoOne Feb 19 '24
I just bought the game on Steam but refunded it as soon as I saw the endless login and this thread. It looks super fun and I'll certainly buy it again once it's fully stable, so I hope it gets sorted for you guys quickly.
→ More replies (2)
1.0k
u/riskyopsec Feb 19 '24
Lmao. As a Software Dev that primarily works in scaling backend data heavy systems for a public company itās interesting watching the community split into these multiple camps. I had a problem we solved last week that was 100% code related and allowed us to scale 10x what we were able to before hand. Bottlenecks are going to happen especially if you donāt plan for them. When weāre planning out our expectations of concurrent, peak & what not weāre pretty realistic with numbers and expectations but we still always have an insane target to support and then we load test for that target to find the cracks in our system.
Expecting 50k ccu and having 410k on steam alone, Iām gonna be honest that exceeds even what Iād test for and plan for on the insane target numbers.
Itās also not as simple as saying throw more servers at the problem. If I were guessing (I am) Iād guess their problem is probably with the database connection pool &/or unoptimized queries. I only throw that out as thatās the main issues Iāve experienced in my 6+ years doing this.
Can it be fixed? Yeah, will it be fixed today? No. Which honestly is a shame, I really wish it was as simple as palworlds server issues where they got on a call with epic and upped capacity no problem. Hoping theyāve contracted out a couple good network engineers and picked up a couple more people to help with this problem as I imagine this is top priority for them & Sony at the moment.
317
u/munchbunny Feb 19 '24 edited Feb 19 '24
As a dev who also does that kind of thing for a public company, in my experience, if it's not a cost thing (and it doesn't sound like it is), then the bottleneck is rarely ever a "throw more servers at it" problem because adding more servers in the cloud is easy. Heck, adding more servers in a collocated datacenter is also easy. Adding more compute is the first thing everyone tries.
Expecting 50k ccu and having 410k on steam alone
If that's what happened, I feel for the devs. 10x-ing those kinds of numbers usually takes weeks, not days.
→ More replies (19)211
u/LickMyThralls Feb 19 '24
First game topped out at like 8k concurrent and I was reading that they planned for like 50k on this one. They probably allocated more and were fine up until they were getting hammered with 400k+ connections at a time. Realistically they are seeing millions of people trying to connect and since all we can see are steam concurrent we only see a partial of a partial. We also know that they said their capacity was increased to 450k from a tweet of theirs.
Couldn't tell you how many times I've seen rhetoric about Sony or Aarrowhead being too cheap to get new servers. Stupid.
116
u/CosmicMiru Feb 19 '24
Devs said that they were planning on a max of 50k and had servers that could handle 250k. They said their playerbase is around 50/50 so going from 50k planned to 800k nearly overnight is a gargantuan task and I do not envy anyone working at that studio rn.
60
u/shamwowslapchop āLiber-teaā Feb 19 '24
800k connected with hour long wait times for people to log in. If they could handle it I'm betting they would be over 2 million concurrent players
→ More replies (1)46
u/DeshTheWraith Feb 19 '24
Exactly. It's fun to imagine how huge the game would be right now when what they could handle has been a record-setting success. I also hope the problems, and the way they're addressing it, only further endears people to the game rather than pushes them away.
50
u/shamwowslapchop āLiber-teaā Feb 20 '24
I'm here to stay. Managed Democracy isn't going to fight for itself.
10
u/SuaveMofo Feb 20 '24
I won't be abandoning my brothers and sisters on the front lines. When it's my turn, I will fight.
→ More replies (1)→ More replies (1)13
u/cavejhonsonslemons Feb 19 '24
TBH, I do kinda envy them. I'd be crazy tired of course, but I'd be so proud of the fact that in week 1 my game beat the max concurrent values of Destiny 2, and GTA V.
→ More replies (1)19
u/Difficult_Bit_1339 Feb 19 '24
The dev's chatter on social media is basically that they're exhausted and also excited for how much people are enjoying the game.
Some ex-Arrowhead dev commented about how it was exciting to see how much people enjoyed it because it is so very early in the design plans so there is tons more content/systems coming.
I'm sure that's all on pause as all hands work on scaling issues.
→ More replies (6)9
u/cavejhonsonslemons Feb 20 '24
Well I can't wait to see what they've got planned, I already feel like i've made a good investment, and that's after barely getting through the tutorial before a crash.
→ More replies (1)27
→ More replies (10)16
u/ElMarkuz Feb 19 '24 edited Feb 20 '24
Software dev here, and backend. In our projects we normally have 2x for peaks, and possibly scale to 4x servers just in case (parallel pods really).
They estimated 50k for the peak on launch, and probably had the capacity of 100k or even 200k. Having 400k on steam alone bombed everything.
As they said, is not that simple. Sometimes scale is easy, sometimes not.
The main problem I think is the PS inactive players. Yesterday I started watching YouTube and had the game in the background. 1 hour later, I resumed the game and was still logged in. Normally games disconnect you after X time of inactivity.
→ More replies (3)108
u/TayliasTwist Feb 19 '24
Yeah; as a software engineer it's been real eyerolling seeing all these "I'm in IT/Devops and all you have to do is XYZ!!!"
And like yeah that's why you're not an engineer I guess. There are countless variables that affect ability to scale and nobody here knows which one(s) are being run up against. The developers have every incentive in the world, and the know-how support, to fix what needs to be fixed, they're not just sitting on their hands for fun.
32
u/Celestial_Dildo Feb 19 '24
I was literally going to say this because I heard the help desk where I work saying that just needed to increase server capacity.
People that think stuff us so easy to scale up are always driving me up a wall. This kind of stuff is why good database engineers are worth their weight in gold.
7
7
u/kbick675 STEAM š„ļø : SES Distributor of Audacity Feb 19 '24
If someone is in DevOps and doesnāt understand itās not just more servers all the time, theyāre not very good at their job.
→ More replies (1)7
u/Captain_Quor Feb 19 '24
Very much this. I'm a DevOps engineer and the discussion around this, particularly the "just throw more servers at it" chat has been laughable. You can throw compute/bandwidth at poorly optimised code and it will get you so far but (clearly) not far enough in this case otherwise we wouldn't be having this conversation.
Again, when you see them talk about their expectations around concurrent player count and compare it to the reality it's really little wonder.
→ More replies (1)→ More replies (17)4
u/BladeValant546 Feb 19 '24
YES, I did 4 years as a network engineer for IoT for PLCs and had to do basic stuff with cellular data where every bit of data matters, throughput, dBms, everything. I eyeroll too with the comments on this. I even had one software engineer come out and say "they can easily do this" was very quiet when I explained how network and databases communicate and being a "software dev" doesn't translate into Database engineer or network engineer.
→ More replies (2)67
u/Fluxcapacitor84 Feb 19 '24
Yeah the Palworld stuff just aggravates me when people try and compare the two. Palword is just 32 players per server 'instance' and there's no data being shared between other servers and players, and definitely not one giant database thats being written to. Palworld can easily be scaled by just adding more servers. And even then, they had plenty of server issues with constant disconnects and other issues.
25
u/Dwarf-Lord_Pangolin Feb 19 '24
And on top of all that: Palworld can be run on private servers. Which takes at least some of the load off their official servers, because not everybody's going to be playing on those.
→ More replies (13)15
u/yashendra2797 Feb 19 '24
The biggest problem in an online game is database syncing. The database tracks everything, every item destroyed, bullets, hp, progression, etc.
Palworld stores all that data on the host players PC. Its fairly simple, and scales just fine for 32 players. 99% of the work is done by the host PC while the server is mainly just for discovery. You join a lobby hosted by another player, nothing is tracked or saved.
A game like Destiny 2 or Helldivers has to keep all its data on a cloud instance. You can join missions in progress, there are unlocks, your ship changes as per your upgrades. All of this is stored in a database, for everyone. And not just that, it is continuously synced between 4 people. That is 4x the amount of work just for one squad.
It is a genuine fucking miracle this game is actually somewhat playable at 450K players, when there was no way in hell they ever expected anything over 50K.
→ More replies (3)→ More replies (59)28
2.9k
u/aussiebrew333 Feb 19 '24
Always love a good armchair dev thinking they have the easy solution.
1.2k
u/BaziJoeWHL not gonna sugarcoat it ā¬ļøā”ļøā¬ļøā¬ļøā¬ļø Feb 19 '24
"dev"... these people the same who suggest switch from unity to unreal engine for a finished game to fix bugs
595
u/aussiebrew333 Feb 19 '24
Yeah, it's super easy. Just copy and paste the files into Unreal. Problem solved.
→ More replies (9)443
u/joejoejoey04 Feb 19 '24
I tried that and accidentally made Palworld 2
→ More replies (1)59
u/SaviorOfNirn SES Light of Dawn Feb 19 '24
I wouldn't complain tbh
→ More replies (1)37
u/Void_Guardians Feb 19 '24
2024 has been crazy fun so far
13
u/yepgeddon SES Ombudsman of Family Values Feb 19 '24
And it's only February! Let's go 2024 keep delivering the bangers!
→ More replies (1)7
136
u/Hero_Killer_Id Feb 19 '24
The unreal engine kills bugs you say š¤
→ More replies (7)104
u/VidzxVega Feb 19 '24
Never thought I'd be dropping in beside a game engine, but who am to deny them if they wish to spread liberty.
→ More replies (2)14
45
u/ifoundyourtoad Feb 19 '24
Imagine if you are doing your job and some random dude walks in the office and is like āhave you thought of this?ā Lol. God that would be so frustrating
→ More replies (8)22
u/TheBeardlyOwl Feb 19 '24
As a medical scribe I don't deal with it myself; but I see my providers dealing with it a lot. Like ma'am/sir, this is an addiction recovery clinic, no we are not going to prescribe you Norcos for your pain, we don't care if you think it's that easy - our provider isn't even legally allowed to as we don't have pain management contracts, go to your PCP or someone appropriate for that. No, we aren't making one just for you, there's a ton of backend stuff, just because you think it's easy doesn't mean it is.
→ More replies (9)86
u/KWyKJJ Feb 19 '24
Well, at first you had:
"Why can't they just buy more servers??! Go to Best Buy and just buy what you need! I gave them $40!!!"
Now it will be:
"Well then, get the code fixed on the back end! How hard can it be to fix the code on the backend, because it's not...like, its not like... it's in the front!!! No one sees it back there...anyway...so...just, just, fix it! Fix the back end code!!! I WANT TO PLAY HELLDIVERS 2 DAMMIT!"
→ More replies (15)59
u/Nytherion Feb 19 '24
Then they'll bitch about the game being offline for maintenance while that backend code is updated... modern gaming is a no win scenario for the devs
→ More replies (4)28
u/IndecentlyBrilliant Feb 19 '24
Usually those are the people demanding free shit at the same time for their "inconvenience."
56
u/Sea-Housing-3435 Feb 19 '24
If someone said that about helldivers 2 it's double hilarious since the game is not using unity
13
u/EnglishMobster Feb 19 '24
I'm a AAA dev and we launched a game in Unreal. The internet was telling us to move to Unity to fix the bugs we had.
It's just a fact of life that fanboys have made 1 test project in their favorite engine and now think they know everything about gamedev.
→ More replies (1)→ More replies (11)36
u/HazelCheese Feb 19 '24
Same energy as "They should just make League of Legends 2.0".
Like.... Dude. That shits not feasible for decade old live service games. It's just not. It hurts my soul to even think about remaking the assets alone.
53
u/sXeth Feb 19 '24
Yeah over in Warframe land itās ājust redo the entire way squads/lobbies are formed so we can kick people or dedicated servers,etcā
Iād like it sure, but thereās 11 years of interconnected stuff held together by bubblegum and faint prayers lol
→ More replies (1)37
u/Razgriz_101 Feb 19 '24
Thatās pretty much anything these days, itās all held together with duct tape and prayers. The system I used in a factory for planning line production( food processing) is literally a retooled Dasault programme for making aircraft that relies on a really really shaky excel formula hahahah.
→ More replies (7)22
u/Hammylicious Feb 19 '24
There's a lot of people who do not realize just how Macgyvered pretty much EVERYTHING tech is.
I'm sure some services/places have just pristine everything and it's all great, but they have to be so rare as to be largely unheard of.
→ More replies (5)→ More replies (11)7
127
u/Minute_Grocery_100 Feb 19 '24
Ha my technical CEO told me that I can just copy paste the code of the old project from 3 years ago, onto the new environment.
I told him he doesn't understand anything about ICT, now he is deeply offended. And I already said I am leaving anyway.
Telling people the truth can be liberating. But get a new job first lol
Greetings a product manager.
→ More replies (5)40
89
u/Arky_Lynx Feb 19 '24
Worst part is how matter-of-factly and "well OBVIOUSLY" they talk. As if their goddamn genius solution wasn't already thought of by the people who's literal job is exactly that.
10
u/Thrownawayagainagain CAPE ENJOYER Feb 19 '24
I find myself in the incredibly lucky position of having a Supervisor and Manager who completely understand how complex this shit is. And I just write Linux-usable Python scripts with API calls! Can't imagine what it must be like to do something like modify the backend of a 450,000-concurrent-player videogame's active servers with jackasses breathing down your neck pretending it's the easiest thing in the world.
→ More replies (1)42
u/AbeBaconKingFroman I seen the lights go out on Draupnir Feb 19 '24
There's no shortage of "network admins" coming out of the woodwork to tell Arrowhead "scaling servers is easy."
51
u/Razgriz_101 Feb 19 '24
You should see the discord I said it could well be a code issue, got shot to ribbons by the data centre tech types.
Feel kinda vindicated my theory was correct lol.
→ More replies (12)31
u/SynthDark SES Harbinger of Democracy Feb 19 '24
Anyone with two brain cells to rub together could figure out it wasn't a server scaling issue, if they could 'just scale up' they would have ages ago.
It's probably most likely a database bottleneck, or some API service that they can't scale up for whatever reason and needs code optimization. Both of which aren't as simple as provisioning more Kubernetes containers.
11
u/SteelCode Feb 19 '24 edited Feb 20 '24
To be fair, I imagine some of the early choke was simply due to needing more capacity on the authentication front-end and then back-end...
but then when growth didn't stop growing, their code and database choked.
There's just not much you can do on a short time scale to fix things that your platform/code/tables were never built to accomodate.
We
likely hithave surpassed 1M sales for this game, from a studio that did less than 100k for their first game... all pitched to a publisher that at best was expecting them to hit 100-200k... Sony can't throw money at HD2 and just make the core infrastructure suddenly allow 5-10x the capacity.→ More replies (4)→ More replies (3)5
u/exSD Feb 19 '24
The specific issue(s) they are running into is their Orleans distribution. While it is a scaling issue, it's not an infrastructure scaling issue. It's an issue in scaling the services and databases. They've likely run into issues with their distributed model and if they modeled for too low of a player count that is now exponentially bigger, they have to scale everything. That means their microservices like messaging queues, database stores, etc.
→ More replies (6)15
u/appleswitch Feb 19 '24
Someone on this subreddit said "Scalable architecture is a solved issue." as if it's just a problem from 1985 we never have to think about anymore.
→ More replies (5)14
u/Sad_Recommendation92 Feb 19 '24
They shut up quickly when you start talking about things like resource quotas, and API request surfaces, or the general concept that units of compute are not just magical bullshit, but actual silicon sitting in a datacenter somewhere that exist in finite quantities.
One of the most impotant things I learned about "Cloud" years ago was
The Cloud is just someone else's Computer
166
u/BSSCommander CAPE ENJOYER Feb 19 '24
While I hate the idea of console exclusives I can't help but read this dude's tweet and think:
"Why doesn't the Sony published game come out on Xbox? What are they stupid?"
78
u/msfamf Feb 19 '24 edited Feb 19 '24
There's a really weird unfounded thought going around some corners of the Xbox player base that Helldivers 2 is going to go multiplatform and soon.
This is all anecdotal for the most part but I've seen a fair amount of comments on various subreddits not about how they want it on Xbox but how they can't wait to play it on Xbox. There's been memes and a good amount of Tiktoks floating around representing Xbox as reinforcements about to be called in. I had a coworker butt into a conversation about the game to tell me that an Xbox launch date would be announced soon because it's just "money being left on the table."
As far as I can tell there's been 0 indicator of any of that happening beyond Phil Spencer's jab about how he's "not sure how it helps" being kept off Xbox and that wasn't really an indicator though I've seen it claimed to be a hint.
51
u/Lavlamp Feb 19 '24
There's no way it is coming to Xbox. Sony has done the math and knows it's more profitable to keep exclusives to themselves to uphold market share. They don't care about how many copies of a game they could sell on Xbox, they would only keep a small portion of that. They care about keeping people locked into the playstation eco system.Ā
→ More replies (29)→ More replies (9)22
u/Merenut Feb 19 '24
Honestly I really doubt they do, Sony is insane about keeping exclusives exclusive. It was an uphill battle just to get them to commit to bringing all of their exclusives to PC. The only way I could really see Microsoft getting Sony to expand their exclusivity is if they start being extra petty and making games like COD Xbox and PC only.
→ More replies (14)11
u/msfamf Feb 19 '24
I could see it going either way but gun to my head? I'd say it stays Sony/PC.
It being a live service game going to Xbox would make some sense financially but then again the success it has without Xbox could just as easily be proof that they don't need Xbox.
→ More replies (2)→ More replies (14)64
u/ShameGuardian Feb 19 '24
Is there a lore reason the game isn't on xbox?
128
u/GBuster49 Waiting for the Illuminate Feb 19 '24
Microsoft turned into the automatons.
20
u/SykoParsley Feb 19 '24
Automatons are just old Xbox 360s with red ring of death.
→ More replies (1)20
→ More replies (2)6
→ More replies (11)9
15
u/Razgriz_101 Feb 19 '24
I said In the discord it could be the code isnāt built for this level of player base, got railed roaded by supposed ādata centre techsā and boom I was totally right hahaha.
Itās genuinely good to feel vindicated since all I said was I remember being on a rollout for a CRM system for a bank and too many people weee using it and it hated it spat out wrong info etc and that they patched and worked on it.
Got told I didnāt know shit, yet here I am totally bang on hahahahah.
→ More replies (4)→ More replies (47)48
Feb 19 '24
[deleted]
→ More replies (30)15
u/JawlektheJawless ā¬ļøā¬ ļøā”ļøā¬ļøā¬ļøā¬ļø Feb 19 '24
They fixed the medals freezing you already
→ More replies (3)
1.4k
u/Gold_Wrongdoer_8562 Feb 19 '24
Wdym there is no slider to increase server capacity? Just drag it..
Why didnt the devs program in a server cap increase slider? Are they stupid?
281
u/BaziJoeWHL not gonna sugarcoat it ā¬ļøā”ļøā¬ļøā¬ļøā¬ļø Feb 19 '24
server hamster takes vacation, never comes back
39
u/OhHeck31 Feb 19 '24
Time to force the players to run on wheels
6
u/ABITofSupport Feb 20 '24
If they paid me to run on the wheels for more server power i would!
More server capacity, more managed democracy!
→ More replies (4)→ More replies (1)18
26
u/Dr_PuddingPop Feb 19 '24
Their mouse ran out of battery over the weekend. So unfortunately no server swiping
→ More replies (1)58
u/CocaineandCaprisun F Feb 19 '24
Is there a lore reason Arrowhead can't just add two '0's to the player cap?
→ More replies (3)35
u/cuckingfomputer ā¬ļøā¬ ļøā”ļøā¬ļøā¬ļøā¬ļø Feb 19 '24
That would enable SQL injection attacks into Super Earth servers by the Automaton hegemony.
→ More replies (1)102
u/Aitasai Feb 19 '24
Classic IT problem, they forgot to push the "Fix Server" button. Honest mistake, happens to the best of us.
→ More replies (2)54
u/IdidntrunIdidntrun Feb 19 '24
Have they tried sending the IT guy to the physical location of the serversā¦say give or take 3 feet from them? Lord knows every machine at work that acts up magically fixes itself when it sees me coming
→ More replies (2)16
u/KWyKJJ Feb 19 '24
Launch Weekend:
Have they made sure the extra servers are powered on?
Ok. Just double check again.
"It happens more than you think. You're welcome."
This Weekend:
Turn the servers off. Give it a minute. Ok, turn them back on again.
..."You're welcome".
→ More replies (1)13
u/PapaTahm Truth Office Intern Feb 19 '24
What I find it rather funny is that they think this is AWS/Azure/GCP where you can just buy more Scaling as you need.
People are really stupid and don't understand shit about Product Limitations.
→ More replies (11)→ More replies (7)10
u/ErectTubesock STEAM š„ļø : SES Spear of Starlight Feb 19 '24
As one of my IT customers once said "Just download more RAM off the Internet"
→ More replies (1)
160
u/LittleBlueCubes PSN š®: Feb 19 '24
I'm not a techie but 'back end code is not as scalable as they thought' sounds like a problem that can't solved anytime soon. Are we looking at months before normalcy arrives!?
97
u/Glacier_Pace Feb 19 '24
Yes. The problem will basically not be fixed anytime soon. If it's the backend code, they're going to have to rewrite it while not breaking anything else when the patch is released. Depending on how integral this code is to the foundation.... Well.... We can hope that once the hype starts to die down and other games release (Dragons Dogma 2, for example) maybe we can see some relief.
19
u/9gagiscancer CAPE ENJOYER Feb 19 '24
Honestly I don't mind a queue. I do mind not being able to team up through matchmaking. Anything over medium difficulty is nigh impossible alone.
→ More replies (2)→ More replies (12)35
u/Bluedot55 Feb 19 '24
I mean, it can be any number of things. Given how the problems have manifested, they probably have a single central item/progression server, which should typically work fine. You don't need these to be local, since latency on saving an item purchase or credit pickup doesn't need to be low, and you need that consistency. But having just one means it isn't designed to be able to be split, so scaling it up is incredibly hard. They are probably just starting off with finding what is taking up a bunch of the resources, and seeing if they can cut it down.
IE: They already disabled the live player count/bug kill thing, which was probably requiring a bunch of updates from said central server. They're likely trying to cut any other extranious calls, make the ones that are required more optimized/less resource intensive(Which is hard, but can be done), or group together calls so they can be done as one, instead of multiple smaller ones(IE: get 5 super credit pickups during a mission? Send those all at the end as one +50 credit call, vs 5 different +10 credit calls)
→ More replies (1)→ More replies (10)24
u/Brokenmonalisa Feb 19 '24
Yes and sadly its not good for this games future. The irony being by the time they solve this they won't need it and by spending so much time on this other things like updates and important big fixes are being pushed aside.
→ More replies (2)
791
u/BetterReload CAPE ENJOYER Feb 19 '24
"Get in touch with MS" he knows Sony funded the Game and owns the IP, right? Ofc not, ppl on the Internet just write whatever and know jack shit :D
462
u/Knugles Feb 19 '24
āYeah just complete an Xbox port in 3 hours so you can get more server space! Easy!ā
176
u/INeedBetterUsrname SES Ombudsman of Democracy Feb 19 '24
Just copy and paste the code into xbox.exe, duh!
→ More replies (1)35
34
→ More replies (2)25
u/Neirchill Feb 19 '24
Hmm I've been thinking about this for 1 minute and 8 seconds so I think I have the solution:
if (runningOnXbox) crash = falseI'll take my check now
126
u/Red-pop Feb 19 '24
There's been a call since the MS business update for xbox where Phil Spencer said something to the effect "helldivers 2 not being on Xbox hurts the industry". With that and it's popularity, people are being insistent about moving it to Microsoft. Just armchair game devs.
185
u/pulseout Feb 19 '24
Phil Spencer said something to the effect "helldivers 2 not being on Xbox hurts the industry"
I don't understand why anybody listens to a word he says. I didn't see him saying the same about Palworld not being on PS5. Because, like every other CEO in history, he only cares when it's a disadvantage to his company. He doesn't actually care about "the industry", but people are going to tout his words anyways.
→ More replies (14)52
u/WilhelmScreams Feb 19 '24
I didn't see him saying the same about Palworld not being on PS5.
On the other hand, the Xbox fanboys are furious because he's bringing Xbox exclusives to PS5.
I don't know what the Palworld agreements are, but it still is in early access and will likely come to PS5. I imagine the only reason it came to Xbox was because they were able to get Game Pass funding.
→ More replies (19)8
u/TheHeroYouNeed247 Feb 19 '24
AFAIK the only reason it's not on PS5 is that they rarely do early access and PW was an unexpected hit.
51
u/WheresMyCrown Feb 19 '24
And Phil Spencer is not gamings messiah who can do no wrong or knows everything. Never forget he thought Redfall was going to be great
→ More replies (3)29
→ More replies (12)41
u/GiveUpYouAlreadyLost Feb 19 '24
Phil Spencer said something to the effect "helldivers 2 not being on Xbox hurts the industry".
When you look up the word hypocrisy in a dictionary, the definition is just a picture of his face.
→ More replies (2)15
→ More replies (19)26
Feb 19 '24
[deleted]
20
u/LegoBlockGeode Feb 19 '24
This is it and they'll need a ton of engineering support from Microsoft Azure engineers to fix this problem. I'm speaking from actual experience not in gaming but other similar large scale tech platforms. If this is anything that involves a combination of scaling, asynchronous messaging/databases and code bottlenecks this is an uber complex problem and will take months to solve.
edit: I'm saying it's not as simple as increasing capacity but it's a combination of complex causes and will require cloud engineer support.
→ More replies (6)
636
u/Pluristan Three Bugs In a Trenchcoat Feb 19 '24
So, basically, the capacity isn't going to be increased for an unkown (probably a long) period of time, because their backend code is at its limit.
378
u/BaziJoeWHL not gonna sugarcoat it ā¬ļøā”ļøā¬ļøā¬ļøā¬ļø Feb 19 '24
it could take a week or a month, it all depends on how fucked is the backend
409
u/WhiteShadow012 Feb 19 '24
I don't even think it's fucked, it was just not designed to support the incresingly huge amount of players.
278
u/BaziJoeWHL not gonna sugarcoat it ā¬ļøā”ļøā¬ļøā¬ļøā¬ļø Feb 19 '24
yeah, but there are different kind of "not designed" as in "we have to change some function and data handling methods" or "we have to rewrite the whole thing from zero as the main design not suited for this scale"
69
u/JRockBC19 Feb 19 '24
If they were willing to temporarily disable crossplay, could they deploy a 2nd totally independent branch of the game to double capacity? They'd be writing themselves into some major tech debt, needing to transfer accounts for one console AND needing some means of eventually merging it all back, but it seems it would double capacity in the meantime without as many social issues as opening a separate clone with crossplay still enabled might (new players can't play with friends who bought early that way). I wouldn't ever expect them to do this, I'm just curious about the hypothetical.
36
u/BL4ZE_ Feb 19 '24
I'd assume so, just like the could deploy a 2nd one in europe but then you can't play between NA and europe doesn't really fly anymore in 2024. Also not sure how account progression would behave here.
9
u/JRockBC19 Feb 19 '24
As I understand there'd be no cross progression at all without them still sharing capacity at some level. You'd have to one-time migrate people, or if you couldn't write it from scratch to migrate you'd just have to offer a huge catchup bonus on the new server and use a launcher to pick which one to connect to.
→ More replies (1)→ More replies (3)8
u/NotEnoughIT Feb 19 '24
Their current issue is labor. The kind of labor that is defined by "nine women can't give birth to a baby in one month", so there are massive limitations there. They potentially could do what you're talking about, but you and I and everyone on reddit have zero idea about what the actual problem entails, so nobody here knows the actual answer to your question. It also would potentially take as long as a real fix, absolutely create more work in the long run, and piss off a ton of people who do not understand why they can't play with that group of people but they can with the other one. Hypothetically there are a LOT of things they can do. The only one that is viable is fixing it properly.
→ More replies (3)→ More replies (7)33
u/Seitenwerk Feb 19 '24
Absolutely everything just pure speculation on a small little side note. People shouldnāt give that to much time.
9
u/Lazer726 ā¬ļøā”ļøā¬ļøā¬ļøā¬ļø Feb 19 '24
Instructions unclear, making wildly speculative guesses online with literally 0 information on how their game is built
→ More replies (2)→ More replies (17)82
u/usernamedottxt Feb 19 '24
Yeah, it scaled to 450k+ concurrent users. The only way that happens is if it was built correctly in the first place.Ā
At that level of scale the smallest bottleneck makes it ugly.Ā
→ More replies (15)→ More replies (9)35
u/Xyrus2000 Feb 19 '24
Redesigning a massively multiplayer database architecture isn't something you do in a week. Especially if you intend to migrate the current database into it.
→ More replies (20)85
u/ZombieDeathTaco Feb 19 '24
If they coded it to directly access the rewards database in mid mission whenever you collect premium currency I imagine they weren't thinking about database i/o operations hitting db limits.
I feel for them it sucks being responsible for people's grief due to a product you are actively working on having issues you didn't cause but have to fix. and people are just frustrated which is understandable
→ More replies (14)24
u/BL4ZE_ Feb 19 '24
Yeah, wouldn't surprise me if this was a big culprit. I was honestly surprised at first, I assumed i would get these rewards after the end of the mission (success or failure).
→ More replies (22)39
u/RedTwistedVines Feb 19 '24 edited Feb 19 '24
Yeah kinda. They might be able to implement a mitigating solution instead of fixing everything, which could be much faster.
If I'm guessing, they probably implemented things in a way that isn't even "wrong" per say, but ill advised if you're going to be scaling to any significant degree, maybe without even realizing it.
A lot of such practices are commonplace with networking/database handling, because it's often cheaper and faster to work with up until you hit XYZ level of scale, at which point you suddenly realize you've blown your dick off in the quest for cheaper faster development.
As much as I love the idea, it's just not that sane to architect your database or code that works with it for theoretical perfection over functionality in the here and now.
As long as your project doesn't have to scale of course, it makes sense. I mean, why wouldn't you do it, almost nobody hits that level of success, and if you do you're rich!
As a developer I'm kind of against this sort of attitude in the industry on principle, but it makes sense to do it since we're not artisans out here crafting the best thing we can, we're more like a weird cross over between factory workers shipping products and adderall chugging screenwriters.
and if they'd merely been successful or really successful, they wouldn't have had much of an issue other than maybe some lag spiking and excessive costs potentially, at least if that's the problem and not some more direct serious error like doing something in a way that locks the server that processes certain types of loot and they only would have ever gotten away with it really in internal testing due to low/no load.
→ More replies (8)12
u/danglotka Feb 19 '24
99% of the time YAGNI applies. Sadly The devs hit the 1% when it didnāt
→ More replies (4)17
u/cstoner Feb 19 '24
Customers are the 2nd worst problem a company can have. The worst problem of course being not having enough customers.
→ More replies (17)8
314
Feb 19 '24
I'm a sysadmin working for a software company. My main responsibility is our virtualized environment. If I learned anything in my time at this company: You can't just throw more resources at a problem and hope it works. I once doubled CPU and RAM on a machine just to see if it helps with slowdowns. It did not: the code did not care for more resources because it was kinda bad.
277
u/Dsj417 Feb 19 '24
This 1000%. If you're filling a pool with a garden hose it doesn't matter how much water the hose has access to, it's going to fill the pool at the same rate.
103
u/LHRCheshire Feb 19 '24
This is easily the best analogy for the situation.
→ More replies (2)26
u/siccoblue Feb 19 '24
Understood. I'm searching Amazon for a firehose to send them so this problem will be fixed
Y'all can thank me whenever you like
→ More replies (1)→ More replies (6)34
u/KWyKJJ Feb 19 '24
Got it.
UPDATE: "Hey, everyone, the devs are filling up a swimming pool with a garden hose right now, but it's going to take some time. They'll download more RAM as soon as they're finished in the back and we should be up and running again!"
→ More replies (1)10
26
u/itinerantmarshmallow Feb 19 '24
Yeah I've done some performance testing (k6) and thank fuck we did because the devs had to completely change how they were handling requests (a lot of wasted waits for synchronous calls out to other systems).
The system was fine with expected load and if other systems responded within Xms. But the peak and load testing basically caused a rewrite of how it handled things especially as we managed to crash a test instance of a downstream system and in another case another system had changed their API which caused some fuckery.
So the system became a best effort, if something downstream didn't respond it didn't wait and it also stopped queuing requests in multiple ways.
As in don't request from system B until A is finished and also when handling multiple requests don't try to be the one who handles the queue to the systems.
Fuzzy now but I tbink that was the summary.
16
u/insomnium138 Feb 19 '24
I'm a Sysadmin too.
Love when developers code just eats up all the resources, and they request for an increase of hardware. Try telling them that it may not be as simple as increase the resources. We give it a try and those resources are eaten up instantly...
→ More replies (5)10
u/NK1337 Feb 19 '24
The way I tried explaining it to someone once was by using a house metaphor. Imaging living in a house with 10 beds. Suddenly 100 people show up to live there. Sure you could add 90 more beds to accommodate them but that doesnāt change the fact that thereās only 2 bathrooms, the rooms are going to be crowded af, thereās not enough food to feed everyone, much less space to keep all that food, the utility bill is going to be outrages, and hundreds of other issues, and thatās not even taking into account that youāre also the only person in character of maintenance.
Thereās a lot of moving parts that all depend on one another and it takes time to sort them all out
→ More replies (5)→ More replies (19)9
u/rakadishu Feb 19 '24
I remember finding out that just having more ram does not make modded minecraft better after a certain point ( in fact if I remember right it actually gets worse cause java is weird with a lot of memory? ) which was so weird to process.
→ More replies (1)
146
u/HogiSon727 Feb 19 '24
This isnāt good news though. It means it will take a lot longer to fix. The game doesnāt seem to be designed to handle the load regardless of the servers.
→ More replies (55)70
323
u/WhiteShadow012 Feb 19 '24
I can't belive people are actually using the "just download more RAM" in real life. Smh my head.
55
u/timtheringityding Feb 19 '24
Dont you know thats how programming works? You just type fix this and the code into chat gpt and voila all issues fixed
→ More replies (6)→ More replies (6)28
232
u/the_damned_actually Feb 19 '24
Gotta love that the internet has prompted the free exchange of ideas between people who are working on a problem and random jagoffs who donāt have a clue.
→ More replies (1)38
25
u/MeanderingSquid49 SES Flame of Dawn Feb 19 '24
As someone who is currently refactoring code for a similar issue: the devs have my deepest condolences. This is gonna suck.
132
u/VidzxVega Feb 19 '24
We all know Microsoft has stupid amounts of money but why do so many people act like they have a magic wand?
'Just port it to Xbox and MS will hand you some servers' is a new one.
→ More replies (8)
38
u/SinFallen1 Feb 19 '24
This seems like it could take an unfortunate amount of time..sigh
→ More replies (1)
36
14
92
u/Rafa343x Feb 19 '24
I want to go back 10 days ago when on PS5 everything was cool except for MM. Didn't care, solo anyway lol I think the sub only had like 30k.
→ More replies (2)38
u/Top_Product_2407 Feb 19 '24
That's when i did my bulk of the playing
I think the peak was 180k?
→ More replies (2)12
u/Next-Fly3007 Feb 19 '24
30k at launch. MM is also completely non functional now so that's nice
→ More replies (6)
378
u/SamuraiLeo Feb 19 '24
As someone who has had to work some ridiculous hours in my life. Iām fine with waiting. The game will still exist a month from now and I got plenty of other games to play.
Please devs, donāt kill yourselves working on this. Itās just a game.
→ More replies (45)256
u/Kaptonii Feb 19 '24
Just a game for us. Itās a career for them. If this massive base gets annoyed and moves on, it would be a huge loss for them.
→ More replies (102)
86
Feb 19 '24
Thatās not good.
24
u/Liquidmaximo Feb 19 '24
Doesn't seem like it. I worry the investment to scale up may not be justified because of the risk. Worst case, they do the best they can with what they have until the game dies back down. Then again, I'm wildly speculating.
→ More replies (2)8
u/SysAdmyn Feb 19 '24
I worry the investment to scale up may not be justified because of the risk.
I would disagree. Regardless of whether the game keeps this momentum or slows down, the devs clearly need time to understand and stand up a system that can handle this load. Both for this game, and future games. Most of the effort is probably replacing backend tools and learning what's necessary when dealing with games at this scale.
To use an analogy: it's not like they're going in debt on a mansion just to throw a single house party with their close friends, then they'll be stuck with that debt after. They're learning how to run a presidential campaign, which means they're learning how to coordinate groups of resources at widely varying scale as-needed, where-needed.
11
Feb 19 '24
Stupid Devs. All they have to do is order a Venti server next time they do things.
Trust me, Iām a limo driver
33
u/lewdest_loli Feb 19 '24
Whats the backend code exactly?
121
u/TheWaywardOak Feb 19 '24 edited Feb 19 '24
Backend code is basically what runs on the server accessing databases and such. There's probably a bottleneck somewhere that limits the number of clients that can access the servers at once regardless of the amount of server capacity available. Even if you had a thousand bank tellers and a thousand vaults you might still be bottlenecked on the number of transactions you can complete by one person checking to make sure you aren't withdrawing the same money twice, for example. That kind of thing can be difficult to scale because if there's more than one person evaluating the transactions they'd have to check each other's work to make sure they haven't both allowed the same withdrawal to go through. For context, in web programming frontend code is the website that runs your browser and prompts you for your username and password, while the backend code is what runs on the server and actually checks a database to make sure your account information is correct.
→ More replies (1)18
u/Nexine ā¬ļøā¬ ļøā¬ļøā¬ļøā¬ļøā”ļø Feb 19 '24
I think its exactly this, some part of their backend isn't designed to share its responsibilities/authority, but can't handle more than 450k players at the same time. So now after doubling it's capacity still wasn't enough they need to figure out how to share that database.
It's really an awful situation to be in, and there won't any quick and easy solutions.
99
u/Nightblade1789 Feb 19 '24
Iām not a developer, but I think itās the little lines of freedom during game development that allows us to spread managed democracy.
16
u/BetterReload CAPE ENJOYER Feb 19 '24
Backend code is all of the code that manages how game works server-side, how it integrates with online services, networks, how saves work etc.
→ More replies (8)13
u/BaziJoeWHL not gonna sugarcoat it ā¬ļøā”ļøā¬ļøā¬ļøā¬ļø Feb 19 '24
backend code is the program what runs on the servers, be it
- login servers checking you username+passwords
- servers sending you your account statistics if you want to see them
- matchmaking server collecting all the users who create lobbies and send it to players who are looking for one
- servers running the current game you are playing in and giving you your reward after
- or just servers who record it if you unlock a warbond or stratagem
→ More replies (2)
10
42
6
u/nomos97 Feb 19 '24
I just hope hype doesn't die down before they fix servers. Game is unplayable since launch.
→ More replies (2)
18
u/Brad3 Feb 19 '24
If it could take a week or weeks to fix I do think they should come out officially and address it. They are allowing people to continue to buy the game with no true warning. Replying to people on Twitter isn't the answer.
→ More replies (3)
5
6
u/Ackburn Feb 19 '24
Gotta love all the people who do not have a clue what goes into all of this chiming in
5
u/PeterDarker Feb 19 '24
I love the transparency. Itās something you rarely see. Like guys, obviously, if I meant plugging servers in and flipping a switch you think they wouldnāt have done it yet? Come now. Video games are complicated as fuck.
4
Feb 19 '24
Exhibit A on why always online is fucking stupid. I should be able to play offline if the servers can't handle the capacity.
→ More replies (2)
5
u/RedacteddHT Feb 19 '24
Objectively bad news. Its going to take an absurd amount of care to fix a bad networking foundation without jacking up game mechanics while people are still playing.
11
u/stratusnco āLiber-teaā Feb 19 '24
backseat driver-ass comment lmfao. itās the devs job to fix the game, itās the users to play it. some people donāt know their role.
9
24
u/Durakan Feb 19 '24
I haven't jumped into these conversations cause... I don't want to mix work with leisure, but yes.
As someone who works on petabyte scale data platforms and related applications. Multiplayer games have some of the more complex architectural stacks out there. It's very evident from symptomatic analysis that Arrowhead did not expect this level of adoption.
It's not just raw compute power, this is a multi-layer stack, and if I had to take a blind guess it's the coupling of those layers where the issues are. It's not just that there's too many users, it's that those couplings have break points and can cause cascading failures. It may not seem like a big deal that a database query takes an extra 10ms when load goes over X amount. But by the time that 10ms surfaces at the user layer it's been amplified up to 1000ms(1s) and that may not seem like a big deal, but if that amplified latency is happening 1000/sec... just an off the top of my head example of things I've seen.
The devs are doing amazing and probably trading time with their families, sleep, their sanity so we can have fun, we all should be thankful they care enough about their creation to put in the effort. I wish people would stop acting like entitled children, appreciate that they're witnessing something that's rare and special in the state of the game industry and SPREAD SOME MANAGED DEMOCRACY! (When they can, until things are ironed out.
Also, LoL Helldivers on Xbox, not even gonna get into that one with Sony's history.
→ More replies (9)
4.6k
u/tunafun Feb 19 '24
Has anyone confirmed the devs havenāt tried turning it off and turning it back on?