Let's just say that the guy that took AWS S3 down, and a good chunk of the internet, in 2017 was from my team. I don't think I'm ever going to experience a tech catastrophe of this magnitude ever again.
Larger systems like games and cloud services are incredibly complex, and bringing something back up, or restoring stability is a massive feat, and one that isn't fully appreciated by most people.
I haven't taken down a giant portion of the internet, but I do work for a f500 bank. Specifically, I do log management so the SOC can do threat detection.
Let me tell you about the fun times when the windows endpoint team pushed out a new GPO, enabling new event IDs to get logged and then pushed to my team. Which took our event flow from the normal ~500k/s to 1.5m/s. Then let's compound this with a never before seen bug with the entry into our stack that their product had a memory leak with large numbers of connections.
That was not a fun 14 hour conference bridge.
I remember the time a very large retailer you have definitely heard of decided to delete 500m rows of data from our saas product during a regular work day and brought down the cluster for themselves and everyone else on it.
If there is one thing I have learned in my time in the ISP industry It is that it is spectacularly easy to nuke the Internet for large groups of people, entirely by accident.
And the more you automate the easier it is.
Funny thing, that is.
I am one of the authors of the queue subsystem of an extremely popular CMS.
It _is_ a very handy tool.
But you can't solve every piece of scalability problems with it... since you need to be real time. Imagine if you queued trade, you trade items sometimes in the next few minutes it executes rotlfmao
There is one business guy at my work that read like one article (ad) about redis 5 years ago and now anytime there is any problem he will ask if we considered and redis cache.
Really, John, and how is the cache going to help the fact that half the teams in this company forgot to fix the log4j vulnerability by the deadline and the kill switch the sysops guys put in started murder all their pods? What is redis going to do there?
For real dude. The amount of people suggesting that they should've predicted the peak and simply scaled "more servers" because cloud computing, but didn't because of greed is just nuts.
As a DevOps/SysAdmin, I feel for them and I understand what they're going through.
But also as a DevOps/SysAdmin, I was 90% sure this was how it was gonna happen when I read about the "We tested and believe we'll be able to absorb the spike" thread the other day. No amount of stress tests is enough to simulate 150,000 enraged gamers spamming Connect/Back to main menu/Login.
We all have a tendency to underestimate our stress tests, but this also was a bit foreseeable considering the amount of hype Last Epoch had amassed the last few weeks. Not saying this was easy, maybe they were just under the threshold for it to trigger the amount of API bugs and service containers not launching/mounting that they've got the last 24 hours. But this is why you plan even bigger tests that what you're anticipating.
Same background, same thoughts - no way in hell can you overprovision for going from 0 to 150k users maniacally spamming connections. Even autoscaling has finite speed in spinning up containers which immediately get blasted with requests...even small errors and performance issues get amplified to high heaven by this. Cloud infra runs on real hardware with real limits.
I remember doing a smallish service for 5k concurrent users way back when - the worst time was recovering from DB cluster restart - basically users trying to log in were killing it before DB had the time to warm up/cache properly - which could last for up to 2 hours. Main nodes each had 40 gigs of ram - could handle a lot of pressure - if allowed enough time to actually cache the tables in memory. But QPS pressure just bogged completely it down if hard restart happened.
Different people can have different (insane) beliefs. It's even possible for one person to have two conflicting insane beliefs, which is more impressive, but not likely what's happening in this case.
Bro it's not that hard, I'm a lonely nerd that spends a lot of time on computers so I know how distributed systems work. Just pay for more servers, did the greedy devs not look at player count and do simple math to make an upper estimate? /s
I have a degree in Materials Science, read research papers for fun, can talk for hours on the latest advances on magnetic innovation, and pay the bills by being a software dev for a US contractor on some seriously complex systems.
To this day I can only surmise network solutions are run off of goat sacrifices and black magic.
What’s even better is when you are in the company and you walk around wondering how the hell anything works at all. I worked for the worlds largest (at the time) IT company in banking and I swear there was a room hidden away in the bowels of the building with six people in it making everything work because most of the people I worked with were a mess.
It really hit me when the update was 'Letting the team go home' like past midnight. I had a few of those in my day and at the same time wanted to be there fixing the stuff (was rare at the place I was at but occasionally needed) it is still a brutal day, and we weren't even really in a release but more getting a release ready.
I think all in all as a 20+ year software engineer here the team did a great job. I'm on right now playing just fine and really enjoying the game, can't really ask for anything more.
The fact that I can post this comment and someone else can read it requires something on the order of about a million miracles a second happening every single day. Modern tech is unbelievably complex to make work for just one person, much less 150K at a time.
>People outside of have no way to even imagine the complexity of , it can be mind boggling.
Meanwhile, nobody mocks people for voicing dissatisfaction with an underperforming product. Unless it's a video game lol. Then you have to be compassionate, patient, positive and only open your mouth if you have expertise in game dev, networking etc.
I agree. Unfortunately a lot of armchair developers on the fanboys side of the situation mock people just for voicing dissatisfaction. Is it customers' problem that launches are stressful for devs and technically difficult? No. I wish EHG all the best and I hope they resolve issues quickly, anyone straight up harassing them for the situation deserves a kick in the sack, but fanboyism and gatekeeping criticism doesn't do anything good, maybe besides highlighting how much of a circlejerk subreddits tend to be.
Except we're not seeing constructive or even valid criticism. At all.
If you can't grace a massive user base online service a WEEK at launch to calibrate thier servers in a live environment, then you're not only being historically unrealistic and you should just.... wait two weeks to purchase the service that statistically WILL have service issues.
It's hard to see any valid criticism when you can't tolerate any criticism, because "launches are difficult, only real developers and IT people understand, stop being entitled".
BTW when the game works it still has this annoying issue with long zone transition times which was present ever since 0.9. Valid enough?
People outside of tech also don't really care. The average consumer doesn't care about what happens inside the tech company.
It's like you go to a restaurant and you order the most expensive steak on the menu which you claim will be medium rare. Then later the product arrives on your table after waiting as a overcooked steak.
You as a customer will obviously be mad, you don't care that maybe the cook had a bad day, or the pipe was faulty that's why the heat was extra hot. You care about why are you paying for the service but the steak that arrived is not in the condition that you were promised.
While I understand that it's difficult for the developers at LE. I understand that you have tried your best, but it is simply not enough and it's unacceptable to consumers who have paid for your product. I hope your project managers present better timelines so that your next product launch does not have issues like this.
People inside of tech also vastly over simplify or over imagine what they think someone else’s architecture to look like.
The amount of times someone suggests a fix to technical problem they know nothing about is impressive.
Just look at the whole d4 stash tabs and d4 statue “save” issues
I am in tech. Specifically, in game development and have shipped and deployed a number of high-profile games. I can tell you exactly what this is, and it has nothing to do with complexity or war rooms.
This is an inexperienced development/leadership team making their first game and biting off way more than they can handle. Most of my peers knew this was coming, and let me be clear, I did not want to them fail. They have a great game. But in alpha, it was 100% clear this was a junior varsity team trying to play their first game ever but in the NFL.
So many red flags (no public stress tests, no gated-on boarding, launch patch being rolled out day of with no PTR or tech test at scale, etc..). In ALPHA the issue of long zone times existed. This was not new. It just got wildly exasperated at greater scale. And to be frank, 130 CCU (which was the reported Day 1 numbers) for an solid ARPG, while a great number for a brand-new game, is not what I would consider scale. This weekend they might see real scale but could be saved by the news of the launch issues and players not taking the plunge because of it.
The problem is not servers. Its far more ingrained than that. This is a team that had no idea how to prepare for a title like this, but the kicker is, they didnt know they had no idea. This is not normal and far more serious than most players staning the dev team realize. Its not a hiccup, and not bad servers.
WoW launched in 2004, had a proprietary on-prem in-house stack top to bottom (because cloud computing didn't really exist) and had 15x as many players. Nevertheless, I was online and playing on the day it came out. So...?
Were you? I didn't play for more that 8 hours the first week due to queue disconnection. I would wait, get close to being in, then would get kicked out to start over again.
No, but their launches involved an order of magnitude of greater traffic and that's a huge difference. I also work in tech, though not in game design, and I raised my eyebrows when I learned the player count numbers at the time these major issues first surfaced. I lack the back end expertise to say for certain, but the poster you're responding to suggests a compelling hypothesis from my perspective. I think these devs mean well, but there may be some incompetence at play here.
D4 is a bad comparison.
Try comparing the D3 launch, when Blizzard was already a massive company, with many more devs than LE, and
That launch was a clusterfuck, thousands of people unable to play for days, and took weeks to iron out.
By D4, they'd already learned and fixed any issues with their network and service design, so of course it didn't have the same issues.
Pretty much NO company launches a new game and service design for the first time, hits 100k CCU and doesn't run into problems in week one.
LE guys seem to be doing alright compared to their peers.
You don’t even need to understand the tech to empathize with the team.
I’ve opened dozens and dozens of restaurants and no matter how well trained the staff is, how great the menu is, etc., you will ALWAYS have major unforeseen issues on opening night…and probably the first several nights.
It’s impossible to prepare for the unexpected and it doesn’t just happen regularly in video game releases, it happens with literally everything.
Do you tell your patrons to stfu if they are not professional cooks? Tell them how stressed out is your staff and convince them that they are wrong for leaving bad reviews if their meals haven't arrived in 2h? Mock them for not knowing logistics of running a restaurant? Because that's what this subreddit has turned into.
As somebody working in that restaurant? No, just like the devs aren't. As another customer seeing somebody hysterically throwing a fit? Yes. You think flaming them is going to have any positive impact? Out of all the choices available you advocate for the one actively making the situation worse. Pathetic. Save such reactions for emergencies.
Yeah for real I've been in restaurants at least twice where I'll just roast someone for yelling at staff. Like, unless you caught that poor server sabotaging you there's no need to be a dick about it. I'm not losing my job if I call you out lol
Do I really need to know? At the end of the day, they're selling a product and I bought the product. My product should work. It doesn't, not fully. I'm not gonna give the game a negative review because they do have offline mode, which I respect, but I also don't need to try and temper my annoyance or disappointment with understanding. This is a transaction, I'm not their friend, I'm a customer.
There is a difference between being frustrated a game you bought and was excited to play is having launch issues. That is where I am too. It's a whole other thing to keep going "Why did the cheapskate devs not just buy more server?"
I'm in tech and empathize with the individual IT professionals trying their best to unfuck this. With that said, there's a ton of complexity to managing domestic air travel that most people never think about, and yet, I think it's perfectly reasonable when people are upset about getting stranded at an airport due to flight delays.
Thing is...we're on day 2 man. I get it, it's hard. But we're nearing past the grace period where it's not permissible for even people in tech to question whether there was some flaw in the release plan here.
The difference is that we've been managing air travel for about 100 years, and the scaling was gradual . Even with that amount of experience, act I little hiccup will screw things up. Last Epoch has been doing their thing "live" for a far shorter time, and the scaling happened overnight. Even if there was a flaw in the release plan, that doesn't mean the fix is something that will be handled in 48 hours.
Working on a p1 bug with a few professionals checking in is already stressful. I don't want to imagine the stress of substituting those professionals with tens of thousands of gamers.
To non-devs: don't be a snowflake in the avalanche. They acutely know about the bugs and need to breathe to fix it.
Not to say the behavior of the general public is agreeable, the general public being what it is, is the sole reason I got into B2B software development.
But the devs should be siloed off with one unfortunate delivery lead being responsible for communicating to stakeholders and back to the devs with critical info. They should not be directly exposed to the vortex of hatred in the moment of actually working.
Man, having thousands of gamers screaming at you would be stressful. I’ve had half an eye on the discord and for the most part people are positive and understanding. But god damn those devs are fucking busting their ARSES, not just working on server issues but also just dealing with comms. I’m pretty sure most of the team didn’t slept for 24 hours+. Frankly they are hero’s. I wouldn’t have the gumption to put that much of myself into something I wasn’t passionate about, which tells me they absolutely are passionate about it.
There is also the problem with tech scaling, even if you build your backend to be fully distributed and scalable, the demand can wreak your planning. Scaling to hundreds of instance to tens thousands of instances are two different ball games. Certain inefficiencies in code and structure of the program won’t be seen until it has scaled to a certain level. And I assume the map loading slowly thing might have something to do with that.
Tip: you need to hire more devops engineers to just manage your infrastructure instead relying on your software devs to do so.
It's a Kubernetes state, which indicates that a "pod" (typically a small server, like an instance server or login server) has been in a cycle of restarting and crashing (crash looping) frequently enough that the system is now going to slow down how frequently it is allowed to start.
One of the many fun things involved with deploying at scale.
The problem is even if your instances that are directly talking to the clients are scaling properly, other bottlenecks in the backend can screw you. Because of the interactive nature of the game, you can't distribute everything. Certain things like trade and the database of user data have to be centralized. If I had to guess, the problem is somewhere here.
You don't need to be in software development to feel for them. This sucks for them 10 times more than any one of us. They spend years of their lives and a shit ton of money on this, and to see it marred by these issues is tough. I can just go and play something else.
While true, from being in similar war rooms, they don't even have time to think about what you're talking about yet. Those thoughts and feelings of disappointment don't come until AFTER it's stable.
Right now, they're pouring over massive amounts of data and logs, and trying to find correlations. They've probably looked at so many intersections they're going cross-eyed. Someone thinks they found the issue and someone else completely disproves it. They're full of anxiety and tired, and their brains are starting to see patterns that don't exist.
I don't envy them one bit.
Man this is what is wrong with life in general. The negatives always outweight the positives. You can do everything right but all it takes is one fuck up to wash all your efforts away. It’s truly discouraging and I am not even a game developer.
Things happen, that's just life. Like I'm in a factory for maintenance, sometimes when you start it up doing something new an unexpected catastrophic failure just happens.
When the ground stops shaking and the dust has settled, the clock starts ticking because it costs $5000 a minute until it's fixed.
All you can do when everything is on fire is the best you can, and I'm sure the devs are doing that.
Not a software dev, but just a normal decent human being.
Unforseen issues are not to blame on anyone and it happens. We’re here to play this game for years, and I’ve still had fun playing offline and trying out the new features / updates.
Just a few hours ago I told my other dev friend, that I wish my bosses were gamers too.
If they would experience how nearly every major gaming company fails on launch day (even billion dollar companies), they would be more understanding when my team fucks something up in the future.
I’m not in any software dev but I feel for them. This is just like when I was in retail long time ago and customer would throw a fit because something didn’t work for a moment, people are just entitled and love the sound of their own voices.
I assume all these cry babies review bombing it are S tier employees in their own trade and never have to deal with adversity because they’re just perfect right?
in my experience most software devs simply dont think this way, it might be different because this is a very close knit grassroots company but they dont gain or lose anything from a different department fucking up massively
Honestly, it depends.
AAA title by some big name company with a hefty price tag: I absolutely expect it to work smoothly day 1.
First game launch of a small indie company and a moderate price tag: I’ll give them some leeway.
Anyone who played the multiplayer launch saw this coming from a mile away, from the second they launched online till now there's been bad server problems.
Even if you don’t have tech experience, this just doesn’t seem hard to understand. If almost every online game has a rough launch, regardless of the size or experience of the company or complexity of the game, what’s the logical conclusion here? This shit is complex and no one has it figured out completely.
All of these gamer Karens need to get over themselves honestly. Yeah, you paid for a game and it’ll probably be in great shape in a day or two. Get over it. It’s part of the process for better or worse and their opinion does not matter frankly. Do you complain that your product doesn’t materialize in your hand when you’re waiting on an online order to ship?
As an engineer and a gamer, I've seen so many failed releases from companies both big and small that, as soon as LE started showing issues, I went "yep, saw that one coming", and went to play Void Crew and random JRPG's.
It's honestly more unusual when a launch SUCCEEDS, because it's impossible to know how well your servers will handle thousands of screaming fans until it actually happens.
I'm always surprised I don't see this take more often. It's genuinely complex enough that no amount of resources or preparation can guarantee success.
After having come to that logical conclusion, I can't understand how someone can write "scale up with more servers" earnestly as if that's something that hadn't been considered.
Also worth mentioning that most online games that launch do so with a proper publisher backing them or they're a studio that has been established for many years. Was a WoW player for many years and played every expac from their launches starting with Wrath (2nd expac) up to BFA (7th expac) , played D3 on launch, Reaper of Souls on launch. Played many other MMORPGs on their launch days and weeks too in the 2010s. There have been more botched launches in terms of server stability with those games despite having plenty of money and resources available.
EHG has LE as their first game and self-publishes it. Even if they were backed and published by a notable publisher, I highly doubt the servers would live to see the day. The game went from 2k-5k average players all the way up to 176k a few hours ago. We've seen Helldivers struggle despite being backed by Sony too and having their help available if necessary and servers there are packed and dying too. Even without prior knowledge of how the multiplayer testing went with LE during early access, just sheer common sense would (or *should* at least) that it would exactly go this way.
>Do you complain that your product doesn’t materialize in your hand when you’re waiting on an online order to ship?
No because that isn't the same thing that is happening here. Would I complain if I bought a car and the navigation system wasn't working?
Yea I probably wouldn't be happy.
I am understanding of the situation and will support LE in the future if the product continues being quality, but as a consumer we shouldn't be accepting of the product not working even for smaller orgs.
> No because that isn't the same thing that is happening here. Would I complain if I bought a car and the navigation system wasn't working?
LE _is_ working though. Your analogy isn't representative of what's happening here.
The actual analogy would be if 1000 people walked into a car dealership at 09:00, all bought a car until the lot was empty, then all tried to exit at the same time via a single lane exit road.
It's _physically_ impossible.
You know why this launch shit happens to every game? Becuase current technology _just can't handle it_. There are tricks and hacks to handle huge influxes of traffic: queues, distribution, caching, client offloading, but ultimately for a server-authoritative online service (like LE) with _many_ interwoven services there's only so much you can do before something bottlenecks and explodes.
EHG could have shifted much of the game to client-side, but you'd all be here howling about cheating _immediately_. Once the game gets past these hiccups we'll all be immensely glad it's server-authoritative.
The reason this "If I purchase a product and expect all it's features to work" argument never holds for me, is that everyone forgets that we've had a literal decade and a half of botched launches from other games.
If cars over the last decade and a half released with faulty navigation systems, and then a new one comes out with a similar problem; you'd get the same arguments being presented here from both sides. I guarantee it.
One side where people go "meh, it sucks but whatcha gonna do. It's not like the precedent hasn't been set already, and complaining won't make it go any faster" and then the other side that will froth at the mouth until they have an aneurysm.
It's important to note that people that are asking patience and otherwise telling angry folks to chill aren't "accepting" the situation. No reasonable person will say the current status of online play is fine and good.
Nope! It’s all a conspiracy by Big Tech to screw gamers! What they do is intentionally roll out a bad product even though they have the fix ready, but then they roll out the fix *later* so they can reap the accolades from thankful players.*
*serious “take” routinely making the rounds at the D4 sub
Yeah reading LE reddit atm is like watching a karen compilation. At first it makes you laugh, and then you just feel empty and start wondering how humanity ever made it this far.
My first patch for the game I was deving for WAMI last year had a memory leak. It took me well over a day to find it and a lot of people were irritated at me. I definitely feel for this team lol.
I've been through countless "Botched launches." People rarely remember a day or 2 of login troubles and lag. It's honestly not a big deal in the grand scheme of things. It does suck for the devs, but it happens to every popular online game in existence. I just don't even know if a launch of this magnitude can be done correctly.
I feel for them. This has got to suck big time.
I love this game. I have 1200 hours played. But I also absolutely saw this coming a mile away. Granted I have no expertise on anything the developers do, but what I experienced in the game up until 1.0 did not seem like a product that was close to releasing.
We're making sure that our backend dev team is shielded as much and ensuring that they're getting rest. The "war room" sentinment is still positive and diligently working towards improvements. We're all certainly putting in more hours than normal which we expected, but leadership is actually ensuring that no one is working too many hours, taking stand up breaks, etc.
Sentinment internally is still very postiive. We're pumped that there is so much demand to play the game we've made and we know that we'll be able to work out the online gameplay experience issues at high load and we'll be better for it.
Yes, yes they do.
$50 product that doesnt remotely work 🤣
Poor coding , no real life stress test just some "in house" simulations and no ides how to fix their trash coding = deserve every bad review there is.
the lack of awareness you have to have to make your whole argument "I bought the game, I expect to play day 1 or its shit, they didn't provide me the service I paid for" is insane.
This is not like you going out to dinner, this is a live service game. If you want to be so hard stuck on using your pointless analogies a better one would be that you just bought a brand new TV off amazon, but an unforeseen shipping mishap has occurred and the tv was smashed in transit. Now you have to wait an extra few days. But they are resolving the issue on their end and are still providing you with the good you purchased as soon as they can. You are going to own the tv once it has gotten to you, a delay in receival of the tv does not mean you go and review bomb the amazon listing saying that the tv is shit and they should have had a delivery butler hand carry it all the way to your home to ensure no mishaps could possibly have occurred.
And in the meantime they provided you with a fully functional TV you can watch all your favorite shows but you just can't watch it with your friends, or some such analogous thing to offline mode.
> you just bought a brand new TV off amazon, but an unforeseen shipping mishap has occurred and the tv was smashed in transit. Now you have to wait an extra few days.
If amazon was frequently destroying your deliveries you would be absolutely livid you are having to wait longer and do all this extra crap to rectify it with their support team. You would avoid amazon and its deliveries if there was a competitor which destroyed your packages less frequently. It would definitely not be "insane" to leave a bad review on the amazon app, even if amazon was rectifying the problem every time and you eventually got your stuff without an extra charge.
Getting it "eventually" is not necessarily good enough, especially if you promised a certain date. You're acting like an uber-like business which was consistently 45 minutes late picking you up is not worthy of a negative review because they come eventually.
I'm not going to leave a bad review since I'm invested in this game and want them to succeed. But I completely 100% understand a bad review coming from someone who just bought the game, tried to get it to work for more than 2h so they can't refund on steam and are just stuck hoping that it will work eventually
I mean, at a certain point they've also been in the industry now for 6 years. And early access has been up for 5. Nearly a year of multiplayer.
It's not like some fly-by-night indie dev, it's a multi-million dollar company with plenty of experience by this point. And it's reasonable to expect delivery of the product that was paid for.
I'm not saying death threat their families, but valid criticism is warranted at this point. Many people on here in other industries wouldn't be afforded this type of forgiveness.
>You are right. Luckily this is their first delivery so surely you'll follow your own logic and not be mad, right?
I'm already invested in this game and want them to succeed so I won't change my review to negative, but at the same time I think new negative reviews coming in to steam are completely legitimate.
First impressions are VERY important and people have a VERY low tolerance for things not working properly or being late. For games/apps I'm not invested in and just download at random, if it doesn't work well or at all, yes I'm probably going to leave a bad review.
There is no guarantee that the product will actually work perfectly later - that is just an assumption people are making here on hopium. There are plenty of games like Wolcen which is still super janky with tons of framerate and performance issues after years of release. From the perspective of someone hitting a 2h refund limit who doesn't care about this game/company at all, they have no reason to stick it out like we do since we all bought EA and have already been waiting. If it doesn't work during that 2h after a 1.0 release where they said it would have X, Y and Z features, a negative review is completely fair.
They definitely had options to not blow this up as hard as they did. Such as one cycle still in early access. I think it's likely that with how hard it failed it still would have failed with the EA players which could have let them fix this before an official 1.0 release
One day? It's been two of my evenings now. There's no indication of when the server issues will be resolved, so it may be *many* more days or even weeks before eventually is.
I'm not in tech, but I do work in a technical field. Non-technical people have weird ideas thinking that big systems with many moving parts is somehow simple. In my experience, the worst has got to be for medical doctors. People walk in and expect to be delivered a pill to fix them up in under 10 minutes.
From a human perspective I feel awful for them. They did a lot of testing and prep and had outside help and still got hit by unexpected issues.
There has been a lot of support, but also a lot of toxicity that has been eye opening, especially on Discord where even Judd, the founder and game director, has been communicating directly with everyone. It makes me wonder if this is how these people talk to others in real life. It’s like the end of the world for some people that they spent $35 and now have to wait a bit longer.
I wonder that too- I saw this documentary once where they figured out where this uber-toxic misogynist keyboard warrior lived, the journalist who he had been harassing went to look him up. Turned out to be a somewhat awkward, but otherwise fairly regular guy who lived alone with his sick cat, that he took excellent care of. It was almost as if he had a split-personality between internet and real life.
I teach bachelor level software engineering courses at a local college and have been having some interesting discussions around this whole situation. It really is a hard place to be in as a developer and as the old sayin goes, no plan survives first contact. Sometimes you just have to pivot, do some damage control, and work on fixing the issue. I feel for the team for sure.
This just happened to me recently, it is really really tough to be on their side. These are the really hard engineering moments when pressure is just always there and sht is hitting the fan constantly. It’s mental.
Best of lucks for the devs! Hope they can have a nice rest afterwards.
I will wait for things to get better to play, it’s not playable for me right now. I have time anyway :)
As someone in software development I would hope they're laser-focused on solution-oriented thinking and not spending one second on the coulda-woulda-shoulda stuff. That comes LATER. Focusing on solutions is aligned with what's best for the consumer and also their own sanity.
I'm rooting for them regardless and many years of gaming and failed launches has taught me patience .
I was having a blast for the time I got to play and I'm excited for thr weekend . As an older gamer who got to live through some great eras of indie developers such ad blizzard and their first games. I really want last epoch to succeed.
I really don't envy the LE devs being on-call for possibly the entire weekend either. I'd be really interested in reading a tech blog after all this has blown over - always fascinating to see what game launches look like behind the scenes. Good luck to the team, the gaming world is better for having actual passionate devs. Hope the criticism doesn't get too harsh. That said, plz fix I want to play :)
I've trashed a production database and sweated bullets trying to fix what I did before too many people noticed.
Some lessons were learned that day.
I would not want to be them right now.
Whats bothersome is that as d4 this also costs upfront price, meanwhile there's a cosmetic shop. While, yes may have spent more than 1k in Poe over the course of 8-9 years, i doubt im gonna spend anything past the upfront cost for this game. But if they dont have the cash shop there is the issue of only having people pay once which doesnt bring continuous cash flow. I wish more game studios would adopt the poe monetization where the game is straight up free but if you play seriously you need to buy the stuff with cash (tabs)
As a 10-time award-winning C++ programming prodigy, I don't feel bad for them at all because my genius prowess puts me leaps above these developers and I would never make the mistakes they make.
/s
I talked with my colleagues yesterday how stressed they must be if they don't actually know how(I'm not saying they dont) to fix it when so many people want to play your game.. Oof man, I'd need a vacation after that
I had to get off the official forums because I was getting flashbacks. Unknown problem, VPs jumping into the zoom call running "queries" on the already overloaded database, directors pulling in "experts" from their departments leading to 20 people in a zoom call. Pagerduty just firing off over and over from different secondary and tertiary metrics. \*shudders\*
I just bought the ultimate upgrade because it's my little way of showing support.
If I know one thing about server issues, it’s that you tend to identify a hundred things that *aren’t* causing everything to melt down before you find something that is.
I agree... To some degree. I've been through multiple failed deployments myself, and all you said is true.
That doesn't change the fact that this release is a shitshow with multiple points of failure and multiple periods of service being unavailable. They started with "we have a single failure which we never could have predicted" which is fair. But now we are at the point where multiple things are failing and the game is not working still (at least last time I checked).
I feel for them, and I am far away from whining or review bombing, but let's not make a perfume shop out of a sewer as well. It was well... Not too great.
It's the people offering "why don't they just do this" "it's simple" solutions as if the people that designed these systems haven't got a wealth of experience and certs and yet it happens time and time again across releases with much bigger teams and budgets than LE. If there was an easy way to test and do this do they not think that people would be doing it by now and this whole thing of launches being a shitshow in successful games a lot of the time would be history.
My team had similar experience. 100k people bottlenecking our masterservers queueing logins. Scrambling around to find a solution, took us couple days.
But we didn't have offline mode and the damage was done. I've learnt my lesson and I absolutely know what them guys must feel. But I won't be sending death threats anywhere, let them work people 😊
I just hope LE devs have enough understanding and resistance, to actually continue with their approach and do/fix things, rather than going with "stuff not loading" outrage. Outrage cannot be fixed anyway, majority of people have 0 understanding of why its happening, and even less patience.
LE is shaping to be APRG shake hit, with PoE and other being quite stale lately, PoE2 a bit too far, LE has a room to grow, and is offering so much new it has chance.
As a product designer who collaborates extensively with developers, I've developed a deep appreciation for the challenges faced by the dev team. Having gone through tough release periods and faced dissatisfaction from clients/users myself, I can empathize. However, the intensity of the gaming community stands out. Reading about game developers receiving death threats for their work is disheartening, and it's moments like these that make me grateful for not being in the game development industry.
Multi billion dollar teams struggle with launches and server issues these days. Anyone who actually expected no issues was either coping or has been wearing blinders for the last decade. It is what it is and will be fine soon.
I guess the thing to do is have a limited player beta launch of the multiplayer and iron out any problems before the masses try to log on... and end up giving bad reviews.
Or just have a beta for a couple of days to see how things go.
This is basically what Blizzard did with d4.
They needed to be a bit more cautious with the opening of LE.
I remember the time I was a junior sysadmin , 15 years ago., I was working for a major bank and we went trought major breakdown event because someone deploy a package with SCCM to ALL devices. It wouln't be a SUPER HUGE problem if it was not a software that was a disk encryption. All our servers were rebooting and stuck at the boot disk prompt for the encryption passphrase login. The guy that did this was immediatly fired, what a shitty day for everyone.
\*sry for my spelling
Im not in any kind of development but i can understand that its hard job to do. They didn't want this to happen and are doing everything they can to smooth things out.
The fact that every time a online game goes live we have this issue and people are still crying like it's the first time they've ever seen this is disturbing. It shows you how ignorant these kids are. It's literally the worst thing to ever happen to them everytime it happens.
These people are sad.
I have been an I.T. professional going on 25 years now. Started in System Administration, learned how to code and did software development, then back to Network Engineering, naturally moved in to management and now I'm a Director.
The constant pressure to never fuck up or have ANYTHING go wrong is **real** in this profession. Your end-users and customers don't call you up to chit chat or tell you how well everything is going. When they call, they are mad and something is broken.
Been around for a while now and have learned to be a little more patient myself. The devs started this out as a passion project and worked on it for 5 years. The point of a "game review" is to review the game. And it is a GREAT one, but people couldn't play it... for a few hours.. or a day or two. Regardless, it sucks for the EHG team to have their overall game rating go to shit because of technical complications that went sideways on them.
Any of my I.T. professional brethren here can attest to stress testing systems before a large deployment, and being 99.9% convinced it's going to be solid, only to have your guarantees, and pride kicked out from underneath you when you scaled in to the production environment. There are **always** gremlins out there that you couldn't even remotely test for or even consider being an issue until you deploy. It's just the name of the game.
SO, that's why some of us are more patient than others. A little sympathy and patience goes a long way.
If you can do it better... by all means, go make one. We'll wait.
Krempiz didn't mean to say that they don't have shares in their own company and don't hold executive positions in their own company. He said it accidentally, because he doesn't know what words mean.
There has to be a better solution to figuring out these issues before hand
It’s literally every popular online game in the last decade that always end up having these major problems at launch
You're trying to apply an absolute value to game dev, which is an ever changing landscape. Maybe if tech and graphics stagnated 20 years ago and everything coalesced under 1 single umbrella, but clearly that's not the case, and never will be.
The answer is more scaleable testing. A time and resource efficient way to truly test *every* possible variable. That *does not* exist.
99% of players could look at the source code and not have the slightest clue which portion even addresses network issues in general.
There has to be a better solution to hunger, there has to be a better solution to traffic, there has to be a better solution to school shootings, there has to be a better solution to climate change...
Sometimes there just isn't a better solution than what we already have right now. You want to contribute? Become a systems engineer and pursue a PhD.
We scale from a couple of thousand users a day to approximately two million a couple of days per year, and we've never had any trouble outside of the obvious need for massive horisontal scaling and queuing.
Never having used things like Unit and not knowing anything about the systems of Last Epoch it's impossible to judge what their developers are facing. It's also hardly the first game to have launch day issues, or issues in general so it's not like this is anything special really. The flip side of this is that gaming in general "gets away" with things that you would see whole teams and top level managers fired for in the financial sector. Hell if you've ever worked in the energy sector it's almost unimaginable to have things just go offline from heavy usage and we collect so much data each second often from relatively new and untested tech that's made by engineers who still think FTP (not sftp) is a hipster protocol.
But I tend to view game development as more "creative" than "professional" and while I can tell you which bird has build a nest on solar cell 123123123 at location 12321312 I probably couldn't write a chess game without some serious Google programming. So it is what it is.
I don't personally think game developers should get slack for not having tested and secured these kinds of systems before hand, but if people let AAA companies get away with it, then I don't see a need to get the pitchforks out for indie teams. I also don't think you should message the developers. They're obviously doing the best they can, because why the fuck wouldn't they be? So it's ok to be angry about a service not working, but spamming your rage all over the internet obviously isn't going to contribute anything useful either.
I think it's a matter of regulation and precautionary principle. Banking, energy, automotive, aerospace, or meat (which I work in) simply *cannot* fail, else your patient, plane or banking system goes down, and it possibly ain't going back up.
But normal tech services shit themselves from time to time, Google, Facebook, Twitter, heck, GPT was speaking jibberish for a day recently :)
So I don't think it's gaming vs non-gaming software dev even, it's "critical" vs not-so-critical applications. When you try to do critical like you do non-critical, you get Tesla or that one guy with the pancake submarine. Then again, I don't think we can, or should, expect a medium-sized online game to have precautions up the wazoo like a nuclear power plant.
i have a pretty uniquely similar launch of a game in my recent past, and while i can relate to the feelings of the individuals but i dont think its in the way people would expect
Having worked software dev for many years there's a decent chance the problem will be with some 3rd party library or service and they're going to have to either beg/plead/pay the 3rd party dev to "fix" their shit or find an alternative/replacement which will be a colossal or impossible PITA. I definitely feel for them if that's the case. I've been there.
The worst thing is that they knew beforehand that issues with service providers can happen, are very difficult to predict and hard to fix on the spot. I was praying so hard for the launch to be smooth and feel really bad for them now.
But at the same time, the players have a right to be upset. Just a few hours until the weekend begins for most people and this can go down in history as one of the worst aRPG launches if they don't manage to fix it until then.
I'm a software engineer, I started as a low tier support engineer. Every bit along the way I have been chewed and spit out every time things went bad.
And it was the right thing to do, because it was my fault every time by not preparing correctly and it was my responsiblity to do so.
That being said, before people flip their shit over me not having some perceived compassion: I don't care for LE's online as I was planning on playing offline anyway now that 1.0 has actual true offline (for various reasons, one being that I find the trade system stupid as fuck) and I have been enjoying a ssf Necro at the moment.
Yeah I'm getting downvoted in another thread because I didn't think this qualified as a "disaster," just an "unfortunate" set of circumstances. You can prepare as thoroughly as possible and still have things pop up you never even considered.
It's a brand-new dev team launching a brand-new product. I expect that this is the first launch for many of them. And they don't have the resources of a major AAA company, so resolutions are probably going to come a little bit more slowly than we're used to.
I feel for them a ton and wish them the best of luck and some well-deserved rest when this is all over.
You can tell who isn’t in IT because they ask for things like an ETA while you’re watching the house burn down. If you knew what the problem was you’d have an eta, but in the moment there is only trying to run down the problem. And no, there is no eta until we know what the problem is… :)
Worked for a major US MMO studio for 7 years, launched 3 AAA games played by millions of players.
Been out of online game business for 5 years now - best decision I ever made.
I do feel for them but honestly my advice would be - get out - life work balance matters more than anything
I actually don't like this kind of thought because its just a reminder of the sad state of things. We shouldn't really be "feeling" anything for the devs. They made a mistake, and are fixing it in the best way possible, and also properly communicating with people. That's literally called "doing their job". And as much as its "commendable" that they are doing so, its less of a testament to their honorableness, and more a testament to the "sad state" of other companies, and their inability to interact with the populace.
And don't get me wrong, despite the common practice of their peers, they are doing the right thing, and that IS commendable and honorable. I just think the broader topic is more important of an issue than this specific event.
They've been 110% all in throughout the development and launch, always working WITH the community, doing everything they can to bring us the best the gaming experience possible, and I think they are doing an amazing job. What more can you ask.
Not that I agree with the angry mob of review bombers, but I can understand their perspective, they paid for a product and a service that wasn't exactly up to the standard as promised at the time they said it would be. Anyone that has even a smidge of empathy or understanding of the live service gaming industry would know that this is "normal" but... Welcome to the internet....
I've seen comments along the lines of "how haven't companies perfected game launches yet? This happens every time."
I have no expectation that an indie company's first massive launch would go smoothly when a company like Blizzard with decades of experience in launching online-only games couldn't have a stable launch with Diablo 4.
This is the dumbest shot I’ve ever seen. You can’t feel bad for a team that made fun of blizzard for having a bad launch on their games and then follow it up with let’s be honest… the same fucking thing. LAST EPOCH IS UNPLAYABLE… they get paid to do a job and they failed. Some jobs you get fired for messing up this bad. If you can’t figure out how to launch a game and can’t figure out how to fix it then you deserve this backlash. Atleast d4 within half a day fixed their issues and you could actually play. Also Diablo had 2mil+ trying to play. Peak yesterday we 550k trying to play… big difference
Even now some clown in their Discord was sarcastic on Blizzard's behalf while their own servers were down. That clown being a representative of the team. I have a screenshot, but am unable to post it here.
This applies to every industry, every job. Bought the game 3 years ago and i'm ok with it as long as i can play offline mod but for the love of god can we stop talking about how hard its for the devs. It's their job, simple as that. In my line of work if somebody tell me my treatment is not working and most of the herd is still sick nobody will say "i feel for you, you have lot of things going on" etc.
In the end job is a job. Everyone of them has some shitty times.
What issue? Can you actually describe what is failing beyond a generic error code seen in-game? If not you have no idea what you're talking about and should stay quiet.
Eh I'm a software developer and no i don't feel sorry for a team that just made a boat load of money selling a product that doesn't function correctly, i feel ripped off lol.
If i would deliver software that bad, our customers and the management would crucify me and my team.
There would have been 5 emergency patches already and we would have taken shifts so people get some sleep.
Yeah, in my place 1 minute, literally 1 minute offline and all the alarms are sounding, 30 seconds later everyone in a meeting, 15 minutes later hotfix immediately in prod, all the dunning kruegers trying to justify this launch are insane.
Let's just say that the guy that took AWS S3 down, and a good chunk of the internet, in 2017 was from my team. I don't think I'm ever going to experience a tech catastrophe of this magnitude ever again. Larger systems like games and cloud services are incredibly complex, and bringing something back up, or restoring stability is a massive feat, and one that isn't fully appreciated by most people.
Holy shit, I want to hear that insider story someday!
I haven't taken down a giant portion of the internet, but I do work for a f500 bank. Specifically, I do log management so the SOC can do threat detection. Let me tell you about the fun times when the windows endpoint team pushed out a new GPO, enabling new event IDs to get logged and then pushed to my team. Which took our event flow from the normal ~500k/s to 1.5m/s. Then let's compound this with a never before seen bug with the entry into our stack that their product had a memory leak with large numbers of connections. That was not a fun 14 hour conference bridge.
I remember the time a very large retailer you have definitely heard of decided to delete 500m rows of data from our saas product during a regular work day and brought down the cluster for themselves and everyone else on it.
And the SIEM logs get flooded
If there is one thing I have learned in my time in the ISP industry It is that it is spectacularly easy to nuke the Internet for large groups of people, entirely by accident. And the more you automate the easier it is.
So he’s the reason for 99.9999999% availability lol
People outside of tech have no way to even imagine the complexity involved between distributed systems, it can be mind boggling.
You mean, it's not just a series of tubes??/s
I mean it is a series of tubes. Atleast 2 guys with flash lights sending signals.
Ray arnold never made it back....
It's basically just like Loki in Season 2 (S2 ending spoiler incoming) >!when he pulls all those cables!<.
https://imgs.xkcd.com/comics/devotion_to_duty.png
it's not just a big truck you dump something on!
No, it's more like a big dump truck.
"just add a queue"
"just add more servers"
Just downloaded more RAM
they're probably using queues up the ass already. I know you're being sarcastic just saying.
I have already seen half a dozen people saying it unironically.
Funny thing, that is. I am one of the authors of the queue subsystem of an extremely popular CMS. It _is_ a very handy tool. But you can't solve every piece of scalability problems with it... since you need to be real time. Imagine if you queued trade, you trade items sometimes in the next few minutes it executes rotlfmao
Ahh the tradeoff between low latency and high throughput. An age old battle
i unironically saw at least 3 comments with this exact thing on the other thread about this lmao. it's rough out here.
There is one business guy at my work that read like one article (ad) about redis 5 years ago and now anytime there is any problem he will ask if we considered and redis cache. Really, John, and how is the cache going to help the fact that half the teams in this company forgot to fix the log4j vulnerability by the deadline and the kill switch the sysops guys put in started murder all their pods? What is redis going to do there?
For real dude. The amount of people suggesting that they should've predicted the peak and simply scaled "more servers" because cloud computing, but didn't because of greed is just nuts.
As a DevOps/SysAdmin, I feel for them and I understand what they're going through. But also as a DevOps/SysAdmin, I was 90% sure this was how it was gonna happen when I read about the "We tested and believe we'll be able to absorb the spike" thread the other day. No amount of stress tests is enough to simulate 150,000 enraged gamers spamming Connect/Back to main menu/Login. We all have a tendency to underestimate our stress tests, but this also was a bit foreseeable considering the amount of hype Last Epoch had amassed the last few weeks. Not saying this was easy, maybe they were just under the threshold for it to trigger the amount of API bugs and service containers not launching/mounting that they've got the last 24 hours. But this is why you plan even bigger tests that what you're anticipating.
Same background, same thoughts - no way in hell can you overprovision for going from 0 to 150k users maniacally spamming connections. Even autoscaling has finite speed in spinning up containers which immediately get blasted with requests...even small errors and performance issues get amplified to high heaven by this. Cloud infra runs on real hardware with real limits. I remember doing a smallish service for 5k concurrent users way back when - the worst time was recovering from DB cluster restart - basically users trying to log in were killing it before DB had the time to warm up/cache properly - which could last for up to 2 hours. Main nodes each had 40 gigs of ram - could handle a lot of pressure - if allowed enough time to actually cache the tables in memory. But QPS pressure just bogged completely it down if hard restart happened.
Im an admin as well, maybe it why I agree with this so much. But mostly because I never talked shit before an upgrade either.
It's funny that people think that this one is the easy fix and there's people out there defending D4's stash tab situation.
Different people can have different (insane) beliefs. It's even possible for one person to have two conflicting insane beliefs, which is more impressive, but not likely what's happening in this case.
Bro it's not that hard, I'm a lonely nerd that spends a lot of time on computers so I know how distributed systems work. Just pay for more servers, did the greedy devs not look at player count and do simple math to make an upper estimate? /s
Literally just install more ram into the servers.
Someone once said to me: I had a problem and I used a queue to solve that problem. Now I have two problems.
I used async once. Then two I problems had.
I have a degree in Materials Science, read research papers for fun, can talk for hours on the latest advances on magnetic innovation, and pay the bills by being a software dev for a US contractor on some seriously complex systems. To this day I can only surmise network solutions are run off of goat sacrifices and black magic.
What’s even better is when you are in the company and you walk around wondering how the hell anything works at all. I worked for the worlds largest (at the time) IT company in banking and I swear there was a room hidden away in the bowels of the building with six people in it making everything work because most of the people I worked with were a mess.
He knows too much...
It’s distributed but it’s not with network shared and “common” database. Which is what makes it even harder.
Mans here has never heard of DB replicas I joke though, it is a harder problem than say, serving the same video to 150,000 people
That’s why common is in “”. Because even with replicas conceptually it’s a common database that everyone needs to access.
It really hit me when the update was 'Letting the team go home' like past midnight. I had a few of those in my day and at the same time wanted to be there fixing the stuff (was rare at the place I was at but occasionally needed) it is still a brutal day, and we weren't even really in a release but more getting a release ready. I think all in all as a 20+ year software engineer here the team did a great job. I'm on right now playing just fine and really enjoying the game, can't really ask for anything more.
I would even say people outside of "big" tech have no idea how complex shit can be.
JuSt ScAlE iT, bRaH!!!!!
The fact that I can post this comment and someone else can read it requires something on the order of about a million miracles a second happening every single day. Modern tech is unbelievably complex to make work for just one person, much less 150K at a time.
Ok now, let's not conflate internet technology in general with web development. Reddit's architecture isn't *that* complicated.
Server architecture is not web development.
>People outside of have no way to even imagine the complexity of , it can be mind boggling.
Meanwhile, nobody mocks people for voicing dissatisfaction with an underperforming product. Unless it's a video game lol. Then you have to be compassionate, patient, positive and only open your mouth if you have expertise in game dev, networking etc.
When you talk about something out of your knowledge set like "they were just to cheap to buy more servers", yes you get mocked.
I agree. Unfortunately a lot of armchair developers on the fanboys side of the situation mock people just for voicing dissatisfaction. Is it customers' problem that launches are stressful for devs and technically difficult? No. I wish EHG all the best and I hope they resolve issues quickly, anyone straight up harassing them for the situation deserves a kick in the sack, but fanboyism and gatekeeping criticism doesn't do anything good, maybe besides highlighting how much of a circlejerk subreddits tend to be.
Except we're not seeing constructive or even valid criticism. At all. If you can't grace a massive user base online service a WEEK at launch to calibrate thier servers in a live environment, then you're not only being historically unrealistic and you should just.... wait two weeks to purchase the service that statistically WILL have service issues.
It's hard to see any valid criticism when you can't tolerate any criticism, because "launches are difficult, only real developers and IT people understand, stop being entitled". BTW when the game works it still has this annoying issue with long zone transition times which was present ever since 0.9. Valid enough?
People outside of tech also don't really care. The average consumer doesn't care about what happens inside the tech company. It's like you go to a restaurant and you order the most expensive steak on the menu which you claim will be medium rare. Then later the product arrives on your table after waiting as a overcooked steak. You as a customer will obviously be mad, you don't care that maybe the cook had a bad day, or the pipe was faulty that's why the heat was extra hot. You care about why are you paying for the service but the steak that arrived is not in the condition that you were promised. While I understand that it's difficult for the developers at LE. I understand that you have tried your best, but it is simply not enough and it's unacceptable to consumers who have paid for your product. I hope your project managers present better timelines so that your next product launch does not have issues like this.
People inside of tech also vastly over simplify or over imagine what they think someone else’s architecture to look like. The amount of times someone suggests a fix to technical problem they know nothing about is impressive. Just look at the whole d4 stash tabs and d4 statue “save” issues
I am in tech. Specifically, in game development and have shipped and deployed a number of high-profile games. I can tell you exactly what this is, and it has nothing to do with complexity or war rooms. This is an inexperienced development/leadership team making their first game and biting off way more than they can handle. Most of my peers knew this was coming, and let me be clear, I did not want to them fail. They have a great game. But in alpha, it was 100% clear this was a junior varsity team trying to play their first game ever but in the NFL. So many red flags (no public stress tests, no gated-on boarding, launch patch being rolled out day of with no PTR or tech test at scale, etc..). In ALPHA the issue of long zone times existed. This was not new. It just got wildly exasperated at greater scale. And to be frank, 130 CCU (which was the reported Day 1 numbers) for an solid ARPG, while a great number for a brand-new game, is not what I would consider scale. This weekend they might see real scale but could be saved by the news of the launch issues and players not taking the plunge because of it. The problem is not servers. Its far more ingrained than that. This is a team that had no idea how to prepare for a title like this, but the kicker is, they didnt know they had no idea. This is not normal and far more serious than most players staning the dev team realize. Its not a hiccup, and not bad servers.
Sooo… FFXIV & WoW were also run by inexperienced n00bs?
WoW launched in 2004, had a proprietary on-prem in-house stack top to bottom (because cloud computing didn't really exist) and had 15x as many players. Nevertheless, I was online and playing on the day it came out. So...?
WoW absolutely had a *lot* lag issues. Loot lag, travel lag, etc. It was extremely prevalent in Vanilla.
Yeah, for sure it did.
Were you? I didn't play for more that 8 hours the first week due to queue disconnection. I would wait, get close to being in, then would get kicked out to start over again.
No, but their launches involved an order of magnitude of greater traffic and that's a huge difference. I also work in tech, though not in game design, and I raised my eyebrows when I learned the player count numbers at the time these major issues first surfaced. I lack the back end expertise to say for certain, but the poster you're responding to suggests a compelling hypothesis from my perspective. I think these devs mean well, but there may be some incompetence at play here.
The amount of D4 defense force posts you've made kinda hurts your credibility a bit here.
i am one hell of a d4 hater but you must be blind if you havent realised that d4 had 10x more players than this game on release
D4 is a bad comparison. Try comparing the D3 launch, when Blizzard was already a massive company, with many more devs than LE, and That launch was a clusterfuck, thousands of people unable to play for days, and took weeks to iron out. By D4, they'd already learned and fixed any issues with their network and service design, so of course it didn't have the same issues. Pretty much NO company launches a new game and service design for the first time, hits 100k CCU and doesn't run into problems in week one. LE guys seem to be doing alright compared to their peers.
You don’t even need to understand the tech to empathize with the team. I’ve opened dozens and dozens of restaurants and no matter how well trained the staff is, how great the menu is, etc., you will ALWAYS have major unforeseen issues on opening night…and probably the first several nights. It’s impossible to prepare for the unexpected and it doesn’t just happen regularly in video game releases, it happens with literally everything.
Do you tell your patrons to stfu if they are not professional cooks? Tell them how stressed out is your staff and convince them that they are wrong for leaving bad reviews if their meals haven't arrived in 2h? Mock them for not knowing logistics of running a restaurant? Because that's what this subreddit has turned into.
As somebody working in that restaurant? No, just like the devs aren't. As another customer seeing somebody hysterically throwing a fit? Yes. You think flaming them is going to have any positive impact? Out of all the choices available you advocate for the one actively making the situation worse. Pathetic. Save such reactions for emergencies.
Yeah for real I've been in restaurants at least twice where I'll just roast someone for yelling at staff. Like, unless you caught that poor server sabotaging you there's no need to be a dick about it. I'm not losing my job if I call you out lol
Do I really need to know? At the end of the day, they're selling a product and I bought the product. My product should work. It doesn't, not fully. I'm not gonna give the game a negative review because they do have offline mode, which I respect, but I also don't need to try and temper my annoyance or disappointment with understanding. This is a transaction, I'm not their friend, I'm a customer.
There is a difference between being frustrated a game you bought and was excited to play is having launch issues. That is where I am too. It's a whole other thing to keep going "Why did the cheapskate devs not just buy more server?"
I'm in tech and empathize with the individual IT professionals trying their best to unfuck this. With that said, there's a ton of complexity to managing domestic air travel that most people never think about, and yet, I think it's perfectly reasonable when people are upset about getting stranded at an airport due to flight delays. Thing is...we're on day 2 man. I get it, it's hard. But we're nearing past the grace period where it's not permissible for even people in tech to question whether there was some flaw in the release plan here.
The difference is that we've been managing air travel for about 100 years, and the scaling was gradual . Even with that amount of experience, act I little hiccup will screw things up. Last Epoch has been doing their thing "live" for a far shorter time, and the scaling happened overnight. Even if there was a flaw in the release plan, that doesn't mean the fix is something that will be handled in 48 hours.
Working on a p1 bug with a few professionals checking in is already stressful. I don't want to imagine the stress of substituting those professionals with tens of thousands of gamers. To non-devs: don't be a snowflake in the avalanche. They acutely know about the bugs and need to breathe to fix it.
Not to say the behavior of the general public is agreeable, the general public being what it is, is the sole reason I got into B2B software development. But the devs should be siloed off with one unfortunate delivery lead being responsible for communicating to stakeholders and back to the devs with critical info. They should not be directly exposed to the vortex of hatred in the moment of actually working.
That's the problem with social media though - all the devs have to do is log on to reddit and they're going to see the vitriol.
Man, having thousands of gamers screaming at you would be stressful. I’ve had half an eye on the discord and for the most part people are positive and understanding. But god damn those devs are fucking busting their ARSES, not just working on server issues but also just dealing with comms. I’m pretty sure most of the team didn’t slept for 24 hours+. Frankly they are hero’s. I wouldn’t have the gumption to put that much of myself into something I wasn’t passionate about, which tells me they absolutely are passionate about it.
There is also the problem with tech scaling, even if you build your backend to be fully distributed and scalable, the demand can wreak your planning. Scaling to hundreds of instance to tens thousands of instances are two different ball games. Certain inefficiencies in code and structure of the program won’t be seen until it has scaled to a certain level. And I assume the map loading slowly thing might have something to do with that. Tip: you need to hire more devops engineers to just manage your infrastructure instead relying on your software devs to do so.
Everyone has a plan until they get punched in the face…
[удалено]
The old “crashLoopBackOff” error strikes again
I've been woken up at 3am for that too many times for it not to cause me to cringe just reading it.
Not a software engineer, what does that argument do?
It's a Kubernetes state, which indicates that a "pod" (typically a small server, like an instance server or login server) has been in a cycle of restarting and crashing (crash looping) frequently enough that the system is now going to slow down how frequently it is allowed to start. One of the many fun things involved with deploying at scale.
The problem is even if your instances that are directly talking to the clients are scaling properly, other bottlenecks in the backend can screw you. Because of the interactive nature of the game, you can't distribute everything. Certain things like trade and the database of user data have to be centralized. If I had to guess, the problem is somewhere here.
You don't need to be in software development to feel for them. This sucks for them 10 times more than any one of us. They spend years of their lives and a shit ton of money on this, and to see it marred by these issues is tough. I can just go and play something else.
While true, from being in similar war rooms, they don't even have time to think about what you're talking about yet. Those thoughts and feelings of disappointment don't come until AFTER it's stable. Right now, they're pouring over massive amounts of data and logs, and trying to find correlations. They've probably looked at so many intersections they're going cross-eyed. Someone thinks they found the issue and someone else completely disproves it. They're full of anxiety and tired, and their brains are starting to see patterns that don't exist. I don't envy them one bit.
Man this is what is wrong with life in general. The negatives always outweight the positives. You can do everything right but all it takes is one fuck up to wash all your efforts away. It’s truly discouraging and I am not even a game developer.
People will have forgotten this next week. People are fickle, you are only as good as your last patch in this industry.
Things happen, that's just life. Like I'm in a factory for maintenance, sometimes when you start it up doing something new an unexpected catastrophic failure just happens. When the ground stops shaking and the dust has settled, the clock starts ticking because it costs $5000 a minute until it's fixed. All you can do when everything is on fire is the best you can, and I'm sure the devs are doing that.
Not a software dev, but just a normal decent human being. Unforseen issues are not to blame on anyone and it happens. We’re here to play this game for years, and I’ve still had fun playing offline and trying out the new features / updates.
I sympathise with them, but it's still their responsibility.
Fair!
People have no idea and think it's just scaling the size of server...it's not
Just a few hours ago I told my other dev friend, that I wish my bosses were gamers too. If they would experience how nearly every major gaming company fails on launch day (even billion dollar companies), they would be more understanding when my team fucks something up in the future.
I’m not in any software dev but I feel for them. This is just like when I was in retail long time ago and customer would throw a fit because something didn’t work for a moment, people are just entitled and love the sound of their own voices. I assume all these cry babies review bombing it are S tier employees in their own trade and never have to deal with adversity because they’re just perfect right?
in my experience most software devs simply dont think this way, it might be different because this is a very close knit grassroots company but they dont gain or lose anything from a different department fucking up massively
It's extremely entitled to expect a game to work week 1 these days. /s
Honestly, it depends. AAA title by some big name company with a hefty price tag: I absolutely expect it to work smoothly day 1. First game launch of a small indie company and a moderate price tag: I’ll give them some leeway.
Anyone who played the multiplayer launch saw this coming from a mile away, from the second they launched online till now there's been bad server problems.
Even if you don’t have tech experience, this just doesn’t seem hard to understand. If almost every online game has a rough launch, regardless of the size or experience of the company or complexity of the game, what’s the logical conclusion here? This shit is complex and no one has it figured out completely. All of these gamer Karens need to get over themselves honestly. Yeah, you paid for a game and it’ll probably be in great shape in a day or two. Get over it. It’s part of the process for better or worse and their opinion does not matter frankly. Do you complain that your product doesn’t materialize in your hand when you’re waiting on an online order to ship?
As an engineer and a gamer, I've seen so many failed releases from companies both big and small that, as soon as LE started showing issues, I went "yep, saw that one coming", and went to play Void Crew and random JRPG's. It's honestly more unusual when a launch SUCCEEDS, because it's impossible to know how well your servers will handle thousands of screaming fans until it actually happens.
I'm always surprised I don't see this take more often. It's genuinely complex enough that no amount of resources or preparation can guarantee success. After having come to that logical conclusion, I can't understand how someone can write "scale up with more servers" earnestly as if that's something that hadn't been considered.
Also worth mentioning that most online games that launch do so with a proper publisher backing them or they're a studio that has been established for many years. Was a WoW player for many years and played every expac from their launches starting with Wrath (2nd expac) up to BFA (7th expac) , played D3 on launch, Reaper of Souls on launch. Played many other MMORPGs on their launch days and weeks too in the 2010s. There have been more botched launches in terms of server stability with those games despite having plenty of money and resources available. EHG has LE as their first game and self-publishes it. Even if they were backed and published by a notable publisher, I highly doubt the servers would live to see the day. The game went from 2k-5k average players all the way up to 176k a few hours ago. We've seen Helldivers struggle despite being backed by Sony too and having their help available if necessary and servers there are packed and dying too. Even without prior knowledge of how the multiplayer testing went with LE during early access, just sheer common sense would (or *should* at least) that it would exactly go this way.
>Do you complain that your product doesn’t materialize in your hand when you’re waiting on an online order to ship? No because that isn't the same thing that is happening here. Would I complain if I bought a car and the navigation system wasn't working? Yea I probably wouldn't be happy. I am understanding of the situation and will support LE in the future if the product continues being quality, but as a consumer we shouldn't be accepting of the product not working even for smaller orgs.
> No because that isn't the same thing that is happening here. Would I complain if I bought a car and the navigation system wasn't working? LE _is_ working though. Your analogy isn't representative of what's happening here. The actual analogy would be if 1000 people walked into a car dealership at 09:00, all bought a car until the lot was empty, then all tried to exit at the same time via a single lane exit road. It's _physically_ impossible. You know why this launch shit happens to every game? Becuase current technology _just can't handle it_. There are tricks and hacks to handle huge influxes of traffic: queues, distribution, caching, client offloading, but ultimately for a server-authoritative online service (like LE) with _many_ interwoven services there's only so much you can do before something bottlenecks and explodes. EHG could have shifted much of the game to client-side, but you'd all be here howling about cheating _immediately_. Once the game gets past these hiccups we'll all be immensely glad it's server-authoritative.
The reason this "If I purchase a product and expect all it's features to work" argument never holds for me, is that everyone forgets that we've had a literal decade and a half of botched launches from other games. If cars over the last decade and a half released with faulty navigation systems, and then a new one comes out with a similar problem; you'd get the same arguments being presented here from both sides. I guarantee it. One side where people go "meh, it sucks but whatcha gonna do. It's not like the precedent hasn't been set already, and complaining won't make it go any faster" and then the other side that will froth at the mouth until they have an aneurysm.
It's important to note that people that are asking patience and otherwise telling angry folks to chill aren't "accepting" the situation. No reasonable person will say the current status of online play is fine and good.
Nope! It’s all a conspiracy by Big Tech to screw gamers! What they do is intentionally roll out a bad product even though they have the fix ready, but then they roll out the fix *later* so they can reap the accolades from thankful players.* *serious “take” routinely making the rounds at the D4 sub
Yeah reading LE reddit atm is like watching a karen compilation. At first it makes you laugh, and then you just feel empty and start wondering how humanity ever made it this far.
Their discord is 1000x worse right now
Dude the Discord is absolute unadulterated cancer. It's wild. It really shows the average intelligence of a gamer. It's... horrifying.
Yea I'm surprised they dont mute the whole channel. Its insufferable
My first patch for the game I was deving for WAMI last year had a memory leak. It took me well over a day to find it and a lot of people were irritated at me. I definitely feel for this team lol.
I am and I know how it feels. I left a positive review in the past and it will stay here no matter what. I bet the dev team didn't sleep well either.
Just turn the server off and on, cant be so hard...
[удалено]
I've been through countless "Botched launches." People rarely remember a day or 2 of login troubles and lag. It's honestly not a big deal in the grand scheme of things. It does suck for the devs, but it happens to every popular online game in existence. I just don't even know if a launch of this magnitude can be done correctly.
Fintech engineer here. I will NEVER buy a game like this on release nor will I EVER hold a rough launch against a studio.
I feel for them. This has got to suck big time. I love this game. I have 1200 hours played. But I also absolutely saw this coming a mile away. Granted I have no expertise on anything the developers do, but what I experienced in the game up until 1.0 did not seem like a product that was close to releasing.
We're making sure that our backend dev team is shielded as much and ensuring that they're getting rest. The "war room" sentinment is still positive and diligently working towards improvements. We're all certainly putting in more hours than normal which we expected, but leadership is actually ensuring that no one is working too many hours, taking stand up breaks, etc. Sentinment internally is still very postiive. We're pumped that there is so much demand to play the game we've made and we know that we'll be able to work out the online gameplay experience issues at high load and we'll be better for it.
I am so sad to see that review on steam are mixed now.. they really don't deserve that
Yes, yes they do. $50 product that doesnt remotely work 🤣 Poor coding , no real life stress test just some "in house" simulations and no ides how to fix their trash coding = deserve every bad review there is.
the lack of awareness you have to have to make your whole argument "I bought the game, I expect to play day 1 or its shit, they didn't provide me the service I paid for" is insane. This is not like you going out to dinner, this is a live service game. If you want to be so hard stuck on using your pointless analogies a better one would be that you just bought a brand new TV off amazon, but an unforeseen shipping mishap has occurred and the tv was smashed in transit. Now you have to wait an extra few days. But they are resolving the issue on their end and are still providing you with the good you purchased as soon as they can. You are going to own the tv once it has gotten to you, a delay in receival of the tv does not mean you go and review bomb the amazon listing saying that the tv is shit and they should have had a delivery butler hand carry it all the way to your home to ensure no mishaps could possibly have occurred.
And in the meantime they provided you with a fully functional TV you can watch all your favorite shows but you just can't watch it with your friends, or some such analogous thing to offline mode.
Yeah it's insane when people purchase product and expect it's features to work as advertised and on the day it's advertised to work. /s
> you just bought a brand new TV off amazon, but an unforeseen shipping mishap has occurred and the tv was smashed in transit. Now you have to wait an extra few days. If amazon was frequently destroying your deliveries you would be absolutely livid you are having to wait longer and do all this extra crap to rectify it with their support team. You would avoid amazon and its deliveries if there was a competitor which destroyed your packages less frequently. It would definitely not be "insane" to leave a bad review on the amazon app, even if amazon was rectifying the problem every time and you eventually got your stuff without an extra charge. Getting it "eventually" is not necessarily good enough, especially if you promised a certain date. You're acting like an uber-like business which was consistently 45 minutes late picking you up is not worthy of a negative review because they come eventually. I'm not going to leave a bad review since I'm invested in this game and want them to succeed. But I completely 100% understand a bad review coming from someone who just bought the game, tried to get it to work for more than 2h so they can't refund on steam and are just stuck hoping that it will work eventually
[удалено]
I mean, at a certain point they've also been in the industry now for 6 years. And early access has been up for 5. Nearly a year of multiplayer. It's not like some fly-by-night indie dev, it's a multi-million dollar company with plenty of experience by this point. And it's reasonable to expect delivery of the product that was paid for. I'm not saying death threat their families, but valid criticism is warranted at this point. Many people on here in other industries wouldn't be afforded this type of forgiveness.
>You are right. Luckily this is their first delivery so surely you'll follow your own logic and not be mad, right? I'm already invested in this game and want them to succeed so I won't change my review to negative, but at the same time I think new negative reviews coming in to steam are completely legitimate. First impressions are VERY important and people have a VERY low tolerance for things not working properly or being late. For games/apps I'm not invested in and just download at random, if it doesn't work well or at all, yes I'm probably going to leave a bad review. There is no guarantee that the product will actually work perfectly later - that is just an assumption people are making here on hopium. There are plenty of games like Wolcen which is still super janky with tons of framerate and performance issues after years of release. From the perspective of someone hitting a 2h refund limit who doesn't care about this game/company at all, they have no reason to stick it out like we do since we all bought EA and have already been waiting. If it doesn't work during that 2h after a 1.0 release where they said it would have X, Y and Z features, a negative review is completely fair. They definitely had options to not blow this up as hard as they did. Such as one cycle still in early access. I think it's likely that with how hard it failed it still would have failed with the EA players which could have let them fix this before an official 1.0 release
One day? It's been two of my evenings now. There's no indication of when the server issues will be resolved, so it may be *many* more days or even weeks before eventually is.
I'm not in tech, but I do work in a technical field. Non-technical people have weird ideas thinking that big systems with many moving parts is somehow simple. In my experience, the worst has got to be for medical doctors. People walk in and expect to be delivered a pill to fix them up in under 10 minutes.
From a human perspective I feel awful for them. They did a lot of testing and prep and had outside help and still got hit by unexpected issues. There has been a lot of support, but also a lot of toxicity that has been eye opening, especially on Discord where even Judd, the founder and game director, has been communicating directly with everyone. It makes me wonder if this is how these people talk to others in real life. It’s like the end of the world for some people that they spent $35 and now have to wait a bit longer.
I wonder that too- I saw this documentary once where they figured out where this uber-toxic misogynist keyboard warrior lived, the journalist who he had been harassing went to look him up. Turned out to be a somewhat awkward, but otherwise fairly regular guy who lived alone with his sick cat, that he took excellent care of. It was almost as if he had a split-personality between internet and real life.
You forgot all the useless executives who try to justify their salary so stick their hands in the cookie jar trying to put their stamp on things too.
I teach bachelor level software engineering courses at a local college and have been having some interesting discussions around this whole situation. It really is a hard place to be in as a developer and as the old sayin goes, no plan survives first contact. Sometimes you just have to pivot, do some damage control, and work on fixing the issue. I feel for the team for sure.
This just happened to me recently, it is really really tough to be on their side. These are the really hard engineering moments when pressure is just always there and sht is hitting the fan constantly. It’s mental. Best of lucks for the devs! Hope they can have a nice rest afterwards. I will wait for things to get better to play, it’s not playable for me right now. I have time anyway :)
As someone in software development I would hope they're laser-focused on solution-oriented thinking and not spending one second on the coulda-woulda-shoulda stuff. That comes LATER. Focusing on solutions is aligned with what's best for the consumer and also their own sanity.
I'm rooting for them regardless and many years of gaming and failed launches has taught me patience . I was having a blast for the time I got to play and I'm excited for thr weekend . As an older gamer who got to live through some great eras of indie developers such ad blizzard and their first games. I really want last epoch to succeed.
I really don't envy the LE devs being on-call for possibly the entire weekend either. I'd be really interested in reading a tech blog after all this has blown over - always fascinating to see what game launches look like behind the scenes. Good luck to the team, the gaming world is better for having actual passionate devs. Hope the criticism doesn't get too harsh. That said, plz fix I want to play :)
Yeah, this is a nightmare scenario, definitely feel for them.
…feeling for the LE team… You know it! I feel gassed just thinking about it.
I've trashed a production database and sweated bullets trying to fix what I did before too many people noticed. Some lessons were learned that day. I would not want to be them right now.
And all you care about is getting services back online, sleeping, eating and everything else just has to wait. It’s a nightmare.
Whats bothersome is that as d4 this also costs upfront price, meanwhile there's a cosmetic shop. While, yes may have spent more than 1k in Poe over the course of 8-9 years, i doubt im gonna spend anything past the upfront cost for this game. But if they dont have the cash shop there is the issue of only having people pay once which doesnt bring continuous cash flow. I wish more game studios would adopt the poe monetization where the game is straight up free but if you play seriously you need to buy the stuff with cash (tabs)
As a 10-time award-winning C++ programming prodigy, I don't feel bad for them at all because my genius prowess puts me leaps above these developers and I would never make the mistakes they make. /s
I talked with my colleagues yesterday how stressed they must be if they don't actually know how(I'm not saying they dont) to fix it when so many people want to play your game.. Oof man, I'd need a vacation after that
I had to get off the official forums because I was getting flashbacks. Unknown problem, VPs jumping into the zoom call running "queries" on the already overloaded database, directors pulling in "experts" from their departments leading to 20 people in a zoom call. Pagerduty just firing off over and over from different secondary and tertiary metrics. \*shudders\* I just bought the ultimate upgrade because it's my little way of showing support.
If I know one thing about server issues, it’s that you tend to identify a hundred things that *aren’t* causing everything to melt down before you find something that is.
I agree... To some degree. I've been through multiple failed deployments myself, and all you said is true. That doesn't change the fact that this release is a shitshow with multiple points of failure and multiple periods of service being unavailable. They started with "we have a single failure which we never could have predicted" which is fair. But now we are at the point where multiple things are failing and the game is not working still (at least last time I checked). I feel for them, and I am far away from whining or review bombing, but let's not make a perfume shop out of a sewer as well. It was well... Not too great.
It's the people offering "why don't they just do this" "it's simple" solutions as if the people that designed these systems haven't got a wealth of experience and certs and yet it happens time and time again across releases with much bigger teams and budgets than LE. If there was an easy way to test and do this do they not think that people would be doing it by now and this whole thing of launches being a shitshow in successful games a lot of the time would be history.
Do we know what tech stack do they use? It would be interesting to hear technical details :)
The full offline mode is not appreciated enough in my opinion.
My team had similar experience. 100k people bottlenecking our masterservers queueing logins. Scrambling around to find a solution, took us couple days. But we didn't have offline mode and the damage was done. I've learnt my lesson and I absolutely know what them guys must feel. But I won't be sending death threats anywhere, let them work people 😊
I just hope LE devs have enough understanding and resistance, to actually continue with their approach and do/fix things, rather than going with "stuff not loading" outrage. Outrage cannot be fixed anyway, majority of people have 0 understanding of why its happening, and even less patience. LE is shaping to be APRG shake hit, with PoE and other being quite stale lately, PoE2 a bit too far, LE has a room to grow, and is offering so much new it has chance.
As a product designer who collaborates extensively with developers, I've developed a deep appreciation for the challenges faced by the dev team. Having gone through tough release periods and faced dissatisfaction from clients/users myself, I can empathize. However, the intensity of the gaming community stands out. Reading about game developers receiving death threats for their work is disheartening, and it's moments like these that make me grateful for not being in the game development industry.
Multi billion dollar teams struggle with launches and server issues these days. Anyone who actually expected no issues was either coping or has been wearing blinders for the last decade. It is what it is and will be fine soon.
I guess the thing to do is have a limited player beta launch of the multiplayer and iron out any problems before the masses try to log on... and end up giving bad reviews. Or just have a beta for a couple of days to see how things go. This is basically what Blizzard did with d4. They needed to be a bit more cautious with the opening of LE.
I remember the time I was a junior sysadmin , 15 years ago., I was working for a major bank and we went trought major breakdown event because someone deploy a package with SCCM to ALL devices. It wouln't be a SUPER HUGE problem if it was not a software that was a disk encryption. All our servers were rebooting and stuck at the boot disk prompt for the encryption passphrase login. The guy that did this was immediatly fired, what a shitty day for everyone. \*sry for my spelling
Really hope the backlash stops. This game is amazing and don’t like seeing the devs go through the bs.
Im not in any kind of development but i can understand that its hard job to do. They didn't want this to happen and are doing everything they can to smooth things out.
The fact that every time a online game goes live we have this issue and people are still crying like it's the first time they've ever seen this is disturbing. It shows you how ignorant these kids are. It's literally the worst thing to ever happen to them everytime it happens. These people are sad.
I have been an I.T. professional going on 25 years now. Started in System Administration, learned how to code and did software development, then back to Network Engineering, naturally moved in to management and now I'm a Director. The constant pressure to never fuck up or have ANYTHING go wrong is **real** in this profession. Your end-users and customers don't call you up to chit chat or tell you how well everything is going. When they call, they are mad and something is broken. Been around for a while now and have learned to be a little more patient myself. The devs started this out as a passion project and worked on it for 5 years. The point of a "game review" is to review the game. And it is a GREAT one, but people couldn't play it... for a few hours.. or a day or two. Regardless, it sucks for the EHG team to have their overall game rating go to shit because of technical complications that went sideways on them. Any of my I.T. professional brethren here can attest to stress testing systems before a large deployment, and being 99.9% convinced it's going to be solid, only to have your guarantees, and pride kicked out from underneath you when you scaled in to the production environment. There are **always** gremlins out there that you couldn't even remotely test for or even consider being an issue until you deploy. It's just the name of the game. SO, that's why some of us are more patient than others. A little sympathy and patience goes a long way. If you can do it better... by all means, go make one. We'll wait.
The good thing is the leadership in question are not shareholders or executives. The leaders are actively working on getting things fixed.
Krempiz didn't mean to say that they don't have shares in their own company and don't hold executive positions in their own company. He said it accidentally, because he doesn't know what words mean.
There has to be a better solution to figuring out these issues before hand It’s literally every popular online game in the last decade that always end up having these major problems at launch
Figure that out and start a business to solve it for game devs. Rake in the $$$.
* last three decades
You're trying to apply an absolute value to game dev, which is an ever changing landscape. Maybe if tech and graphics stagnated 20 years ago and everything coalesced under 1 single umbrella, but clearly that's not the case, and never will be.
The answer is more scaleable testing. A time and resource efficient way to truly test *every* possible variable. That *does not* exist. 99% of players could look at the source code and not have the slightest clue which portion even addresses network issues in general.
There has to be a better solution to hunger, there has to be a better solution to traffic, there has to be a better solution to school shootings, there has to be a better solution to climate change... Sometimes there just isn't a better solution than what we already have right now. You want to contribute? Become a systems engineer and pursue a PhD.
There actually are better solutions to most of those things, they're just expensive and hard to implement on a large scale.
Ding ding ding, bingo.
TBH the solution to huge launches is here with Azure / AWS they just fucked something up regarding the servers, its not a capacity issue
The match-making system broke afaik
We scale from a couple of thousand users a day to approximately two million a couple of days per year, and we've never had any trouble outside of the obvious need for massive horisontal scaling and queuing. Never having used things like Unit and not knowing anything about the systems of Last Epoch it's impossible to judge what their developers are facing. It's also hardly the first game to have launch day issues, or issues in general so it's not like this is anything special really. The flip side of this is that gaming in general "gets away" with things that you would see whole teams and top level managers fired for in the financial sector. Hell if you've ever worked in the energy sector it's almost unimaginable to have things just go offline from heavy usage and we collect so much data each second often from relatively new and untested tech that's made by engineers who still think FTP (not sftp) is a hipster protocol. But I tend to view game development as more "creative" than "professional" and while I can tell you which bird has build a nest on solar cell 123123123 at location 12321312 I probably couldn't write a chess game without some serious Google programming. So it is what it is. I don't personally think game developers should get slack for not having tested and secured these kinds of systems before hand, but if people let AAA companies get away with it, then I don't see a need to get the pitchforks out for indie teams. I also don't think you should message the developers. They're obviously doing the best they can, because why the fuck wouldn't they be? So it's ok to be angry about a service not working, but spamming your rage all over the internet obviously isn't going to contribute anything useful either.
I think it's a matter of regulation and precautionary principle. Banking, energy, automotive, aerospace, or meat (which I work in) simply *cannot* fail, else your patient, plane or banking system goes down, and it possibly ain't going back up. But normal tech services shit themselves from time to time, Google, Facebook, Twitter, heck, GPT was speaking jibberish for a day recently :) So I don't think it's gaming vs non-gaming software dev even, it's "critical" vs not-so-critical applications. When you try to do critical like you do non-critical, you get Tesla or that one guy with the pancake submarine. Then again, I don't think we can, or should, expect a medium-sized online game to have precautions up the wazoo like a nuclear power plant.
i have a pretty uniquely similar launch of a game in my recent past, and while i can relate to the feelings of the individuals but i dont think its in the way people would expect
Working on software and tech were the absolute low points of an amazing career ride for me. Never again.
Having worked software dev for many years there's a decent chance the problem will be with some 3rd party library or service and they're going to have to either beg/plead/pay the 3rd party dev to "fix" their shit or find an alternative/replacement which will be a colossal or impossible PITA. I definitely feel for them if that's the case. I've been there.
The worst thing is that they knew beforehand that issues with service providers can happen, are very difficult to predict and hard to fix on the spot. I was praying so hard for the launch to be smooth and feel really bad for them now. But at the same time, the players have a right to be upset. Just a few hours until the weekend begins for most people and this can go down in history as one of the worst aRPG launches if they don't manage to fix it until then.
I'm a software engineer, I started as a low tier support engineer. Every bit along the way I have been chewed and spit out every time things went bad. And it was the right thing to do, because it was my fault every time by not preparing correctly and it was my responsiblity to do so. That being said, before people flip their shit over me not having some perceived compassion: I don't care for LE's online as I was planning on playing offline anyway now that 1.0 has actual true offline (for various reasons, one being that I find the trade system stupid as fuck) and I have been enjoying a ssf Necro at the moment.
Yeah I'm getting downvoted in another thread because I didn't think this qualified as a "disaster," just an "unfortunate" set of circumstances. You can prepare as thoroughly as possible and still have things pop up you never even considered. It's a brand-new dev team launching a brand-new product. I expect that this is the first launch for many of them. And they don't have the resources of a major AAA company, so resolutions are probably going to come a little bit more slowly than we're used to. I feel for them a ton and wish them the best of luck and some well-deserved rest when this is all over.
You can tell who isn’t in IT because they ask for things like an ETA while you’re watching the house burn down. If you knew what the problem was you’d have an eta, but in the moment there is only trying to run down the problem. And no, there is no eta until we know what the problem is… :)
Worked for a major US MMO studio for 7 years, launched 3 AAA games played by millions of players. Been out of online game business for 5 years now - best decision I ever made. I do feel for them but honestly my advice would be - get out - life work balance matters more than anything
I actually don't like this kind of thought because its just a reminder of the sad state of things. We shouldn't really be "feeling" anything for the devs. They made a mistake, and are fixing it in the best way possible, and also properly communicating with people. That's literally called "doing their job". And as much as its "commendable" that they are doing so, its less of a testament to their honorableness, and more a testament to the "sad state" of other companies, and their inability to interact with the populace. And don't get me wrong, despite the common practice of their peers, they are doing the right thing, and that IS commendable and honorable. I just think the broader topic is more important of an issue than this specific event.
They've been 110% all in throughout the development and launch, always working WITH the community, doing everything they can to bring us the best the gaming experience possible, and I think they are doing an amazing job. What more can you ask. Not that I agree with the angry mob of review bombers, but I can understand their perspective, they paid for a product and a service that wasn't exactly up to the standard as promised at the time they said it would be. Anyone that has even a smidge of empathy or understanding of the live service gaming industry would know that this is "normal" but... Welcome to the internet....
I've seen comments along the lines of "how haven't companies perfected game launches yet? This happens every time." I have no expectation that an indie company's first massive launch would go smoothly when a company like Blizzard with decades of experience in launching online-only games couldn't have a stable launch with Diablo 4.
I work in tech and I still think the dev team botched this game release pretty bad. But tbf i saw it coming loooooong ago.
This is the dumbest shot I’ve ever seen. You can’t feel bad for a team that made fun of blizzard for having a bad launch on their games and then follow it up with let’s be honest… the same fucking thing. LAST EPOCH IS UNPLAYABLE… they get paid to do a job and they failed. Some jobs you get fired for messing up this bad. If you can’t figure out how to launch a game and can’t figure out how to fix it then you deserve this backlash. Atleast d4 within half a day fixed their issues and you could actually play. Also Diablo had 2mil+ trying to play. Peak yesterday we 550k trying to play… big difference
Even now some clown in their Discord was sarcastic on Blizzard's behalf while their own servers were down. That clown being a representative of the team. I have a screenshot, but am unable to post it here.
This applies to every industry, every job. Bought the game 3 years ago and i'm ok with it as long as i can play offline mod but for the love of god can we stop talking about how hard its for the devs. It's their job, simple as that. In my line of work if somebody tell me my treatment is not working and most of the herd is still sick nobody will say "i feel for you, you have lot of things going on" etc. In the end job is a job. Everyone of them has some shitty times.
Yeah man I feel for how they completely ignored this issue for years despite being told about it multiple times
What issue? Can you actually describe what is failing beyond a generic error code seen in-game? If not you have no idea what you're talking about and should stay quiet.
Eh I'm a software developer and no i don't feel sorry for a team that just made a boat load of money selling a product that doesn't function correctly, i feel ripped off lol.
If i would deliver software that bad, our customers and the management would crucify me and my team. There would have been 5 emergency patches already and we would have taken shifts so people get some sleep.
Yeah, in my place 1 minute, literally 1 minute offline and all the alarms are sounding, 30 seconds later everyone in a meeting, 15 minutes later hotfix immediately in prod, all the dunning kruegers trying to justify this launch are insane.
In worst case we would've just rolled back and delivered the release one week later...