Introduction

First off, THIS ARTICLE IS INTENDED FOR NON-GAME DEVELOPERS(the player's essentially).

To give an explanation for why this occurs in games while keeping it simple to understand. This doesn’t cover everything, only general explanations and reasons for why this happens in the real world.

The goal of this article is to explain why a game server would crash while you’re playing it. The reason for this article came from me watching a twitch stream where the streamer was dealing with the server crashing in their game, getting frustrated, and then the twitch chat was making up reasons for why that occurs.

Then the streamer started talking about why the developer needed to have dedicated servers for the gamemode they were playing and talked about the reasons for why that would make sense, and none of it was true or accurate to why a game developer would implement these features.

At this point I got annoyed and decided, instead of fighting the good fight and responding in twitch chat (which we all know would cause lasting and sweeping change throughout all gaming communities), I decided it would be better to write this article.

Ok that's cool and all but why should I listen to you?

What is a server?

First off for this whole article there is a distinction I want to make; there are Game Servers and Backend Servers.

• Game Server(Commonly referred to as Frontend Server): Runs the application that all player’s(clients) are connected to for sharing data across player machines safely and securely without the player’s directly connecting to each other.
• Backend Server: A general purpose name for servers that players don’t directly connect to(usually), these servers are usually databases, player data, matchmaking, authentication, analytics, etc.

For this article I’m referring to Game Servers unless I specify Backend Server.

With that out of the way, let’s clearly understand what a server is. A game server for games is basically a computer running the game, everybody else in the lobby/session/game is connected to that server. The server is the actual host of the game session(session is when you’re playing the game, usually described as a match in the game). If the server dies then everybody disconnects from the session. One person disconnecting shouldn’t affect the other player’s connected to the server’s session.

Game state is all of the state for the game, so not just the player's inventory but things like object's positions, physics, animation's, meshes, game logic flow, match score, etc.

Basically it’s a LOT of data.

The server will run the game logic and shares the state of the game to everybody that’s connected, so if we’re playing together I can see what you’re doing without directly connecting to your machine.

See how much more secure that is than Peer to Peer? Which is directly connecting to all other players.

There are 2 types of game servers:

• Dedicated Server
• Listen Server

Dedicated Server

This is a server running in a warehouse that the game developer pays for to run the game session for the connected player(s). This is more secure because nobody can access another person’s machine, everything goes through the server.

This is commonly referred to as “source of truth game logic/game state”.

This allows the server to have security code/software running on it that you can’t get installed on a player’s machine without them getting mad about privacy or whatever else reason they would get mad about it. It can help avoid personal information being stolen since it all is happening on the developer’s side of things and they can detect somebody hacking.

The developer can specify the exact hardware of that server and optimize performance. BUT this does not mean the server will be more performant; it just means they know what they’re working with.

Another aspect is how the dedicated server can talk to backend servers for player profile data (your progression level, inventory, unlocks, store and currencies, etc.) without having to worry about somebody faking player data since the environment for the dedicated server is already secure and in control of the developer.

Made in Draw.IO

This can be expensive if not properly architected, which is on the fault of the developer’s and not the player’s, which is a scalability issue.

It's a common misconception that player's think that they are the host because they're hosting a party for a lobby. This is incorrect because it's just game logic of deciding who picks the game session details and such.

• Dedicated Servers can totally support hosting a lobby, that doesn’t mean the player is the server.
• Again server = runs the source of truth game logic.
• Lobby host = picks the game session details. (Can be the player, for example: custom matches or the dedicated server which is configured by the developer)

Listen Server

This is a server being hosted by one of the players. Still technically more secure than Peer to Peer BUT players are still connecting to another player.

The hardware of the host player can be whatever the player is using (on console it’s much more predictable), but it makes it difficult to optimize for the developer.

Well why can't the developer just scan my hardware and software to account for that?

They legally cannot do this in all countries without permission by the player, and some countries don’t allow it outright.

There are some permitted pieces of information they can retrieve from the operating system like the hardware, but that’s about it.

Made in Draw.IO

If you ever noticed on listen servers when the host/server (because the host is usually the server in this format) disconnect’s, the game pauses for everybody. Then they either all disconnect from the game session or attempt “Server Migration”.

Server Migration

Server migration is a process where the game will try to transfer all of the game state to a newly picked player to host the game session. This is unreliable because that requires figuring out who has the most accurate version of the game before the previous host left and forcing everybody into that version of the game state. Lots of variables to account for and very faulty (so it’s easier to just disconnect everybody).

This is something the developer has to build for their game specifically by the way. It is not some sort of checkbox.

So at this point it should be clear listen server’s can be dangerous because one player can send a virus to the other person’s machine and possibly spread it to other machines. Hacking is also much easier because they can find out what hardware the server player is using and ping back their data. Not just for hacking gameplay but also getting personal data on that machine, which is even more scary!

So at this point you’re seeing why dedicated servers are so common in games today. The developer is primarily trying to combat cheaters, hackers, have consistent performance, and have their online stores not get robbed(so they can make a profit and keep making games).

Notice how I haven’t really talked about network connection from your ISP? Because that’s not a factor for servers, your network connection strength can affect it but only for that individual doesn’t necessarily cause the server to crash. Also if you have a terrible internet connection from your ISP, usually that’s not just to the game but to everything else.

This would be the only case where “buy more servers” would actually help because maybe the player’s ISP is too far from the server, so they need a closer server to connect to. Which leads into:

Why buying more servers doesn’t fix the issue

When you buy more servers all that does is add more servers to the warehouse, or maybe buying servers in a warehouse that has a few servers in it (so you can host more sessions).

Buying a new server doesn’t make existing ones magically faster or more stable. Since again a server is responsible for running the game’s logic and sending it out to other players.

So why not spread the game logic across multiple servers?

GLAD YOU ASKED! This is a very technical problem that is still trying to be solved at the time of writing this. This is a technique called distributed servers or server meshing. Star Citizen is the main pioneer currently regarding this technology if you’re interested in it.

Loading video…

Moving on, another question you may have is "Do they use a single server for each game session?" yes and no… It’s very common to have multiple game sessions running on a single server, and the processor is spreading that computation across multiple instances of the game running on that single server.

Which means depending on how many are running at a time, performance can be worse than if a single instance was running on that server. This is a load balancing issue that the developer is responsible for fixing to make sure the server is performant.

The team at Riot games for Valorant did a bunch of great articles on the topic that I recommend if you’re interested:

Why the server will crash

So at this point you hopefully have enough context of what a server is and how its generally setup for games where we can go into more detail of what could cause them to crash(causing you to disconnect).

First things first, a crash causes complete loss of data. When it happens there is no recovery, there is no save data to rely on when a crash occurs. A crash means the application stops functioning unexpectedly and exits abruptly/closes, not disconnects.

The developer can save game state prior to the crash but not knowing when its going to crash "So how do they know when to save?" great question! That’s part of the game design of when to save the game to account for crashes, this is why we have checkpoints and moments to save the game and such.

When you join a game session in the middle of it, what you’re doing isn’t loading in the save data but just reading the current game state for the currently running session.

So when an individual disconnects due to an error but nobody else disconnects, it could be from the server, but it could also be on the individual’s machine.

Now lets cover the different reasons a crash could occur on the server(causing everybody to disconnect).

Crash Type: Game Logic/Code

This is one of the most common types of server crashes, the developer did not write game logic/code properly that causes the game application to close.

This can be gameplay logic being incorrect, something not valid checked properly, something didn’t load properly, etc.

This is the same sort of crash that would occur for the player’s machine if they also ran into an significant error.

Regarding this type of crash, it’s impossible for me to tell you the reason for every game because it’s whatever is going on in the game at that time of the crash. Usually it’s because something wasn’t valid checked though in the code causing a null pointer exception.

The developer usually can diagnose this using log files on the server and seeing the reproducible situation. So when you fill out a crash report, actually explain what your doing don’t just say “Fix this game!” You want it fixed? Help them diagnose it.

Why do I have to help them? They should be finding these before releasing the game/update! Pay the QA team's and listen to them!

Trust me, they do. Games are massive, very complicated things to make. The larger the game and team, the harder it is to get a handle of every possible bug. They do pay their QA teams, and they do treat them with respect (from my experience at least, it would be messed up if they’re mistreating their QA teams!).

-Side note-

Most of the time QA can just help flag the bugs and provide info to the engineers/designers/artists to go fix them. Their job isn’t to fix bugs.

An important aspect is that QA isn’t restricted from helping and trying to fix bugs. The problem really is that it’s usually outside their skill set and not stopped because of gatekeeping.

Another factor of this is that more bugs are found when a game launches or has a major release because the player base is large enough to report all these problems, as the developers fix these bugs the player base either stays or leaves to go play other games.

Crash Type: Data is not loaded into memory

This can go hand in hand with the game logic crash type, usually this occurs when something isn’t loaded into memory yet. Again, the developer is at fault to account for this and requires valid checking and ensuring something is loaded into memory before you try to use it.

So you’re aware, dedicated servers don’t have a screen to render anything. Because of this developers strip out anything shader related from dedicated server builds of the game. This is to optimize what’s running on the server and to lower the size of the game for the server, so anything it has to load is just the bare minimum.

The only case where you would see a server dealing with shaders is in certain UGC(User Generated Content) games where they need to validate that the shader data being shared CAN run for other players safely. But this is not common.

You can’t predict this one as well without having reproducible steps, logs from the server, and being able to see what is/isn’t included in the game when they create the executable package.

So hopefully next time you get a crash and it asks what was happening when the crash occurred, you can just tell them reproducible steps instead of saying, “You need to buy more servers!” (That’s the nice version of feedback I’ve seen in the past… we do read them, and we do have feelings )

The developers are responsible for managing this as well.

Crash Type: External system failure

This one is less likely but can be a hardware or network failure since most dedicated server warehouses (known as data centers) have tools and systems to catch this stuff ahead of time, also being done during server maintenance. For those that hate the dreaded steam server maintenance every Tuesday, this is to avoid hardware/network failures.

When this does occur, the solution is equivalent to host migration but with dedicated servers unless the developer didn’t build server migration into their game, then they just disconnect everybody from the session.

Conclusion

Hopefully this helped shed some light on why game servers crash and understand what’s going on when you keep getting server disconnects.

There are a lot of details regarding this that does not make somebody reading this article with zero/little knowledge on the subject suddenly become an expert at all.

This was just to provide a primer and something based on the truth regarding server crashes as a concept.

Server crashes will almost always happen, and it’s the responsibility of the developer to curve them and try to keep the game stable.