Player Expectations

Imagine you're playing a split screen cart racer game at home on the couch with a friend. The race starts, you push the stick forward, and your cart immediately starts pulling away. The game is responsive to your input. But you take the first corner too slowly, and your friend speeds past you. You both see the overtaking manouevre, your race position drops to 2nd, and theirs switches to 1st. The game is consistent for all players. You and your friend control your vehicles with split-second reflexes, until eventually one of you crosses the finish line just a moment before the other. The game plays in real-time.

These are the three key expectations that we have of most (not all!) video games:

• They will be responsive to our inputs, acting quickly to take the action that we have asked the game to take.
• They will be consistent, meaning that every player experiences the same game state, even only seeing different parts of it and from different angles, and maybe even at slightly different times.
• The game runs in real-time, so that we have a constant input-process-output loop that allows us to respond to the challenges set by the game and our opponents.

Information moves slowly

This works well when everyone is sat in front of the same machine. Within the tiny duration of a single rendered 'frame' of the game - maybe 17 milliseconds for a 60Hz game - the software is able to query all the game pads, apply the input in each case to the specific player carts, perform any AI routines for non-player drivers, run the physics simulation to move all the carts, calculate any game-specific logic (such as on-track collectables, boosts, etc), create all the visuals and sounds, and send them to the appropriate output devices.

The fact that this can be done literally faster than the blink of an eye is a technological marvel, and it's made possible by all the hard work being done within a single console or computer where everything is connected and can send billions of bits of information around the computer every second.

But what if we put those players in front of separate computers, miles apart from each other, connected only by The Internet? You'd be right in thinking that they're still technically 'connected', but the problem is, these connections are slow - glacially so, compared to the insides of a computer. Internet bandwidth continues to go up and up over time, but that is mostly about making the virtual pipes 'wider' so that more data can travel at once. The actual speed at which any one given message travels has not increased much, and cannot get much quicker. Network traffic - and indeed, all information - is limited by the speed of light, and that is not changing any time soon.

We see people talking about 20ms or 30ms being a good 'ping' for online gaming, and that means it takes 20ms to get a message - even an absolutely tiny one - to the recipient and back. But if it takes 20ms to get a reply, that's too long for the 17ms frame times we mentioned above, so we can't process the input in the same frame it was generated. Even if we accepted a 33ms frame duration (for a 30Hz game) it still means that the response to your input would arrive about two-thirds of the way through the frame time, leaving little time left to do the actual calculations and rendering.

This led to one of the earliest lessons learned by online game developers. The simplest way to implement networked multiplayer would have been to run the game on a server, and have each player's computer run a much simpler program (sometimes called a 'thin client') that only handles input and rendering, along with the network messages required to send the input to the server and to get the rendering data back from that server. This is the model used by online text games such as MUDs.

However, those text games do not run at 30 or 60 frames per second, and are not concerned with complex physics and rendering either. For modern action game developers, even with messages travelling at near to light speed, it still took too long to get them across the internet to allow the usual input-process-output loop within a single frame duration. This meant that simply could not work for a typical action-oriented game. A different approach was needed.

Shared simulations

The answer was to have the 'client' - that is, the program running on the player's computer, rather than on some server somewhere - to run the actual game, rather than just handling the input and output. This allows it to do all the usual calculations that a game normally has to make, without waiting 20ms or longer to get the necessary information.

The interesting thing is that while this was new for developers of racing games and shooting games, the strategy gaming world had already been doing this. A classic article named "1500 Archers on a 28.8" refers to how Age of Empires was able to have massive battles playing across the internet back in an age when a computer could not even send 28 kilobytes of data per second. The trick they used was to have each player run the whole game separately, and exchange network messages to ensure they all play out the same way, i.e. 'in sync'. In their words:

the expectation was to run the exact same simulation on each machine, passing each an identical set of commands that were issued by the users at the same time. The PCs would basically synchronize their game watches in best war-movie tradition, allow players to issue commands, and then execute in exactly the same way at the same time and have identical games.

Problem solved! Or was it?

Keeping the simulations perfectly synchronised required some very careful programming, including working to a schedule of "communication turns", each 200ms long. The idea here is that each of those turns could contain orders from each player, about which units they wanted to move and to where, and they could be sent out to all players and processed at the same time. However, for them to all be processed at the same time meant that all players had to wait a little while for everyone else's orders for that turn to arrive. The end result was several hundred milliseconds of latency between a player clicking to give an order, and seeing that order start to play out on their screen. In terms of our earlier expectations, this was real-time (in that the game world keeps marching on as we watch it) and it was consistent (in that all players were seeing that same world) but it was not especially responsive, at least not in a way that you would need from a shooter or racing game.

The Core of the Problem

This starts to show us that the concept of information taking time to travel has unavoidable implications for our games. We essentially have these 3 ideas - responsiveness, consistency, and the need for each player to run the game in real-time - and it turns out that it's basically impossible to achieve all three simultaneously without any compromises.

• A real-time game that is instantly responsive to local player input is making changes for each player that will take some time to reach the other players, meaning the simulations can't be kept consistent - events could happen in different orders or an event that made sense on one player's machine might not make sense on another player's by the time that information arrives there.
• A real-time game that guarantees consistency must wait for consensus about the result of any action before it advances, whether that consensus is in the form of synchronised input (as with the RTS example) or in the form of authoritative game state broadcast out from a server. This means the simulations cannot be instantly responsive to local inputs.
• A game that is responsive and yet also needs to be consistent must allow the input made by one player to reach and be applied by other players before the game can proceed, to ensure that actions taken by the other players don't somehow contradict the action we already permitted to happen. This means it cannot run in real-time.

Each of these permutations has its place. We saw above how real-time strategy games (at least in the early days of the internet) would choose to be real-time and consistent at the expense of responsiveness. Compare that to a turn-based strategy game, anything from Magic: The Gathering to online chess. By giving up the real-time requirement, and only moving the game on when a player submits their turn, they can be responsive and instantly show the results of a player's move locally.

There are also online games that are real-time and responsive but less concerned about consistency. Many RPGs - especially the 'massively multiplayer' variety - are like this. It rarely matters if players are slightly out of position or playing slightly different animations as it has little effect on the actual gameplay, which is usually resolved by game rules on the server that do not have to be responsive. Many MMORPG players have seen enemies hit by a spell when they appeared to be safely outside the area of effect, or had their character die even though they just drank a healing potion, because the server had resolved the death before it received the input about the potion. While frustrating at times, it's a reasonable technical tradeoff that the genre can make.

Another category that makes concessions on consistency is that of physics-based games like Fall Guys, which inevitably end up processing client-side physics slightly differently for all players. This results in discrepancies between player positions that are usually minor, but occasionally visible and game-changing. It also includes some racing games which extrapolate vehicle positions to give the impression of zero network latency, but which means cars can 'teleport' when that extrapolated position turns out to be wrong due to player input or collisions. Sometimes it even means revising the rankings at the end of a tightly-contested race.

In practice, it's rare that any game totally commits to two of the three values while fully sacrificing the third. It's more realistic to compromise on all three to a lesser or greater extent, or even to make different trade-offs for different features.

Predicting the Future

So, we know that we have to compromise on at least one of our three core expectations, but we'll obviously want to mitigate that as far as possible. Just what sort of mitigation would a game need, in each case?

With real time strategy games, they tend to 'fake' responsiveness by playing visual and audio cues to let you know immediately that your order has been accepted by the system, even if that order doesn't really start being executed until quite some time later.

Similarly, with the games that compromise on consistency, they still attempt to minimise the discrepancies, often by having a central server which acts as a tie-breaker, instructing clients to blend or teleport any out-of-position players back towards the spot that the server thinks they should be at.

What about games like our cart racer, or most shooters, or indeed fighting games? We find ourselves in a position where the genre dictates the requirement for the game to be real-time, and the competitive nature means consistency is important. But these are fast-paced action games where responsiveness is essential too. How do we solve that contradiction?

The usual answer is that such games live in a twilight zone where they essentially fake responsiveness and consistency, in such a way that it seems like the player is experiencing both, while arguably experiencing neither. This is done in the following way:

• Each player's inputs are immediately processed locally to give the appearance of responsiveness, while also being sent to a server or to the other clients;
• When they arrive on the server or other clients, the inputs are processed, and the results are sent back to the original player;
• Upon receiving the results, the original player's client checks to see if they are consistent with what happened locally (usually a few frames earlier), and if not, has to make corrections to bring the client's simulation in line with what the 'real' state should be.

The first step, where inputs are applied locally without waiting to see the exact effect, is often called "client prediction". It might be better to think of it as client assumption, because the input is applied locally on the assumption that things will play out exactly as the local simulation shows. Often, that is exactly what happens. But, when a response comes from the network and disproves the assumption, a correction needs to be made before the game proceeds, or the inconsistency between the simulations will grow to the point where the game is unplayable.

So, how are those corrections made?

Reconciliation

Some sources refer to this as "server reconciliation", the idea being that a client has to reconcile its idea with what it assumed would happen with the actual events that the server has told the client about. However, the broader concept doesn't actually require a server, as it can apply in a peer-to-peer game as well.

This is arguably the most complex part of the whole process, for two reasons. Firstly, it potentially has significant effects on how the whole game is implemented because it can become necessary to store past world states, to store past inputs, to be able to do arbitrary resimulation, maybe to be able to blend entity states (or rendered states), and so on. Because the correction is usually responding to something that happened a few frames ago (since it takes time to send the inputs across the internet and get a response), it's not as simple as just overwriting some data with the 'correct' values - the local effects of the client prediction, which have been building upon the incorrect values, also have to be taken into account.

Secondly, much of this is constrained by the rules of the game and a player's specific expectations for the genre, because there is not necessarily a single correct way of resolving an inconsistency.

For example, a modern fighting game is likely to "rollback" and resimulate the whole game every frame, starting from the most recent verified state and re-applying the player's input since then. The players expect extremely precise hit calculations and will not tolerate being hit when their screen clearly shows them out of range of their opponent. As a result, most fighting games historically operated what was known as 'delay-based netcode', not too dissimilar to the real-time strategy example above, giving full consistency at the price of reduced responsiveness. In recent years this has given way to 'rollback netcode' which gives that responsiveness back, at the price of a little visual consistency when corrections have to be made, plus the extra CPU power needed to be able to perform several simulation steps within a single rendering frame.

First-person shooter games have been doing the rollback-and-resimulate approach for much longer, since at least the late 1990s, because that was the only way to get the shared simulations working in a responsive way, once the 'thin client' approach was abandoned. Unlike fighting games, they don't necessarily need to do the resimulation every frame, because direct contact between entities is very rare and the exact positioning only really matters when resolving shots fired or projectile trajectories. Similarly, even in cases of resimulation, it's not always necessary to resimulate every single entity as most will be unaffected. (However, determining whether this is true or not is a hard problem, and sometimes it's better just to err on the side of a complete resimulation to be on the safe side.)

In most cases, this process of performing a rollback and a resimulation means consistency - every client ends up back in the same state as all the others, excluding the local prediction of course. But that's not the end of the story.

Arbitrary decision-making

The first person perspective used for these games, in conjunction with the 'prediction' aspect, means that there is often a very visible lack of consistency, with the classic example being where Player 1 has quickly ducked behind cover, but the information about that movement hasn't reached the Player 2 yet, who has just pulled the trigger to shoot Player 1 who was right in the crosshairs. Should the game award the kill, in a way that Player 2 would think fair? Or should the game say the shot missed, in a way that Player 1 would think fair? Games choose to handle this differently - the Overwatch developers chose to "favor the shooter" and explained this in their video. This is controversial to some, but is not a new concept - Valve were doing it in their Source engine over two decades ago.

Other types of game can be more lenient and don't need to attempt the rollback-and-resimulate approach. It was mentioned above that RPGs might be real-time and responsive but less concerned about consistency, and one effect of this is that their approach to applying corrections might just be to teleport a character back into position, or to blend its position over time. Positional blending is also popular with some racing games (though some with simpler physics models might also opt for the rollback approach if practical), and can be almost invisible to the player due to everything already being in rapid motion. But since such corrections take time to blend the entity towards the expected position, it trades off even more consistency during the blending period than if a resimulation had happened.

A hybrid approach is also not uncommon - some games might perform the rollback-and-resimulate to calculate the 'official' corrected position of an object, but apply a slow blend towards that position when rendering the object. This looks better than an abrupt correction, at the price of potentially visible glitches if the rendered position is clearly inconsistent with the gameplay (e.g. characters temporarily intersecting visually, despite their collision volumes being correctly separate).

Finally, any game which runs a physics model on each client has to make a decision about how the input received from the network should affect the outcome from the local physics simulation. It's not uncommon for the network to send information that positions entities in a way that causes overlaps, causing a powerful (and unwanted) repulsion force, or for it to end up causing instability as a physics constraint that is usually stably solved in one iteration of the physics engine instead gets unstably iterated four times a second across the internet.

What about the rest of the netcode?

The interesting thing about everything covered above is how very little of it is about the low level technical implementation, or the things that programmers tend to talk about. Writing the code to have the player clients communicating with each other across the network is deep and complex, potentially involving any or all of the following:

• socket programming
• event loops
• reliable vs unreliable messaging
• ordered vs unordered messaging
• congestion avoidance and management
• latency measurement
• clock synchronisation
• packet fragmentation and reassembly
• message serialization
• state snapshots or actor replication
• remote procedure calls (RPCs)
• ...and more!

The good news is that modern engines and frameworks have done a lot to make the above easier, which is why they're not mentioned above.

The bad news is that a developer will usually still need to understand a fair bit about some of these areas, unless they are lucky enough to be working with a genre that has a networking framework designed specifically for the type of game they are making. And chances are, they will still need to consider some aspects of the real-time/responsiveness/consistency trade-offs if they come to implement new features or attempt to improve the player experience.

So while it's true to say that making a networked multiplayer game is easier than ever, it is still far from easy. To those attempting it, good luck!

Let's have a look at how we can do this, starting with the worst case scenario and then presenting some options to improve the 'tweakability' of the values.

Examples are in C#-like pseudocode with the Unity engine in mind but should generalise to most programming languages.

Worst case - magic numbers in code

Let's imagine we have a simple RPG-like game where the player can attack a monster and deal some amount of damage to it. Let's also imagine that we have an magician in the game who can cast a shield spell on monsters, meaning they take less damage from attacks. A very simple implementation could look like this:

public float CalculateMeleeDamage()
{
    // Calculate the normal amount of damage for this weapon
    float damage_dealt = current_weapon.BaseDamage();

    // Decrease it if the monster is shielded
    if (monster.HasShieldSpell)
    {
        damage_dealt *= 0.85f;
    }
    
    return damage_dealt;
}

The best thing about this code is that it was quick and easy to write! However, it has an obvious disadvantage - the hard-coded 'magic number' of 0.85. Writing the literal value inline is fine when the code is as simple as this, but this function is likely to get more and more complex as functionality is added, and it may no longer be clear why there is a 0.85 in there. Worse, if we need to perform damage calculations in multiple places we could end up with this value in several places in the code, leading to subtle bugs if we ever change it in one place but forget to change it in the other.

Constants in code

The simple improvement here is to make it a named constant - this will make the algorithm more readable in future and also allows a programmer to change the value in a single place, even if it is used by more than one routine.

private const float ShieldSpellDamageMultiplier = 0.85f;

public float CalculateMeleeDamage()
{
    // Calculate the normal amount of damage for this weapon
    float damage_dealt = current_weapon.BaseDamage();

    // Decrease it if the monster is shielded
    if (monster.HasShieldSpell)
    {
        damage_dealt *= ShieldSpellDamageMultiplier;
    }
    
    return damage_dealt;
}

This is a simple fix, but still has limitations. To change the value, the programmer needs to stop the program, rebuild it, and run it again. That might not be the most convenient thing if you were trying out different multipliers to see which one gave the best game balance. What can we do?

Statics in code

By converting the constant value to a static value, we gain the ability to change it in the debugger. This lets a programmer tweak the value during a session by running the game in the debugger, setting a breakpoint, and then altering the value in the debugger at runtime.

private static float ShieldSpellDamageMultiplier = 0.85f;

public float CalculateMeleeDamage()
{
    ...as above...
}

The only change here is literally one word on line 1, but for a programmer this could potentially save them a lot of time in future.

But what if it's not a programmer who wants to change the value, but a designer, or any other team member who doesn't have a debugger handy (or isn't comfortable using one)?

Editor Properties

In an engine like Unity, it is relatively easy to expose a value like this in the editor so that it can be easily changed by anyone on the development team. A very naive implementation might look like this, assuming the code below is part of a MonoBehaviour or ScriptableObject:

[SerializeField]
[Range(0.0f, 1.0f)]
private float ShieldSpellDamageMultiplier = 0.85f;

public float CalculateMeleeDamage()
{
    ...as above...
}

Here we're also using the capability to restrict the range of the value in the editor and displaying it as a convenient slider, like so:

In Unreal Engine 4 you can achieve a similar effect with a UPROPERTY, in Godot you can export a variable, and so on. With this capability anyone on the development team has the ability to edit these variables and tweak them.

But it's not always obvious where to find the variable you need, especially if you have hundreds of objects/scenes/prefabs/etc, each with their own set of values.

Even if you fix that - perhaps by having a central object that stores all the tweakable values - it's still only developers who can do this, because it requires access to the editor (or the ability to edit the sometimes-esoteric serialization format that the data is stored in). It's not ideal for QA who might be working with the built executables.

Can we do better?

Settings/configuration files

The next step is to consider loading values like this from the game settings, so that not only can they be changed in the debugger, but they can be set before runtime by editing a simple human-readable file.

[CombatValues]
ShieldSpellDamageMultiplier = 0.85

static float ShieldSpellDamageMultiplier = 0.85f;

public void LoadSettings()
{
    var settingsFile = new SettingsFile("gamesettings.ini");
    if (settingsFile.HasSection("CombatValues")
    {
        var combatValuesSection = settingsFile.GetSection("CombatValues");
        if (combatValuesSection.HasValue("ShieldSpellDamageMultiplier"))
        {
            ShieldSpellDamageMultiplier =
                combatValuesSection.GetFloatValue("ShieldSpellDamageMultiplier");
        }
    }
}

public float CalculateMeleeDamage()
{
    ...as above...
}

This example (using a fictitious .INI loading library) will load in the gamesettings.ini file and looks for a CombatValues section, and inside there it looks for the ShieldSpellDamageMultiplier value. If the value is not found, it will continue to use the default as set in code, but if it is found, the value in the file will override it. Depending on personal preference you might want to use a different format such as JSON, YAML, or XML - each will allow you to set up some named variables grouped conveniently into sections.

(In practice, you probably wouldn't have each part of your code opening the settings file itself - you'd open it in one place and then pass it to whichever objects needed to configure themselves from that file.)

So now you have a system where programmers can set defaults in code and any developer can override the values in the settings file. All they have to do is edit the file and restart the program. Better still, you might even be able to set it up so that the program can re-read the settings file during play at the press of a button, allowing for some very quick testing and iteration!

Not only is this good for developers in the course of them iterating on features, but it also allows balance changes to be done simply by deploying a new configuration file, without needing to recompile the executable. And it allows for easy implementation of 'feature flags', by putting boolean values into the configuration and checking those values in code to decide which path to take. This is useful when testing experimental features, doing A/B tests, or perhaps just leaving your options open to disable something later.

But we can still do a bit better. This approach requires us to know the exact variable names that we expect to see in the settings files. (The alternative of simply putting all variables in the settings from the start gets unwieldy quickly as the file will need to change frequently and developers with locally-modified copies will find themselves merging it on a regular basis.) It also has no simple protection against us inserting erroneous values.

In-game console

What would be ideal is to be able to tweak these values directly from inside the game. It's still useful to load them from a settings file but for transient changes it helps to be able to change them directly in-game without using an external editor.

A common approach is to add a "debug console" which looks like a chat or terminal window where you enter commands and see results in text form. For our purposes we might want a "SET" command which allows us to change values.

// Globally-shared set of configurable values
Dictionary<string, float> FloatValues = new Dictionary<string, float>();

void ProcessConsoleCommand(string command)
{
    // Split the command text up at each space
    string[] commandWords = command.Split(' ');

    if (commandWords[0] == "SET")
    {
        string variableName = commandWords[1];
        string variableValueString = commandWords[2];
        if (float.TryParse(variableValue, out float variableValue))
        {
            FloatValues[variableName] = variableValue;
            ShowDebugConsoleOutput($"Set {variableName} to {variableValue}.");
        }
    }
    ... etc ...
}
 

public float CalculateMeleeDamage()
{
    // Pull this value from the shared values collection
    float ShieldSpellDamageMultiplier =
        FloatValues["ShieldSpellDamageMultiplier"];
    
    ... rest of the code unchanged ...
}

In this example we would allow the user to type something like SET ShieldSpellDamageMultiplier 0.5 into the debug console and the appropriate entry in the FloatValues dictionary would be set, meaning other parts of the code can immediately see that new value.

Obviously you're not limited to float values - you can have separate dictionaries for strings, bools, and so on, or keep everything in one dictionary with a variant type. An even cleaner way to handle it might be to have your game systems register relevant values with the debug console and for the console to tell the system when to update them, but implementation is left as an exercise for the reader!

Often we can go one further and integrate a scripting language like Lua into the game and expose it to the console. This means that rather than hardcoding commands like "SET" and being restricted to rigid syntax, we can write more complex expressions, and add more commands easily. This can be a lot of effort but is worthwhile, especially if the scripting language can be used in other ways (testing, automation, in-game scripted events, etc).

Still, this might be too complex for your needs. And even if you do have a debug console, it's not necessarily the most friendly user experience for non-technical users. Adding auto-complete and a way to see a list of valid commands can help, but it only goes so far.

In-game UI

What might be better is to have a debug UI which exposes these values in a more visual form. This means no more remembering the syntax, quicker editing of values, and the ability to group and display the tweakable aspects in a way that suits you.

The classic approach here is to use a so-called 'immediate-mode' GUI library. Immediate mode UI is a poor choice for your game's main user interface because it ties the visuals tightly to the code, and is not suited to interfaces which are complex, need custom graphics, or have animations. However, they are very quick for programmers to implement and as such make a perfect way of implementing debug user interfaces.

In Unity the old IMGUI system is adequate for this task, and provides a quick way to not only tweak in-game values but to limit those tweaks to reasonable ranges.

private float ShieldSpellDamageMultiplier = 0.85f;

void OnGUI () 
{
    var windowPositionRect = new Rect (25, 25, 250, 200);
    GUI.Window(1, windowPositionRect,
               ShowTweakableValuesWindow, "Tweakable Values");
}

private void ShowTweakableValuesWindow(int id)
{
    // Show label
    var labelPositionRect = new Rect (25, 25, 200, 75);
    GUI.Label (labelPositionRect, "Shield Spell Damage Multiplier");

    // Show slider
    var sliderPositionRect = new Rect (25, 50, 200, 75);
    const float MinValue = 0.0f;
    const float MaxValue = 1.0f;		
    ShieldSpellDamageMultiplier =
        GUI.HorizontalSlider(
            sliderPositionRect,
            ShieldSpellDamageMultiplier,
            MinValue, 
            MaxValue);

    // Show value
    var valuePositionRect = new Rect (175, 70, 200, 75);
    GUI.Label (valuePositionRect, ShieldSpellDamageMultiplier.ToString());
}

This gives output a bit like this:

With this sort of capability, backed up by the ability to change defaults via configuration files, we make our game about as convenient as possible to tweak in real-time, making development quick and easier. And even after the game is shipped, we can still test and deploy some rapid changes merely by altering the configuration file. For most games this will be enough.

But what if you have an online multiplayer game, and you have not only a game client to think about, but a game server too? Often you'll need to make changes to the game servers, but taking the servers down for a code change causes downtime that annoys and loses players. You can load values in from your settings file but even if you have a live reload facility it's not usually the case that your team will have direct access to the settings files once deployed.

You can even have a way to log into the game and perform debug commands from a trusted account, if you are confident about being able to correctly authenticate and authorize such an account, but this is harder to make secure and awkward in that it requires someone to log directly into the game.

In-game web server

A better way might be for the game to expose the various tweakable values via a HTTP API. This allows you to create a simple UI in HTML and Javascript which connects to the game server and requests the changes directly.

This might sound like a huge undertaking but these days is quite easy to set up. Most languages come with rudimentary HTTP server functionality built in, and this is enough to implement a simple web interface that lets you change variables while the game is running. Not only that, you can implement commands (as with the debug console) that allow arbitrary actions while the program is running, such as respawning enemies, toggling a player's invulnerability mode for testing, etc.

And why stop there - you could extend the interface to provide any diagnostic information you care about such as frame rates, player locations, bandwidth usage, etc., and render them out conveniently in the browser using HTML or Javascript.

The code for this is a little longer but still very manageable - usually it's just a few lines to set up a HTTP server and then a function to handle requests by examining the URL and query string, and then to return a rudimentary response. This is a Unity code example but the only Unity-specific parts are the MonoBehaviour and the Debug.Log calls.

using UnityEngine;
using System;
using System.Net;

public class HTTPServer : MonoBehaviour
{
    private float ShieldSpellDamageMultiplier = 0.85f;
    
    HttpListener listener = new HttpListener();
    
    // Start is called before the first frame update
    void Start()
    {
        // Set the listening addresses
        listener.Prefixes.Add("http://127.0.0.1:8888/");
        listener.Prefixes.Add("http://localhost:8888/");

        listener.Start();       
        
        listener.BeginGetContext(new AsyncCallback(ListenerCallback), this);
    }
    
    private static void ListenerCallback(IAsyncResult result)
    {
        try
        {
            Debug.Log("Handling HTTP request.");
            
            // Get the data from the request
            var thisObject = (HTTPServer) result.AsyncState;
            HttpListener listener = thisObject.listener;
            HttpListenerContext context = listener.EndGetContext(result);
            HttpListenerRequest request = context.Request;
            HttpListenerResponse response = context.Response;
            
            // Set up the context again for next time
            listener.BeginGetContext(
                new AsyncCallback(ListenerCallback), thisObject);
            
            // Perform the required action
            var endpoint = context.Request.Url.LocalPath;
            if (endpoint == "/set")
            {
                // Retrieve the value from the request and 
                // store it in the variable.
                string newValue =
                    request.QueryString.Get("ShieldSpellDamageMultiplier");
                thisObject.ShieldSpellDamageMultiplier =
                    float.Parse(newValue);
            }
            else
            {
                // Assume a 'get' operation, which is covered 
                // by the default response below
            }
            
            // Construct a response, showing the new value
            string responseString = "<html><body>" +
                "ShieldSpellDamageMultiplier=" +
                thisObject.ShieldSpellDamageMultiplier.ToString() +
                "</body></html>";
            byte[] buffer =
                System.Text.Encoding.UTF8.GetBytes(responseString);
            response.ContentLength64 = buffer.Length;
            System.IO.Stream output = response.OutputStream;
            output.Write(buffer,0,buffer.Length);
            output.Close();
            
            Debug.Log("Completed HTTP request.");
        }
        catch (Exception e)
        {
            // Catch and log errors thrown from listeners,
            // as Unity silently suppresses them
            Debug.LogError(e);
            throw;
        }
    }
}

This is not production-quality code - you'd want to have different endpoints for different functionality, look up values dynamically, add validation and error-checking, etc. It is also more common to return data in JSON format rather than HTML so that Javascript can process it effectively. But, this gives an indication of how you can start building such a system. If you add this script to a GameObject in a Unity scene and run the game, you can connect to http://127.0.0.1:8888/ in a web browser and see the current value of ShieldSpellDamageMultiplier. And if you enter http://127.0.0.1:8888/set?ShieldSpellDamageMultiplier=0.7 into your browser instead, it will change the value in-game.

Summary

Today's game engines and programming languages give us a variety of ways to change values during play and get to quickly test things without needing to recompile the game or even to restart it. Hopefully with the ideas above you can implement one or more of these into your game today and start to reap the benefits immediately.

Anyone who's followed game development for a while will know there's nothing new here. 'Crunch time' has been a staple of game development for as long as the industry has had major projects, and was already common knowledge when the famous 'EA Spouse' blog entry was posted 17 years ago. Despite faster computers, better tools, more powerful languages, and growing experience, things don't seem to have improved much in game development.

And yet in the rest of the software development world, things have changed a lot. Perhaps the biggest change in development practices has been the move towards agile development. Agile means many things to many people but the core principles are basically that you involve the customer and/or the user more, in a collaborative and iterative process that discovers requirements as you go and responds to change. It is supposed to lead to a more manageable project.

One commentator claimed that Cyberpunk failed primarily because they did not follow agile project management processes, and particularly did not practice the specific act of 'Continuous Delivery', where the software is being 'delivered' on a regular basis (e.g. daily) to be assessed. It's reasonable to say that "bad software engineering" played a part in the problem. However, the claims that this is a failure to adhere to agile principles don't add up. The developers CD Projekt Red - like most major game developers and quite a lot of smaller ones - do in fact employ people to perform continuous delivery and 'devops thinking'. It just didn't really make a difference.

I mentioned in my introduction post that I think iterative, 'agile' development can actually cause problems for game development projects, despite being the overwhelmingly dominant paradigm. Why?

The myth of people over processes

'Agile' is a vague term and therefore any criticism of it is often batted away by resorting to some variant of the No True Scotsman fallacy. And there's some merit to that, because while the Agile manifesto talks about "individuals and interactions over processes and tools", what we actually got are Scrum and Sprints and Standups and Epics and Kanban and so on - a variety of ways in which management seeks to have the benefits of Agile without giving up its array of tools and processes to impose some sort of structure on it.

The promise was that we could follow the Agile principles and thereby enable agility among the development team. The reality is that developers work under a new set of constraints and agility is demanded of them.

How Agile helps most software

Maybe it's a price worth paying? For most software, there's a good argument that Agile makes sense, especially the principle "Welcome changing requirements, even late in development". In the world of traditional software development where a program or system is being written for a client, this principle solves a big problem: programmers have to make software that meets the needs of non-technical users and customers. As Patricia Aas said, "The users rarely know how to express what they need. You don’t have the context needed to understand what they say. Only once you’ve built something do you have sufficient understanding to ask the right questions". It's inevitable that the first draft will need extensive reworking, so Agile development embraces this reality - produce that draft as early as possible and have both sides work together in a process of gradual improvement based on early and frequent feedback.

All very sensible. But the explicit assumption here is that users or customers can't adequately express what they need, or provide any sort of detailed specification of the software, therefore this is the only reasonable route that remains. And the implicit assumption that follows is tied to the cost/scope/time triangle above - i.e. changing requirements will mean that some functionality will either fall out of scope or increase the project time and cost, and that the customer can simply select their chosen trade-off.

Do these assumptions hold true in a game development context?

Effective up-front design is possible

The first assumption should be untrue. Game development is rarely performed only by programmers delivering a program for a non-technical outside client. Usually it includes a whole game design team, who shape the vision for the product in conjunction with a publisher or management. Even if you consider the customer to be the publisher rather than the design team, it's still the case that a game development team has designers on staff whose role is to be experts in analysing, designing, and planning video games.

In particular, a designer should be able to take a particular feature and break that down into clear deliverables with relatively unambiguous specifications, such that a sufficiently competent software engineer is able to implement something very much like that at the first attempt. The need for iteration to discover requirements and solicit feedback should be greatly reduced by the presence of this team of domain experts.

This role played by a 'semi-technical' design team has parallels in other industries - for example, the work done by an architect when designing buildings. By coordinating both with the client and with the teams responsible for the eventual 'construction', they can formulate a fairly comprehensive plan that delivers on the eventual user's requirements while staying within all constraints and creating a plan for engineers to follow.

One snag is that the games industry has, on the whole, failed to train designers to fill this role, and instead focuses them on content creation (mapping the world, writing lore, placing spawn points, specifying items, assigning character attributes, etc) and creative direction (choosing and maintaining design 'pillars', identifying references in other games, selecting key features, communicating the overall vision to artists and programmers, etc). Too often, there's a void in the middle where the actual design would be. It's like having city planners and construction workers, but no architects.

One game I worked on (which will remain nameless) revolved heavily around combat. Senior designers had specified that there would be various weapon types and damage types. Other designers were tasked with implementing these via scripts. It was only after many months and several bug reports that anyone realised that there was no combat or damage system specified anywhere - just various disparate scripts attached to weapons that applied the damage in inconsistent ways. Eventually someone was forced to come up with a coherent system, the necessary engineering work happened, and the various weapons were re-implemented that way.

Problems like this are incredibly wasteful from a project management point of view because time is expensive! Although software is much easier to change than, say, a building, it is not free to do so. Worse, this process causes 'technical debt' since programmers may cut corners to find the quickest route to implementation if they know there's a good chance the designers will request future amendments anyway. To quote one commenter on the Continuous Delivery video above, "All delay problems, all software quality problems were coming from basically one source: constant changes by game designers". I wouldn't go that far, as there are many problems in game development that software engineers have entirely inflicted upon themselves, but the impact of designer changes on the timeliness and quality of game software is severe.

So while the statement "the users rarely know how to express what they need" is true in many industries, it's not in ours. For example, we know that a combat-oriented game needs a combat system, and we can know what aspects make up a combat system. It's our job as game developers to understand these concepts.

You can't avoid an iteration or feedback loop entirely. Even the best plans have flaws, or a change gets forced upon you for technical or business reasons. You need to be adaptable and you can't eliminate change, but planning and foresight can reduce it and save you time. A changeable plan is better than no plan.

In particular, the cost of design time is usually much lower than the cost of programmer time, so you want the changes to happen as close to the design end as possible. Iterating mentally and verbally will always be cheaper than iterating in documents or spreadsheets. Iterating in documents will always be cheaper than iterating in designer prototypes. Iterating in designer prototypes will always be cheaper than iterating in engineer prototypes, which in turn are cheaper than iterating during production. Changes in production should be the last resort.

Sadly, this focus on agility has emphasised responding to change, and de-emphasised the need to plan things in advance. So it's no surprise that Cyberpunk 2077's development saw people say "we'll figure it out along the way" and that poor planning is blamed for many of the problems. This is endemic to the industry.

The project is rarely as agile as the methodology

So, to the second assumption - just how flexible are the corners of that project management triangle above?

In an Agile project, the idea is that stakeholders regularly evaluate the state of the software, provide feedback, and adjust priorities. As change requests are made and new requirements emerge, less important features might get delayed or shelved entirely, and if development is otherwise going well, the budget might increase to accommodate these features later. The nature of the process assumes that new requirements are going to be discovered during development and therefore either some old requirements fall out of scope or the budget stretches to include the new ones as well.

This is somewhat at odds with the way that many game development projects work. Typically, the publisher is promised a certain set of features, often delivered in a fairly well-defined order, in return for funding to pay for a team to deliver them over a known timescale. These 'milestones' during development might correspond to broad phases - e.g. preproduction, pre-alpha, alpha, beta, etc - or they may be broken down further still, for example to require a monthly deliverable, each with certain features delivered to an agreed level of completeness.

The milestone system means that even though the project is delivered frequently over several iterations (just as agile proponents would want) it's largely decided ahead of time what gets delivered and when. This reduces risk for the publisher and gives them a very clear overview of the project's progress. However, from the developer's point of view, this incurs a lot of the dangers of a traditional model where you need to hit a fixed deadline with fixed resources, but with much of the overhead of an Agile model where the software must be ready for continuous delivery and evaluation at all times. The worst of both worlds?

It's easy to blame publishers for not being willing to be more flexible. But historically most games have not been profitable. It is a risky business where a handful of weaker releases might get subsidised by 1 or 2 hits, and survival as a publisher - and therefore, survival as a game developer funded by that publisher - can revolve around whether the publisher is able to keep costs down. And the publisher, unlike most customers who are buying software, can't just decide to cut their losses and accept the current state of the software when things start to overrun, because they need a playable and polished game for retail purposes. The Agile principle of "deliver working software frequently" can only go so far.

One of my favourite games, Deus Ex: Mankind Divided, saw the publisher come in for serious criticism from YouTuber Jimquisition over publisher 'meddling' and 'cash-grabs'. Yet we also know the game was delayed at least once and even when it shipped, was criticized for being 'cut short'. Given that sales were disappointing, and that development was already over budget, it's not hard to see why the publisher may have insisted on a reduction in scope and some controversial revenue-generating features given that they probably suspected they were going to lose money on this. Granting the developers an extension of an extra year or two to make the game they truly wanted to make would have made a better product but probably a greater financial loss too. When development can cost several million dollars a month, there has to be a deadline.

All this adds to the stress on engineers. Each bit of work thrown away because a designer wanted to 'iterate on it' rather than specify it in more detail is time not available to hit these milestones. A related issue (which I want to talk about more in future) is where designers or project managers trickle out features to engineers sequentially over time. What the designer might gain from leaving the subsequent specifications to a later date is often more than lost on the engineering side as key systems have to be reworked to accommodate the new requirement, which often could have been done properly from the start in less time.

The last problem I want to mention is that games are relatively complex systems, even ignoring the software engineering aspect. It's rarely easy to reduce the scope of a game without there being significant knock-on effects. Imagine a crafting system where players create their own items. Disabling that is only the start of the process, because any other system that assumed players had access to crafted items will now need re-assessing - will players get those items elsewhere? Do we need to add more merchants into the game? Do defeated enemies need to drop more loot? Does the inventory management need change now that there are fewer items to consider? It's rarely as simple as merely dropping a few features to hit a launch date.

What should we do?

Would I agree with those who say we should abandon agile development entirely? Probably not, although the Agile term carries both too much baggage and too little clarity, so it is essentially meaningless now. Still, much of the philosophy is common sense and has clear benefits, and some degree of agility is beneficial.

So, my specific recommendations are:

1) We need to be less pessimistic about what can be effectively planned in advance. We don't need 'Big Design Up Front' or hundred-page documents before development starts. But we should be front-loading much more product design, doing more work on (digital) paper first, and when we do hand things off to engineers, do it with clear specifications from designers rather than an expectation that the work will be re-done based on feedback later. Less 'Find the Fun', and more 'Design The Fun'.

2) We need to rein in the heavy-handed project management aspects that crept into Agile development that obstruct developers, and get back to the original promise of a 'light touch' that allows engineers to be more effective. Replace the micro-management of tasks with clearer instructions to engineers and sufficient information for them to be able to carry them out well. (In return developers probably have to get better at estimating task durations, but that's for another day.) A good project manager or producer is worth their weight in gold, but that gold is too easily spent on admin and bureaucracy to 'manage upwards' while impeding actual work.

3) The games industry needs to refocus the design discipline on game system development and the production of good feature specifications. This might need more support from both academia and industry veterans to ensure that we have the effective tools and vocabulary for this, since few of the theories or concepts discussed in the past appear to have gained traction.

Conclusion

Game software development has some unique constraints that other software development often lacks - but it also employs designers who, if effective, can not just improve the product but significantly aid in planning the project. As such, Agile development - at least in the form that it usually ends up taking - is not ideal for games, and an approach that borrows from more traditional "plan-first" methods is likely to be better, especially if the industry learns to play to its strengths.

I believe that video game development has only 2 major problems. Unfortunately, the first is that we're not very good at writing software, and the second is that we're not very good at designing games! It's only through the dedication and hard work of the developers, combined with the public's insatiable appetite (and thus, funding) for new games that we're somehow able to keep cranking out new ones despite all the odds.

Elaborating on these two points is the primary purpose of this blog, so for now I'll just provide an overview and set the stage for what to expect from future entries.

Over the years, software development has followed a curious course. The continual advance of free or cheap third-party libraries of software means that developers can deliver more and more functionality while writing less and less code. Many modern apps are mostly existing components bolted together with a thin layer of application-specific 'glue' code. But while this paves the way for increasing numbers of creators able to make useful and entertaining applications, it comes at a cost - fragile software that breaks in ways that the creator does not understand, insecure software that leaks personal data or exposes security flaws, inefficient software that uses more of your computing resources in order to save the original programmer time. This is going to get worse before it gets better - a wave of self-taught programmers are using these components to achieve great things with unmaintainable code, many embracing a culture that dismisses fusty old software engineering formalisms in favour of the "move fast and break things" ethos popularised by the world of tech startups.

The games industry has been spared the worst of this, but not all of it. The dominance of the two major games engines (Unity and Unreal) is a similarly two-edged sword, allowing developers to achieve great success with less training or rigour. And while the games industry was never a bastion of perfect code, poor software engineering practices are arguably increasing now that more developers are self-taught and engines isolate them from their errors in ways that previously could not be ignored. As with non-game applications, we're seeing an explosion of new programs that do more than ever before but which are increasingly fragile.

Game design has a different problem. It's a far older craft and most of what we know about games and play in general can be transferred to the digital medium. We may not have developed the same sort of 'game components' that can be bolted together, but at least there is this rich history of past games - digital or otherwise - plus our understanding of related media such as cinema and prose, which can guide and inform us as we make new works.

But if this is so, and we have decades if not centuries of game and narrative design to draw upon for designing video games - why are we so bad at it? As veteran game designer Daniel Cook said recently, "an expert game designer is 20x more effective than a newbie. They are correct 20% of the time instead of 1%". We tend to assume that good games are discovered through a process of iteration rather than designed explicitly, and this is the accepted wisdom - at least in video game design, where the themes of iterative and 'Agile' development processes are applied to the design of the game as well as the software implementing that game. But I would argue that not only is this excessively iterative approach unnecessary, but it's inefficient, ill-suited to game development, and actually adds to the trouble we have when trying to successfully complete a video game project.

I think it's no coincidence that many - perhaps most? - game designers I've met were not really 'designing games' in the sense that someone making a card game or board game might design a game. Instead, they were usually creating and tweaking content for systems that already existed, often without ever choosing that system or planning its parameters. But if we were to take a more systems-oriented view of game design, as designer and author Raph Koster does by temporarily setting aside the aesthetics and considering the core "mechanics, inputs and processes, rules and tokens and actions", we should be able to use our experience to structure a game that effectively delivers something close to our final intent earlier on, reducing the amount of subsequent iteration time needing to be spent on it. The aesthetics are still essential, but to some extent these two strands can be managed separately, with the interface between the two arguably being easier to manage when there is less iteration and therefore less of a moving target.

Thankfully, I think both of these issues - how we approach software development (especially for games) and how we approach game design (especially for software) are both fixable. I don't have all the answers, but in the spirit of Thomas Edison, I do know a lot of ways that don't work, and I hope to share various suggestions based on those experiences over the weeks and months ahead.