The real-time bug that only appears once you run two pods.

The symptom

Half the dashboards froze the day we added a second server

A live dashboard makes a small promise. The number on the screen is the number right now.

Every server the studio rents out has one. Players online, memory, load, all streamed live to the browser as it happens. It ran fine on a single pod of that service.

Then we scaled the service to a second pod to keep up with traffic. Half the dashboards froze. Old numbers, sitting still. A refresh might bring them back. Might not.

The console on the same page kept working. Only the live numbers stopped. That split is the first real clue.

Ruling out the network

The network gets blamed first. It wasn't the network

The network is the first thing everyone blames. More copies, more hops, more places for a message to get lost or held up.

So I checked it properly. I followed the same live connection across every hop it takes to reach the browser: the edge, the CDN and the reverse proxy, and the service behind them. Every message showed up, at every hop, in order, on time.

The network was clean. That still helped. It meant the fault was on our side, inside the service.

The real cause

The numbers lived inside one pod of the service

With one pod of the service, this is simple. Every browser talks to that pod. It has the numbers, it sends them out, everyone sees them.

Run two pods to share the load, and each browser is handed to one of them and stays there for the whole visit. That part is fine. The trouble is who gathers the numbers.

You do not want both copies polling every server at once and doubling the work. So one copy is picked to do the gathering. Call it the collector. It grabs fresh numbers every few seconds and pushes them to the browsers connected to it.

Connected to it, and only it. The browsers on the other copy sit on numbers nobody ever gathered for them, waiting for an update that will never come. Half your users end up there: parked on the quiet copy, watching a screen that will not move.

The console never had this problem, for a reason that only looked obvious afterward: it already sent its output through a shared channel every copy reads, not through one copy's private memory. Only the live numbers took the shortcut through a single copy, and only the live numbers broke.

The fix

Put the numbers where both copies can read them

The fix is to stop letting one copy keep the numbers to itself.

So the collector stops handing numbers straight to browsers. It drops each update onto a shared stream instead. Picture a noticeboard that both copies watch.

Each copy reads the board and passes the update to its own browsers. One copy gathers. Both copies deliver. Every screen starts moving again.

For the engineers: the board is a Kafka-style topic, and what matters is that each copy reads it under its own consumer-group name. A shared name splits the updates between the copies, which quietly rebuilds the original bug. Separate names mean separate full copies of every update, one per copy.