Skip to content

The bug that lived between two codebases

7 min readDebugging · Teams · Flutter

The hardest bug I worked on in a recent engagement was not hard because the code was subtle. It was hard because it sat on the line between a Flutter client and a backend service, and each side could look at its own logs and conclude, honestly, that it was behaving correctly. Nothing moved until somebody produced evidence that belonged to neither codebase.

The shape of a bug with no owner

A cross-boundary bug is one where both systems satisfy their own contract and the product is still wrong. The client sends what it believes it should send, the server responds as specified, and the user sees the wrong thing. Neither team is lying and neither team can fix it alone.

These are common in any app where the mobile client is not the only writer of its own data. Assignment flows, order states, anything where a background job or another user's action can change a record while somebody is looking at it — all of them produce this shape.

What makes them expensive is not the diagnosis. It is that the default outcome of an unowned bug is that nothing happens. Each team has a backlog of bugs that are unambiguously theirs, and those get worked first, forever.

Why both sides are telling the truth

Each team is looking at a different slice of the same event and seeing a coherent story in it. The client log shows a request and a valid response. The server log shows a request served correctly. Neither log contains the thing that actually went wrong, which is usually ordering, timing, or an assumption about state that only one side holds.

That is why the conversation stalls at 'works on our side'. It genuinely does. The disagreement is not about facts, it is about scope — two correct partial accounts of one incorrect whole.

The trap here is arguing about whose fault it is. That argument cannot be won with the evidence either side has, and attempting it costs you the working relationship you are about to need.

Produce the artifact neither team has

The move that unblocks this is to build a single trace that crosses the boundary: one correlation identifier attached to the request on the client, carried through the backend call, and logged on both sides — so that one timeline shows both halves of the same event in order.

Before that trace exists, you have two logs and an opinion. After it exists, you have one artifact that shows the sequence, and the sequence usually makes the answer obvious to everyone at once. In my experience the discussion ends within minutes of that timeline appearing, because there is nothing left to disagree about.

Do the same for reproduction. A cross-boundary bug that only appears in production is unfixable by argument; the same bug with a written set of steps that reproduces it on a staging environment is a normal ticket. Getting from the first state to the second is often the entire job.

This is not sophisticated engineering. It is deliberately cheap work whose purpose is to move a bug from 'contested' to 'assigned', and it is worth doing early rather than after two weeks of exchanges.

Fix it on the side where it is cheapest, not where it is fairest

Once the trace shows what happened, there is usually still a choice about where to fix it — and the answer is not always the side that caused it. Ask which fix is smaller, which ships sooner, and which one leaves the system easier to reason about afterwards.

Sometimes the honest verdict is that the client is compensating for something the server should guarantee, and the right long-term fix belongs to the backend while the client carries a guard in the meantime. Say that out loud, including the fact that the guard is temporary, and write down what would let it be removed.

The failure mode to avoid is a permanent client-side workaround that nobody records as a workaround. Two years later it is indistinguishable from intentional design, and the underlying behaviour is now load-bearing.

What this costs, and why it is worth naming

The real cost of these bugs is elapsed time rather than engineering time. The fix is often an afternoon. Getting to the point where two teams agree what the fix is can take considerably longer, and that gap is invisible in any estimate.

I would rather say this plainly than pretend otherwise: on the engagement that prompted this post, the delay was not technical. It was the time between the bug being reported and anybody being able to prove where it lived. What I would do differently is build the cross-boundary trace on day one of the investigation instead of reaching for it after the log-comparison conversation had already happened twice.

If you are hiring someone to work on an app whose backend belongs to another team, this is a large part of what you are hiring for. Reading unfamiliar code matters less than being able to establish, without blame, where a problem actually is.

Related

I build AI features into production mobile apps, and stabilise the apps underneath them. See what working together looks like.