A test fails once in three hundred runs. That is not weather

What it costs in production

A race silenced by a retry does not disappear — it moves to production, where the schedule is chosen by load rather than by your laptop. There it costs money.

The first row is the most expensive and the most common. It is exactly what idempotency in payment systems exists for, and exactly what an ordinary test misses: the abandoned request lands after the test has already walked away.

Why retry(3) is not a fix

A retry replaces the question "why did it fail" with "how many restarts until it passes". That works right up to the day a red build stops surprising anyone — and from that day the suite no longer protects anything.

• A retry hides a class of bugs, not a flake: every genuine race looks the same and gets "fixed" the same way.
• Build times grow while trust falls: people re-run before they read the log.
• The bug stays in the code, waiting for the load that turns a rare interleaving into a frequent one.

Taking the schedule away from the OS

The idea behind deterministic simulation is simple: if nondeterminism comes from something you do not control, take control of it. Inside a run, setTimeout, setInterval, setImmediate, Date, performance.now and Math.random are replaced. Every way your code can wait ends up in one virtual timer queue, so "what happens next" has exactly one owner.

When several callbacks come due at the same virtual instant, the order is drawn from the seed and written down. The same seed produces the same run, byte for byte, on any machine. Invariants are re-checked between steps, so a violation is caught at the interleaving that caused it rather than whenever the test next happens to look.

One side effect is hard to overstate: virtual time is free. A retry policy that backs off for six hours is tested in microseconds, and a hung system is distinguishable from a slow one — because the simulator knows there is nothing left to wake anyone up.

What it looks like in code

We wrote an open-source tool for this — unflake. The test looks ordinary; the body simply runs in a controlled world, and the condition is re-checked after every scheduling step.

import { check } from 'unflake';
import { it } from 'vitest';

it('never double-leases a connection', async () => {
  await check('a connection is never leased twice', async (sim) => {
    const pool = createPool(sim, { size: 2 });
    const held = new Map<string, number>();

    // Re-checked after every scheduling step, not just at the end.
    sim.invariant('no connection is held twice', () =>
      [...held.values()].every((n) => n <= 1),
    );

    await sim.parallel(4, async () => {
      const conn = await pool.acquire();
      held.set(conn, (held.get(conn) ?? 0) + 1);
      await sim.io('query', { latency: [1, 6] });
      held.set(conn, (held.get(conn) ?? 0) - 1);
      pool.release(conn);
    });
  }, { runs: 200 });
});

If one of the two hundred schedules breaks the invariant, the failure is shrunk to the smallest schedule that still breaks it and printed with a timeline: who held what, who was waiting, and the instant at which nothing could ever wake up again. It replays from the recorded plan rather than the seed — which means it replays on someone else’s machine too.

Sampling and proving are different things

Two hundred random schedules is fuzzing: a clean result means "not found", not "cannot happen". For a small enough test you can get something stronger — enumerate the decision tree instead of sampling it. What that difference amounts to in practice is covered in the piece on model checking.

const report = await explore('a connection is never leased twice', body);
// { ok: true, schedules: 24, exhaustive: true }

exhaustive: true means the space was covered completely, and then a pass reads as "no violation exists" — for every execution the model can produce. The bound is honest: the tree grows multiplicatively, a wide latency: [1, 25] is a 25-way branch at every I/O, and exhaustive coverage stays the privilege of small tests.

What happened when we pointed it at other people’s code

A tool that finds nothing invites suspicion — so we ran it against third-party packages and published the whole result: 19 documented contracts across seven widely used libraries (p-limit, p-queue, async-mutex, async-sema, generic-pool, p-retry, bottleneck), hundreds of schedules each.

Every contract held. That is the honest headline, and it is worth saying out loud rather than quietly not mentioning. It is also what should have been expected: these packages have millions of weekly downloads and their core guarantees are exercised by every user daily. A tool that "breaks" the concurrency bound of p-limit on its first evening is telling you about its own calibration, not about p-limit.

One finding looked real: bottleneck starting two jobs 9ms apart under minTime: 10. It reproduced off the simulator too — but the gap was exactly 1ms across all four hundred schedules, which is start-up cost rather than a scheduling defect. Filing that report would have cost a maintainer an afternoon over one millisecond. We did not file it, and we wrote down why.

What the method cannot do

A testing tool that oversells its guarantees is worse than no tool at all. The boundaries are these.

• It only sees what it controls. A real socket, file or native driver is invisible to the scheduler, and a run with nothing left to schedule looks exactly like a deadlock. Route real I/O through the simulator or fake it.
• No partial-order reduction. The tool cannot see which operations touch shared state, so it cannot prove two orderings equivalent and skip one. Rust’s loom can, because the code under test uses loom’s own primitives.
• Not about parallelism. Node is single-threaded and so is the simulator: races between workers, processes or inside native addons are out of scope.
• Microtask ordering is left alone. Promise resolution is already deterministic; reordering it would invent races the engine cannot produce, and a false positive costs more than a missed bug.

How to start in an hour

If the system in question counts money, start with the shape of the operations rather than the tests: the details are in the write-up on payment idempotency and on the fintech development page. The rest of our engineering write-ups live in the Engineering section.