Engineering
How the systems you cannot switch off are built. Payment cores, live database migrations, regulatory requirements, performance under load. We write about decisions we made ourselves — including what they cost and what we would do differently.
Tests do not prove the absence of a bug. This does
Short version: a test suite is a search. It can say “found” and it cannot say “none”. For most code that is fine — but not for a payment protocol, an order state machine, or anywhere two tasks touch shared state: there the bug is a rare combination of steps. Model checking visits every reachable state and answers the question tests cannot answer at all.
Isolation levels: what your database actually admits
Short version: an isolation level is not a performance setting, it is the list of outcomes your application is allowed to see. Between “read committed” and “serializable” sit anomalies with names and with a price: a balance driven below zero, a shift with no doctor on it, a document numbered twice. Reading in the docs that the database “supports snapshot isolation” is not enough — that is a property of executions, not of text.
The master dies: what actually happens to your data
Short version: “we run a three-node cluster” describes a configuration, not a property. The property appears once somebody has checked that a network split does not make two nodes believe they are in charge at the same time, and that a write already acknowledged to a client does not disappear along with the failed leader. That is checked by simulation, not by pulling a plug.
A test fails once in three hundred runs. That is not weather
Short version: a test that fails once in three hundred runs has already found a real race — you simply cannot repeat it. What is random is not the bug but the interleaving the operating system picked. Take that choice away from it, and the failure stops being an anecdote and becomes an address.
A retried request must not charge the customer twice
In short: the network does not tell you whether a payment went through — it simply goes quiet. The customer taps the button again, the mobile app retries on its own, and without protection you charge twice. Here is how idempotency works at the level of tables and code, and where it usually gets broken.
LCP of 6.4 seconds caused by an overlay nobody saw
In short: a walkthrough of our own site. The text of the first screen arrived from the server in 10 milliseconds and appeared on screen six and a half seconds later — hidden behind an intro overlay that only loaded JavaScript could remove. After the fix, LCP went 6.4 → 2.2 s and the performance score 73 → 98. Below is how we found it, including the hypothesis that turned out to be wrong.
Moving a live database without losing writes
In short: “we will do it overnight on Sunday” is a bet, not a plan. The working technique is different: for a while the system writes to both stores at once, history is backfilled in the background, the data is reconciled, and only then does reading switch over. Up to the last step you can always go back.