Why a maintenance window is a bad plan
The task comes up whenever a company moves off an old system onto a new one (CRM or ERP) or when a platform outgrows its store. The infrastructure side lives on our Cloud & DevOps page. The plan “we will take it down for four hours at night” looks simple and almost always fails the same way: the transfer runs longer than estimated, halfway through it turns out some rows fail validation, and rolling back is no longer possible — the old system is stopped and the new one is half-populated. By six in the morning the decision is being made by exhaustion, not by an engineer.
Worse, a maintenance window forces you to verify the result at the moment of maximum stress. A field-mapping mistake will surface a week later, when the data has already changed and there is nothing left to restore it from.
The industrial technique is called parallel change: first expand the system so old and new can coexist, then migrate, then remove the old (expand — migrate — contract).
The five phases
- 01
Dual write
The application writes to both stores and still reads from the old one. From this moment the new store receives every fresh change — and we can take our time with the history.
- 02
Backfill
A background process moves old rows in batches, newest first. The backfill must be idempotent and interruptible: it will be stopped, and more than once.
- 03
Reconcile
Compare the contents: checksums over ranges first, then row by row on the ranges that disagree. In parallel, turn on shadow reads — read from both stores, compare the answers, serve the old one.
- 04
Switch reads
Reading moves to the new store behind a flag, starting with a fraction of traffic. Writes still go to both — that is exactly what keeps the rollback instant.
- 05
Decommission the old store
Only after the new one has run under full load long enough to have seen monthly and quarterly scenarios. Then the dual-write code is deleted — otherwise it stays forever.
Dual write: where it breaks
The naive version of dual write is two calls in a row, and it contains a bug: if the second write fails you already have a divergence, and the user got an error on data that was in fact saved. The answer to “which store is authoritative” must be unambiguous for the whole transition.
async function save(record: Record) {
// The source of truth during the transition is the old store.
await legacy.save(record);
try {
await next.save(record);
} catch (err) {
// Do not fail the request: the user should not suffer for a migration
// they know nothing about. Record the divergence and repair it later.
metrics.increment('migration.dual_write.failed');
await outbox.enqueue({ type: 'migration.resync', id: record.id });
}
}Reconciliation: checksums, not spot checks
“We opened ten records and they all matched” proves nothing. The technique that works is checksums over ranges: split the data into intervals, compute an aggregate over each in both stores, compare. A divergence is localised immediately, and only the suspicious interval needs a row-by-row comparison.
select
date_trunc('day', created_at) as bucket,
count(*) as rows,
sum(amount) as total,
md5(string_agg(id::text || ':' || amount::text, ','
order by id)) as checksum
from operation
where created_at < :cutoff
group by 1
order by 1;Shadow reads add a second, independent signal: the system serves the request from the old store, issues the same query against the new one in parallel and compares the results, logging any difference. This catches what data reconciliation cannot — divergence in the query logic: a different sort order, different behaviour on NULL, different rounding.
What actually breaks
Rows almost never go missing wholesale — their meaning does. The list below is drawn from what really turns up during reconciliation.
- Empty string versus `NULL`. In the old database “no value” was written as
'', in the new one asNULL. Filters stop finding rows and reports quietly change their numbers. - Time zones. Timestamps were stored without a zone and implied local time; the new database uses
timestamptz. Operations shift by several hours, and it is only visible at day boundaries — which is to say, in reports. - Money rounding.
floatin the old system andnumericin the new one differ by fractions of a cent, and that accumulates until the totals stop matching. - Auto-increments. The new store starts numbering at one and collides with the migrated identifiers. The sequence is set explicitly, before writes are enabled.
- Uniqueness. The old database accumulated duplicates that the new schema will not accept. They have to be resolved before the transfer — and that is a business decision, not a developer’s.
What a rollback plan looks like
A rollback plan is not “restore from backup”. While dual write is running, rolling back means flipping the read flag back: the old store has stayed complete and current the whole time. That is precisely why the dual-write phase is not wound down the moment reads switch — it is cheap and it buys the ability to return.
- The flag that switches reads, and a tested procedure for flipping it back.
- Reconciliation matching across every interval, not on a sample.
- Shadow reads running under real load with no divergences.
- Monitoring that compares key business figures before and after — not just technical metrics.
- A named person who decides on a rollback, and the threshold at which they do it.
Frequently asked questions
How long does a transition like this take?
From two weeks for a single mid-sized table to several months for a system with hundreds of millions of historical rows. Most of the time goes not into the transfer but into reconciliation and into resolving what it finds: that is where mismatched field meanings surface.
Can we skip dual write?
If the system may be stopped and the data volume is small — yes, and it will be cheaper. Dual write is for cases where stopping is impossible or where the volume makes the transfer take hours. Fintech, healthcare and any system with external obligations almost always fall into that category.
What do we do with the divergences reconciliation finds?
Sort them by type rather than fixing them one by one. A single divergence is nearly always the consequence of a rule that behaved differently; fix the cause and hundreds of rows close at once. Editing individual records before the cause is understood guarantees a repeat.
How do we know the old store can be switched off?
When the new one has run under full load long enough to have captured the rare scenarios — month-end close, quarterly reporting, traffic spikes. We do not advise switching off before a complete reporting cycle: that is usually where a rounding mismatch finally shows up.
Sources
Every claim here can be checked: below are the primary sources, not a retelling.
- 01Martin Fowler — Parallel Change (expand/contract) — the technique the whole transition is built on
- 02PostgreSQL — Transaction Isolation — what the database guarantees when both schemas are written in parallel
These articles are written by engineers working on the projects — but they are not signed by name. The reason is the same one that keeps client logos off this site: nearly every project runs under an NDA or white-label, and a byline on a piece about a payment core points at the client as clearly as a logo would. Instead of names, we stand behind the text with rules — except for articles about our own open-source code, which carry the author’s name.
How we write and what we verify