Case study: one wrong price, and what a system should do about it

The observation (a human, reading his own page)

On August 14, reviewing my paper portfolio, one row stopped me: QBTS recorded as bought yesterday at $16.21 — a price I remembered from weeks earlier. The stock had traded above $20 for days. No alert had fired. The desk ran ~50 automated checks at the time; every one was green.

The diagnosis (mechanism, not blame)

The recorded fills were real closes from earlier bars, stamped under later dates. Two distinct incidents, separable in the data:

• A feed-wide stale refresh: four unrelated tickers (GFS, HUM, TSLA, MOH) all recorded on 2026-07-29 at exactly their 2026-07-27 closes. Four names landing on one shared prior bar is a whole-feed event, not coincidence.
• A per-name stall: QBTS served one cached value ($16.21, its 07-24 close) on three later dates while the real price moved 35%.

The trading engine was innocent — it faithfully recorded what the feed said. The feed even self-healed days later. But a written row is never re-read, so the error fossilized in a permanent record. Root cause: the desk validated every layer of analysis (benchmarks, clustering, survivorship, effective sample size) and had never validated the raw inputs underneath them.

The correction (and the mistake I made making it)

Fourteen rows were restated to the tape, with every before/after logged. Then I found that my restatement had violated the desk's own standing rule: a closed row keeps the number it was scored with; corrections may be added as new columns, never overwritten. Nobody caught it but the record itself, a day later. I reverted every scored number and carried the tape's values in companion columns (*_tape), so the book now shows both what was scored and what was true — and I amended the integrity check, which as written demanded the impossible (a closed row matching the tape), into the achievable and sharper demand: a divergence must be declared; an undeclared one fails.

The generalization (the part that matters)

The wrong price was never the finding. The finding was a class of defect that then surfaced four more times in one week, wearing different clothes:

1. The price check read entry fills only — the largest error in the book was on an exit (a trade booked −0.01% where the tape said +23.3%).
2. The refresh pipeline had silently omitted its data fetchers for three weeks — every weekly “no drift” verdict had been arithmetic on a frozen panel.
3. The front page's headline figures were hand-typed SVG that nothing recomputed; the “figures are stamped and current” check was green because it audited three other figures.
4. The automation registry was missing five running jobs — alive, unwatched, invisible to the weekly audit if they died.

The fixes (structural, each proven by breaking it first)

• auto_reconcile — every session re-checks the last session's fills against the feed as it has since settled. Open rows repair in place; closed rows only ever gain a declared companion column (the standing rule holds by construction). Proven by corrupting a fill and watching it get caught.
• A frozen-bar gate — entries refuse any name whose last three closes are identical: measured across ~36,900 real bars, that signature occurred zero times naturally, so the gate refuses only what is provably stalled. Proven by freezing a name's closes and watching the run print the refusal by name.
• A coverage audit with a ratchet — an inventory of what every registered check actually reads, which published surfaces no check touches, and which automations the registry doesn't know. It fails if the blind spot ever grows; demanding zero would be permanently red, and permanent red teaches people to ignore red. On its own first run it caught three of its own bugs — including counting its output file as a new blind spot and tripping itself.
• Close-only scoring, expiry-before-check-date guards, vintage-correct attribution joins — the same input-first discipline applied to every other book the week's audit reached.

Outcome

The corrections moved the desk's headline numbers against the strategies being promoted that week (two books had been overstated by the bad fills), and were published that way. Verified prices: 1,750 across both legs, zero undeclared divergences. The check count grew from ~50 to 53, with the new ones pointed at inputs — and at the checks themselves.

What I'd tell a team about this

• Input validation is not beneath analytical work; it is load-bearing for it.
• A test that can never pass is not a standard, and a test that always passes may be pointed at nothing. Both need auditing — mechanically, not by resolve.
• Corrections belong beside records, not over them. An audit trail you can overwrite is a story, not a record.
• The human noticing “that price can't be right” is the last line of defense. The goal of the week's work was to stop being needed there — and the next three input defects were caught by the machine before I saw them.