The deploy script that deleted its own releases
For an unknown stretch of the build, shipping and having shipped were different things, and nothing in the system said so.
The site deploys the ordinary way. Files sync into a timestamped release directory, a symlink flips to point at it, and old releases are pruned so the disk does not fill. Roughly fifteen lines of shell. It ran dozens of times without complaint.
The pruning line sorted releases by modification time and deleted everything past the fifth.
Directory modification times are not creation times, and rsync -a preserves timestamps by design. That is the entire bug. A brand new release directory was stamped with the modification time of the source directory it was copied from, which was frequently older than the releases already on disk. Sorted newest first, the release we had just created sorted last, fell past the cutoff, and was deleted.
The symlink pointed at a directory that had been removed a second earlier, by the same script, in the same run.
Why nobody noticed
Because every individual step succeeded. The sync worked. The symlink flipped. The prune removed exactly what it was told to remove and exited zero. There was no error to see, no alert to fire, and no log line that would look wrong to a person reading it.
The observable symptom was subtle in the worst way: a change would ship, and the page would look the same. The obvious first suspicion is a browser cache. So you hard-refresh, and sometimes it appears, because sometimes the source timestamp happened to sort favourably and the release survived. An intermittent failure that resolves when you retry is one of the most expensive shapes a bug can take, because it trains you to retry instead of to look.
It was found while investigating something else entirely, by listing the release directory and noticing the newest release was missing and the symlink was dangling.
The fix, and the better fix
The immediate fix is one line: sort by name rather than by time. The directories are named with a UTC timestamp, so the name is the only field guaranteed to reflect creation order. We also touch the directory after sync, so its timestamp stops lying about what it is.
The better fix is not about sorting. It is that a deploy which cannot be verified is not a deploy. The script now asserts that the symlink target exists and that a known marker in the shipped files is actually being served over HTTPS afterwards, and fails loudly if not.
What we changed
| Before | After |
|---|---|
| Prune by modification time | Prune by name, with an explicit touch after sync |
| Exit zero if each step exits zero | Verify the symlink target still exists |
| No post-deploy check | Fetch the live URL and confirm the release actually serves |
Why this is on the research page
Because it is the same failure mode the venue itself is built to avoid, in a much cheaper setting.
A trading engine that reports a price without reporting how much it trusts it has exactly this property. Every step succeeds. Nothing errors. The number appears and looks entirely normal, and the fact that it is being derived from one stale source instead of three fresh ones is invisible until the moment it costs somebody a position. That is why confidence, source count, quote age and the leverage derivation are printed on the screen instead of held privately in a risk system.
The failures worth engineering against are not the ones that throw. They are the ones that return success.
We got a free lesson from a deploy script. The equivalent lesson from a margin engine is not free, and the people who pay for it are not the ones who wrote it.