When the pager goes off, this is the page you open

A venue that breaks quietly is worse than one that breaks loudly. Every condition below wakes somebody up on purpose. For each one: what the watchdog actually measured, what it means for a person holding a position, and the first three things to do. Nothing here needs a decision made at three in the morning.

How paging works

A watchdog runs on the venue host every two minutes. It checks seven families of condition, and each condition it finds is an alert with a stable name. The rules are deliberately boring.

• New alerts page once. An alert that was already open sends nothing, however long it stays open.
• Open alerts remind every six hours. Long enough to sleep through, short enough that nothing is forgotten by Monday.
• Cleared alerts say so. When the condition stops being true, one message arrives with how long it lasted, and the alert is forgotten.
• Alert names carry no numbers. The name is disk_high and the message says 91%. A name with the number in it re-fires on every wobble, which is how an alerting system teaches people to ignore it.
• Anything that can blip gets confirmed once. A single timed-out request is the network. Two, eight seconds apart, is an outage.

Pages go to the same Telegram channel as the rest of the operator's alerting, so there is one place to look. The watchdog itself is a single file with no dependencies, and it exits non-zero when it cannot send, which is what makes a silent pager visible in the cron mail.

api_down

What it measured. Two attempts, eight seconds apart, at /api/v0/health failed, or answered with something other than 200 and JSON.

What it means. Nobody can trade. The terminal shows stale prices and every order fails. This is the loudest thing the watchdog can say.

1. systemctl status openfield-api and journalctl -u openfield-api -n 200. A crash loop names itself in the first twenty lines.
2. If the service is running and the port is closed, check nginx: nginx -t then systemctl status nginx. A bad config after a deploy is the common cause.
3. If the box itself is unreachable, check the provider console before touching anything. A host that is being live-migrated comes back on its own and a restart in the middle makes it worse.

service_down

What it measured. systemctl is-active openfield-api returned something other than active.

What it means. The unit stopped or is restarting. This usually arrives a moment before or after api_down, and it is the more specific of the two.

1. journalctl -u openfield-api --since "-20 min" and read from the bottom. The exit reason is on the last line before the restart.
2. If it exited on a bad release, roll back: the release directory holds the previous build, and deploy/ has the symlink swap.
3. If it exited on state, check the database and the journal chain before restarting: a torn write is repaired at boot, a corrupt one has to be restored from the nightly backup.

engine_behind

What it measured. The matching, risk and funding sweep runs every three seconds and publishes when it last finished. That timestamp is more than thirty seconds old.

What it means. The process answers HTTP and looks alive from outside, while orders sit unmatched, margin is never re-checked and nobody is liquidated. This is the failure mode that no uptime monitor catches, and it is the reason the heartbeat exists.

1. Read /api/v0/health directly. rss_mb climbing with the sweep age is a memory problem; a flat rss_mb with a growing sweep age is a blocked event loop.
2. Look for a synchronous call that got slow: a large journal scan, a disk that has stopped answering, an external fetch without a timeout. The last deploy is the first suspect.
3. Restart the service. A stalled engine is worse than a cold start, and the journal replays on boot. Then find the cause with the log you kept.

engine_recovered

What it measured. On boot the engine found journal records newer than the last state flush and replayed them onto the live accounts. The alert carries how many records, and what it could not apply.

What it means. The process died between two flushes (they run every five seconds) without the shutdown hook. Cash, positions, resting orders and brackets from the gap are back; TWAP schedules from the gap are not, and an account whose opening record fell in the gap is listed as unrecovered.

1. Read recovery on /api/v0/health. skipped should be empty or only TWAP records.
2. Read /api/v0/journal/rebuild. Zero diverging accounts means the replay landed exactly; otherwise the rebuild_diverging alert follows and names them.
3. Find why the process died: journalctl -u openfield-api around the boot. An OOM kill or a segfault here is the real incident; the recovery is only its receipt.

chain_broken

What it measured. The journal recomputed from genesis does not match its own hashes, or a sequence number is missing.

What it means. A record was rewritten, dropped or torn in the middle of a day file. This is the one alert that questions the venue's own word; nothing else is trusted until it is explained.

1. Read /api/v0/journal/verify: broken.at is the sequence, reason is hash_mismatch or sequence_gap.
2. Compare that day file with the nightly backup. A torn final line is tolerated on boot; a torn interior line is not.
3. Do not edit the file. Restore the day from the backup, restart, verify again, and write the incident up on the status page.

money_leak · funding_leak

What it measured. The cash identity on /api/v0/invariants (what accounts and funds hold against what was granted, minus fees and net funding, plus realised profit) has a residual over a dollar, or accounts paid funding that the fund did not receive beyond the frozen legacy figure.

What it means. Money appeared or vanished without a record. Funding runs on one clock for the venue, so a growing unmatched figure means a code path charged one side without the other.

1. Note the residual and whether it grows between two runs. A constant residual is a historical artifact; a growing one is live.
2. Look at the newest of_fund records and the last hour's funding records for a market with one-sided payments.
3. Stop new risk if it grows: set the breaker with the admin console, find the path, fix, and let the identity settle before reopening.

rebuild_diverging

What it measured. Accounts rebuilt from the journal alone differ from the live state by a dollar or more.

What it means. Either the store missed a record the chain has (a crash before the catch-up existed, or a record type the replay does not know) or the chain missed something the store did. The rebuild names the accounts and the size.

1. Read the rows on /api/v0/journal/rebuild. Old accounts can disagree by their pre-v0.11 close-out fees; that is documented and does not grow.
2. A new divergence after a boot recovery means the replay skipped a record type; the health recovery.skipped map says which.
3. Correct the live state by hand only from the chain, and record the correction as its own event.

bad_debt · fund_empty

What it measured. The bad debt paid by the funds since genesis rose since the last watchdog run, or the funds are empty.

What it means. A liquidation ended below zero and the waterfall reached the fund. Expected on a gap; worth reading every time.

1. Read the of_fund records with kind: bad_debt and the account's liquidation steps before them.
2. Check the market's regime and the jump that caused it on the divergence page; if the maintenance band was too thin for the move, that is a parameter change, on the chain.
3. If the fund is empty, the next shortfall is deleveraged onto winners. Say so on the status page before it happens.

The watchdog itself

Every run writes its state file; the API publishes its age as watchdog_age_s on /api/v0/health and openfield_watchdog_age_seconds on /api/v0/metrics. The operator's hourly healthcheck on the other box pages when that age passes fifteen minutes, so a dead cron here is noticed there.

feeds_stale

What it measured. Two or more price sources are disconnected, or have sent nothing for three minutes, out of the seven the venue subscribes to.

What it means. Confidence falls, markets slide toward Divergence and Degraded, and leverage drops for everybody holding a position. The venue is behaving correctly here. It is degrading on purpose, which is the whole design, and the alert exists so that the cause gets fixed rather than the symptom tolerated.

1. Open /status/ and read the per-venue health. One exchange dark is an exchange problem, five dark at once is our network or our egress IP.
2. Check whether the venue is rate limiting us or has changed its symbol list. The error field on the feed carries the last socket message.
3. Do not force confidence back up. If markets need to keep trading with fewer sources, that is a config decision made awake, in daylight, and written down.

markets_unavailable

What it measured. At least one market is in the Unavailable regime, which closes trading on it.

What it means. Somebody with a position on that market cannot reduce it. Usually it follows feeds_stale on a thin instrument where only one or two sources ever quote.

1. Find which markets, from /status/ or the console overview.
2. Check whether a degradation drill is still active in the console. A drill left injected looks exactly like an outage.
3. If the underlying source has gone away for good, the market needs a new reference rather than an override.

error_rate

What it measured. More than twenty error lines in the service journal in ten minutes. The page names the three most common.

What it means. Something is failing repeatedly and being swallowed. A venue that logs a thousand handled exceptions an hour has a bug that is one bad input away from being an outage.

1. Read the three named errors. A repeated TypeError in one code path is a bug shipped in the last release; a repeated network error is a dependency.
2. journalctl -u openfield-api --since "-30 min" | grep -c Error to see whether it is rising or flat. Rising with traffic is usually one caller looping.
3. If it started at a deploy, roll back first and diagnose after. The chain and the database survive a rollback.

disk_high and memory_high

What it measured. The root filesystem is 85% full, or 90% of memory is in use.

What it means. Both end the same way. A full disk stops the journal appending, which stops the venue accepting anything, and it takes the backups with it. Memory pressure ends in the kernel choosing which process to kill, and it usually chooses the biggest one.

1. df -h / and du -xh --max-depth=2 /var /opt | sort -h | tail -20. Old releases and the journal day files are the usual growth.
2. Check for deleted but still open files: lsof +L1 -nP | awk '$7+0 > 1e9'. A daemon holding an unlinked file keeps the space allocated while du shows nothing.
3. For memory, compare rss_mb in /api/v0/health against its usual figure. A slow climb across days is a leak and needs a restart plus a heap snapshot, rather than a bigger box.

cert_expiring, cert_expired and cert_unreadable

What it measured. A TLS handshake with the site and with the app host, reading the expiry from the certificate that was actually served.

What it means. Ten days of warning is deliberate. Automatic renewal has already had several chances by then, so an alert here means renewal is broken rather than late. An expired certificate makes the venue unusable in every browser, with no way for a visitor to click past it.

1. certbot renew --dry-run. It names the failing challenge, which is usually a DNS or a webroot problem after a config change.
2. Confirm the served certificate is the renewed one: openssl s_client -connect openfield.markets:443 -servername openfield.markets </dev/null | openssl x509 -noout -dates. A renewed file that nginx never reloaded is a common and invisible failure.
3. cert_unreadable on its own, with the API healthy, usually means the handshake was blocked rather than the certificate missing. Check the firewall and the proxy before touching the certificate.

Mail, which does not page

Deliverability lives in the operator console rather than the pager, because it degrades over weeks instead of minutes. The mail panel in the console shows sends, bounces, complaints and every suppressed address. Two numbers matter: a bounce rate above 2% and a complaint rate above 0.1% are the thresholds where providers start filing the domain under junk.

• An address that hard bounces or reports a message as spam is suppressed permanently, and the venue stops mailing it. Only the operator can undo that, from the console, and the undo is written into the chain.
• Every address has a cap on how much mail it can receive per hour and per day, so a loop in a caller stops at the cap instead of at the recipient's patience.
• While the sending domain is new, bulk mail lives under a daily ceiling that widens on a schedule. Mail a person is actively waiting for, a verification code or a security notice, goes out regardless.

If the pager has been quiet for a suspiciously long time

A silent alerting system and a healthy venue look identical from the outside. Once a month, run --test and confirm the message arrives. It costs one line and it is the only way to tell the two apart.