Trust · Policies

DOC-21

Continuity, backup and restore

What this service does to survive disruption, what it copies, how a copy is proven to restore, and what it honestly cannot do.

Record for
  • 27001 Annex A.5.29
  • 27001 Annex A.5.30
  • 27001 Annex A.8.13

Review. Reviewed after every restore drill, after any failed scheduled copy, and otherwise at least once every 90 days. Reissued by any deployment that changes what is copied or how often.

What disruption means here

This service is offered with no availability commitment, and that is stated to players rather than buried. So continuity here is not about keeping the game up. It is about not losing the evidence — the receipt chain, the action log, the incident record and the profiles — because the evidence is what the register is built on and the game is not.

Two disruptions are planned for. A platform outage takes the game and every evidence route down together, and nothing this service can build would change that; it is carried by the provider and recorded as risk R-04 with the residual accepted. Loss or corruption of the shared durable instance destroys evidence, and that one is this service's own problem. It was, until this version, entirely untreated.

What is copied

Everything the shared durable instance holds, in one export covering both places that state actually lives: the object's keys, and the two tables beneath them holding the action log and the receipt chain. An export covering only one of the two would restore to a service that looked intact and had quietly lost its evidence, which is a worse outcome than an obvious failure.

The export is digested over a canonical form of the state it carries — keys sorted at every depth, rows in primary-key order — and deliberately not over the moment of capture. That is what makes two exports of unchanged state hash identically, which is the property the restore drill depends on entirely.

Schedule, retention and failure

A copy is written daily to the bound object storage bucket by a scheduled trigger, under a timestamped key, alongside a most-recent copy at a fixed key. Thirty dated copies are retained; older ones are pruned in the same run. The copies carry every profile row and are therefore operator-only, as recorded in DOC-18.

Each run receipts its own outcome into the receipt chain — the same chain the copy protects. A failed copy writes a receipt saying it failed and surfaces the reason on the public status page. A scheduled job that fails silently is how a backup gap goes unnoticed for months, and it is the specific failure this arrangement is built to make impossible.

The restore drill, which is the part that makes this evidence

A copy nobody has restored is a claim. The drill reads the most recent copy back out of object storage, restores that copy into a scratch instance, exports the instance, and compares its digest against the copy's. Equal digests mean the stored copy reconstitutes the state it was taken from rather than merely something like it — a comparison of every key and every row at once, not a spot check.

The stored copy is deliberately the thing under test. If no bucket is bound, or there is no copy in it, the drill fails and says which, rather than falling back to exporting the live object and restoring that. That fallback is what an earlier version of the drill did, and it proved only that the object could round-trip its own state — nothing about object storage — while this document and the status page both said the most recent copy was what had been restored. The drill also reports whether the copy still matches live state, which is the age of the copy rather than a fault, so it is stated and not failed on.

Live state is never written to, so the drill is safe to run while the game is up. The scratch instance is wiped afterwards whether the drill passed or failed, because it holds a full copy of every player profile and an orphaned copy of personal data is not made acceptable by having been created to test a backup.

The first drill failed, and it failed for a real reason: the digest covered the moment of capture as well as the state, so no two exports could ever have matched and no drill could ever have passed. That is recorded as nonconformity N-05 and the failing drill sits on the public chain immediately before the passing ones. A restore capability whose first recorded test is a pass should invite the question of whether it was ever tested at all.

The result of the most recent drill — pass or fail, when, and what it covered — is published at /status/#backup, in shape and timing only. No exported content appears on a public route.

Recovery, and what is still not claimed

Recovery is deliberate rather than automatic. The restore path exists as an operator action against a named copy; nothing restores by itself, because an automatic restore triggered by a misread signal would overwrite good state with old state.

No recovery time or recovery point objective is committed to, and none should be inferred. What can be stated factually is the recovery point the schedule produces — at most twenty-four hours of state, being one day's action-log rows, receipts and profile changes — and that the restore of a copy of the current state has been measured in single-digit milliseconds against a local instance. Neither figure is a commitment, and neither has been exercised against production.

Still not claimed: this has never been executed as a real recovery, only as a drill. There is no second region and no failover, so a platform outage remains untreated by design. Redundancy of the underlying processing facilities is the provider's, not this service's.

What this changes about leaving

The exit strategy in DOC-19 previously described shutting the service down, and — because nothing exported state anywhere — it amounted to a plan to abandon the evidence at the moment someone would most want to read it. That is no longer the case. The same export that backs up state is a complete, portable, digested copy of it in a documented format, so the evidence can leave with the service rather than ending with it.