Recovery Ordering for Multi-Service Estates

Restoring one database is a well-defined operation with a measurable duration. Restoring an estate of forty services is not, because the services depend on each other and the order in which they come back determines whether the result is a working system or a collection of processes that are running and failing.

The dependencies are not the ones drawn on the architecture diagram. Those show which service calls which during normal operation, and they are usually complete for that purpose. A restore exercises a different set of edges: which service must be able to answer before another can start, which must have data before another can serve, and which must issue a credential before another can authenticate.

The graph that restore actually follows

Consider what a single service needs in order to start and serve. It needs a resolvable name, which requires DNS. It needs a certificate that a client will accept, which requires a certificate authority and a clock within the certificate’s validity window. It needs to authenticate its callers, which requires an identity provider or a shared secret. It needs its own configuration, which may come from a configuration service or a mounted file. It needs its data store, which needs its own storage, its own credentials, and its own initialization.

Each of those is a node in the restore graph, and each has its own restore procedure with its own duration. The critical path is the longest chain, and the recovery time objective applies to the end of that chain, not to any individual step. A database that restores in twenty minutes behind an identity provider that takes two hours to restore gives a two-hour-plus recovery, and the database’s speed is irrelevant.

The declared dependency graph and the restore graph also disagree in the other direction. A service may declare a dependency on a message broker because it subscribes to a topic at startup, but be perfectly able to start and serve reads without it. Treating the declared graph as the restore order produces a serialization that is longer than necessary, which is a cost paid on every drill.

Cycles that real designs create

A topological order exists only for a directed acyclic graph, and real estates contain cycles that are not accidents.

The most common is authentication. An application authenticates users against a directory service, which stores its data in a database, which uses the directory for administrative logins. Restoring the database requires an account in the directory; restoring the directory requires the database. The cycle is not a design error; it is the natural consequence of centralizing identity.

The second is naming. Internal DNS is served by a cluster that runs on the same infrastructure it names, and the nodes that host it need names to find each other. The names may resolve through a public zone that is also managed by a service inside the cluster.

The third is secrets. The secret store is itself a service with a database, a certificate, and an unseal mechanism, and its own credentials have to come from somewhere that is not the secret store.

Cycles are broken, not solved. The standard breaks are bootstrap credentials that exist outside the cycle: a local administrative account on the database that is not federated, a static hosts file or a secondary resolver that does not depend on the cluster, an offline copy of the secret store’s unseal material. Each break is a deliberate weakening of the normal authentication model, which means each one is a standing security exception, and the exception has to be scoped to something that cannot be used for general access.

The break also has to be tested. A bootstrap account that was disabled during a routine cleanup is discovered during a restore, at which point the cycle is closed again and the recovery stops.

The credentials needed to restore are inside the restore

The most consequential instance of the cycle is the backup system itself. Its credentials to read the repository, the keys to decrypt the data, and the configuration describing what to restore all tend to be stored in the environment that the restore is meant to rebuild.

This is not only a disaster-recovery problem; it is the ordinary failure mode of a compromised environment, where the attacker has the same access as the restore procedure and can delete both the data and the backups. The mitigation is a copy of the restore-critical material held outside the blast radius: a printed or offline-stored key, a separate account with access only to the backup repository, and a restore procedure that does not require the production identity provider to authenticate its operator.

The same argument applies to the cloud control plane. Restoring into a new account or region requires credentials that can create resources there, and those credentials must not be reachable from the environment that failed. Where the encryption keys live in a managed key service, the ability to decrypt backups depends on that service’s availability and on the key still existing, which is the problem described in key escrow for encrypted backups.

Migrations impose a second ordering

Data restore and code deployment are two sequences that must be interleaved correctly, and the correct interleaving is not the obvious one.

A schema migration that drops a column is incompatible with the code version that still reads it. A migration that adds a non-nullable column without a default is incompatible with the code version that inserts without it. The compatible pattern is expand and contract: add the new structure in a way the old code ignores, deploy code that writes both and reads the old, backfill, then deploy code that reads the new and writes both, then remove the old structure. Each step is independently deployable, and the database is compatible with two adjacent code versions at every point.

During a restore, this ordering reappears as a constraint on what can be done at all. Replaying a log that includes the contraction step onto a base backup taken before it produces a schema that the restored application binary may not match, depending on which binary version was restored. The restore is only coherent if the data and the code are restored to points that were compatible when they were live, which means the recovery plan needs a record of which code version was running at which time, not just a list of backups.

Batch jobs and message brokers add a third ordering. Restoring a broker to a point before the application consumes messages that the application has already processed produces duplicates on the next start; restoring it to a point after loses the messages that were in flight. Since the commit of a message consumption and the side effect it triggers are usually in different systems, no restore point is exactly right. The design response is idempotency at the consumer, keyed on something stable in the message, so that a duplicate is harmless; without it, the restore ordering has a failure mode that cannot be fixed by choosing a better timestamp.

Making the order executable

An ordering that exists as a document is a hypothesis. An ordering that exists as a script is a testable claim, and the difference shows up in the first drill.

The executable form does not need to be sophisticated. It needs to encode the partial order rather than a total sequence, so that independent components restore concurrently and the critical path stays as short as the true dependencies allow. It needs to verify readiness rather than elapsed time, because a service that starts in two minutes on an idle machine may take twenty on a restoring one. It needs to be re-runnable, because a restore that fails at step nine is attempted again from step one or from step nine, and both have to be safe.

The ordering also has a clock component that is easy to omit. DNS answers are cached for their time-to-live, so a client that resolved a name before the failure continues to use the old address for up to that many seconds after the new one is published. A restore that changes addresses is not complete when the server answers; it is complete when the caches have expired. Certificate validity is bounded at both ends, and a restored system whose clock is wrong will reject its own certificates. Neither of these is a dependency between services, and both behave like one during a recovery.

The number of components that must be restored before the estate serves traffic is the number that determines the recovery time, and it is usually larger than the number of components in the backup plan. That gap, rather than the speed of any single restore, is where the recovery objective is won or lost. Verifying that the restored estate actually works, rather than merely starting, is the drill described in checksum verification and restore drills.