Rebuild, Don't Merely Restore: Attesting the CI/CD Path to Recoverable Software

Recovery plans tend to privilege the data tier. They describe backup cadence, restore order and reconciliation checks, then assume that application code will somehow be available when needed. In many UK enterprises that assumption is weaker than it looks. The source may be intact, but the pipeline that turns it into a deployable artefact can be tied to the same site, cloud region, identity provider or supplier that has just failed. The recovery question is therefore not only whether backups can be restored. It is whether the organisation can rebuild and verify its software from source in a clean environment, with the same dependency set, build logic, signing authority and runtime configuration it relied on before the incident.

The recovery question behind the restore question

A restored virtual machine or database is a starting point, not a recovered service. If the application binary must be rebuilt after a site loss, several dependencies come into play at once: source repositories, package mirrors, runner capacity, build images, artefact repositories, secret stores and code-signing keys. Each can be lost, degraded or contractually constrained. A pipeline that worked continuously in normal operations may never have been tested as a recovery capability.

This is a distinct control from backup restoration. Backups prove that bytes were held. Reproducibility proves that a known-good software product can be recreated and verified. For UK technology leaders, the distinction matters because operational resilience commitments are framed around important business services, not around infrastructure components. A service that depends on an unreproducible build is not resilient merely because its database can be recovered.

What reproducibility means in evidence terms

Reproducibility should not be reduced to a developer convenience. In a recovery context it means the organisation can demonstrate four things: the source revision is identifiable; the dependency set is pinned and obtainable; the build environment is reconstructable; and the resulting artefact can be verified against expected provenance.

The evidence should be concrete. A build manifest might record the commit hash, lockfile digest, builder image digest, build arguments, environment variables that affect output, and the identity of the pipeline that produced the artefact. A software bill of materials supports dependency visibility, but it is not a substitute for proving that the dependencies can actually be fetched during recovery. Signed provenance attestations are useful only if the verification keys and trust roots remain available when the primary estate is not.

The practical test is a clean-room rebuild: take the recorded inputs, rebuild in an isolated environment that does not depend on the failed site, and compare the output or its attested provenance with what production expects. That test should be recorded, timed and reviewed. It is the closest evidence an enterprise can produce that its software supply chain is recoverable.

Five domains to attest

Source and dependency availability. Repositories and package registries are often hosted externally, but access may depend on single sign-on, network paths or contractual entitlements that are disrupted in a recovery scenario. Internal package feeds and mirrored registries should be tested for availability, completeness and checksum integrity. Evidence should show that locked dependencies can be resolved without reaching an unavailable public service or an untested mirror.

Runner and orchestration capacity. CI runners are frequently treated as elastic and therefore invisible. In recovery, capacity may be constrained by cloud quotas, network egress, licence limits or the loss of a self-hosted runner estate. A fallback runner pool should be defined, with the container images, plugins and toolchains it requires. Evidence should include a capacity test at the recovery location and a record of the build duration achieved under realistic load.

Artefact repositories and signing. Artefact stores need retention, immutability and replication appropriate to recovery. More critically, signing and verification keys need custody arrangements that survive the loss of a primary key management service. Hardware security modules, offline root keys and break-glass procedures should be documented and tested. An artefact that cannot be verified after recovery is of limited operational value.

Pipeline secrets and identity. Build pipelines use service accounts, tokens, certificates and federated identities. These are often bound to a directory service or secrets manager in the primary estate. The evidence pack should identify which pipeline identities must be recreated, who can approve their recreation, and how secrets are provisioned without weakening controls. A break-glass route that bypasses normal approval may be necessary, but it should be designed and tested rather than improvised.

Clean-room rebuild execution. The final domain is the test itself. A clean-room rebuild should use only the recorded source, dependency mirrors and build definitions, in an environment isolated from the production pipeline. The record should capture date, scope, participants, build duration, verification result and any manual interventions. Manual steps are not automatically unacceptable, but they must be known, owned and rehearsed.

A UK evidence pack for the board

For UK regulated enterprises, the evidence pack should be concise enough for governance and detailed enough for engineers. A useful structure includes the scope of services covered, the build manifest standard, dependency mirror attestations, runner capacity results, key custody statements, clean-room rebuild records and an exception register with remediation dates.

Ownership should sit with platform engineering, but accountability should be shared with the resilience function. The pack should map to important business services and to third-party dependencies, because CI/CD platforms, artefact repositories and package registries are often suppliers in their own right. UK operational resilience expectations increasingly require firms to understand dependencies beyond the immediate tier one, and the build pipeline is a dependency that is easy to overlook because it sits behind the application.

Metrics should be honest rather than flattering. Useful measures include the percentage of tier-one services with a successful clean-room rebuild in the last twelve months, median rebuild time against recovery time objectives, dependency mirror coverage, signing verification success after restore, and the age of unresolved exceptions. A low score is not a failure of the report; it is the reason the report exists.

Cadence and consequence

A clean-room rebuild should be performed at least annually for critical services and after significant changes to build tooling, identity providers, artefact repositories or key custody. The test should not be run only by the team that designed the pipeline, because familiarity can conceal undocumented steps. It should include a verification stage that confirms the rebuilt artefact is the one the recovery plan expects.

The conclusion for UK leaders is straightforward. Service recovery depends on a pipeline that can be reconstructed, not merely on backups that can be restored. A backup proves that data was retained. A reproducible build proves that the organisation can recreate the software that makes the data useful. In 2026, with operational resilience under sustained board scrutiny, that distinction deserves a place in the evidence pack.