Backup, Restore & Disaster Recovery Runbook

Recover Cloud Health Office without guessing: identify the authoritative data, select a safe recovery point, restore into isolation, reconcile asynchronous work, validate healthcare correctness, and record evidence against approved RTO and RPO.

1. Recovery Principles

1. Declare the source of truth. Recover durable business records before caches, projections, queues, or generated output.
2. Restore into isolation first. Never overwrite the only surviving copy while investigating corruption or accidental deletion.
3. Preserve the incident boundary. Record the last known-good UTC time, first known-bad write, release SHA, configuration, tenant scope, and affected identifiers.
4. Prefer replayable state over queue recovery. Service Bus is transport, not the system of record. Rebuild work from persisted events or an approved reconciliation source when messages cannot be recovered.
5. Prove application correctness. A successful provider restore is incomplete until claim versions, member coverage, benefit configuration, payments, audit history, tenant isolation, and message reconciliation pass.
6. Separate recovery from failback. Stabilize the recovered environment before returning traffic or attempting to restore the original topology.

2. Approve RTO and RPO Before the Incident

RTO is the maximum acceptable time to restore service. RPO is the maximum acceptable data-loss window. The values below are planning fields, not product guarantees or contractual service levels.

Capability	Business owner	RTO	RPO	Recovery source
Claim intake and adjudication	________	________	________	Claims database, append-only claim events, source 837 files, Service Bus state
Membership and coverage	________	________	________	Member database plus accepted 834 or upstream enrollment source
Benefits, providers, and pricing	________	________	________	Configuration database plus approved source files
Payments and remittance	________	________	________	Financial records, immutable audit evidence, bank or payment reconciliation
Platform configuration and secrets	________	________	________	Git, infrastructure as code, Key Vault, approved configuration register

3. Capture the Protection Inventory

Complete this inventory for every environment. A blank or assumed cell is a no-go for production traffic.

Asset	Repository posture	Production evidence required
Local MongoDB	Single MongoDB 6.0 StatefulSet, one PVC, no scheduled backup job in the local bootstrap	Development export when needed; do not present this as production DR
Cosmos DB for MongoDB	Free-tier benchmark module is single-region with automatic failover disabled; backup policy is not declared in that module	Observed backup mode and retention, latest restorable timestamp, restore permissions, completed isolated restore
Cosmos DB for NoSQL	Current module is single-region with automatic failover disabled; backup policy is not explicitly declared	Observed backup mode, region design, recovery point, network/RBAC reconstruction, completed restore
Azure Service Bus	Standard tier in the main template; claim messages have TTL, duplicate detection, retry, and DLQ controls	Entity inventory, replay source, DLQ evidence, regional recovery design, reconciliation procedure
Kubernetes	Deployments and services are reproducible from Git and deployment automation; some workloads use PVCs	Release SHA, image digests, manifests/configuration, cluster backup scope, alternate-cluster restore test
Azure Key Vault	Application vault enables soft delete, 90-day retention, and purge protection	Recovery permissions, secret/key inventory, rotation source, recovery test without exporting secret values to evidence

NAMESPACE='cloudhealthoffice'
EVIDENCE_DIR="recovery-evidence-$(date -u +%Y%m%dT%H%M%SZ)"
mkdir -p "$EVIDENCE_DIR"

git rev-parse HEAD > "$EVIDENCE_DIR/release-sha.txt"
kubectl get deployment,statefulset,pvc -n "$NAMESPACE" -o wide \
  > "$EVIDENCE_DIR/kubernetes-inventory.txt"
kubectl get configmap -n "$NAMESPACE" \
  > "$EVIDENCE_DIR/configmap-inventory.txt"

Do not export Kubernetes Secret objects or decrypted credentials into the evidence directory.

4. Recover Local or Self-Managed MongoDB

For local development, create a logical export before a destructive experiment. Production self-managed MongoDB requires an approved replica-set, backup, encryption, retention, monitoring, and off-cluster storage design beyond the local manifest.

Create a local development export

NAMESPACE='cloudhealthoffice'
MONGO_POD="$(kubectl get pod -n "$NAMESPACE" \
  -l app=mongodb -o jsonpath='{.items[0].metadata.name}')"

kubectl exec -n "$NAMESPACE" "$MONGO_POD" -- sh -c \
  'mongodump --archive=/tmp/cho-mongo.archive.gz --gzip \
    --username "$MONGO_INITDB_ROOT_USERNAME" \
    --password "$MONGO_INITDB_ROOT_PASSWORD" \
    --authenticationDatabase admin'

kubectl cp \
  "$NAMESPACE/$MONGO_POD:/tmp/cho-mongo.archive.gz" \
  "./cho-mongo.archive.gz"

Encrypt the archive, move it outside the cluster, record its checksum and UTC creation time, and delete the pod copy. A backup that remains only on the same laptop or cluster does not cover device or cluster loss.

Restore into an isolated target

test -n "$ISOLATED_MONGO_URI"
mongorestore \
  --uri "$ISOLATED_MONGO_URI" \
  --archive="./cho-mongo.archive.gz" \
  --gzip \
  --drop

Use a dedicated recovery identity, rotate any credential exposed during the exercise, and retain only sanitized command output as evidence.

5. Recover Azure Cosmos DB

Cloud Health Office supports both the Cosmos DB native SDK and Cosmos DB for MongoDB through the MongoDB driver. Recovery is an account capability, not a driver behavior. Query the deployed account rather than assuming its backup mode from the application connection string.

az cosmosdb show \
  --resource-group "$RESOURCE_GROUP" \
  --name "$COSMOS_ACCOUNT" \
  --query '{
    kind:kind,
    backupPolicy:backupPolicy,
    locations:locations[].locationName,
    automaticFailover:enableAutomaticFailover,
    multipleWrites:enableMultipleWriteLocations,
    publicNetworkAccess:publicNetworkAccess
  }'

Required production posture

• Choose periodic or continuous backup deliberately and document its retention, storage redundancy, cost, and RPO impact.
• For point-in-time recovery, enable an approved continuous-backup tier and confirm the latest restorable timestamp before every drill.
• Grant restore discovery and execution permissions to a controlled break-glass identity, then test them before an incident.
• Restore to a new account or use an explicitly approved same-account restore path; preserve the source until validation is complete.
• Reapply settings that point-in-time restore does not reproduce, including applicable firewall, virtual-network, private-endpoint, region, and data-plane RBAC configuration.
• Rebuild secrets and application configuration to reference the recovered endpoint without committing connection strings.

az cosmosdb update \
  --resource-group "$RESOURCE_GROUP" \
  --name "$COSMOS_ACCOUNT" \
  --backup-policy-type Continuous \
  --continuous-tier Continuous30Days

Changing backup policy affects cost and recovery behavior. Review it through change control, then follow Microsoft’s current continuous-backup account restore procedure or same-account restore procedure. Record the source account, recovery timestamp in UTC, target account, restored resources, and job result.

6. Recover Azure Service Bus and Asynchronous Work

The main infrastructure template provisions Standard tier. Standard namespaces do not provide native multi-region Geo-Replication or metadata Geo-Disaster Recovery. If production RTO/RPO requires regional messaging recovery, design and fund that capability explicitly.

1. Freeze producers when safe. Prevent new work from widening the recovery boundary.
2. Capture entity state. Record active, scheduled, transfer, and dead-letter counts plus entity configuration.
3. Recover the namespace. Reprovision entities and filters from infrastructure as code, or invoke the approved Premium failover procedure.
4. Repoint identities and endpoints. Restore RBAC, private endpoints, DNS, and application secret references; validate Send and Listen separately.
5. Reconcile transport with durable state. Compare accepted claim versions and other source records against completed outcomes.
6. Replay only missing work. Preserve deterministic message identity and persisted-result checks. Never bulk-replay a DLQ before correcting its cause.

az servicebus topic subscription show \
  --resource-group "$RESOURCE_GROUP" \
  --namespace-name "$SERVICEBUS_NAMESPACE" \
  --topic-name claim-version-events \
  --name adjudication-orchestrator \
  --query '{
    active:countDetails.activeMessageCount,
    deadLetter:countDetails.deadLetterMessageCount,
    transferDeadLetter:countDetails.transferDeadLetterMessageCount,
    maxDelivery:maxDeliveryCount,
    lockDuration:lockDuration
  }'

Use the 837 Intake & Adjudication Operations Runbook for claim-level tracing, retries, dead letters, deterministic message IDs, and recovery verification.

7. Recover Kubernetes Configuration and Secrets

Rebuild stateless workloads from the reviewed Git SHA, pinned image digests, infrastructure templates, and deployment automation. Back up Kubernetes cluster state only where it contains authoritative resources or persistent volumes that cannot be reproduced safely.

• Record the release SHA, container digests, Helm values or manifests, feature flags, replica counts, autoscaling limits, and environment-specific configuration.
• Use Azure Backup for AKS only after its backup extension, vault, policy, identity permissions, supported CSI volumes, and alternate-cluster restore have been validated.
• Keep database-native recovery as the primary plan for databases; do not assume a crash-consistent PVC snapshot is application-consistent.
• Restore network policies, ingress, certificates, workload identity, private DNS, monitoring, alerting, and policy assignments before traffic.
• Keep secret values in Key Vault or the approved secret system. Evidence should contain secret names, versions, expiry, and recovery status—not values.

Key Vault recovery checks

az keyvault show-deleted --name "$KEY_VAULT_NAME"
az keyvault recover --name "$KEY_VAULT_NAME"

az keyvault secret show-deleted \
  --vault-name "$KEY_VAULT_NAME" \
  --name "$SECRET_NAME"
az keyvault secret recover \
  --vault-name "$KEY_VAULT_NAME" \
  --name "$SECRET_NAME"

The application Key Vault module enables soft delete with 90-day retention and purge protection. Recovery still requires the correct control-plane permissions, restored network access, workload identity, and application verification.

8. Execute an Incident Recovery

Phase	Required actions	Exit criterion
Declare	Open incident record; name commander, recovery lead, data validator, security lead, and business decision maker; start UTC timeline.	Authority and scope are explicit.
Contain	Stop unsafe writes, preserve source data and logs, restrict access, identify last known-good time and affected tenants.	Recovery boundary is no longer widening.
Select	Inventory available recovery points, quantify potential data loss, obtain business approval, and choose an isolated target.	Recovery point and expected RPO breach are recorded.
Restore	Recover data first, then platform configuration, secrets, networking, workloads, and asynchronous transport.	Isolated environment is technically healthy.
Validate	Run structural, domain, tenant, financial, and message reconciliation checks; security reviews access and exposure.	Named validators sign the evidence.
Cut over	Freeze recovered data, apply final delta if designed, update endpoints, run smoke tests, gradually resume traffic, watch rollback triggers.	Business owner approves service restoration.
Close	Rotate temporary access, preserve evidence, monitor for delayed failures, document actual RTO/RPO, and assign corrective actions.	Incident review and follow-up owners are scheduled.

9. Validate Healthcare Data After Restore

Use counts as a starting point, not the final proof. Sample records across tenants and lifecycle states, and compare them with independent source evidence.

• Structural: expected databases, collections or containers, indexes, partition keys, schema versions, and migrations are present.
• Claims: claim version chains are complete; accepted 837 claims reconcile to persisted outcomes; paid or adjusted claims retain audit history.
• Membership: members, coverage spans, identifiers, and accepted enrollment transactions agree without impossible gaps or overlaps.
• Benefits and providers: effective dates, network assignments, fee schedules, authorization rules, and configuration versions match the recovery point.
• Financial: payment batches, remittances, totals, ledger references, and external settlement records reconcile; no payment is duplicated.
• Messaging: durable accepted work is classified as completed, pending, safely replayable, or explicitly excepted; DLQs contain no unexplained messages.
• Tenancy and privacy: tenant scoping, encryption, audit access, and representative authorization checks pass before user traffic.
• Operations: health checks, dashboards, logs, traces, alerts, backup jobs, and the next scheduled recovery point are working.

10. Run and Record a Restore Drill

1. Choose one production-like but non-production environment and publish the scenario, scope, approved disruption, and safety boundaries.
2. Start the RTO clock, capture inventory, and select a recovery point without assistance from undocumented knowledge.
3. Restore data and platform components into an isolated target using the same identities and controls expected during an incident.
4. Run the complete validation checklist, reconcile asynchronous work, and exercise the cutover decision without routing real production traffic.
5. Record actual RTO, achieved RPO, manual steps, access failures, missing telemetry, correctness exceptions, cost, and operator handoffs.
6. Assign every gap an owner and due date. Repeat the failed portion; a document review alone does not close a failed restore control.

Evidence	Value
Scenario and drill ID	________________________________________
Source and isolated target	________________________________________
Recovery point in UTC	________________________________________
RTO target / actual	________________________________________
RPO target / achieved	________________________________________
Validation result and exceptions	________________________________________
Approvers and evidence location	________________________________________
Next drill date	________________________________________

Related Guides