Production Readiness Checklist
Turn a technically successful deployment into an evidence-backed go-live decision with named owners, tested recovery, production-sized dependencies, measurable service objectives, and a rehearsed rollback.
A successful local Kubernetes or Million Claim Challenge run is valuable engineering evidence, but it does not prove production identity, network, backup, restore, privacy, support, or change-management controls.
How to Use This Checklist
Copy each item into the customer’s change record or launch workbook. Record an owner, evidence link, verification date, and exception decision. An unchecked required item blocks go-live unless the accountable risk owner signs a time-bounded exception.
| Gate | Meaning | Required evidence |
|---|---|---|
| Required | Must pass before production traffic. | Test result, configuration export, dashboard, approval, or recovery record. |
| Conditional | Required when the named capability is enabled. | Capability-specific security, support, and failure-mode evidence. |
| Observed | Validated during the launch observation window. | Timestamped operational metrics and named reviewer. |
1. Ownership, Scope, and Change Control
- Production scope is written down. Tenants, lines of business, trading partners, claim types, APIs, payment functions, and optional integrations are explicitly in or out.
- Every critical component has an owner. Application, Kubernetes, database, Service Bus, identity, network, clearinghouse, security, compliance, and claims operations contacts are named.
- Service objectives are approved. Availability, intake timeliness, adjudication completion, queue age, RTO, and RPO targets are measurable.
- The release is immutable and traceable. Commit SHA, container digests, infrastructure version, Helm values, and tenant configuration are attached to the change record.
- The deployment window and communications plan are approved. Business owners, support, on-call responders, vendors, and rollback decision makers know the schedule.
- Open defects are dispositioned. No unresolved critical issue remains; accepted risks have an owner, expiry date, and mitigation.
2. Infrastructure and Network
- Subscription and regional quota are confirmed. AKS node families, public IPs, load balancers, storage, Cosmos DB or MongoDB, and Service Bus can scale to the tested target.
- Production is isolated from development and UAT. Resource groups, identities, secrets, databases, queues, storage, DNS, and telemetry do not share mutable production state.
- Resource requests, limits, probes, and disruption budgets are set. Critical services tolerate node maintenance and avoid uncontrolled eviction or noisy-neighbor pressure.
- Ingress, DNS, TLS, and certificates are validated. Public and private endpoints match the intended trust boundary; expiration monitoring is active.
- Network access defaults to deny. Required east-west service paths, Azure private endpoints, egress destinations, and administrative access are documented and tested.
- Autoscaling has safe bounds. Minimum availability, maximum cost, downstream capacity, and scale-up behavior have been tested rather than inferred.
- Infrastructure What-If or equivalent shows no unexpected deletion. The reviewed plan is attached to the release approval.
3. Identity, Secrets, and Privacy
- Human access uses the production identity provider. Microsoft Entra ID tenant settings, redirect URIs, MFA, role mappings, break-glass access, and termination procedures are tested.
- Workloads use scoped identities. Service-specific Workload Identity or managed identity replaces shared administrator credentials wherever supported.
- Secrets originate in Azure Key Vault. No placeholder, local-development, trial, or example credential remains in a production pod or pipeline variable.
- Secret permissions follow least privilege. Each service can read only the keys and external resources it needs; production operators cannot casually export plaintext secrets.
- Rotation is rehearsed. Database, Service Bus, SFTP, API, encryption, signing, and third-party credentials can rotate without an undocumented outage.
- Encryption is verified in transit and at rest. Certificates, database settings, storage, backups, queues, and external connections meet the customer security baseline.
- PHI-safe logging is reviewed. Request bodies, X12 payloads, member identifiers, tokens, and secrets do not appear in routine logs, traces, screenshots, or alerts.
- Security and dependency scans are dispositioned. Critical findings block launch; accepted findings carry an owner and remediation date.
- Audit retention and access are approved. Identity, deployment, application, Key Vault, database, and operator actions are queryable for the required period.
4. Data Stores, Backup, and Recovery
- The production backend is explicit. MongoDB or Cosmos DB for MongoDB is configured consistently across services; no unintended Cosmos fallback or local in-memory store remains.
- Driver compatibility is proven against the selected service. Queries, indexes, transactions, retry behavior, serialization, and concurrency semantics have run against the exact production database type and version.
- Capacity is based on a representative test. Free-tier results are not used as a production sizing baseline; throughput, storage, working set, connections, and tail latency have headroom.
- Tenant isolation is verified. Partition keys, repository filters, indexes, and support queries prevent cross-tenant reads and writes.
- Indexes and retention policies are deployed. Required unique, compound, TTL, and audit/event indexes exist before traffic begins.
- Backups are active and monitored. Frequency, retention, encryption, geographic scope, and failure alerts meet the approved RPO.
- A restore drill has passed. A recent backup was restored into an isolated environment, validated at the application level, timed against RTO, and documented.
- Schema and data migrations are reversible. Cutover, compatibility window, validation counts, and rollback source are recorded.
- Redis and caches are treated as disposable. No authoritative claim, enrollment, financial, or audit state exists only in cache.
5. Messaging and External Integrations
- Azure Service Bus is selected in production. Startup logs confirm the expected backend; an in-memory fallback is never accepted as production evidence.
- Topics, queues, subscriptions, and filters match infrastructure code. TTL, locks, duplicate detection, delivery counts, dead-letter behavior, and access policies are reviewed.
- Senders and receivers use least-privilege authorization. Administrative connection strings are not distributed to application pods.
- Dead-letter alerts and ownership are active. Operators know who investigates, preserves evidence, approves replay, and verifies recovery.
- Duplicate and redelivery behavior is tested. Deterministic message IDs and persisted-result guards prevent double adjudication or duplicate side effects.
- Clearinghouse connectivity is certified. SFTP host keys, credentials, folders, naming, acknowledgments, retry expectations, and support contacts are confirmed with each partner.
- FHIR and external API trust is validated. OAuth scopes, certificates, rate limits, timeouts, CORS, and trading-partner identifiers match production agreements.
- Optional metered integrations are deliberate. AI examination, payment gateways, email, and other billed services are enabled only with owner, budget, alert, and failure-mode approval.
6. Observability and Support
- Health checks reflect real dependencies. Kubernetes readiness and liveness behavior has been observed during database, Service Bus, and downstream-service failure.
- Dashboards cover the business flow. Intake, accepted claims, active messages, completions, p95/p99 latency, pends, denials, retries, dead letters, payment output, and API errors are visible.
- Distributed traces cross service boundaries. A claim can be followed from API request through Service Bus delivery and downstream HTTP calls without exposing PHI.
- Alerts are actionable and tested. Every page has threshold, owner, runbook, escalation, and silence policy; synthetic alert tests reached the on-call responder.
- Log and metric retention meet policy. Access controls, export, redaction, and cost limits are approved.
- Support has safe diagnostic access. Operators can identify tenant, claim ID, stage, delivery count, release, and dependency health without direct uncontrolled data access.
- Operational runbooks are linked from alerts. Include the 837 Operations Runbook and Pended Claims Operations Guide.
7. Application and Tenant Configuration
- The environment is production. Development-only authentication, local stubs, demo users, permissive CORS, debug output, and sample secrets are removed.
- Tenant routing and adapter modes are approved. Replace, augment, and legacy-system connections match the signed implementation scope.
- Enforcement policy is explicit. Network, credentialing, provider-integrity, NCCI, COB, prior-authorization, and AI modes are recorded and tested.
- Reference data is current. Benefit plans, plan-code mappings, fee schedules, NCCI/MUE tables, terminology, networks, provider participation, and credentialing effective dates cover launch traffic.
- Trading-partner identifiers are production values. ISA/GS sender and receiver IDs, payer identifiers, group mappings, and file conventions are not copied from synthetic fixtures.
- Financial controls are approved. Payment accounts, segregation of duties, approval limits, 835 generation, reconciliation, and reversal paths are validated before real disbursement.
- AI remains off unless approved. If enabled, use
BestEffort, confirm the production key source, meter and alert usage, preserve human authority, and test graceful degradation.
8. Capacity, Resilience, and Rollback
- A representative UAT load test passed. Arrival pattern, claim mix, tenant mix, dependencies, database, Service Bus, and network path resemble production.
- Capacity settings are recorded together. Replicas, concurrent consumers, raw-file parallelism, node size, database size, cache, and downstream limits form one reproducible baseline.
- Scale does not compromise correctness. Result accuracy, tenant isolation, payment agreement, eventual completion, and zero unexplained dead letters hold at target load.
- Dependency failures were exercised. Database latency, Service Bus interruption, downstream timeouts, pod termination, and redelivery recover without silent loss or duplicate payment.
- The rollback procedure was rehearsed. Previous application and infrastructure versions, configuration, data compatibility, decision authority, and verification steps are known.
- Rollback preserves data safety. No rollback assumes destructive database reset; forward-only migrations have an explicit compatibility or restore strategy.
- Cost guardrails are configured. Budgets and alerts cover AKS, database throughput, storage, Service Bus, telemetry, external APIs, and optional AI usage.
9. Final Go-Live Gates
| Decision | Minimum condition |
|---|---|
| Go | All required gates pass; conditional capabilities have evidence; rollback and restore are proven; accountable business, security, operations, and engineering owners approve. |
| Conditional Go | Only documented noncritical exceptions remain, each with risk owner, mitigation, monitoring, expiry date, and rollback trigger. |
| No-Go | Unknown data recovery, unresolved critical security issue, production placeholders, unverified tenant isolation, unexplained DLQ messages, failed correctness tests, or no executable rollback. |
- Freeze the approved release, configuration, and infrastructure plan.
- Confirm current backups and capture the pre-deployment baseline.
- Notify stakeholders and open the incident bridge or launch channel.
- Deploy through the approved production gate.
- Run health, identity, tenant-isolation, FHIR, raw 837, adjudication, pend-resolution, and remittance smoke tests.
- Compare metrics with the pre-deployment baseline and UAT evidence.
- Announce go, conditional go, or rollback with the named decision maker.
Go-Live Evidence Packet
| Artifact | Accountable owner | Evidence |
|---|---|---|
| Release and infrastructure manifest | Engineering / Platform | Commit, image digests, What-If, Helm values, approvals |
| Security readiness | Security | Scan results, identity review, Key Vault access, logging review |
| Recovery readiness | Data / Platform | Backup status, restore-drill result, measured RTO/RPO |
| Operational readiness | SRE / Support | Dashboards, tested alerts, on-call roster, runbooks, escalation |
| Claims correctness | Claims operations | Representative adjudication, pend, payment, and reconciliation results |
| Business approval | Product / Customer | UAT sign-off, launch scope, exception acceptance, go/no-go record |
10. After Launch
| Window | Required review |
|---|---|
| First 30 minutes | Pod health, authentication, API errors, Service Bus backlog, database latency, claim completion, pends, denials, and dead letters. |
| First 24 hours | Trading-partner acknowledgments, queue age, payment/remittance reconciliation, alert quality, support contacts, cost anomalies, and tenant-isolation evidence. |
| First 7 days | SLO trend, capacity headroom, recurring manual work, false pends/denials, security events, backup success, incident actions, and accepted-risk closure. |
Close the launch only after temporary access, diagnostic logging, elevated capacity, demo integrations, and relaxed alert thresholds have been removed or formally accepted as production configuration.
Related Guides
Deployment Guide
Provision infrastructure, configure Kubernetes, manage secrets, and run the release pipeline.
Open deployment guide →837 Operations Runbook
Monitor asynchronous adjudication, diagnose backlog, and recover retries or dead letters.
Open operations runbook →Backup & Disaster Recovery
Restore data and platform services, reconcile work, validate correctness, and record RTO/RPO evidence.
Open recovery runbook →Observability & Incident Response
Define service signals, test alerts, coordinate incidents, and prove healthy recovery.
Open incident runbook →