Observability, Alerting & Incident Response Runbook

Detect failures from business impact, follow a claim or transaction across asynchronous boundaries, protect PHI in telemetry, page the right owner, restore service safely, and prove the platform is healthy again.

1. Know the Current Observability Posture

Capability	Repository evidence	Operator must verify
Application metrics	Shared OpenTelemetry setup exposes /metrics; custom HTTP, claims, EDI, PAS, and compliance-oriented instruments exist	Every production service is scraped or exported, labels stay bounded, gaps alert, and retention meets policy
Distributed traces	ASP.NET Core, HttpClient, Redis, MongoDB, Service Bus, and Cloud Health Office activity sources are registered	Observability__OtlpEndpoint reaches a production collector/backend and cross-service traces are searchable
Telemetry privacy	A mandatory span processor removes prohibited attributes; identifiers supplied to the shared activity helper are hashed	Logs, exception messages, baggage, custom activity sources, and exporter access also meet the approved PHI policy
Health checks	Shared /health, /health/live, and /health/ready endpoints can test self, MongoDB/Cosmos DB, Redis, and named HTTP dependencies	Each workload calls the shared mapping, probes use the correct endpoint, and readiness covers critical dependencies without creating cascading load
Prometheus and Grafana	Local scrape configuration, a claims dashboard, Helm monitoring dependency, and example alert rules are present	Service discovery, ServiceMonitors, rule loading, Alertmanager routing, dashboard queries, and notification delivery work in the target cluster
Azure monitoring	Infrastructure provisions Application Insights and four Azure Monitor workbooks; AKS templates include Log Analytics integration	Application exporters and diagnostic settings are connected; workbooks return current data; ingestion, sampling, retention, RBAC, and cost controls are approved
OTel Collector	The Kubernetes collector manifest is explicitly marked as a scaffold with a debug exporter	Select and configure the production backend before deployment; do not treat the scaffold as a production telemetry pipeline

2. Start with Service Objectives and User Impact

Page on symptoms that threaten an approved objective. Use component signals to diagnose the cause. Thresholds must be derived from tenant traffic, contractual obligations, and measured baselines rather than copied unchanged from this guide.

Service-level indicator	Good event	Failure symptom	Diagnostic signals
API availability	Authorized request returns the expected non-5xx response	Elevated error ratio or failed synthetic workflow	HTTP rate/errors/duration, ingress, pod readiness, dependency spans
837 acceptance	Valid file produces accepted claim IDs within the intake objective	Parser or submission failures, slow upload, partial acceptance	EDI count, raw-import latency, validation outcomes, claims persistence
Claim completion	Accepted claim reaches an auditable final or pended disposition within the objective	Submitted claims age beyond the window or outcomes stop advancing	Service Bus active/DLQ counts, consumer rate, stage latency, database contention
Correctness	Outcome, allowed amount, payment, and audit trail match deterministic rules and reconciliation	Unexpected denial/pend shift, duplicate payment, missing version, unexplained mismatch	Outcome ratios, answer-key or control totals, event history, payment reconciliation
Eligibility and authorization	Transactions complete within applicable processing objectives with traceable decisions	Timeouts, stale decisions, backlog, or partner failure	EDI/PAS counters, dependency latency, partner status, queue age

Record the approved objectives

Indicator	Objective	Window	Owner	Page threshold
API availability	________	________	________	________
837 acceptance latency	________	________	________	________
Claim completion latency	________	________	________	________
Queue age	________	________	________	________
Correctness / reconciliation	________	________	________	________

3. Use Health Checks and Synthetic Workflows Correctly

Signal	Question answered	Do not use it for
/health/live	Is this process running?	Dependency checks that cause Kubernetes to restart a healthy process during an external outage
/health/ready	Can this instance safely receive work with its configured dependencies?	Proving an end-to-end healthcare workflow or measuring historical reliability
/health	What is the combined current health-check report?	A public unauthenticated diagnostic surface with sensitive dependency detail
Synthetic transaction	Can a controlled user journey complete across real boundaries?	Submitting PHI, triggering real payment, consuming metered AI unintentionally, or polluting production reporting

NAMESPACE='cloudhealthoffice'

kubectl get pods -n "$NAMESPACE"
kubectl get deployment -n "$NAMESPACE"

kubectl port-forward -n "$NAMESPACE" \
  service/claims-service 18080:80

curl --fail --silent http://127.0.0.1:18080/health/live | jq
curl --fail --silent http://127.0.0.1:18080/health/ready | jq
curl --fail --silent http://127.0.0.1:18080/metrics | head

Use the actual service port exposed by the target deployment. A production synthetic should use a dedicated tenant, synthetic member and provider identifiers, a nonpayable claim path, a unique correlation value, and explicit cleanup or retention rules.

4. Correlate Metrics, Traces, and PHI-Safe Logs

Metrics: detect and size

The shared meter is CloudHealthOffice. Core instruments include cho.http.request.duration, cho.claims.processing.duration, cho.edi.transactions.total, cho.claims.adjudication.outcome.total, and cho.telemetry.scrub.total. Prometheus renders dotted names with underscores.

# p95 claim-processing latency
histogram_quantile(
  0.95,
  sum(rate(cho_claims_processing_duration_bucket[5m])) by (le)
)

# adjudication outcomes in the last hour
sum by (cho_outcome) (
  increase(cho_claims_adjudication_outcome_total[1h])
)

# PHI attributes removed before span export
sum by (service_name, attribute_name) (
  increase(cho_telemetry_scrub_total[15m])
)

A nonzero telemetry-scrub rate is a privacy defect signal: the processor prevented export of a prohibited span attribute, but the emitting code still attempted to add it.

If Azure Monitor is the selected backend, review Microsoft’s current Application Insights OpenTelemetry configuration before choosing exporters, sampling, ingestion limits, and cost controls.

Traces: locate the slow or failing boundary

1. Start with the service, route, environment, release, and UTC interval from the alert.
2. Follow the trace across ASP.NET Core, outbound HTTP, MongoDB/Redis, Service Bus, and Cloud Health Office business spans.
3. Compare a failing trace with a successful trace from the same transaction type and release.
4. Search by approved hashed identifiers or incident-safe correlation fields. Never paste a raw member ID, MBI, SSN, patient name, payload, or authorization header into telemetry tools.
5. If the trace ends at an asynchronous send, pivot to deterministic message ID, persisted event, queue state, and consumer logs.

Logs: explain decisions without copying the record

• Log stable event names, service, environment, release, tenant scope where approved, trace/span IDs, hashed business identifiers, decision codes, duration, retry count, and exception class.
• Do not log raw X12, FHIR resources, request or response bodies, names, dates of birth, member or subscriber identifiers, addresses, tokens, secrets, connection strings, or payment account details.
• Restrict telemetry access by role, audit access, encrypt in transit and at rest, define retention, and prevent unreviewed exports or support copies.
• Test custom logging and activity sources separately. The mandatory span processor protects exported span attributes; it does not automatically sanitize every log message or external telemetry path.

5. Monitor Claims and Azure Service Bus Together

Submission rate is not completion rate. Watch accepted claims, persisted dispositions, queue state, delivery failures, and dependency pressure on the same timeline.

Signal	Interpretation	Response
Incoming exceeds outgoing briefly	Normal burst or consumer lag	Watch active-message count, oldest-work age, and completion rate
Active messages rise continuously	Consumers cannot sustain arrival rate	Identify slow stage or dependency before increasing parallelism
Dead-lettered messages > 0	Work exhausted delivery policy or expired	Page the owning team, preserve evidence, fix cause, then replay deliberately
Throttled requests or server errors	Namespace/service pressure or provider failure	Correlate with client retries, handler latency, delivery count, and tier limits
Queue clears but submitted claims remain	Missing trigger, filtered message, persistence inconsistency, or observation gap	Reconcile durable claim events and statuses; do not infer success from an empty queue
Pend/deny ratio shifts	Data, configuration, dependency, or code behavior changed	Compare by tenant, claim type, rule/stage, and release before declaring capacity failure

Azure Monitor exposes Service Bus message and request metrics including Active Messages, Dead-lettered Messages, Incoming/Outgoing/Completed/Abandoned Messages, Server Errors, User Errors, and Throttled Requests. Counts can be point-in-time and incoming/outgoing totals need not match because of filters, duplicates, expiration, dead-lettering, and consumer lag. Use the current Service Bus monitoring data reference when implementing rules.

For the operational procedure, use the 837 Intake & Adjudication Operations Runbook.

6. Monitor Databases and Kubernetes as Capacity Boundaries

Layer	Watch	Evidence of saturation or failure
MongoDB	Operation latency, connections, CPU, working set, storage latency/capacity, locks/queues, replication and backup status where applicable	Broad stage slowdown, timeouts, version conflicts, connection pressure, eviction, or storage stalls
Cosmos DB	Request rate, normalized RU consumption, throttled requests, latency, availability, data size, region/replication health, backup status	HTTP 429s, rising request charge/latency, hot partitions, exhausted throughput, or region issue
Kubernetes workload	Desired vs. available replicas, readiness, restarts, OOM kills, CPU throttling, memory working set, pending pods, HPA behavior	Unavailable replicas, crash loops, repeated restarts, scheduling failure, or resource saturation
Kubernetes cluster	Node readiness, allocatable CPU/memory, disk pressure, networking, DNS, ingress, certificate expiry	NotReady nodes, pod eviction, network errors, DNS latency, or expiring certificate
Telemetry pipeline	Scrape success, exporter/collector errors, dropped spans, ingestion delay, cardinality, retention, alert evaluation and notification	Missing series, stale dashboards, collector backlog, sudden ingestion loss/spike, or undelivered test alert

For AKS, use the current Azure Kubernetes Service monitoring guidance to select Container Insights, managed Prometheus, Grafana, and cluster alert coverage for the deployed topology.

More claims-service replicas or concurrent handlers can make database contention worse. Scale one dimension at a time and require completion-rate improvement—not only higher submission rate.

7. Design Alerts That a Responder Can Act On

Every page needs a condition, sustained window, severity, owner, user impact, dashboard, runbook, first three actions, escalation path, and resolution signal. Route warnings to a work queue when immediate human action is unnecessary.

Severity	Use when	Expected response
SEV-1	Widespread outage, active data-integrity or payment risk, confirmed cross-tenant exposure, material security event, or recovery required	Page immediately; incident commander and business/security leadership engaged
SEV-2	Material degradation, sustained objective breach, growing claim backlog, critical dependency loss, or contained tenant impact	Page primary owner; establish incident channel and mitigation plan
SEV-3	Early warning, reduced redundancy, nonurgent drift, capacity trend, or isolated recoverable failure	Create owned work; escalate if impact or duration increases

Starter conditions to tune with real baselines

• Failed end-to-end synthetic transaction for two consecutive runs.
• Sustained API 5xx ratio or latency consumes the approved error budget.
• Accepted claims age beyond the completion objective or accepted-versus-disposition reconciliation diverges.
• Any unexplained Service Bus dead letter; sustained throttling, server errors, or growing active-message age.
• Available replicas fall below required capacity; repeated restarts, OOM kills, pending pods, or NotReady nodes persist.
• Database throttling, connection pressure, latency, or storage capacity breaches the tested operating envelope.
• cho.telemetry.scrub.total increases in production, telemetry stops arriving, or an alert-delivery canary fails.
• Any correctness, duplicate-payment, tenant-isolation, secret-exposure, or audit-integrity signal.

8. Respond to an Incident

Role	Responsibility
Incident commander	Owns severity, priorities, decision cadence, roles, escalation, and final service-restoration decision
Operations lead	Triages telemetry, executes mitigations, and records exact changes and UTC timestamps
Application/data lead	Protects correctness, defines reconciliation, evaluates replay/rollback/recovery, and signs data validation
Security/privacy lead	Handles suspected exposure, evidence access, containment, legal/compliance escalation, and notification process
Communications lead	Provides factual stakeholder updates with impact, scope, mitigation, and next-update time

1. Declare. Record alert, UTC start, severity, affected capability/tenants, release, commander, responders, and next update.
2. Confirm impact. Use a safe synthetic and durable business records; distinguish monitoring failure from service failure.
3. Bound the incident. Identify first known-bad time, last known-good result, blast radius, data/payment/privacy risk, and whether the boundary is expanding.
4. Stabilize. Stop unsafe writes or producers if necessary; preserve logs and durable evidence; avoid destructive cleanup and uncontrolled replay.
5. Diagnose. Move from service objective to metrics, trace, logs, queue, dependency, database, and cluster. Change one controlled variable at a time.
6. Mitigate. Roll back, scale, disable an optional dependency, fail over, or invoke the Backup & Disaster Recovery Runbook using named authority.
7. Validate. Reconcile accepted and completed work, run correctness controls, verify tenant isolation and payments, clear unexplained DLQs, and confirm alerts recover.
8. Close. Remove temporary access/capacity/logging, preserve evidence, record actual impact and objective breach, and schedule the review.

Incident ID:
UTC start / detected / mitigated / resolved:
Severity and decision maker:
Affected capabilities and tenants:
User and business impact:
Release SHA and configuration changes:
Primary signals and trace references:
Data, payment, privacy, and compliance assessment:
Mitigations with owner and UTC time:
Reconciliation and recovery evidence:
Outstanding risk and next update:
Follow-up owners and due dates:

9. Verify Recovery Before Declaring Resolution

• The original symptom and the underlying cause are both absent for a defined observation window.
• End-to-end synthetics pass from more than one observation point where applicable.
• Error rate, latency, queue age, completion rate, database pressure, and Kubernetes health return to the expected envelope.
• Accepted claims and other transactions reconcile to durable, auditable outcomes; no unexplained messages remain in a DLQ.
• Outcome distributions, allowed amounts, payments, coverage, and tenant boundaries pass targeted correctness checks.
• Telemetry is complete, PHI-scrub counters are investigated, alerts evaluate, and a notification reaches the expected responder.
• Rollback remains available until the business and technical owners approve closure.

10. Test the Operating System, Not Just the Dashboard

1. Choose one scenario: stopped consumer, unavailable dependency, injected HTTP error, growing queue, pod restart, telemetry loss, DLQ message, or restore/failover exercise.
2. Declare safety boundaries, synthetic tenant/data, expected signals, maximum impact, abort condition, and cleanup owner.
3. Start the scenario and measure detection time, notification time, acknowledgment time, diagnosis time, mitigation time, and verified recovery time.
4. Require the responder to use the alert, dashboard, trace, and linked runbook without undocumented assistance.
5. Verify no raw PHI or secrets entered telemetry, chat, tickets, screenshots, or drill evidence.
6. Fix missing/noisy signals, unclear ownership, broken links, permissions, and unsafe instructions; repeat the failed portion.

Drill evidence	Result
Scenario, environment, and safety boundary	________________________________________
Expected / observed signals	________________________________________
Detected / notified / acknowledged	________________________________________
Mitigated / verified / resolved	________________________________________
PHI and access review	________________________________________
Gaps, owners, and due dates	________________________________________
Next drill date	________________________________________

Related Guides