Million Claim Challenge ยท Part 8

Running a Healthcare Claims Platform Locally in Kubernetes, Part 8: The Clean 100,000-Claim Run

Evidence scope: This engineering field note describes local Kubernetes development and validation. It is not a production-cloud capacity claim. Exact results and limitations are preserved in the linked evidence artifact.
View evidence View article source

Part 7 moved the Million Claim Challenge evidence out of terminal logs and into an operator console.

Then the correctness bar changed.

Payment accuracy became a gate instead of a diagnostic. Expected-pay and expected-deny claims were checked for false pends. Zero-limit accumulator placeholders stopped distorting expected payments. Generated members were isolated so one corpus segment could not contaminate another. Validation providers were moved into role-separated synthetic NPI namespaces so scoreable scenarios could not collide with stale provider-integrity state in a long-lived local tenant.

Then the evidence bar moved again.

Prior-authorization scenarios that had been partially unsupported became scoreable. Provider-exclusion denial codes were normalized so B7, CARC_B7, and PROVIDER_EXCLUDED represented the same expected business outcome instead of producing false mismatches.

Only after those changes did Cloud Health Office move back to 100,000 claims.

The result was clean: every claim processed, no platform failures, no scoreable workflow mismatches, no unexpected pends, and every comparable payment within one cent.

The run also exposed the next local scaling question. Docker Desktop was not out of CPU or memory, and fixture preparation was no longer a two-hour wall-clock problem. The platform finished timed adjudication in 30 minutes and 5 seconds, with the full tracked lifecycle taking about 38 minutes.

That is exactly the kind of result this series is supposed to produce: correctness held, the evidence became inspectable, and the next performance question became sharper.

What changed before 100K

The previous 50K result proved broader outcome scoring, but it still left important questions open.

A claim could reach the expected paid disposition while paying the wrong amount. Expected-pend claims were observed, but expected-pay and expected-deny claims were not swept for accidental pends. Placeholder accumulator limits could be interpreted as real zero-dollar limits. Reused generated member identities could let fixture state cross scenario boundaries.

Those are not cosmetic benchmark concerns. They affect whether the answer key and the platform are actually evaluating the same claim context.

The validator now includes:

Several final hardening passes mattered before the final 100K rerun.

PR #927 anchored validation providers as run-scoped fixtures. A follow-up 50K verification then found 10 scoreable TxStarInpatientNoAuth mismatches: those claims should have denied for missing prior authorization, but provider-integrity evaluation treated the generated rendering provider as excluded. The current provider record was approved and in-network, which pointed to stale or colliding validation-provider identity state in the long-lived local tenant.

PR #928 moved scoreable validation providers into wider, role-separated synthetic NPI namespaces: billing providers in the 93... range, adjudicatable rendering providers in 94..., and intentionally excluded rendering providers in 95....

Then PR #931 wired prior-authorization validation into adjudication, PR #932 made seeded prior-auth evidence scoreable, and PR #933 fixed the scoring path so expired-authorization and wrong-procedure scenarios could be counted honestly instead of left unsupported. PR #934 normalized provider-exclusion denial codes so B7, CARC_B7, and PROVIDER_EXCLUDED no longer disagreed as labels when they represented the same exclusion outcome.

The post-#934 50K verification processed 50,000 claims cleanly: 5,927 of 6,500 workflow checks matched, zero workflow mismatches, 460 of 460 expected-pend claims observed, zero unexpected pends across 5,467 scoreable non-pend claims, and 1,000 of 1,000 payment comparisons within tolerance with a $0.00 maximum payment delta.

That hardened fixture and scoring model is what the final 100K run used.

The 100K result

The local Docker Desktop Kubernetes run processed 100,000 deterministic synthetic claims with parallelism set to 10.

The final summary reported:

The pend numbers have two scopes. The run ended with 924 persisted pended outcomes overall. The validator specifically observed 920 expected-pend claims as pended, then swept 10,934 scoreable expected-pay and expected-deny claims and found zero unexpected pends. That distinction matters because unsupported scenarios remain separate from the scored non-pend set.

Unsupported scenarios also remained separate from passes and failures. They were concentrated in behavioral-health carve-out, Medicaid spend-down, the prior-authorization wrong-provider variant, retroactive coverage change, and subrogation workflows.

Those are named product gaps, not hidden successes.

Performance did not saturate the machine

The timed claim-processing phase completed in 30 minutes and 5.013 seconds.

That produced:

The provider-pool fixture work changed the local lifecycle story. Earlier 100K preparation had been dominated by run-scoped provider creation. In this run, the validator reduced 199,847 provider references to 13,742 distinct NPIs, reused 186,258 provider assignments, and preserved 2,800 provider-sensitive claims. It created 13,383 synthetic providers instead of trying to create a provider record for every repeated reference.

That is why the tracked lifecycle was about 38 minutes instead of more than two hours.

The run still did not scale linearly from smaller benchmarks, and Docker Desktop showed plenty of CPU and memory headroom while the run was active. That points the next investigation away from raw host capacity and toward internal throughput limits: service concurrency, writeback behavior, MongoDB pressure, HTTP client settings, and validator parallelism.

That is not a production capacity conclusion. It is a local scaling signal worth investigating.

Timed throughput and preparation are different

The tracked lifecycle took 38 minutes and 5.792 seconds. The timed adjudication phase took 30 minutes and 5.013 seconds.

The remaining tracked time was mostly preparation:

The validator seeded 10,614 synthetic members, reused 88,977 existing members, aligned 99,591 member statuses, found 920 COB coverage rows already present, and found both required provider networks already present.

Both numbers matter. The platform adjudicated at 55.40 claims per second during the measured phase. The full local evidence run, including fixture preparation and observation, took about 38 minutes.

There was one reproducibility caveat in the local wrapper. The shell command that launched the Kubernetes job timed out while waiting for the job condition, but the Kubernetes job continued running, completed successfully, and published the completed Mass Adjudication summary. That was a local orchestration timeout, not a platform failure. The wrapper now supports a configurable JOB_TIMEOUT so larger local runs can wait long enough without changing validator scoring behavior.

What the console should make obvious

A useful operator console should not show only the best-looking number.

For this run, it should make clear:

The run also exposed a live-progress display issue. During processing, expected-pend scenarios can look like workflow mismatches until the post-processing observation pass confirms the persisted pended state. In this run, the console showed 920 live workflow mismatches while processing claims. After expected-pend observation completed, that number correctly resolved to zero.

That is not an adjudication failure, but it is a product lesson: the live console should distinguish unresolved expected-pend checks from true mismatches.

The next local optimization target

The next target is not immediately 250K.

First, the team should run controlled 100K parallelism sweeps with the same seed, fixture model, and correctness gates. The p10 run used only a fraction of the allocated Docker Desktop CPU and memory, so p12 and p16 were natural experiments.

The first of those sweeps is now complete. At p12, the same 100,000-claim gate stayed clean: 100,000 processed, zero platform failures, zero workflow mismatches, 920 of 920 expected-pend claims observed, zero unexpected pends across 10,934 scoreable non-pend claims, and 2,000 of 2,000 payment comparisons within one cent.

Throughput improved from 55.40 to 58.55 claims per second. Tail latency moved the other direction: P95 increased from 324 ms to 416 ms and P99 increased from 480 ms to 580 ms. That is a good benchmark result because it is not a slogan. It says p12 buys some throughput, but the latency curve is starting to push back.

The p16 run answered the next question. It also stayed correct: 100,000 processed, zero platform failures, zero workflow mismatches, 920 of 920 expected-pend claims observed, zero unexpected pends across 10,934 scoreable non-pend claims, and 2,000 of 2,000 payment comparisons within one cent.

But p16 did not improve the local result. Throughput fell to 55.11 claims per second, P95 rose to 518 ms, and P99 rose to 683 ms. That makes the p16 run useful evidence in the other direction. The local system had enough raw Docker CPU and memory, but adding validator concurrency pushed internal service, persistence, or writeback pressure past the efficient point.

The acceptance criterion stays the same: performance work cannot weaken the correctness gates.

That suggests several concrete experiments:

Unsupported scenarios should become their own product-evidence track. The first likely candidates are the prior-authorization wrong-provider variant, followed by retroactive coverage change, Medicaid spend-down, behavioral-health carve-out, and subrogation. They should be converted into scoreable behavior deliberately, not swept into the same PR as a benchmark evidence update.

What comes after the local ceiling

The Million Claim Challenge is still aimed at one million claims.

But the local series should continue until the limiting resource is reproducible and explained. It may be service concurrency, persistence, MongoDB, Docker networking, fixture preparation, or a combination of them.

Once that ceiling is understood, the natural follow-up is a cloud scaling series using the same corpus and gates across Azure Kubernetes Service, Amazon Elastic Kubernetes Service, and Google Kubernetes Engine.

The comparison should not ask only which cloud is fastest. It should ask how much infrastructure, time, and cost each environment needs to preserve zero platform failures, zero scoreable mismatches, zero unexpected pends, payment accuracy, and inspectable evidence.

Part 5 made the benchmark repeatable.

Part 6 made the scoring honest.

Part 7 made the evidence visible.

Part 8 proves that the stronger system can process 100,000 claims cleanly, with the fixture model hardened and the proof visible in the console.

All MCC articlesEvidence archive