A modular evaluation runner built on one shared provenance, streaming-telemetry and report spine. Run reproducible experiments across point-of-care EHR chart assistants (ChartSearchAI), iterative SQL and cohort analytics (Catalyst), multi-source FHIR R4 summarization (WA Health Summary), or any OpenAI-compatible model or agent endpoint.
Every evaluation in the OpenClinAI Validation Harness separates fast, deterministic provenance and execution checks from blinded multi-family autorater juries and human clinician calibration:
Layer 1 · Deterministic Oracles
Clinical, Provenance & Execution Gates
Zero-LLM checks that run on every turn before any autorater is invoked:
Provenance & Citation Recall: Exact [N] source index resolution, claim-level grounding (sources.v1), and abstention on absent records.
Execution Oracles: Live read-only SQL execution & schema-boundary audit (Catalyst) and CQL / FHIR `$apply` parity against WHO SMART L2 decision tables.
Layer 2 · Pinned System-1 Encoders
Non-Autoregressive Entailment & Drift Control
Pinned encoder scorers that audit atomic claim support and embedding stability:
Atomic Claim Entailment: Sentence-level premise-hypothesis entailment scoring across retrieved chart records and generated claims.
System-1 Gate Evaluation: Precision/recall and Expected Calibration Error (ECE) for Phase 0a context-admission and Phase 0b review-error triggers (see Lab).
Embedding Drift Control: Pinned SHA-256 encoder digests across QueryStore and evaluation runs.
Layer 3 · Blinded Multi-Family LLM Judges
Cross-Family Autorater Panels
Blinded multi-model juries that prevent self-preference bias when grading open-weight models:
Cross-Family Jury:Claude Opus 4.6 + GPT-5.4 / Gemini 3.1 Pro grading randomized, anonymized model outputs.
Prediction-Powered Inference (PPI): Combines large-scale autorater runs with gold clinician-adjudicated subsets to compute rectified 95% confidence intervals and Cohen’s $\kappa$.
One evaluation spine across four target modalities
The harness separates shared evaluation infrastructure—immutable run manifests (run_manifest.json), append-only event logs (events.jsonl), streaming latency and token-cache telemetry, Prediction-Powered Inference (PPI) human calibration, and self-contained HTML reports (report_shell)—from target adapters (harness.adapters) that speak each application’s native protocol:
OpenMRS REST & SSE stream (/ws/rest/v1/chartsearchai/chat/stream) under provisioned clinical roles
Compares built-in (bundled) vs. Med Agent Hub profiles; scores chart-record citation precision/recall (score-citations) and claim-level support (sources.v1).
FHIR R4 bundle condensation (/v1/wahs/condense) and downstream clinical Q&A across 1k–8k token budgets
Evaluates deterministic vs. hybrid LLM condensation, International Patient Summary (IPS) section coverage, clinical entity retention, and grounded abstention & provenance gates.
General OpenAI-compatible LLM & agent evaluation
harness-cli validate (transport="openai-compatible" or "med-agent-hub", OpenAICompatibleClient / MedAgentHubClient)
Any /v1/chat/completions server (vLLM, llama-server, LiteLLM, Ollama, custom agent gateways) via JSON or SSE streaming
Evaluates arbitrary models and prompts without requiring an EHR host: per-backend endpointUrl routing, plain-text or JSON responses, deterministic turn checks, multi-judge Scout rubrics, and multi-session concurrency goodput.
How the validation architecture works
01
Measure the live stream and stage milestones
Drive each target’s real interface—OpenMRS SSE streams, Catalyst notebook turns, WAHS condensation calls, or OpenAI SSE completions—and timestamp every frame as it arrives. Time to First Token (TTFT), main answer completion (answer_done), and post-answer review or In-Depth completion (turn_done) are recorded separately rather than collapsed into one wall-clock number.
02
Isolate model capability from runtime artifacts
Use wire-level capture (engine-tap) and explicit transport modes to separate underlying model quality from application-layer prompt assembly, reference-date anchoring, SQL retry loops, or sampler rules (such as conditional DRY repetition penalties on long patient charts).
03
Stress-test multi-session concurrency and goodput
Run N=1, 2, 4+ simultaneous sessions—isolated across distinct OpenMRS clinical accounts or parallel API workers—so concurrent turns do not preempt one another’s conversation state. Report Service-Level Objective (SLO) goodput alongside p50/p95 latency and prefix-cache hit ratios.
What each run records: quality, grounding and streaming telemetry
Evaluating clinical and analytical AI requires both domain correctness (whether the output is faithful to patient records, database state, or privacy rules) and interactive responsiveness (how quickly a user sees the first token and completed answer). Every harness run captures five standardized layers:
Evaluation layer
Recorded metrics
How it applies across modalities
Interactive streaming milestones
ttft_ms (first reasoning_delta, answer_delta or OpenAI SSE delta), answer_done_ms, turn_done_ms, inter-token latency (itl_p50_ms, itl_p95_ms) and Time Per Output Token (tpot_ms).
Separates perceived answer latency from background verification or In-Depth synthesis across OpenMRS streams, Med Agent Hub profiles, and arbitrary OpenAI-compatible SSE endpoints.
Token & prompt-cache efficiency
tokens_in, tokens_out, cached_tokens, cache_hit_ratio and turn-1 vs. turn-2+ TTFT.
Counts only requests that meet interactive latency targets under N-user load, exposing queueing saturation that raw throughput averages hide.
Deterministic & domain grounding
Scenario turn checks (must_contain, must_not_contain, must_cite), chart citation precision/recall (ChartSearchAI), live DB SQL reconciliation (Catalyst), and IPS/privacy redaction checks (WAHS).
Pairs fast, zero-LLM deterministic verification with modality-specific source checks (EHR observations, SQL result sets, or FHIR R4 bundle resources).
Multi-judge rubric & human calibration
Scout LLM-jury rubric scores (accuracy, completeness, relevance, harm, temporal) calibrated with human adjudication via Prediction-Powered Inference (PPI 95% CI and Cohen’s κ).
Combines scalable automated grading across model comparison sets with clinician- and analyst-anchored confidence intervals and explicit safety/harm flags.
Workload and hardware sizing guide
Deployment sizing depends on the prompt context length per turn and number of concurrent active sessions (N). Condensed WA Health Summary views (1k–4k tokens) and Catalyst SQL generation turns require modest prefill compute, whereas ChartSearchAI with fullChart enabled on a longitudinal HIV record sends roughly 12,200 input tokens per turn. Using the ~12.2k-token longitudinal chart workload as an upper-bound sizing benchmark, the matrix below summarizes what implementers can expect across four hardware tiers:
TTFT is 25–60 s on a 12.2k-token chart, and N≥2 queues serially (>60–120 s). Not recommended for longitudinal fullChart HIV clinics without an external GPU host.
Tier 1: Entry single GPU / edge workstation
NVIDIA L4 / RTX 4090 (24 GB VRAM) or Apple Silicon M-series (32–64 GB unified)
llama-server --parallel 2 -cb or vLLM (Gemma 4 E4B or Gemma 4 12B Q4_K_M)
At N=1..2, warm prefix-cached TTFT is 0.8–2.5 s (cold 3–7 s) and answer_done is 4–10 s. Recommended with Med Agent Hub (single-12b-bare or single-12b-checked).
Parallel prefill and decode across N=4 concurrent clinicians with TTFT of 0.3–1.2 s, answer_done of 2–5 s, and TPOT of 12–25 ms/tok. Recommended for full checked and team profiles.
Sub-second TTFT and horizontal slot scaling with site-level governance, audit logging and de-identification policies.
Shared benchmarks and open-source tooling we build on
Rather than reinventing evaluation metrics or load-testing primitives from scratch, the Validation Harness adopts conventions from active clinical NLP benchmarks and open-source LLM serving tools:
Benchmark / tool
Focus area
How OpenClinAI uses or adapts its ideas
ArchEHR-QA & EHRNoteQA (PhysioNet / BioNLP)
Patient-specific multi-note EHR question answering and sentence-level evidence grounding
Separates evidence identification (whether the assistant retrieved and cited the right OpenMRS chart records in score-citations) from answer-evidence alignment (whether each clinical claim is supported by its cited record in sources.v1) across ChartSearchAI and WA Health Summary.
EHRSQL & Spider 2.0 / BIRD-SQL
Clinical text-to-SQL accuracy, unanswerable-query abstention and execution-grounded database verification
Informs Catalyst notebook validation: live read-only SQL reconciliation against ground-truth database queries, schema-boundary auditing, and explicit abstention checks when a schema lacks requested clinical fields.
HealthBench (OpenAI) & MedHELM (Stanford CRFM)
Multi-turn clinical conversations, physician-authored rubrics and LLM-jury evaluation
Combines scenario-specific deterministic turn checks (expectations.py) with multi-axis rubric grading and explicit clinical harm flags, anchored to human reviews via Prediction-Powered Inference (PPI) across all four target modalities.
Adopts streaming milestone decomposition, turn-position prefix-cache tracking (turn_1_ttft_p50_ms vs. turn_2plus_ttft_p50_ms, cache_hit_ratio) and SLO goodput in harness-cli validate concurrency across OpenMRS, Med Agent Hub, and general OpenAI-compatible endpoints.
A worked example
Example validation records
Published ChartSearchAI reports show questions, answers, streaming and stage timings, and their review context. Dated Catalyst reports preserve query results, findings and the methods used to evaluate them.
Research findings are not product certifications. Model evaluation is separate from proving that a person can complete an application workflow.
Explore how the Validation Harness evaluates multi-clinic FHIR R4 patient summaries and WHO SMART Guidelines L2→L3 knowledge compilation across our applied projects.