2nd Place (Software) · ACSEF · March 2026

Verifying LLM Clinical-Trial Decisions with Linear Probes

Trials are delayed or abandoned when they cannot recruit; LLM screening is fast but unverifiable.

Venue: Alameda County Science & Engineering Fair (Hitachi Science & Engineering Fair), Software — 2nd Place, High School Category Award
Fair project: HS-SOFT-371 — “Blackbox to Glassbox: Interpretable Clinical Trial Matching via LLM Probing”
Model: Qwen3-8B, no fine-tuning
Data: Five ClinicalTrials.gov trials; TREC 2021 pooled set

Research poster (PDF) Award certificate (PDF)

89.9%probe accuracy, mean of 5 trials
96.8%GPT-4o, direct
84.0%Claude Haiku, direct
70.4%GPT-4o-mini, direct

The problem

Clinical-trial matching pairs patients with eligibility criteria. Trials that cannot recruit are delayed or abandoned, and language models can screen candidates quickly — but they are unreliable and opaque, and a screening decision no one can check is hard to use in a clinical setting.

Approach

I reframed matching as a verification problem: given a trial’s criteria, a patient’s record, and a proposed assessment, is that assessment consistent with the evidence? Each patient appears twice in the data, once paired with the correct assessment and once with the incorrect one, so a probe has to read the reasoning rather than memorize the patient. The dataset covers five trials from ClinicalTrials.gov — iron deficiency, osteoporosis, kidney disease, coagulation, and lupus — with 50 training and 100 held-out test rows per trial.

I ran each example through Qwen3-8B, extracted the residual-stream activation at every one of its 36 layers, and trained a lightweight linear probe per layer, keeping the best layer for each trial. The model itself was never fine-tuned.

Results

The best probes verified decisions at a mean of 89.9% across the five trials on unseen patients (iron 95.5%, coagulation 93.0%, lupus 88.0%, osteoporosis 87.0%, kidney 86.0%), and they consistently came from layers 21–24, late in the network. For comparison, direct classification by frontier models scored 70.4% (GPT-4o-mini), 84.0% (Claude Haiku), and 96.8% (GPT-4o). An 8-billion-parameter open model plus a linear probe beat two of the three at a fraction of the cost.

Two failure tests, kept separate

Two further experiments ask where this breaks, and they are different experiments. First, a probe trained on one trial and transferred to a different, unseen trial fell to 55%. Second, a single probe trained on 500 pooled trials from the TREC 2021 benchmark fell to 50%, which is chance. Accuracy degraded step by step with diversity: three trials combined gave 87.5%, five gave 72.0%. My reading is that the probes learned each trial’s eligibility logic rather than eligibility in general, so per-trial probes are the deployable unit.

Limitations

Single architecture — Qwen3-8B only. The frontier-model baselines were run on 25 samples per trial, so that comparison is indicative rather than definitive. Cross-trial transfer is low, which is a property of the method as much as a flaw in it.

Where it led

The second half of the same fair project asked why the model decides what it decides, and became the six-head decision circuit.

Three results tables: trial diversity versus accuracy, per-trial probe results on unseen patients, and probe versus frontier language models — Fig. 1. Results: trial diversity vs. accuracy, per-trial probe results on unseen patients, and probe vs. frontier models.

Dataset construction: the five clinical trials with their key criteria, and the paired-row design in which each patient appears with a correct and an incorrect assessment — Fig. 2. Dataset construction: five ClinicalTrials.gov trials and the paired-row design.

Preview of the ACSEF 2026 research poster, Blackbox to Glassbox — Fig. 3. The ACSEF 2026 poster (HS-SOFT-371). Click to open the PDF.