From Given to Gathered Evidence:
Agentic Learning for Longitudinal Medical Reasoning

Minye Shao1,2Chaohui Yu2,3,*Yixuan Wu2Fan Wang2Ling Shao4Yang Long1,*

Many thanks to contributors: Jing Wang2,3, Hangjie Yuan2,3, Shang Liu2,3

1Durham University2DAMO Academy, Alibaba Group3Hupan Laboratory4University of the Chinese Academy of Sciences

*Corresponding authors

Paper Code Hugging Face Docker

TL;DR We explore how a compact 8B vision–language agent can learn to seek longitudinal clinical evidence — deciding which visit, organ, MRI sequence, slice and tool to inspect — and introduce a longitudinal multimodal benchmark built on UK Biobank to evaluate it.

Overview of the CASE longitudinal evidence-seeking benchmark: fixed clinical targets from UK Biobank paired with patient environments containing clinical context and baseline and follow-up multi-sequence MRI.
Figure 1. The CASE longitudinal evidence-seeking benchmark. Fixed clinical targets are paired with heterogeneous patient environments.

01Abstract

Foundation models can serve as clinical agents through tool-use harnesses. However, conventional medical benchmarks assess reasoning over preselected evidence rather than the ability to seek it across clinical records and longitudinal imaging. We propose CASE: a series of role-specific Clinical Agents for Seeking Evidence, together with a tool-use harness and an agentic post-training framework for compact vision–language policy models. We further introduce a longitudinal multimodal benchmark built on UK Biobank, comprising 50,401 clinical questions derived from real-world ICD-10-coded diagnoses of 4,739 participants. Each question links to a patient-specific environment containing clinical context and multi-sequence MRI from baseline and follow-up visits, where agents autonomously select which visits, organs, modalities, slices, and specialist tools to inspect and compare. Supervised fine-tuning transfers evidence-seeking workflows from 147,434 frontier-model interaction trajectories, followed by agentic reinforcement learning on the learner's own environment interactions. Privileged on-policy self-distillation and rubric-based LLM feedback refine evidence-to-conclusion reasoning without prescribing tool sequences. Experiments show that CASE moves beyond question–answer imitation toward transferable investigation policies, strengthening evidence-grounded longitudinal reasoning. Under matched evaluation conditions, our Qwen3-VL-8B based agent achieves over 16% and 10% relative improvements in answer accuracy over GPT-5.4 and Claude Opus 4.8.

02Given vs. gathered evidence

Conventional medical benchmarks hand the model preselected evidence and ask for an answer. In real longitudinal care, the evidence must be found: across clinical records, visits, organs, MRI sequences and slices. CASE evaluates whether an agent can decide what patient evidence to inspect before it concludes.

Given

Fixed-context QA

  1. Preselected evidence packet
  2. Question
  3. Answer
  • Evidence is chosen in advance
  • One pass, no follow-up inspection
  • Tests reasoning over what is shown

Gathered

Evidence-seeking agent

  1. Question + initial clinical context
  2. Decide visitorgansequenceslicetool Observe → repeat
  3. P1–P5 synthesis → Answer
  • Agent selects what to inspect and compare
  • Baseline and follow-up MRI across visits
  • Tests investigation, not only recall

03Contributions

1

Rethinking longitudinal reasoning as evidence seeking

Agents must decide what patient evidence to inspect before concluding, rather than reason over evidence that has been preselected for them.

2

Clinical agents as evidence-seeking investigators

Role-specific agents, a patient-level tool-use harness, and a longitudinal multimodal benchmark built on UK Biobank.

3

Learning beyond teacher imitation

From teacher trajectories to RL on the policy's own interactions without direct answer supervision; rubric feedback and privileged self-distillation strengthen evidence-to-conclusion synthesis.

04The CASE benchmark

clinical questions
50,401
participants
4,739
test questions / participants
4,850/455
task–format strata
10
organ regions / sub-modalities
5/7
median inter-visit interval
2.26y

Questions are derived from doctor-diagnosed conditions and ICD-10 records in UK Biobank. Each recorded target is paired with cohort-derived distractors, spanning 7 clinical families and 10 task–format strata in three formats: binary, single-select and multi-select.

Every question links to a patient environment with clinical context and baseline plus follow-up MRI across 5 organ regions (heart, brain, pancreas, liver, whole body) and 7 sub-modalities. Tools expose MRI views, Vista3D-derived measurements and overlays, and MedGemma-27B reports.

The metric is first-pass exact answer-set accuracy, reported as Micro (all questions weighted equally) and Macro (all 10 strata weighted equally).

Question formats

  • Binary 17,502
  • Single-select 19,790
  • Multi-select 13,109

Benchmark and training data are under UK Biobank review. Data are returned to UK Biobank and accessible to approved researchers via UKB-RAP.

05Three evidence-seeking roles

One shared policy instantiates three workflows, differing only by role prompt and tool registry. Every role ends with the same five-phase synthesis.

Anatomist

4 calls

Organ and station measurements, longitudinal volume change, and segmentation overlays.

  • vista3d_segment
  • vista3d_compare_stations
  • vista3d_compare_visits

Radiologist

6 calls

Selects acquisitions, then inspects and compares matched views across visits.

  • list_available_images
  • view_image
  • compare_visits

Consultant

12 calls

Combines images, volumes, MedGemma reports and skill memory; doubts and verifies claims. Uses all 10 tools.

  • all six above
  • list_available_reports
  • get_report
  • recall_skills
  • write_skill
  1. P1 Plan
  2. P2 Baseline / Review
  3. P3 Change / Verify
  4. P4 Insight
  5. P5 Answer

06Method

CASE training and execution diagram: two-stage training (RSFT, then agentic RL with outcome reward, OPSD and RLAIF), one shared policy with three role workflows, and the common contract of evidence decisions and tool observations ending in P1 to P5 synthesis.
Figure 2. CASE training and execution. (1) Two-stage training: RSFT then agentic RL with outcome reward, OPSD and RLAIF. (2) One shared policy instantiates three evidence-seeking workflows. (3) A common contract alternates evidence decisions and tool observations until the P1–P5 synthesis.

Stage 1

RSFT — rejection-sampled fine-tuning

  • Qwen3.8-Max records role-conditioned trajectories, with reference answers withheld.
  • Kimi K2.6 scores evidence linkage; trajectories with ≥ 1 tool call and score ≥ 0.4 are kept.
  • Qwen3-VL-8B-Instruct is fine-tuned as a full model, including the vision encoder, with loss on assistant tokens only.

Stage 2

Agentic RL — GRPO on own interactions

  • GRPO with G = 8 rollouts per prompt on the policy's own tool interactions.
  • Tools replay precomputed observations; the policy still chooses tool, arguments, order and stopping.
  • Outcome reward is refined with RLAIF rubric feedback and OPSD.

Trajectory reward

L_j = b_j (0.5 c_j − 0.3 (1 − c_j)) · B_j = r^ans_j + 0.01 r^fmt_j + 0.6 s_j + L_j · R_{n,j} = sg[ B_j − 2 D^CKL_{n,j} ]
r^ans
+1 exact, −1 wrong, −2 unparsed.
s_j
Kimi K2.6 rubric score of the Phase-4 insight: evidence specificity, logical progression, answer support, completeness.
b_j
Indicates a top-2 of 8 insights in the group. Linkage rewards a strong insight only when the answer is correct and penalizes it otherwise.
D^CKL
Privileged self-distillation causal-KL penalty (OPSD), detached and applied before group normalization.

OPSD

Privileged on-policy self-distillation

The same frozen snapshot scores the trajectory under an ordinary view and an answer-privileged view. The reverse KL over the Phase-4 region (top-128 tokens plus one residual bucket) becomes a detached, trajectory-level penalty applied before group normalization. It reorders whole trajectories by synthesis quality without prescribing tool sequences.

Why: in mid-to-late RL the policy stalls in Phase 4, restating findings instead of committing. OPSD acts as a brake on this repetition.

RLAIF

Rubric feedback with top-2 linkage

Kimi K2.6 grades each Phase-4 insight on a rubric. Among the 8 rollouts of a group, the top-2 insights are linked to correctness: a strong insight is rewarded only when its answer is correct, and penalized otherwise, so persuasive but wrong synthesis is not reinforced.

07Results

+16%relative over GPT-5.4

+10%relative over Claude Opus 4.8

Under matched evaluation conditions, the Qwen3-VL-8B based CASE agent reaches 76.1 Micro / 75.1 Macro accuracy, ahead of every frontier model evaluated.

Table 1. Overall role-averaged accuracy (%). Best in bold.
MethodMicroMacro
Base model
Qwen3-VL-8B (base)46.047.1
Medical models
Lingshu-32B49.950.3
Lingshu-7B45.546.7
MedGemma-27B27.827.5
MedGemma-1.5-4B40.541.4
General multimodal foundation models
Kimi K2.642.842.0
Kimi K370.269.5
Qwen3.7-Plus67.067.7
Qwen3.8-Max (teacher)59.758.3
Gemini-3.5-Flash56.356.3
GPT-5.465.365.7
Claude Opus 4.868.665.8
CASE (8B, ours)76.175.1
Micro accuracy (%)
  • CASE (8B, ours)76.1
  • Kimi K370.2
  • Claude Opus 4.868.6
  • Qwen3.7-Plus67.0
  • GPT-5.465.3
  • Qwen3.8-Max (teacher)59.7
  • Gemini-3.5-Flash56.3
  • Lingshu-32B49.9
  • Qwen3-VL-8B (base)46.0
  • Lingshu-7B45.5
  • Kimi K2.642.8
  • MedGemma-1.5-4B40.5
  • MedGemma-27B27.8

Efficiency

AgentAvg. callsMedian latencyFailure rate
CASE-8B5.134.2 s2.6%
GPT-5.44.346.1 s5.7%
Claude Opus 4.84.144.3 s5.2%

08Ablation

Table 2. Role-averaged results from a shared RSFT initialization.
VariantMicroMacroInsight [0,1]
RSFT only65.765.00.43
GRPO69.368.20.47
GRPO + RLAIF70.669.50.63
GRPO + OPSD73.872.40.57
GRPO + RLAIF + OPSD CASE-8B76.175.10.68

Each component adds on top of GRPO. RLAIF lifts insight quality the most on its own, while OPSD contributes the larger accuracy gain; combining both gives the best accuracy and insight score.

Late collapse without OPSD. In mid-to-late RL the policy tends to stall in Phase 4, restating findings instead of committing to an answer. Without the OPSD brake, GRPO+RLAIF's correctness reward collapses late in training (shaded band in Figure 3), whereas CASE-8B keeps improving.

Line plot of correctness reward from step 1,000 to 1,912 for CASE-8B, GRPO plus OPSD, and GRPO plus RLAIF, with a shaded band marking GRPO plus RLAIF's late collapse.
Figure 3. Correctness reward, steps 1,000–1,912, for CASE-8B (teal), GRPO+OPSD (red), GRPO+RLAIF (blue). The band marks GRPO+RLAIF's late collapse without the OPSD brake.
Two panels of full-epoch training dynamics from step 1 to 1,912: (a) mean return and (b) correctness reward.
Figure 7. Full-epoch training dynamics (steps 1–1,912): (a) mean return; (b) correctness reward.

09Evidence dependence

Accuracy depends on what the agent can gather. Removing adaptive acquisition, or any evidence source, lowers accuracy — the largest drops come from answering the question alone or from initial context only.

Table 16. Full-policy Micro accuracy under evidence conditions.
Evidence conditionMicroDrop (pp)Avg. calls
Question only38.637.50
Initial clinical context52.723.40
Fixed evidence packet66.010.10
Adaptive acquisition, all sources76.1—5.3
Adaptive, without MRI views65.910.24.7
Adaptive, earlier visit only64.611.53.9
Adaptive, without reports71.74.43.7

10Agent in action

Interactive trajectory walkthrough — coming soon. Below is a static summary of one Consultant trajectory.

ConsultantMulti-selectHepatic

Which hepatic conditions apply to this participant?

  • A Hepatic steatosis
  • B Benign hepatic lesion
  • C Primary liver malignancy
  • D Liver cirrhosis
  • E None

Reference answer: {A, B}

  1. 1
    recall_skills(liver)

    “Stable lesions without invasive characteristics are less suggestive of malignancy.”

  2. 2
    list_available_images

    Liver IDEAL, whole-body Dixon, pancreas.

  3. 3
    compare_visits(IDEAL multiecho, V1 vs V2)

    Diffuse fat-related signal increase; persistent focal lesion without enlargement.

  4. 4
    view_image(Dixon 8_W, V1, slices 10/22)

    Well-circumscribed lesion, smooth contour, no invasion.

  5. 5
    compare_visits(Dixon slice 22)

    Lesion stable; steatosis progression.

  6. 6
    vista3d_compare_visits(liver)

    1150.8 ml → 1263.4 ml +9.8%

  7. 7
    get_report(liver, impression)

    Steatosis, hepatomegaly, focal lesion needing characterization.

  8. 8
    get_report(wholebody, findings)

    No confirmed liver cancer.

  9. P3
    Verify
    • A verified
    • B supported
    • C not confirmed
    • D contradicted
    • E rejected
  10. P4
    Insight

    The decision is whether the lesion is malignant — stable, no invasion, no oncological history → benign lesion plus steatosis.

  11. P5
    Answer: A, B

    Matches the reference {A, B}.

  12. +
    write_skill

    Records the lesson for future cases.

Animated explainer

Coming soon.