Small model, careful pipeline

Over 30 versions and 16 blind review rounds, the small model rose from 0.647 to 0.846 on a weighted gold checklist (0.879 when the best of three runs is picked by a reference-free verifier). The frontier reference scored 0.782. In blind review by an LLM supervisor the two could not be told apart at this sample size: the mean paired difference was +0.05 (SD 0.49). The same, unchanged reference brief received between 3.50 and 4.03 from that supervisor across 13 rounds, so the judge's own noise turned out to be a finding in its own right.

Contents

We used one such task as a testbed: a structured brief that a CBT therapist can read in about three minutes before a session. It lists core beliefs with session counts, rules, recurring distortions, typical ABC episodes, tested predictions, the last session and points the therapist may want to bring to today's session. By session 20 the material is around 130,000 words.

The task suits a methods question because it has a known structure (Beck's cognitive model), a natural notion of verifiable evidence (the patient's verbatim words) and several traps: a belief that was first worded by the therapist, or a metaphor that is not a statement about death.

Two synthetic patients

Both patients were generated with our layered synthetic-patient method (profile sheet, therapeutic arc, session plan and voice guide first, then sessions generated one by one as ASR-style transcripts). Anita: 20 sessions, 131,498 words, J. Beck-style CBT. Markus: 12 sessions, 83,978 words, with a therapist who works partly in schema therapy, so the brief has to translate the material back into CBT terms. For each patient a gold reference was written independently from a full reading of the transcripts. No real patient data were used.

Best of three, without looking at the answer

Run-to-run differences within one version were often larger than differences between versions. So the brief-level pipeline runs three times, and a verifier picks one: it takes the 20 most salient patient sentences and asks the classifier whether each is reflected in the brief. The verifier never sees the gold reference or the checklist.

• Two metrics. A gold checklist of 25 weighted statements per checkpoint, graded by the classifier, and a blind review by an LLM supervisor from a different model family, scoring seven criteria on a 1–5 scale.
• A fixed anchor. The supervisor always also sees one unchanged frontier reference brief, under a random letter, so the judge's own drift is measured in every round.
• One change at a time. Failures present in all three runs and the supervisor's objections drive one targeted change per round, informed by a literature search.
• A fixed stop rule. Two consecutive rounds without improvement end the loop. It stopped at v28.
• A fixed cost ceiling. No model more expensive than the small model or the classifier may be used inside the pipeline.

Content coverage: the gold checklist

The small model started well below the reference (0.647 vs 0.782), crossed it at v25 and reached 0.846 at v28. All three v28 runs (0.818, 0.842, 0.879) lie above the reference. Two later versions did not improve on v28, and the loop stopped.

	Anita, before S13	Anita, before S20	Markus, before S12
v28, mean of 3 runs	0.963	0.800	0.776
v28, best-of-3 (verifier)	0.963	0.836	0.836
GPT-5.5 reference (1 run)	0.963	0.655	0.727

Means over checkpoints: v28 0.846 · best-of-3 0.879 · reference 0.782.

Blind review: within the judge's noise

In the final rounds the best small-model version and the reference scored 3.77 vs 3.87 (R28) and 3.77 vs 3.57 (R29). Across six same-round pairs the difference was −0.4, +0.2, −0.1, −0.5, +0.9, +0.2: mean +0.05, SD 0.49. Neither system reached the supervisor's target average of 4.0. By criterion, v28 was ahead on session trace and calibration, behind on accuracy (agreement with the gold reference) and quotes, and level on CBT levels.

Cost

One v28 brief costs about USD 0.03 in section writing and takes 2.5–3 minutes; a best-of-3 brief about USD 0.11. Processing a new session adds about USD 0.07, mostly the classifier pass over every sentence. At list prices the frontier reference brief costs about 19 times a single v28 brief. The entire project used USD 5.57 of small-model tokens.

What helped, and what did not

Three things helped: fixing defects in deterministic code that the supervisor found, a high reasoning-effort setting for the section writers (the highest setting cost more and scored lower), and moving the suggested agenda, the prediction bar and the most recent risk-related statement into code. Several plausible additions hurt: fusing three drafts, a code-imposed belief timeline, salience hints with a coverage-repair pass, and rubric-guided self-revision. Each of them added a generation step in which a small model can lose or distort information. Every change that removed work from the model, or selected among outputs without rewriting them, was safe.

Verifier selection improved the checklist score for seven of eight versions; for v28 from 0.846 to 0.879, close to the oracle (0.885). Because the verifier and the checklist grader are the same classifier, part of this gain may reflect shared preferences.

What stayed hard

Persistent failures fell from 25 to 15 of 164 weight units between v22 and v28. What remains are the subtlest judgments of the cognitive model: the state of a belief after it is reactivated, belief ranges that stop before the latest material, and scope qualifications. Lines about risk were the hardest component: both missing a recent statement and turning a metaphor, an exercise or an emotion rating into a signal were penalised. The best design kept only the most recent literal statement and the session in which the therapist last asked about it directly, and left every interpretation to the therapist.

Across 13 rounds the identical reference brief received a round mean of 3.71 with SD 0.18 and a range of 3.50–4.03 on a 1–5 scale. On single criteria the SD reached 0.46, including for quotes that were script-verified and identical every time. That noise is about as large as the whole improvement the loop produced on the same scale.

• Synthetic data only. Two synthetic patients, three checkpoints, three runs per version. Descriptive statistics only, no significance tests. Transfer to other tasks and to real material is argued, not shown.
• Model-based evaluation only. No clinicians or other human raters took part, by design. Scores describe agreement with a gold checklist and an LLM supervisor's judgment, not clinical quality, safety or any effect on therapy.
• Tuned on the evaluation set. All versions were developed against the same checkpoints and judges, so final scores are optimistic. A held-out patient would be the proper test.
• Indistinguishable is not better. In blind review the small model and the frontier reference could not be told apart at this sample size; the differences lie within the judge's own noise.
• Shared classifier. The checklist grader, the sentence scanner and the verifier are the same model.
• A modest reference. The frontier reference is one earlier, unoptimised pipeline run once with GPT-5.5, not a frontier pipeline tuned with the same loop.

What is in the ZIP

• paper/: the paper as PDF
• figures/: nine figures plus the numbers behind them
• scores/gold_checklists.json: 3 × 25 weighted statements
• scores/checklist_results.json: per-item probabilities for 198 graded briefs
• scores/blind_review_R28.json, R29: seven criteria, verdicts and objections, with letter keys
• example-briefs/: three final v28 briefs (in Polish, like the transcripts)

Not included: synthetic transcripts, prompts and code. They are available from the authors on request for research use.

How to cite

@misc{glowacki2026smallmodel,
  title  = {Small Model, Careful Pipeline: An Autoresearch Method for Closing the Gap to Frontier LLMs on Long, Multi-Document Summarisation},
  author = {G{\l}owacki, Bart{\l}omiej and G{\l}owacka, Joanna and G{\l}owacki, Bohdan and Cylwik, Ewa},
  year   = {2026},
  note   = {Preprint, AITHENTICA P.S.A. / Therapy Support},
  url    = {https://aitherapy.support/research/small-model-careful-pipeline/}
}