How it works

A plain-English walkthrough of what this is, why it exists, and how the test is set up. The technical detail lives on the Architecture page.

What this is

Most people who work with AI today reach for one general-purpose model (ChatGPT, Claude, Gemini) and ask it complicated, multi-domain questions. That works surprisingly well, but it has two costs: the data leaves your machine, and the model is a generalist — strong everywhere, a specialist nowhere.

This project tests an alternative: a small council of four specialized models running entirely on a laptop. One model leads the conversation; three others — fine-tuned on healthcare, law, and finance respectively — contribute their domain expertise. Then we put the council's answer side-by-side with what frontier Claude (Opus 4.7) produces on its own.

Why fine-tunes matter

A useful analogy: you have a complicated business question that touches medicine, law, and money. You can call one extremely smart generalist, or you can convene three actual specialists plus a chair who synthesizes their input.

The generalist will give you a good answer fast. But the specialists will use precise jurisdictional vocabulary, name the right regulatory bodies, distinguish standard of care from emerging practice, and call out where the analysis depends on assumptions vs facts. Those are properties of training data, not of intelligence.

This experiment asks: do small specialist fine-tunes — running on your own machine — capture enough of that domain texture to be worth the operational complexity?

How the test works

The same question runs through five pipelines simultaneously — three baseline pipelines and two later additions that test a different model architecture (Mixture-of-Experts) without leaving the laptop:

• Local Council — four small models on the laptop. A 14-billion-parameter Lead model decomposes the question and dispatches sub-questions to whichever specialists are relevant; the specialists answer; the Lead synthesizes the final response, explicitly surfacing tensions across the three domains.
• Opus single-shot — one call to Claude Opus 4.7 with a neutral system prompt. The "frontier model on its own" baseline.
• Opus-as-council — the exact same orchestration loop as the local council, but with Opus playing every seat. This isolates the question of model vs architecture: same prompts, same reasoning structure, only the underlying weights differ.
• gpt-oss single-shot — one call to gpt-oss-20B, OpenAI's first open-weights release. A Mixture-of-Experts model that runs locally on the laptop with the same neutral system prompt as Opus single-shot. The "what does a strong local open MoE produce on its own" baseline.
• gpt-oss as-council — gpt-oss-20B playing every seat in the same orchestration loop. Lets us ask "does the council architecture buy anything on top of a strong open MoE generalist?" without spending Opus money or leaving the laptop.

For a plain-English explanation of what Mixture-of-Experts is and why it's worth a separate comparison surface, see the Architecture page's MoE section. In short: MoE models trade memory for compute by storing more total knowledge but only activating a fraction of it per token — a genuinely different architectural category from the dense transformers in the existing cabinet.

The five test cases — design and cutoff alignment

The case battery is intentionally divergent: each prompt stresses a different way the council can go wrong. Three of the five cases also sit at or past the local cabinet's training cutoffs in load-bearing ways, which matters because the cutoffs aren't uniform — Saul anchors to ~Aug 2023, Med42 to Dec 2023, Phi-4 to Jun 2024, and only Qwen-Finance reaches into 2025 (see the Architecture page cabinet table). Today is mid-2026, so each case sits 14–32 months past the relevant backbone.

For each case below: what failure mode it's designed to provoke, why it was structured that way, the specific knowledge each domain needs to surface, and a frank assessment of whether that knowledge lives inside the local cabinet's training window or has drifted past it.

Case 1 — AI clinical decision support rollout

Failure mode tested: synthesis under competing recommendations.

Rationale. The most legible three-domain case. A 12-hospital health system is rolling out a sepsis-risk CDS tool via per-seat annual subscription. Each specialist has well-established doctrine to draw on (alert-fatigue tolerance, FDA SaMD classification, subscription-vs-usage pricing). The case tests whether the Lead can integrate three legible specialist contributions into a coherent v1 design — not whether any individual seat can dig up obscure knowledge.

Verdict — ✅ Within boundaries. No load-bearing dependency on post-2024 rulemaking. Even Saul's Aug 2023 corpus covers FDA SaMD doctrine fully. The case stresses synthesis, not recency. A correct local-council answer is fully achievable in principle; if it fails, the failure points at the architecture (or the Lead's integration), not at stale training data.

Case 2 — Cross-border digital therapeutic launch (US / UK / DE)

Failure mode tested: jurisdictional vocabulary discipline.

Rationale. A US-cleared prescription SaMD launching in the UK and Germany is the textbook case for keeping multiple regulatory regimes distinct (FDA vs MHRA vs BfArM; UK GDPR vs EU GDPR post-Brexit; UK MDR 2002 vs EU MDR 2017/745; DiGA reimbursement vs NHS commissioning vs US payer billing). Models that conflate these — or that treat FDA clearance as carrying any direct regulatory weight in the EU — fail visibly. The case is designed to reward jurisdictional precision and penalize translation-layer shortcuts.

Verdict — ⚠️ Edge. The core framework (DiGA, GDPR, MDR) is all well within cutoffs and answerable accurately. The new wrinkle is the EU AI Act × MDR intersection: an adaptive ML smoking-cessation DTx triggers high-risk AI Act obligations on top of MDR conformity assessment, and that intersection only became operative in late 2024 — past both Healthcare and Legal seat backbones. opus-council surfaces it explicitly; the local council cannot.

Case 3 — Capitated Medicare Advantage risk contract

Failure mode tested: quantitative framework discipline.

Rationale. A 40-physician primary-care group is being offered a $1,180 PMPM full-risk capitated contract for a 25,000-member panel. The case tests whether each seat can do (and show) actual quantitative work — actuarial reserves, breakeven utilization, stop-loss layering — rather than hand-wave with adjectives. It's also the case most exposed to currently-evolving CMS rulemaking, which makes it the case where stale training data can produce confidently wrong numbers most easily.

Verdict — ⚠️ Problematic. The V28 risk-adjustment transition is the operational reality for any current MA capitated contract — actuarial reserves built on V24 weights will be wrong by single-digit-percent margins that matter at scale. A local-council answer using V24 logic could pass the rubric's quantitative-discipline bar (shows math, breaks even, layers stop-loss) while being confidently wrong for a 2026 contract. This is the case where adding a recency-honesty rubric item would close the silent failure mode.

Case 4 — GLP-1 employer coverage decision

Failure mode tested: recency / training-cutoff honesty.

Rationale. A 5,000-employee self-insured employer decides whether to extend GLP-1 coverage from diabetes-only to obesity, and under what utilization-management criteria. The clinical, legal, and financial facts on this question move in quarter-by-quarter increments: new SELECT-class outcomes data, compounding-pharmacy enforcement shifts, EEOC obesity-as-disability activity, CVS Caremark exclusion announcements, oral-GLP-1 approvals, list-vs-net price drift. This is the case designed explicitly to test whether each seat flags what its knowledge can't cover — not whether it knows the post-cutoff events themselves.

Verdict — ✅ Intentional. The case lands partly outside the cutoffs by design. The rubric does not reward citing the post-cutoff events — it rewards each seat explicitly disclaiming that its knowledge on net pricing, recent rulings, and post-2024 label expansions may be stale. Failure here is confident assertion, not missing information.

Case 5 — Nonprofit community hospital → private-equity conversion

Failure mode tested: adversarial cross-domain tension.

Rationale. The hardest case in the battery. A 90-bed rural nonprofit being acquired by a PE sponsor through a conversion + sale-leaseback structure looks routine to each specialist individually (the lawyer focuses on AG approval; the accountant focuses on lease economics; the clinician focuses on service-line continuity). The case is engineered so the tension between the three is the real finding: the sale-leaseback that maximizes sponsor return creates fragile operating economics that destabilize clinical service continuity and invite state AG scrutiny under the charitable-trust doctrine. A model that produces three sensible single-domain answers and then writes a synthesis that elides the conflict has failed.

Verdict — ⚠️ Edge. The analytical frame (sale-leaseback collapse risk, charitable-trust constraints, state AG review) is timeless and fully within Saul's training corpus. But the load-bearing contemporary case study (Steward), the live legislative response (CA AB 3129 and copycats), and the federal enforcement posture (FTC PE-healthcare focus) all sit at or past the Legal seat's backbone. A correct-shaped answer is possible using the Hahnemann (2019) playbook; a current-aware answer is not.

Summary — case battery against the cutoffs

The strongest pending change is Case 3: as written, a local-council answer with V24-anchored actuarial reasoning would pass the existing quantitative-discipline rubric while being materially wrong for a 2026 contract. That's a silent failure mode the rubric currently can't catch and probably should.

What we're measuring

Four properties, no single metric of "winning":

• Privacy. The local council never sends a byte off the machine. The Opus columns send the prompt to Anthropic. For sensitive material, that's a load-bearing difference.
• Cost. Local: zero per query after the one-time hardware investment. Opus: about 18 cents per query at current rates (verified against published pricing, with a hard $10/run budget cap that aborts before exceeding).
• Quality. Did the answer stay in lane? Did it surface real tensions instead of just listing concerns? Did it flag training-cutoff uncertainty when relevant? Did it preserve modeled assumptions instead of laundering them as facts? The five canonical test cases each target a specific failure mode.
• Latency. Local: about 5 minutes per question (sequential mode on a fanless laptop). Opus: a few seconds for single-shot, about a minute for the full council architecture.

Honesty about limits

Three things this experiment doesn't try to do:

• It doesn't claim local fine-tunes beat frontier models on raw capability. They almost certainly don't.
• It doesn't run multiple trials per case for variance — one run each, low temperature, subjective human review. That's enough to spot consistent patterns, not enough for statistical claims.
• It doesn't auto-grade the outputs. Scoring is a person reading each answer against a per-case rubric (alert fatigue mentioned? FDA pathway flagged correctly? UK and EU GDPR kept distinct?).

What we found

Final findings land here after Phase 4 review. For now, see the the A/B page (local only) to run the experiment yourself.