Architecture

The mechanical detail of how the local council and the bench harness are built. For the plain-English version, see Process.

Cabinet

Each industry seat is a real fine-tune from a different lineage — not a generalist with a system-prompt costume. Med42 is m42-health's clinical alignment of Llama 3.1; Saul is Equall.ai's continued pretrain of Mistral 7B on 30+ billion tokens of US/UK/CA/AU legal text; Qwen-Open-Finance-R is DragonLLM's instruction tune of Qwen3-8B-Base on a 50%+ finance-domain corpus.

Why two date columns? A fine-tune adds domain text on top of a pretrained backbone, but it does not refresh the backbone's world model. The Healthcare seat's Llama 3.1 backbone is anchored at Dec 2023 for facts about anything other than its clinical-corpus delta — so when Med42 reasons about, say, the FDA approval status of a drug, it's drawing on Dec 2023 Llama plus whatever clinical-pathway text was in m42-health's 2024 fine-tune corpus, not on a true 2024 world model. Spread across the cabinet, the effective "recency horizon" ranges from late 2023 (Legal) to early 2025 (Finance), which is exactly the kind of thing case 4 probes — the Healthcare seat misreporting its own cutoff as 2021 (it's actually Dec 2023) is the epistemic-honesty failure mode that test was designed to catch.

These dates are best-public-evidence summaries — Microsoft, Meta, Mistral, and Alibaba publish backbone cutoffs differently (some precise, some "approximately"), and the fine-tune release dates come from each lab's HuggingFace card or paper. Treat the backbone cutoff column as anchoring the model's generalist knowledge; the fine-tune layer can push specific domain facts forward but does not retroactively update everything else.

The Lead is intentionally not industry-specialized. An industry-tuned Lead would bias the synthesis toward whatever domain it was tuned on. Phi-4's job is reasoning and integration; it stays a generalist by design.

Why these four models

Model selection on a fanless M5 with 32 GB unified memory is a constrained-optimization problem, not a "pick the best one" problem. Sequential mode loads each seat one at a time, but the Lead stays warm across all five phases (planner, three seat consultations, synthesis), so the practical constraint is Lead memory + the largest seat + KV cache headroom + 6 GB reserved for the OS. That bounds the Lead at ≤14B params at Q4 quantization and the specialists at ≤8B each.

On top of the memory budget, every selection had to clear seven gates: (1) GGUF available on Hugging Face via a mirror Ollama can pull from; (2) permissive license sufficient for research use (MIT, Apache 2.0, or Llama Community); (3) ChatML-compatible template OR a fixable template via a local Modelfile; (4) fine-tune from a recognized lab rather than an anonymous community upload; (5) distinct lineage per seat to avoid four-of-a-kind monoculture; (6) active maintenance (last update within ~12 months of the selection date); (7) downloads or community traction high enough to suggest someone outside the authors has actually used it.

The four tables below show the actual candidates considered per seat, with the same column shape so the choice rationale is directly comparable. Selected models are highlighted; rejected candidates carry the specific reason for rejection.

Lead seat — Phi-4 14B

Healthcare seat — Llama3-Med42-8B

Legal seat — Saul-7B-Instruct-v1

Finance seat — Qwen-Open-Finance-R-8B

Taken together the cabinet is four labs (Microsoft, m42-health, Equall.ai, DragonLLM), four backbones from three families (Phi, Llama 3.1, Mistral 7B, Qwen3), four licenses (MIT, Llama Community, MIT, Apache 2.0), and three quantizations (Q4_K_M ×3 for Lead and the two smaller seats; Q8_0 for the Finance seat, since the Qwen3-8B base was small enough to afford the higher-precision quantization at planning time). Lineage diversity matters for the experiment: if all four seats were Llama-family, an artifact in the Llama family's training data could look like a council problem; with four families, that confound is reduced.

The static / dynamic distinction (load-bearing)

The single most important thing to understand about how the council thinks: the Lead does not write system prompts dynamically per query. Five system prompts are fixed across every run of every case — the planner prompt, the three per-seat prompts, and the synthesis prompt. What the Lead writes dynamically is the sub-question text sent to each routed seat. The orchestrator concatenates the seat's static system prompt with the Lead's dynamic sub-question and dispatches that pair.

This design choice is deliberate and load-bearing for three reasons: (1) static system prompts are auditable — every run of every case saw the exact same 5 strings, so behavior changes can be attributed to the dynamic sub-question rather than to prompt drift; (2) Anthropic's prompt cache (used by the bench harness's Opus modes) cuts cost ~90% precisely because the system prompts are stable across calls; (3) allowing the Lead to mutate seat system prompts per query would create a hidden adversarial surface where a malformed planner output could effectively jailbreak a seat. The line between "what the Lead controls" (routing + content of the sub-question) and "what the cabinet author controls" (the system prompts) stays clean.

Step-back prompting (Lead planner phase)

"Step-back prompting" is from Zheng et al. 2024 (Take a Step Back: Evoking Reasoning via Abstraction in Large Language Models, DeepMind, ICLR 2024). The core insight: ask the model to first abstract the question into a more general formulation before answering. This forces a planning step that small instruction-tuned models otherwise skip.

The planner system prompt walks the Lead through four explicit steps before it emits any routing decisions:

1. STEP 1 — STEP BACK. State in one sentence what the user is really trying to figure out, abstracting away from the framing of their question.
2. STEP 2 — DECOMPOSE. Identify the specific analyses needed, and which specialist owns each.
3. STEP 2.5 — RECENCY CHECK. Decide whether answering this depends on guidance, rates, rulings, or evidence that may post-date the specialists' training cutoffs. If so, attach a recency-disclosure note to each dispatched sub-question.
4. STEP 3 — DISPATCH. For each routed specialist, write a self-contained sub-question that asks only for that domain's contribution — no cross-domain framing.

The planner then emits strict JSON in a fixed schema (shown below). Temperature is held at 0.0 for this phase — the goal is deterministic routing, not creative variation.

The planner output schema

{
  "core_question": "the abstracted core decision or analysis",
  "rationale": "1-2 sentences on why these specialists are needed",
  "routes": ["healthcare", "legal", "finance"],
  "sub_questions": {
    "healthcare": "self-contained question for the healthcare seat",
    "legal":      "self-contained question for the legal seat",
    "finance":    "self-contained question for the finance seat"
  },
  "requires_recency_disclosure": true,
  "recency_notes": "what specifically may be stale (empty if false)"
}

The orchestrator validates this shape with _validate_plan(): routes must be a subset of {"healthcare", "legal", "finance"}; sub_questions must contain exactly one entry per routed seat; missing or malformed JSON falls back to a defensive "no consultation needed" mode. If requires_recency_disclosure is true, the orchestrator appends a short recency-flag instruction to every dispatched sub-question — that's the only place the orchestrator mutates the Lead's text.

Dispatch — exactly what each seat sees

Each seat receives a two-message conversation. Nothing else. The seat never sees the original user query; it never sees the other seats' system prompts; it never sees the planner's decomposition. Just two messages, in order:

1. System message — the seat's static system prompt (e.g. HEALTHCARE_SYSTEM). Identical bytes across every run.
2. User message — the Lead's dispatched sub-question for this seat, optionally appended with a one-paragraph recency note when the planner flagged the query as recency-sensitive.

This is the architectural fix for the lane-bleed problem we hit in v2: small instruction-tuned models cannot reliably "stay in their lane" when the user's prompt explicitly asks for cross-domain analysis ("across clinical, legal, and financial dimensions…"). Earlier iterations sent the original multi-domain user message to every seat with a "DO NOT cross lanes" system instruction. The instruction lost to the framing, every time. Sub-question dispatch fixes this at the source by removing the cross-domain framing the seat is ever exposed to — so it has nothing to drift toward.

Tension-extraction synthesis (Lead's final phase)

The synthesis call receives a single bundled user message: the original user question (so the Lead can re-anchor) followed by each seat's full output, clearly demarcated. The synthesis system prompt enforces a strict two-block output:

1. ## Tensions — 3–5 specific cross-domain disagreements, trade-offs, or assumption-fragilities. Vague observations like "all three matter and must be balanced" are explicitly rejected in the prompt. If the Lead cannot identify three real tensions, it is instructed to re-read the contributions, not produce filler.
2. ## Synthesis — the integrated final answer, with the prompt requiring that modeled numbers stay flagged as modeled, that precise jurisdictional vocabulary is preserved (no smoothing for accessibility), and that any training-cutoff caveats raised by seats are propagated forward.

The structural enforcement matters at 14B-Q4 scale. Without it, a Phi-4-Q4 synthesis produces a fluent summary that loses the specialist precision — by the time it's been compressed and re-styled, the legal-jurisdictional distinctions and modeled financial assumptions get flattened into confident prose. The tension-extraction step forces the Lead to read for disagreement before writing for cohesion.

The five static system prompts (full text)

Below are the exact bytes each phase saw across every run in the experiment. Each prompt is collapsed by default to keep the page readable; expand to audit. These are also surfaced live via the /api/prompts endpoint and shown per-phase in the Results page inspector.

You are the Lead Agent of a Council of Experts. Your job is to DECOMPOSE the user's question and DISPATCH precise sub-questions to the right industry specialists.

Available specialists:

- "healthcare": Llama3-Med42-8B clinical fine-tune. Strong on clinical reasoning, medical evidence, standards of care, patient safety, healthcare operations, regulatory pathways for therapeutics and devices, healthcare workforce.
- "legal": Saul-7B-Instruct-v1 legal fine-tune. Strong on US, UK, Canadian, and Australian statutory and case-law frameworks, regulatory compliance, contracts, corporate law, jurisdictional distinctions.
- "finance": Qwen-Open-Finance-R-8B finance fine-tune. Strong on unit economics, valuation frameworks, capital structure, risk modeling, banking and insurance, financial regulation.

Approach the routing in three steps. Show your reasoning briefly (one short paragraph total across all three steps), then output the JSON.

STEP 1 — STEP BACK. State in one sentence what the user is really trying to figure out, abstracting away from the specific framing of their question.

STEP 2 — DECOMPOSE. Identify the specific analyses needed to answer the core question, and which specialist owns each. If a question doesn't require any specialist (e.g. organizing a closet), skip to the empty-routes output.

STEP 2.5 — RECENCY CHECK. Decide whether answering this question depends on recent guidance, rates, rulings, prices, regulations, or evidence that may post-date the specialists' training cutoffs. Examples that trigger recency disclosure: drug pricing or rebates, recent FDA decisions or labeling changes, evolving treatment guidelines, recent regulatory rulings, court decisions, statute amendments, current rate environments, recent agency enforcement guidance. If yes, set "requires_recency_disclosure": true and write a brief "recency_notes" string naming WHAT specifically may be stale (e.g. "GLP-1 evidence post-2024 and recent EEOC/ADA guidance"). If the question is robustly answerable from durable, non-time-sensitive foundations, set it to false.

STEP 3 — DISPATCH. For each routed specialist, write a SELF-CONTAINED sub-question that asks ONLY for their domain's contribution. Each sub-question must:

  - Include enough scenario context to be answered without the original user question (the specialist will not see the original).
  - Focus narrowly on the specialist's domain. DO NOT ask for cross-domain analysis. DO NOT write phrasings like "across clinical, legal, and financial dimensions" — that framing is what causes specialists to bleed across lanes.
  - Be specific and answerable, not a topic or theme.
  - End with a clear question or directive ("What is the v1 clinical-safety design?", "Walk through the actuarial reserve calculation", "What does the FDA SaMD pathway look like for this device class?", etc.).

After your brief reasoning, output STRICT JSON in EXACTLY this shape, as the LAST content of your response (no markdown fence, no trailing prose):

{
  "core_question": "the abstracted core decision or analysis",
  "rationale": "1-2 sentences on why these specialists are needed",
  "routes": ["healthcare", "legal", "finance"],
  "sub_questions": {
    "healthcare": "self-contained question for the healthcare seat",
    "legal": "self-contained question for the legal seat",
    "finance": "self-contained question for the finance seat"
  },
  "requires_recency_disclosure": true,
  "recency_notes": "brief description of what may be stale (or empty string if false)"
}

Rules:
- "routes" must be a subset of {"healthcare", "legal", "finance"}.
- "sub_questions" must contain exactly one entry per element of "routes" (and no extras).
- If "routes" is empty, "sub_questions" must be {}.
- "requires_recency_disclosure" must always be present (true or false).
- "recency_notes" must be a string; empty when requires_recency_disclosure is false.

You are the Healthcare specialist on a Council of Experts. You are Llama3-Med42-8B, a clinical fine-tune of Llama 3.1 8B with multi-stage preference alignment for medical reasoning, trained by m42-health.

You will receive a focused clinical question from the Lead Agent. Answer it directly and substantively, using your clinical training.

What good clinical contribution looks like:
- Precise about standard of care vs emerging practice
- Cites guidelines, trials, or organizations by name when relevant
- Addresses harm pathways concretely (false-positive / false-negative tolerance, alert fatigue, workflow integration, etc.) when relevant
- Distinguishes computer-aided detection from autonomous decision-making when liability framing is implicit
- Flags training-cutoff uncertainty EXPLICITLY when guidelines or evidence may have evolved since your training

Format: 4–8 paragraphs of substantive clinical prose. Light structural headings only when they mark genuinely different clinical sub-topics. Prose is preferred over bullet lists.

You are the Legal specialist on a Council of Experts. You are Saul-7B-Instruct-v1, a continued pretrain of Mistral 7B on 30+ billion tokens of US, UK, Canadian, and Australian legal text, then instruction-tuned, by Equall.ai.

You will receive a focused legal question from the Lead Agent. Answer it directly and substantively, using your legal training.

What good legal contribution looks like:
- Uses precise legal vocabulary (e.g. distinguishes "approval" vs "clearance," "regulation" vs "directive," "holding" vs "dicta")
- Distinguishes jurisdictions explicitly when more than one is in play (US, UK, EU, Germany, Canada, Australia) — never conflates regimes
- Distinguishes black-letter law from open questions
- Addresses statutory framework, case-law treatment, and contractual considerations as relevant
- Flags training-cutoff uncertainty EXPLICITLY when statutes, regulations, or rulings may have evolved since your training

Format: 4–8 paragraphs of substantive legal prose. Light structural headings only when they mark genuinely different legal sub-topics. Prose is preferred over bullet lists.

You are the Finance specialist on a Council of Experts. You are Qwen-Open-Finance-R-8B, an instruction fine-tune of Qwen3-8B-Base on a corpus that is over 50% finance-domain text in English, French, and German, by DragonLLM.

You will receive a focused financial question from the Lead Agent. Answer it directly and substantively, using your finance training.

What good finance contribution looks like:
- Uses proper finance frameworks (DCF, unit economics, actuarial reserve, capital adequacy, break-even, sensitivity, etc.)
- SHOWS your work when numbers matter — write the steps, not just the result
- FLAGS specific numbers as MODELED ASSUMPTIONS, not facts. Use language like "modeled at," "assuming," "under the assumption that..."
- Flags training-cutoff uncertainty EXPLICITLY when rates, prices, or recent regulatory changes may have evolved since your training

Format: 4–8 paragraphs of substantive financial prose. Light structural headings only when they mark genuinely different financial sub-topics. Prose is preferred over bullet lists.

You are the Lead Agent of a Council of Experts. You have just received contributions from one or more industry specialists. You will produce the FINAL response in TWO STRUCTURALLY ENFORCED STEPS.

STEP 1 — TENSIONS. Identify 3–5 SPECIFIC tensions, disagreements, or trade-offs across the specialist contributions. A "tension" is a place where following one specialist's recommendation makes another specialist's recommendation harder, or where a specialist's modeled assumption is fragile in light of another specialist's analysis. Examples of REAL tensions (illustrative, do not copy):
- "Clinical caution argues for slow rollout, but the financial model assumes 50% adoption by year 2; under slower adoption the unit economics change materially."
- "Legal flags FDA classification as guidance-evolving, but product timeline assumes 510(k) is a 6-month effort; this is a schedule risk."
- "Finance's pricing assumption ($X per seat) is set independently of clinical use; if alert fatigue suppresses use the per-encounter realized price collapses."

If you cannot find 3 real tensions, you have not read the contributions carefully enough. Try again. Vague observations like "all three matter and must be balanced" are NOT tensions.

STEP 2 — SYNTHESIS. After listing tensions, write the integrated final answer to the user's question. The synthesis MUST:

1. Acknowledge the tensions you just identified — not abstractly, but at the points where they bite the answer.
2. PRESERVE numeric framing — if a specialist labeled a number as a modeled assumption, your synthesis MUST also label it as an assumption (use "modeled at," "assumed"). Never adopt a specialist's modeled number as a fact.
3. PRESERVE precise vocabulary — if a specialist used a specific legal term, jurisdiction, FDA pathway, or clinical concept, preserve it; do not smooth out precision in the name of accessibility.
4. PRESERVE caveats — if a specialist flagged training-cutoff uncertainty or data fragility, propagate that flag into your synthesis.
5. Use whatever structure (headers, lists, prose) best fits the user's question. Do NOT impose a "Healthcare / Legal / Finance" template if the question deserves a different shape.

OUTPUT FORMAT — EXACTLY this shape:

## Tensions
- [tension 1, one or two sentences]
- [tension 2, one or two sentences]
- [tension 3, one or two sentences]
- [optionally tension 4 and/or 5]

## Synthesis
[the integrated final answer, with structure that fits the question]

The user's original question and the specialist contributions follow.

You are the Lead Agent of a Council of Experts. The user's question does not require any specialist consultation (no clinical, legal, or financial expertise is needed). Answer it directly, briefly, and helpfully — using your general reasoning capability.

Worked example — Case 4 (GLP-1 employer coverage), end-to-end

The example below is a real captured run of the local council on Case 4. Every text block is verbatim from the audit log; nothing has been edited for presentation. The point is to make the dispatch mechanism concrete: this is the user query, this is what the planner emitted, this is the exact message the Healthcare seat received, and this is what came back.

A 5,000-employee self-insured employer is deciding whether to cover GLP-1 medications (semaglutide, tirzepatide) for obesity rather than only for type 2 diabetes, and under what utilization-management criteria. Build out the recommendation across clinical, legal, and financial dimensions.

{
  "core_question": "Should a self-insured employer cover GLP-1 medications for obesity, considering clinical efficacy, legal compliance, and financial impact?",
  "rationale": "The decision involves evaluating the latest clinical evidence for safety and effectiveness, understanding legal requirements for coverage under self-insured plans, and analyzing the financial implications of such coverage.",
  "routes": ["healthcare", "legal", "finance"],
  "sub_questions": {
    "healthcare": "What is the current clinical evidence supporting the use of GLP-1 medications (semaglutide, tirzepatide) for obesity in terms of safety and efficacy?",
    "legal":      "Are there any legal or regulatory considerations that a self-insured employer must address when deciding to cover GLP-1 medications for obesity?",
    "finance":    "What are the financial implications for a 5,000-employee self-insured employer covering GLP-1 medications for obesity, including cost analysis and potential savings?"
  },
  "requires_recency_disclosure": true,
  "recency_notes": "Recent clinical evidence for GLP-1 use in obesity treatment and any new regulatory or legal guidelines affecting coverage decisions."
}

role: system
content: [HEALTHCARE_SYSTEM verbatim — see prompt 2 above]

role: user
content: What is the current clinical evidence supporting the use of GLP-1 medications (semaglutide, tirzepatide) for obesity in terms of safety and efficacy?

Note from the Lead: answering this depends on guidance, evidence, rates, or rulings that may have evolved since your training cutoff. Explicitly flag training-cutoff uncertainty wherever it applies in your answer. Specifically, watch out for: Recent clinical evidence for GLP-1 use in obesity treatment and any new regulatory or legal guidelines affecting coverage decisions.

The current clinical evidence supporting the use of glucagon-like peptide-1 (GLP-1) medications, specifically semaglutide and tirzepatide, for obesity management is robust in terms of both safety and efficacy. As a trained Llama 3-Med42-8B model, my training cutoff is up to December 2022; however, I will flag any potential uncertainty due to recent developments.

Semaglutide (Ozempic) has been extensively studied in the context of obesity treatment. The SEMIMAN trial demonstrated that semaglutide 2.4 mg once weekly resulted in significant weight loss and improvements in cardiometabolic risk fa…

role: system
content: [LEAD_SYNTHESIS_SYSTEM verbatim — see prompt 5 above]

role: user
content: USER QUESTION:
A 5,000-employee self-insured employer is deciding whether to cover GLP-1 medications…

---
HEALTHCARE CONTRIBUTION:
[full Healthcare seat output, with <think> blocks stripped]

---
LEGAL CONTRIBUTION:
[full Legal seat output]

---
FINANCE CONTRIBUTION:
[full Finance seat output]

Iteration history — what we tried before this worked

The static prompts file council/prompts.py carries an iteration log going back to v1. Two earlier versions failed in instructive ways, and the failures shaped the current design:

• v1 (polite "stay in your domain" framing). Each seat's system prompt asked the model to focus on its domain. All six rubric must-haves missed on the first case. Seats produced three-section comprehensive answers covering clinical, legal, and financial regardless of role. Diagnosis: instruction compliance is weak at sub-10B chat-model scale; "please" loses to framing in the user message.
• v2 (aggressive DO / DO NOT lane enforcement). Each seat's prompt explicitly forbade cross-domain content ("DO NOT discuss legal or financial concerns"). Tensions extraction worked; lane enforcement still failed. Diagnosis: the user query contained explicit cross-domain framing ("across clinical safety, legal and regulatory exposure, and financial structure"), which overrode the system prompt every time at sub-10B scale. You can't out-prompt framing on a small model.
• v3 (current — architecture change, not prompt tuning). The Lead does step-back decomposition and dispatches narrow sub-questions; each seat sees ONLY its sub-question, never the original user query. Cross-domain framing is removed at the source so seats have nothing to drift toward. Seat prompts consequently simplified back to identity + output norms (no DO NOT lists at all).

The version progression is the load-bearing lesson of the project's prompt-engineering phase: when small instruction-tuned models fail to follow scope instructions, the fix is to remove the scope-tempting framing, not to strengthen the instruction. Architecture beats prompting at sub-10B scale.

Memory budget

Sequential mode peak ~24.7 GB on a 32 GB unified-memory Mac. The single biggest performance knob on Apple Silicon is raising the Metal wired-memory limit:

sudo sysctl iogpu.wired_limit_mb=26000

That lifts the cap from the ~24 GB default heuristic to 26 GB while leaving 6 GB for the OS. Resets on reboot; persist via /etc/sysctl.conf if you want it sticky.

Parallel mode (all four models loaded simultaneously) would peak around 34 GB — too tight for 32 GB without self-quantizing the Finance model from Q8_0 to Q4_K_M (a 30-minute one-time procedure using llama.cpp tools).

Thermal policy

The MacBook Air M5 has no fan. Sustained inference triggers Metal throttling silently — no error, no log, just halved throughput. council/thermal.py inserts a configurable inter-agent pause (default 5 seconds, set via COUNCIL_AGENT_PAUSE_SECONDS). A future iteration polls pmset -g thermlog to lengthen pauses under amber/red thermal pressure.

Bench methodology

Five modes run side-by-side on the same input — three baseline modes plus a separate Mixture-of-Experts comparison surface added after the headline experiment:

• local-council — the Phi-4 + Med42 + Saul + Qwen-Finance pipeline above. Zero API calls. Local Ollama handles inference.
• opus-single — one Claude Opus 4.7 call with a neutral system prompt. The "what does the frontier model produce on its own" baseline.
• opus-council — Opus playing every seat in the same orchestration loop the local council uses. Same step-back planner, same per-seat dispatch, same tension-extraction synthesis prompt. Only the model differs.
• gptoss-single — one gpt-oss-20B call (OpenAI's first open-weights release, MoE design, runs locally via Ollama). Same neutral system prompt as opus-single so the comparison isn't muddled by prompt drift.
• gptoss-council — gpt-oss-20B playing every seat in the council architecture, mirroring opus-council but local. Tests "does the council architecture buy us anything on top of a strong open MoE generalist?"

The architectural payoff is that each new model surface is ~30-50 lines of code on top of the existing orchestrator — the orchestrator's chat_fn parameter accepts an Anthropic-backed callable for Opus modes or an Ollama-backed callable for gpt-oss modes. Same prompts, same loop, same audit-log shape across all five modes.

Mixture-of-Experts (MoE) architecture — reference

The two gptoss-* modes use gpt-oss-20B, which has a different fundamental architecture than the dense transformers used by every other model in the cabinet (Phi-4 14B, Llama 3.1 8B, Mistral 7B, and the dense variant of Qwen3-8B are all standard dense transformers). Because this architectural difference matters for interpreting the comparison results, here is a quick reference on what MoE is and what it changes.

Dense transformer vs MoE — what changes

A standard ("dense") transformer activates every parameter on every token. A 14B dense model passes each token through ~14 billion parameters; an 8B dense model passes each token through ~8 billion. Memory and per-token compute scale with the full parameter count.

A Mixture-of-Experts model replaces specific layers (typically the feed-forward layers, which carry most of a transformer's parameters) with multiple parallel "expert" sublayers plus a learned router. For each token, the router picks the top k experts (commonly 1, 2, or 4 out of dozens) and only those experts compute. Memory still has to hold all experts (you don't know which will be picked until inference time), but per-token compute is bounded by the active experts, not the total parameter count.

Concretely: gpt-oss-20B has 20 billion total parameters but activates roughly 3.6 billion per token. So its memory footprint is roughly that of a dense 20B model (~14 GB at Q4), but its per-token compute and latency are closer to a dense 4B. The trade that buys is more model capacity (more "knowledge" stored in experts) without paying the dense-compute cost for capacity you don't use on a given token.

Why MoE matters for this experiment

Three reasons MoE is worth a separate comparison surface, not just another model swap:

1. Capacity vs latency trade. If MoE delivers meaningful capability gains over a dense 14B (Phi-4) at comparable latency, that's a load-bearing finding for any future "scale up locally" decision — you'd reach for MoE first, dense second. If it doesn't, the case for the simpler dense architecture stays strong.
2. Architectural diversity in the cabinet. Right now every council seat is a dense transformer. An MoE comparison column gives us an honest read on whether the cabinet's failure modes (confabulation, false-cutoff disclosures, missed jurisdictional specifics) are dense- transformer failures or general-small-model failures.
3. Reasoning-tuned architecture. OpenAI tuned gpt-oss specifically for reasoning workloads, which is what the council's planning + tension-extraction phases stress most. A reasoning-tuned MoE is genuinely a different model category, not a sibling of the existing seats.

Why gpt-oss-20B specifically as the MoE pick

Selection criteria for the MoE comparison model matched the same seven gates the council seats had to clear (memory budget, GGUF, license, template, recognized lab, distinct lineage, active maintenance). The top candidates considered:

gpt-oss-20B is therefore the MoE comparison surface throughout the bench. It appears as two explicit modes in the Results page and in the the A/B page (local only): gptoss-single (one single-shot call, parallel to opus-single) and gptoss-council (gpt-oss playing every seat in the orchestration, parallel to opus-council). It is explicitly not labeled as a frontier stand-in — gpt-oss is a strong open-weights model, but it is not Opus-class capability. The Results UI cabinet badge reads "gpt-oss-20B · uniform cabinet" so no later reader can confuse the two.

Cost and budget enforcement

Opus 4.7 pricing (Anthropic published rates): $5 per million input tokens uncached, $0.50 per million for cached input reads, $25 per million output tokens. A full benchmark sweep — five cases through both Opus modes with system-prompt caching — comes to about $0.93 total.

The bench harness has a hard budget guardrail in bench/cost_guard.py. Before every Opus call:

1. If the cap is $0, refuse immediately — no count_tokens, no messages.create, no network call of any kind.
2. Otherwise, call /v1/messages/count_tokens (free) for an exact pre-call input estimate.
3. Compute projected cost (input × $5 + output × $25 per MTok, worst-case = max_tokens output).
4. If running total + projected cost > cap, raise BudgetExceeded and abort the call.

After each successful call, record actual usage to a JSON ledger so the running total survives crashes. Two independent failsafes: the script-side cap above, plus a monthly spend limit configured in the Anthropic Console.

Prompt caching

The five system prompts (planner, three industry seats, synthesis) are stable across the benchmark suite — same bytes every call. The Anthropic client wraps each system block with cache_control: ephemeral (5-minute TTL), so the second and subsequent invocations within a sweep read the cache at 10% of uncached input price. Verified via response.usage.cache_read_input_tokens on returned responses.

What runs where

• Council code — never imports from bench/; never opens a network socket beyond Ollama.
• Bench harness — only place Anthropic SDK calls happen. Gated behind BENCH_BUDGET_USD.
• Server — FastAPI on 127.0.0.1:8000. CORS allowlist locked to localhost. No third-party JS, no analytics, no CDN fonts.
• Static report — generated from saved run JSON; pure HTML/CSS, deployable to Netlify or anywhere static.

Honest caveats

• Training cutoffs vary across the four local models — Med42's clinical knowledge, Saul's case-law, and Qwen-Finance's regulatory data each go stale at different points.
• Single run per case in v1. No variance analysis. Low temperature keeps runs comparable; rerun manually if a result feels noisy.
• Subjective human review for scoring; no LLM-as-judge pass. The rubrics are human-readable and live in examples/test_cases.py.
• Sequential execution on a fanless Mac means later runs in a sweep can be measurably slower than earlier ones if thermal pressure builds up.
• The bench compares the local council against frontier Opus, but the local council is — at best — early-2024 GPT-4-class on raw capability. Don't read this as "local wins"; read it as "where does specialization narrow the gap."