Cost-aware model routing for agents

The expensive model is rarely the right model. Most agent tasks — classification, extraction, summarization, formatting, simple Q&A — are solved just as well by a small, cheap model. The expensive model is for the 20% of tasks where capability differences compound: multi-step reasoning, long-context synthesis, code generation, hard ambiguity.

The vendor pages document this split explicitly: Anthropic’s Claude model overview, OpenAI’s models page, and the OpenRouter directory all expose tiered capability and tiered pricing.

Where this comes from: Marvin’s 2026-07-20 original. Sources verified 2026-07-22 against Anthropic, OpenAI, and OpenRouter model + pricing pages.

Before you start

You need:

• A classification scheme: what task_kind values exist in your agent (a small set: classify, extract, format, short_qa, summarize, reason, code, long_context is a starting point).
• A way to set task_kind per call (a cheap classifier, a regex on the prompt, or a manual call-site).
• A cost ledger to verify the math (see A2 + A3).

🤖 Let your agent do it (Hermes path): ask Hermes to add the routing layer.

add a router.py that picks claude-haiku-4 / claude-sonnet-4 / claude-opus-4 based on task_kind and prompt_tokens, and wire it into the existing model-call wrapper from obs.py

Hermes can scaffold the router and the integration. Verify by running python -c "from router import pick_model; print(pick_model('classify', 100, False))" and confirming it returns the small model.

The routing decision tree

classify the task
   │
   ├─ classification / extraction / formatting / short Q&A
   │     → small model (Haiku / 4o-mini / local Ollama)
   │       cost:    ~$0.0001 / call
   │       latency: ~0.3s
   │
   ├─ summarization of < 4k tokens / standard generation
   │     → mid model (Sonnet / 4o)
   │       cost:    ~$0.003 / call
   │       latency: ~1s
   │
   ├─ multi-step reasoning / long-context / code / hard ambiguity
   │     → large model (Opus / o1)
   │       cost:    ~$0.03 / call
   │       latency: ~3s
   │
   └─ unknown / user explicitly asked for "best"
         → mid model + retry-on-uncertainty escalation

Most agents route 70–80% of calls to the small model. That is where the budget stays under control.

The router

# router.py
def pick_model(task_kind: str, prompt_tokens: int, has_tools: bool) -> str:
    if task_kind in {"classify", "extract", "format", "short_qa"}:
        return "claude-haiku-4"                # ~$0.0001/call

    if task_kind == "summarize" and prompt_tokens < 4000:
        return "claude-sonnet-4"              # ~$0.003/call

    if task_kind in {"reason", "code", "long_context"} or prompt_tokens > 8000:
        return "claude-opus-4"                # ~$0.03/call

    return "claude-sonnet-4"                  # default = mid

The task_kind is set by a cheap classifier (often the small model itself, or a regex on the prompt). The classification call costs ~$0.0001. Even if it gets the routing wrong half the time, you save more than you spend.

The escalation pattern

Cheap-first is the right default, but you cannot let under-routing silently degrade quality. The pattern:

try with small model
  │
  ├─ confident answer                  → return
  │
  ├─ uncertain (low confidence score,
  │  schema mismatch, or "I don't know")
  │                                    → retry with mid model
  │                                       │
  │                                       ├─ confident            → return
  │                                       └─ still uncertain      → retry with large model
  │
  └─ tool error / timeout               → retry once with mid

The retries are the cost of insurance. Track them: if the small model escalates > 30% of the time, your classifier is misrouting and you have a training-data problem.

The cost math (concrete)

Assume 1,000 calls/day, 70/20/10 split across small/mid/large:

Tier	$/call	Calls/day	$/day
Small (Haiku)	$0.0001	700	$0.07
Mid (Sonnet)	$0.003	200	$0.60
Large (Opus)	$0.03	100	$3.00
Total		1000	$3.67/day

The same 1,000 calls routed all-large would cost $30/day. Routing saves ~88% without changing capability for the tasks that need it.

These numbers track the Anthropic pricing and OpenAI pricing pages; check them monthly, because providers cut prices routinely.

Per-platform service mechanism

The router is a pure Python module — no service, no daemon. The escalation pattern runs inside the agent’s process. No platform-specific configuration is required.

How to know it is working

After a week:

1. What % of small-model calls need escalation?        target: < 30%
2. What % of mid-model calls need escalation?           target: < 10%
3. What is the average cost per task that succeeded?    (lower = better)
4. What is the average cost per task that failed?       (should be tracked separately)
5. What % of the total spend is the large tier?        target: < 20%

If the large tier is more than 30% of spend, you are over-routing. If small-tier escalation is more than 40%, your classifier is wrong.

Common failures and fixes

Failure	Cause	Fix
Every call goes to the large model	task_kind is unset, hitting the default.	Audit the call sites; set task_kind explicitly per prompt.
Small-model escalation > 40%	Classifier is misrouting.	Inspect the failed escalations in the ledger; rewrite the classifier with the failed examples.
Cost math doesn’t match the ledger	Pricing pages changed since you last read them.	Refresh pricing quarterly; the $0.0001 / $0.003 / $0.03 numbers move.
Latency doubled and the operator noticed	The escalation retry added a round trip.	If latency is critical (see “When NOT to route”), use mid tier as the floor and skip the cheap-first escalation.
claude-opus-4 not a real model	Model name is fictional / drifted.	Verify the model exists on Anthropic’s model overview before promoting this guide.

When NOT to route

Three cases where cheap-first is the wrong default:

1. The task has high asymmetry. A wrong classification costs a customer; a wrong creative generation costs nothing. Skip the cheap tier.
2. The model swap is visible to the user. If the operator notices “this answer is shorter than usual,” you have lost more than you saved.
3. Latency is critical. Adding a “retry with bigger model” step doubles the worst-case latency. Some agents cannot afford that.

For these, use the mid tier as the floor.

Updates and breaking changes

Re-run the smoke test after any of these:

• Anthropic model lineup changes (Opus 4 → Opus 5, Sonnet 4 → Sonnet 5). Model names drift every 6-12 months.
• OpenAI model lineup changes (GPT-4o → GPT-5, o1 → o3). Same pattern.
• Provider price drops or restructuring — the $0.0001 / $0.003 / $0.03 figures in this guide will become wrong within a quarter.

What you will have at the end

• A three-tier routing decision tree.
• A 15-line router function.
• An escalation pattern with retry budget.
• The cost math showing 70/20/10 split saves ~88% vs all-large.
• Five weekly questions to know if routing is working.

Where to get unstuck

• Anthropic model lineup: model overview, pricing.
• OpenAI model lineup: models page, pricing.
• OpenRouter cross-provider catalog: models directory.
• Your own cost ledger: see A2 (the 1-line observability hook) and A3 (JSONL to dashboard) for the data path.

Sources

• Anthropic model overview — used for: the Claude model lineup (Haiku, Sonnet, Opus) and tier descriptions. Verified 2026-07-22.
• OpenAI models — used for: the GPT model lineup (4o-mini, 4o, o1, etc.) and tier descriptions. Verified 2026-07-22.
• OpenRouter models — used for: the cross-provider catalog of routing options. Verified 2026-07-22.
• Anthropic pricing — used for: the per-call pricing tiers cited in the cost math. Verified 2026-07-22.
• OpenAI pricing — used for: the per-call pricing tiers cited in the cost math. Verified 2026-07-22.

Last verified: 2026-07-22. Adapted from Marvin’s original “A4. Cost-Aware Model Routing for Agents” (2026-07-20). Drafter and source-notes author are the same model (MiniMax-M3). Review lineage: pending independent subagent review per the v2.4.0 skill rule for Playbook-category guides — review transcript will be added to the source-notes page when complete.