Agent, Know Thyself! (and bid accordingly)

Abstraction: Markets to route AI agent tasks via self-assessment

Key points:

• Proposes MarketBench + a market scaffold where models bid to complete tasks; principal allocates to best bid (Hayekian: markets aggregate dispersed private info).
• Core finding: models can't bid to reflect true capabilities — the bottleneck is metacognition/self-assessment, which must be a training target.
• Ran 93 SWE-bench Lite tasks across 6 frontier models (Claude Opus 4.5, Claude Sonnet 4.5, Gemini 3 Pro Preview, GPT-5.2, GPT-5.2-pro, GPT-5-mini). Actual pass rates cluster 75–81%, but stated confidence spans 61–93%.
• Gemini is dramatically overconfident and wins 84.6% of auctions despite not being most capable; GPT family systematically under-confident. Median token estimate/actual ratio is 0.2 (Gemini 0.02 — ~50x underestimate).
• Giving each model a "performance card" prior improves Brier scores but barely moves auction outcomes — calibration alone isn't enough; need task-level discrimination.
• In matched live scaffold, a centralized LLM router (GPT-5.2-pro) beat the market (27/50 vs 23/50); most market advantage came from model diversity, not the mechanism. Agents = bundles of model + scaffold + execution path.

Connections: AI Agents · Model Routing · Metacognition · Swe Bench · Multi Agent Systems

Source: https://www.strangeloopcanon.com/p/agent-know-thyself-and-bid-accordingly?utm_campaign=post&utm_medium=web