The A.I. work. Pup
Every leaguemate now turns up with an assistant. This site is built so those assistants can read it — dated payloads, stated sources, read-only tools — and that half is real and already running. The half that is not built is the interesting one: a model that emits a probability is a forecaster, and forecasters here get receipts. The plan is to log what several models say before an event and grade them on the same board as nfelo, 538 Classic and the market.
What the models cost, right now
Every model OpenRouter currently serves, with its published price, context window and — where the vendor supplies one — Artificial Analysis’ composite intelligence index. Prices are dollars per million tokens.
| Model | Context | $/M in | $/M out | Intelligence | Reasoning |
|---|
Source: openrouter.ai/api/v1/models, fetched client-side. The intelligence index is Artificial Analysis’ composite, not ours — one vendor’s weighting of one set of benchmarks, carried through OpenRouter’s payload. It ships with no error bar, which is precisely why the tie-aware leaderboard is not built yet. 🤖 For machines
Price against intelligence
Each dot is a model: what it costs per million output tokens, against its composite score. The models on the frontier are the ones nothing else beats on both at once — everything above and to the left of a frontier point is strictly a worse deal.
Coming next: the same axes against Epoch AI’s benchmark data, fitted over time, so the rate of decline can be read off rather than asserted. Published estimates of that rate currently disagree by two orders of magnitude, and the disagreement is entirely about method — which benchmark you hold fixed, whether you price the cheapest model clearing a bar or the frontier one, and whether you count reasoning tokens.
Does a better prompt make a better forecaster?
We are about to find out, and we are telling you how before we start.
Frontier models get the same set of live prediction-market questions, twice. Once cold, with no system prompt at all. Once with a fixed standard reasoning prompt — base rates, explicit priors, a stated posterior, an outside-view check. Same questions, same timestamp, same temperature, same output format. The system prompt is the only thing that differs.
Then every forecast gets scored three ways: against what actually happened, against the market price at the moment the model saw the question, and against that price recalibrated using the curve on our own Markets page. That third one is the interesting test. If a model only beats the market by the amount the market’s known miscalibration already explains, it did not find anything — it rediscovered a bias we had already published.
The three rules that keep it honest
1. No tools, no search, either arm. A model with web access reads the market price instead of forecasting, and a prompt that tells a model to go look things up would make the two arms differ in information rather than in reasoning. Turning tools off is what keeps this a test of the prompt.
2. The same forced output line in both arms. A reasoning prompt asks for ranges and posteriors; scoring needs one number. Without an identical output contract, the disciplined arm gets punished for following its own instructions and we would be measuring our parser.
3. Forward-looking questions only. Every question resolves after the forecast date and after the model’s training cutoff. A model recalling an outcome is not forecasting it.
Written down before any model was run
Probability the disciplined prompt wins at all: 55–70%. That it wins by more than 0.02 Brier: 20–35%. That it is actively worse: 20–30% — long reasoning chains can talk a model out of a well-anchored prior. That any model beats the recalibrated market: under 20%, and we would treat such a result as a bug until a second cohort reproduced it.
What would change our minds about the whole design: if the cold arm already produces long, structured, base-rate-citing reasoning without being asked, then the two arms are nearly the same thing and a null result means “the prompt was redundant,” not “prompting does not work.” That gets checked on the first ten questions, before scaling.
Status: pre-registration, not results. Nothing has been run. When the first cohort is collected the raw model outputs get published alongside the scores — the reasoning bodies, verbatim, including whatever they hallucinate.
Method, and what is missing
What is live. The model board and the frontier chart, both computed in your browser
from one OpenRouter request at page load. No cached mirror exists under /data/ for
this page yet, which cuts both ways honestly: the numbers cannot go stale, and they are not
reproducible — you and I will see slightly different figures depending on when we looked.
What is deliberately not here.
- A tie-aware leaderboard. This was specced and then cut, because the intelligence index in OpenRouter’s payload arrives with no confidence interval. Collapsing statistically indistinguishable models into one block is the whole point of the tool, and inventing an interval to do it would be worse than not shipping it. It needs LMArena’s published Bradley-Terry bounds, which are a separate fetch.
- Epoch AI benchmark data. Their CSVs are CC-BY and current, but the domain sends no CORS header, so a browser cannot read them directly. Needs a Worker proxy.
- METR’s time-horizon revision audit. The most interesting chart we have specced — how much of the apparent acceleration is models improving versus the measuring instrument being changed underneath them. It is gated: it does not ship until the logistic fits are recomputed from METR’s public repository rather than read off their blog post. Publishing an accusation of a measurement artifact on the strength of secondary numbers would be the exact failure this site exists to avoid.
Tier: Pup. Live and useful, not validated. The board is a faithful rendering of somebody else’s catalogue — if OpenRouter’s prices are wrong, ours are wrong in exactly the same way, and we have not independently checked a single one against a provider’s own pricing page.
This section, for machines
One card per row in
/data/surfaces.json carrying
domain: "ai". The count is the file’s count, not a number typed
into this page. Every tool here computes in your browser from a third party’s
public API, so none publishes a machine surface of its own — each card says exactly that,
and the panels above name the upstream every number comes from.