A single model is not enough
We measured six AI readers against a person's sign-off on the lab's public signals. This is the first report.
How it was measured
Data as of November 11, 2026.
Every public signal in the lab is read by six different models, with the same text and the same question: does the verbatim quote from the source support the fact? They can answer Supports, Does not support or Not enough to decide. Then a person signs their own reading.
At the time of the report, 111 signals have panel readings and 71 also carry a person's sign-off. Those 71 are cases where the panel was split, not a random sample of the news.
The raw agreement rate misleads. A script that always said Supports would match the person 49.3% of the time without reading anything. That is why we also look at how much each reader agrees when the person did decide, and how much it agrees once chance is discounted.
Doubting a lot is not reading better
The three commercial readers (Claude Haiku, Claude Sonnet and Muse Ember) answer that there is not enough to decide between 77% and 79% of the time. When the person did take a side, they agree with the person only between 15% and 17%.
Llama 3.1 8B does the opposite: it answers Supports in 98% of its readings. Its raw agreement looks good because the person also said Supports on many signals, not because it reads better.
Always saying yes is not understanding the text
A reader that answers Supports to almost everything agrees with the person every time the person also said Supports, and that is not reading well. Llama 3.1 8B answers Supports in 98% of its readings. After discounting chance, CLM-8B is the reader that agrees most with the person (0.35) and Llama 3.1 8B drops to 0.01.
No reader fully agrees with the person. That is why the lab does not rely on a single one: each reader is measured against a person, with the same text and the same question, and only then is it decided what each one is good for.
Which reader measures best for each type of signal
There are few signed signals per category. Read it as guidance, not as a final assignment.
| Signal types | Signed signals | Best-performing reader |
|---|---|---|
| Litigation, Labor disputes, Supply chain and geopolitics, Environmental and ESG damage, Regulatory change | 68 | CLM-8B |
Not enough signals yet: Reputation crisis, Consumer boycott.
Limits of this measurement
- The signals signed by a person are almost all cases where the panel was split. They are not a random sample of the news.
- One of the open-weights readers was tuned by looking at these same sign-offs. Its figure is optimistic and is not comparable like for like with the others.
- With few signals per category, the best reader for each one may change as cases are added.
What this means
Delegating signal reading to a single AI is operating blind. With one model there is nothing to compare against: it is unclear how often it doubts or in which cases it is wrong. REVsign measures each reader against a person and leaves every reading in plain view, so that each signal is read by the model that measures it best.