Intent
Idea Lab exists to change what we reward in public speech. Today a piece of writing is judged on two things: is it true, and which side is it on. Idea Lab adds a third question and tries to make it the most important one: how was this built? Was it built to find out, or built to convince?
The founding belief is simple. Free speech is not negotiable. But free speech without any shared idea of what good speech looks like is a game with no rules, where picking a story first and fitting the facts to it costs nothing. Idea Lab wants that move to have a cost, not through censorship, but through visibility. If readers can see the foul, the foul stops working.
The framework is Tim Urban’s Ladder: Scientist, Sports Fan, Attorney, Zealot. Idea Lab’s job is to translate the Ladder from a description of how people think into a set of rules you can apply to a text, and then build things on those rules.
The long-term goal is a widely recognised standard for honest writing and conversation, the way nutrition labels became a standard for food. The near-term goal is much smaller: prove the rules can be applied consistently, and get a few thousand people to start noticing the fouls.
The gap
Several serious organisations already rate the news. None of them rates what Idea Lab wants to rate.
| Who | What they measure | Level | What they miss |
|---|---|---|---|
| Ad Fontes Media | Political bias (left to right) and reliability, by trained analysts, three per article | Article, rolled up to outlet | Whether the reasoning is honest. A one-sided but accurate article scores well |
| NewsGuard | Editorial practices: corrections, transparency, separation of news and opinion | Outlet | Individual articles. A good outlet can run an Attorney piece and the score does not move |
| Ground News | Aggregates the above, shows which outlets covered a story and which did not | Outlet | Same as its sources. Factuality rating applies to the outlet, not the article |
| Fact checkers (Snopes, FactCheck.org, etc.) | Is a specific claim true | Claim | Selection and framing. Every claim can be true and the piece still misleads |
The common thread: they answer “is it true?” and “which team?”. Both are useful. Neither catches the most common foul in credible media, which is picking the story first and then choosing true facts to fit it.
Two things make the gap more open now than in 2020. First, a language model can read an article and check it against a written rubric in seconds and at near-zero cost, so per-article scoring is no longer a labour problem. Second, trust in media is low enough that there is appetite for a tool that does not sort by team.
Principles every product must follow
- Judge the method, never the position. No product ever says a conclusion is right or wrong. It only says how the conclusion was built. The moment Idea Lab is seen to favour a side, it is finished.
- Symmetry is the proof. Every rubric is tested on texts from opposing sides of the same issue. If the fouls only ever land on one side, the rubric is broken, not the side.
- Show the evidence, not just the score. A score alone is an Attorney move. Every rating points to the exact sentences that triggered it, so the reader can disagree.
- The rules are public and fixed in advance. Anyone can read the rubric before an article is scored. Rules change in the open, with a changelog, never quietly.
- Hold Idea Lab to its own standard. Everything Idea Lab publishes about itself is scored by the same rules. If it fails, say so.
- Plain language. The rules, the scores and the articles are written so a fifteen-year-old can follow them. Jargon is a way of hiding.
- No censorship, no takedowns, no deplatforming. Idea Lab adds information next to speech. It never removes speech.
Possible products
All of these sit on the same core: a written rubric that turns the Ladder into checks you can run on a text, and a model that applies it and points to the sentences that triggered each check. The products differ in who sees the result and how.
| Product | What it is | Who it is for | Effort | Main risk | Why it matters |
|---|---|---|---|---|---|
| The Rubric | The public document that defines the fouls and credits, with worked examples. Versioned, with a changelog | Everyone; it is the standard itself | Low to build, high to get right | Rules too vague to apply consistently | Nothing else works without it |
| Weekly Ladder Report | Take the most-read articles of the week on one story, score them, publish side by side with the triggering sentences highlighted | Readers who already distrust media | Low | Being accused of bias on the first contested story | Cheapest way to make the fouls visible and test the rubric in public |
| Score this text | A page where anyone pastes a URL or text and gets a Ladder rung, the fouls found and the exact sentences | Readers, students, writers checking their own drafts | Medium | People use it to attack opponents rather than check themselves | Turns the rubric into a habit and generates the data to improve it |
| Browser extension | Shows the score next to an article as you read it, with one click to see why | Daily news readers | Medium to high | Distribution; extensions are hard to get people to install | The only format that reaches people at the moment they read |
| Rewrite to the top rung | Paste an Attorney article, get it back as a Scientist would have written it: facts separated, missing counter-facts added, confidence lowered to match the evidence | Writers, editors, readers who want the fair version | Medium | The model inserts its own bias while rewriting | Shows what good looks like, not just what bad looks like |
| Writing platform | A place to publish where every post is scored before it goes live and carries its score. Authors can see fouls and fix them | Writers who want the badge | High | Empty-room problem; a platform with no readers | The long-term home for the standard |
| Idea Lab badge | A certification outlets or writers can display if their last N pieces scored above a threshold | Outlets and independent writers | Medium, mostly process | Becomes a rubber stamp if too easy, ignored if too hard | Creates an incentive on the supply side, not just the reader side |
| Education module | A short course or school material on the Ladder with the maps and exercises | Teachers, students | Low to medium | Slow to show results | The only route to changing the next generation’s defaults |
Suggested order
The order follows one rule: do not build anything that depends on a rubric nobody has tested yet.
- Publish the Ladder article. Done. It sets the vocabulary and is the thing every later product links back to.
- Write the Rubric, version 0.1. Start with the five most mechanical Attorney fouls: conclusion before evidence, missing counter-facts, no steelman, loaded verbs, confidence above evidence. Leave the harder Zealot signs for later. Each foul gets a definition, two examples that trip it and two that don’t.
- Test it by hand on one story. Take five or six high-traffic articles on the same event from outlets across the spectrum. Score them, twice, by two different people or two separate model runs. If the scores don’t agree, the rule is too vague. Fix the rule, not the score.
- Publish the first Weekly Ladder Report. Side by side, triggering sentences highlighted, both political sides scored on the same story. This is the moment the project becomes real and the moment it gets attacked. Both are good.
- Ship “Score this text”. Once the rubric survives a few public rounds, open it up. Every paste is a data point for improving the rules.
- Then decide between the extension and the platform based on what the first users actually ask for.
The rewrite tool and the badge can come at any point after step 3. The education module is a separate track that needs a partner school to be worth doing.
Credit
The Ladder, the two-mind model, the Idea Spectrum and the term Idea Lab come from Tim Urban’s What’s Our Problem? A Self-Help Book for Societies (2023). Idea Lab is his name for a culture where people argue about ideas without attacking each other, and where saying “I don’t know” earns respect. The chapter that defines the four rungs is free to read at waitbutwhy.com. This site is an independent project and is not affiliated with him.