The Benchmark Illusion: Why AI Evaluation Is Moving On-Chain

Tracing the invisible ink of protocol logic. Scott Wu, CEO of Cognition, just declared that every AI model has saturated every public benchmark. MMLU, HumanEval, GSM8K—dead. The industry, he claims, is shifting to proprietary evaluations that measure real-world application. This isn't just a tech update. It's a narrative inflection point. And for anyone watching the crypto space, it echoes our own obsession with metrics that eventually lose signal. TVL, TPS, even hash rate—each became a proxy until it didn't. Now AI faces the same decay. But the response—going private—is a dangerous experiment in trust. Let me explain why. Context: Public benchmarks have been the backbone of AI progress. They allowed researchers, investors, and users to compare models. Similar to how Ethereum gas efficiency was once the holy grail, then became a footnote as L2s proliferated. But here's the rub: saturation doesn't mean stagnation. It means the tests are too static. In crypto, we saw this with the TVL metric in DeFi. Projects would inflate liquidity through token incentives, creating a false signal of health. The market eventually learned to look beyond. Now AI is at that same crossroads. Wu's argument is technically valid—LLMs do hit 90%+ on most standard tests. But the conclusion that proprietary evaluation is the answer ignores the fundamental lesson from Web3: closed systems compound information asymmetry. Core: The mechanism here is fascinating. Benchmarks like MMLU are open books. Every team knows the questions, so they overfit. In crypto, we call this the 'known exploit'—when everyone knows the vulnerability, it gets patched. But moving to proprietary evaluation creates a new problem: verification. Who audits the auditor? When a company like Cognition states its agent Devin outperforms others in 'real-world tasks,' there's no public ledger to verify. Liquidity is not a resource; it is a behavior. In the same way, evaluation is not a score—it is a trust mechanism. If we accept that benchmarks are dead, we must also accept that we're trading transparency for narrative. Sentiment analysis from early 2025 shows that 70% of crypto-native AI projects still rely on public benchmarks for their pitches. They're caught in the past. The signal is clear: the next generation of AI evaluation will be gated. And that gate is proprietary. But here's where it gets contrarian: What if the solution is on-chain? Decentralized, permissionless evaluation tasks that anyone can submit, verified by consensus. Think of it as a proof-of-work for model capability. Instead of a private lab determining 'real-world' performance, we create a public, open environment—a benchmark DAO where models compete in verifiable, randomized tasks. Smart contracts can enforce randomness and prevent leakage. I audited a similar concept for a decentralized compute project in 2023. The math works. The infrastructure exists. Decoding the cultural syntax of digital ownership—we already have the tools to build a transparent evaluation layer. The industry is just choosing not to use them because proprietary evaluation creates moats. Moats are good for incumbents, bad for innovation. Sifting through the noise to find the signal: What Wu is really saying is that the old scoreboard is broken. He's right. But replacing it with a private scoreboard is like replacing a public blockchain with a private database. It might be faster, but it's not trustless. The crypto industry learned this the hard way with centralized exchanges and opaque reserves. AI evaluation risks the same trap. The next bull run in AI won't be driven by better models alone—it will be driven by better truths. Mapping the topology of decentralized trust: the ultimate benchmark may not be a score at all, but a protocol that lets users verify model outputs for themselves. Everything else is just noise.