To continue improving knowledge work, AI systems need either a reliable mechanism for autonomous self-correction or human evaluators capable of catching errors and generating high-quality feedback. The industry has invested heavily in the former. It’s almost like giving no idea what’s going on with the other person.
I argue that we need to treat the human assessment problem with the same rigor and investment that we put into building the model capabilities themselves. New graduates have been hired in major tech companies Decreased by half since 2019. Document review, first-pass research, data cleaning, code review: models now handle these. Economists who track this call it displacement. Companies that do this call it efficiency. Nor are they paying attention to future problems.
Why are there limits to self-improvement in knowledge work?
The obvious pushback is reinforcement learning (RL). AlphaZero learned Go, chess, and shogi at a superhuman level without human data, and devised new strategies in the process. Move 37 in the 2016 match against Lee Sedol, a move that professionals said he would never have played, did not come from human commentary. It emerged from AI self-play.
What enables this is environmental sustainability. Move 37 is a unique move within the fixed state space of Go. The rules are complete, clear and permanent. More importantly, the reward signal is perfect: win or lose, and immediate, with no room for interpretation. The system always knows whether a move was good or not because the game eventually ends with a clear outcome.
Knowledge work has none of these qualities. The rules in any professional field are dynamic and constantly rewritten by the humans who work within them. New laws are passed. New financial instruments have been invented. A legal strategy that works in 2022 may fail in a jurisdiction that has changed its interpretation since then. Whether the medical diagnosis was correct may not be known for years. Without a stable environment and a clear reward signal, you can’t close the loop. You need humans in the evaluation chain to continue teaching the model.
formation problem
The AI systems being built today were trained on the expertise of people who went through exactly the same formation. The difference now is that entry-level jobs that develop such expertise were previously automated. Which means the next generation of potential experts aren’t accumulating a type of decision This makes it worth including a human evaluator in the loop.
There are examples of extinction of knowledge in history. Roman concrete. Gothic construction techniques. Mathematical traditions that took centuries to recover. But in every historical case, the cause was external: plague, conquest, the collapse of the institutions that harbored knowledge. What is different here is that no external force is required. Regions may be decimated not by disaster but by thousands of individually rational economic decisions, each of which is individually sensible. This is a new mechanism, and we don’t have much practice at recognizing it when it’s happening.
When all the fields go quiet
At its logical limit, this is not just a pipeline problem. This is a decline in demand for expertise itself.
Consider advanced mathematics. It doesn’t atrophy because we stop training mathematicians. It is eroding as organizations cease to need mathematicians for their day-to-day work, the economic incentives to become mathematicians disappear, the population of people doing marginal mathematical reasoning diminishes, and the field’s ability to generate novel insights is quietly destroyed. The same logic applies to coding. Our question is not “will AI write the code” but rather “if AI writes all the production code, who develops the deep architectural intuition that produces truly innovative system designs?”
There is an important difference between an area being automated and an area being understood. We can automate large amounts of structural engineering today, but the abstract knowledge of why certain approaches work lives on in the minds of those who previously spent years getting it wrong. If you eliminate a practice, you don’t just lose the practitioners. You lose the ability to know what you have lost.
Advanced mathematics, theoretical computer science, deep legal reasoning, complex systems architecture: when the last person who deeply understands a subfield of algebra retires and no one replaces them because the funding has dried up and the career path has disappeared, that knowledge is unlikely to be rediscovered any time soon.
He’s gone. And no one noticed because models trained on their task still performed well on benchmarks for another decade. I think of this as hollowing out: superficial competence persists (models can still generate expert-looking outputs) while the underlying human ability to validate, extend or correct that expertise quietly disappears.
Why are rubrics not completely replaced?
The current approach is rubric-based assessment. Constitutive AI, reinforcement learning from AI feedback (RLAIF), and structured criteria that let models score models are serious technologies that meaningfully reduce reliance on human evaluators. I am not dismissing them.
Their limitation is this: A rubric can only capture what the person writing it knew to measure. Push hard against this and you’ll get a model that does a very good job of satisfying the rubric. This is not the same as the model that is actually correct.
Rubrics measure the explicit, clear part of the decision. The deeper part, the instinct, the felt feeling that something is wrong, doesn’t fit the rubric. You can’t write it down because you have to experience it before you know what to write.
What does this mean in practice
This is not an argument for slowing development. The efficiency gains are real. And it’s possible that researchers will find ways to close the evaluation cycle without human judgment. Maybe synthetic data pipelines are good enough. It may be that models develop reliable self-correction mechanisms that we cannot yet imagine.
But today we don’t have them. And meanwhile, we are destroying the human infrastructure that currently fills the gap, not as a deliberate decision but as a byproduct of thousands of rational decisions. The responsible version of this change is not to assume that the problem will resolve itself. This valuation gap is meant to be treated as an open research problem with the same urgency that we bring to capacity gains.
The thing AI needs most from humans is our focus on preserving it the least. Whether it is permanently or temporarily true, the cost of ignoring it is the same.
Ahmed Al-Dahle is the CTO of Airbnb.
<a href