no way. You want to know what’s really loaded. What did it do for customers, for revenue, for reliability. Lines of code, PR matters… We’ve spent a few decades learning that these are blatantly bad ways to measure a developer, to the point where it’s ridiculous to suggest them today.
A lot… What the industry has put on the billboards this year:
Each one is a big claim. “Percentage of code written by AI” is just lines of code with a better promoter. (Any doubters of me editing this draft might want to point out that it’s no coincidence that all of these are AI vendors of some sort, therefore promoting adoption.) Beautiful Important to them.)
we claimed results
Turn back a few years and the title numbers varied not just in size, but in type. GitHub’s key claim was that developers completed tasks 55% faster with Copilot. Say what you like about that study (many did), but this one was outcome claim. Bold, falsifiable about value. If it was wrong, you could show that it was wrong.
The claims of 2026 cannot fail. This is his talent; “75% of our code is AI-written” may be true, and will continue to grow, even if anything gets improved (faster delivery, fewer incidents, happier customers, etc.). A volume number can only disappoint you when adoption stops, and adoption is something most of us agree is real. 📈
So the claims became bigger and less was said. What happened in between?
no one puts up billboards
Result: The evidence became complicated, that’s what happened.
The strongest pro-adoption result is still Cui et al. ; Nearly 5,000 developers completed the task, +26%, with junior developers making the biggest gains. Not really in dispute. But then GitClear showed that as Copilot adoption deepened, code churn increased and refactorings collapsed. Then METR ran the study, which has been cited by many: Experienced open-source developers were 19% Slower With AI in their own codebase, assuming they were 20% faster.
But! Hold my beer…METR effectively withdrew it in February 2026: their follow-up estimates dropped to a speed up (With enough error bars to ride a Moto Guzzi with panniers!), and they abandoned the study design altogether – because the developers refuse to do the work anymore. Without AI, and cannot reliably self-report time on agentic tasks. His latest position: AI will probably outpace developers in 2026, and we can no longer clearly measure by how much.
Meanwhile, at the company level, an NBER survey of ~6,000 executives found that 69% of companies are actively using AI and nearly nine in ten reported no measurable productivity impact. Cross-study consensus sits around 10% of organizational gain. Nothing Still extremely useful! Buuuuut, not even the “you don’t need developers anymore” area.
And if you’re skeptical and still quoting “19% slower”, you’re also cherry-picking. Research continues to be updated; The industry simply changed what it matters.
Vanity metrics, now in AI flavor
To be fair this is not just an AI vendor’s claim. Carnegie Mellon’s SEI and Accenture launched an AI Adoption Maturity Model just a few days ago: five levels, eight dimensions, marketed to almost 95% of organizations without any return. Steve Yegge’s “8 Levels of AI-Assisted Development” ranks you based on what tools you run and how much supervision you give them. And every tool vendor now offers a maturity ladder whose top rung is, typically, “use our product more.” These stairs measure the intensity of adoption and call it maturity. Same replacement, good packaging.
My favorite data point in this whole genre: Augment surveyed 219 engineering leaders and asked them to define “AI-native engineering.” They got 219 different answers. 🫠
And the award for holding both ends of the rope goes to Anthropic, which gave us the “8 times more code sent” claim And One of the more rigorous studies of the year: an RCT found that AI-assisted developers scored 17% lower on comprehension of the code they just shipped, without any statistically significant productivity gains. I use the cloud every day (it recommends half the links I read for this post, so the irony is not lost on me), the products are really excellent, and their research branch updates while their marketing branch counts the volume. Both things are true together, that’s the thing.
Why do I really care?
Because these numbers are not decorative. They put forward budgets, performance expectations and headcount plans. In February, Jack Dorsey cut more than 40% of Block’s workforce (4,000+ people) with AI as the clear core thesis: “A significantly smaller team, using the tools we’re building, can do more and do it better.” A few weeks later, Atlassian cut jobs by 10% (~1,600 people), while admitting that “it would be dishonest to pretend that AI doesn’t change the mix of skills we need or the number of roles we need”. And one important detail strikes me: Dorsey said, in the same announcement, that the business was strong and gross profit was growing.
When a company says, “AI has made everyone more productive, so we need less people”, I want to see evidence – and I don’t believe it exists today. Show me that x% of your workforce is actually idle (or even underutilized) because the work can now be done by fewer people. Still: I’ve never seen a product/SaaS company that didn’t have an endless roadmap. If you’ve got an increase in free headcount essentially overnight, why wouldn’t you use it to provide exponentially more value to your customers? It should show in the form of MAU, Conversions, Revenue. Choosing layoffs instead tells me that the productivity claim is doing PR work for a decision that was already made for other reasons (over-hiring, investor pressure, take your pick).
Look, every business has some fat, and I can accept efficiency-driven cuts as a thing that legitimately happens sometimes – change happens at every turn in this industry. But when it does, try to do this using the personal display system you already have running to keep track of who’s traveling and who’s quarantined. Tokens don’t count. Not “% of AI-written code” or one’s level on the maturity ladder. If your selection evidence is a vanity metric, your selection is a lipstick-wearing lottery.
where i land
As I’ve said in previous posts, don’t read this as anti-AI. I believe that every engineer Needed Use AI every day. Call it AI-first, AI-efficient, whatever you want. Be curious, try new tools, test the latest models. It would be foolish not to do so. I’ve watched the industry absorb high-level languages, IDEs, autocomplete, agile and devops, and there were always crusty hold-outs reminiscent of the good old days before X came along and ruined everything. The hold-outs eventually (usually) came on board. The difference this time is speed: you can delay adopting “the cloud” by a few years and survive. With AI you might get a few months. The way we work has already changed, and as far as I can tell it’s not changing back.
but have to adopt starting lineNo scoreboard. We already know how to measure whether engineering is working: Dora metrics, reliability, rate of meaningful change, and ultimately revenue and customer value. Battle-tested, crisp stuff. Why are we throwing all this nonsense for AI vanity scores? (I may be wrong about a lot in this post, but I don’t think I’m wrong about that.)
So here’s a question to sneak into your next vendor pitch, executive review, or LinkedIn doom-scroll: Is it the results, or the quantity? It’s amazing how quickly a situation or statement defuses when you ask that.
The change is here to stay and the tools are good. The silver lining is that we already know how to measure what matters (and none of it counts in tokens).
Be AI-first in how you work, but battle-tested in how you measure it.
to encourage,
dave
<a href