Two headlines made it look like agents had failed. An MIT NANDA study said 95% of enterprise pilots delivered no ROI at six months. Coverage of Karpathy's interview sounded like a eulogy. The narrative practically wrote itself. Billions of dollars down the drain. Another tech bubble popping in slow motion.
And the numbers seemed to back up the panic. A RAND study interviewing 65 experienced AI practitioners cited external estimates that more than 80% of AI projects fail, twice the rate of traditional IT projects. The technology itself appeared to be the problem.
But neither of those stories actually said what the headlines claimed.
What actually happened points to something that matters far more than whether agents are "good" or "bad."
What Karpathy Actually Said
Media headlines paraphrased agents as "slop." When you go back to the actual interview transcript, here's what Karpathy said: "It's the decade of agents."
Not exactly a damning indictment.
His critique wasn't that agents can't work. It was that we're "overshooting the tooling with respect to present capability." The gap is between what people promise today and what the technology can reliably deliver right now.
He explained that agents today lack continual learning and the reliability needed to function as reliable coworkers. The missing piece is the model improving itself over time, not external memory: storing and retrieving context across sessions is a solvable engineering problem that has already started to ship. Working through the whole bundle of missing capabilities, not enough intelligence, weak multimodality, no computer use, and continual learning among them, is what Karpathy figures takes about a decade of engineering. Not because agents are broken, but because robust systems take time.
The full quote about reinforcement learning is even more revealing: "RL is terrible... but everything we had before it is much worse." That's pragmatism from someone who knows the territory.
His actual position? We're in the early innings of a multiyear build cycle. The infrastructure isn't mature yet. Success requires better memory, better reliability, and better learning. All solvable problems, just not solved yet.
The MIT Study That Wasn't
Now let's talk about that 95% failure rate, because this one requires even more unpacking.
The number does appear in the MIT NANDA preliminary findings. But when you read past the headline, the story gets more complicated. The study defined success as beyond pilot deployment with measurable KPIs, with ROI measured at six months. The authors explicitly note this may understate longer term success.
That's a very particular bar to clear.
Translation: 95% of projects didn't achieve fast, measurable returns in the exact way this study was measuring. That's not the same thing as "95% of all AI fails." Not even close.
The 95% is a 6 month ROI measure, not "project failure." That window may understate longer implementations.
Several analysts called this out. The Marketing AI Institute ran a piece titled "That Viral MIT Study Claiming 95% of AI Pilots Fail? Don't Believe the Hype." Because what the study actually documents is an execution gap, not a fundamental technology failure.
The real finding? About 5% of organizations are integrating AI in ways that generate measurable value quickly, and they're doing it by focusing on learning systems and workflow integration, not flashy demos.
So What's Really Happening?
Here's where it gets interesting. When you look at what's actually working, a pattern emerges.
Take Klarna. They deployed an AI assistant that initially handled two-thirds of their customer support chats, doing the work of 700 full time agents. Resolution time dropped from 11 minutes to under 2. The numbers looked incredible.
Then something interesting happened. A few months later, Klarna started reassigning engineers and marketers to customer support. Their CEO said "really investing in the quality of the human support is the way of the future for us."
Wait, what happened?
They'd over indexed on automation without building in the right escalation paths, quality monitoring, or human fallback systems. When edge cases piled up and customer satisfaction started slipping, they had to backfill with humans. Not because agents can't work, but because they'd skipped the architecture part.
Contrast this with Lyft. They deployed Claude for customer care and reduced resolution time by 87%, handling thousands of requests daily. The difference? They designed for human escalation from the start, emphasizing human handoff for complex issues like safety and fraud. They scoped the agent tightly around common issues. They monitored quality metrics continuously. They kept the feedback loop tight.
Or Amazon, which uses autonomous agents to upgrade tens of thousands production applications, saving $260 million annually. Not by replacing humans wholesale, but by automating narrow, well defined tasks within a larger system that humans still oversee.
This isn't magic. It's tight scope, heavy instrumentation, and specific workflows. They have guardrails. They have observability. They have clear success metrics tied to actual business operations.
In other words, they have architecture.
The Klarna story is instructive precisely because it illustrates the point. They got great initial results by deploying agents aggressively. But without the right architecture for escalation, quality monitoring, and human backup, those results weren't sustainable. When they hit edge cases their agents couldn't handle well, they had to course-correct by bringing humans back in.
That's not a failure of agents. That's a failure to design for the messy reality of production systems. Lyft avoided this by building human escalation and tight scope into their design from day one. Amazon avoided it by keeping humans in the oversight loop even as agents handled the routine work.
This is exactly what both Karpathy and the MIT report are pointing to, once you get past the distorted headlines. The winners aren't just throwing AI at problems and hoping for the best. They're building systems with memory, verification, human fallback, and learning built in from the start.
Why AI Projects Fail at Twice the Rate
Here's a number worth sitting with: RAND cites external estimates that more than 80% of AI projects fail, compared to 40% for traditional IT projects. That's from the RAND study that interviewed those 65 experienced practitioners across industries and company sizes.
But here's what matters: it's not because the AI doesn't work. In RAND's interviews, 84% of practitioners cited leadership-driven causes as a primary reason for failure. Teams optimizing for the wrong metrics. Projects abandoned before they could deliver results. Engineers instructed to apply machine learning to problems that could be solved with simple rules.
Leadership is the top failure mode. In RAND's interviews, 84% cited leadership driven causes as a primary reason.
The pattern kept repeating. Business leaders would set vague objectives without understanding how those translated into technical requirements. Data science teams would build models optimized for accuracy when the business actually needed speed. Projects would get halfway to completion and then get shelved because leadership shifted priorities.
The technical challenges exist. Data quality problems, infrastructure gaps, talent shortages. But those are solvable with time and investment. The real problem is that AI projects require something most organizations haven't built: architecture that bridges the gap between what business leaders need and what the technology can actually deliver.
Which brings us back to Klarna versus Lyft. Same technology. Same use case. One had to backfill with humans, the other is thriving. The difference wasn't the AI. It was everything around it.
The Reliability Problem Nobody Wants to Talk About
Here's the math that should terrify anyone deploying agents without proper architecture: if each step in a process has a 1% error rate, by step 100 you have a 63% chance that something went wrong somewhere.
This is why reliability isn't just a nice to have. It's the entire game. You can't scale agents that fail most of the time. The unit economics don't work. The trust doesn't work. Nothing works.
But here's the thing: this isn't a model problem. It's an architecture problem. You need systems that reduce per-step error rates and catch mistakes before they compound. You need verification at every layer. You need ways to recover gracefully when things go wrong.
Which brings us back to Karpathy's actual point: agents need architecture to become robust.
Memory Is the Primitive Everyone Forgot
One thread connects both stories: the importance of memory and learning.
The MIT report specifically calls out that most GenAI systems "do not retain feedback, adapt to context, or improve over time." Karpathy's decade timeline is built around the same broad observation. He applies that roughly ten-year horizon to the whole bundle of missing capabilities, not enough intelligence, weak multimodality, no computer use, and continual learning among them, saying plainly it will take "about a decade to work through all of those issues." Continual learning is one piece of that, not the whole of it.
Here two problems get blurred that are worth pulling apart: external memory and continual learning are not the same thing. External memory has already shipped. Research systems like MemGPT demonstrated multi-tier memory hierarchies that agents read and write to, and production tools now carry the same idea: GitHub Copilot Memory, currently in public preview, stores repository facts and user preferences and feeds them back in later sessions. That mitigates session amnesia through engineered recall, stored context fed into the next run, not by retraining the model. Continual learning, where the model updates its own weights from experience, is still open research (see the 2025 ACM survey on continual learning of LLMs). The recall piece exists today. It just needs to mature and scale.
The organizations getting value from agents today are the ones who've figured this out. They persist what happened, store it in structured ways, and feed it back into the next run, instead of starting fresh with every conversation. That is an architecture choice, not a breakthrough in the model.
What This Means for Your Next Project
If you're planning to deploy an agent anytime soon, the path forward is clearer than the headlines suggest. You need a few things that aren't optional:
Start with a narrow scope. Pick one workflow, one set of tasks, one measurable outcome. The organizations succeeding aren't trying to build general AI assistants. They're solving specific, well defined problems.
Build in verification from day one. Every step needs a way to check if it worked. Every action needs rollback capability. Every decision needs a confidence threshold. This isn't paranoia; it's math.
Design for human escalation before you need it. Klarna's course correction happened because they didn't build robust escalation paths from the start. Lyft got it right by designing tight agent scope and clear handoff protocols upfront. Your agent should know when it's out of its depth and gracefully hand off to humans.
Implement memory that persists across sessions. Your agent needs to remember what happened, what worked, what didn't, and why. That means structured storage, not just conversation history. Event logs, curated memory tables, reflection loops that compress and synthesize. This is recall you engineer, not learning the model does on its own.
Instrument everything. Success rates by task type. Retry patterns. Human handoffs. Cost per completed task. You can't improve what you don't measure, and you can't trust what you can't observe.
Connect it to real business metrics. Not "we deployed an AI agent" but "we reduced support ticket resolution time by X" or "we cut code maintenance costs by Y." If you can't tie it to a P&L line, you're building a demo, not a system.
The Actual Story
So here's what really happened when those headlines hit:
Karpathy didn't say agents can't work. He said they need better engineering to work reliably, and that building that engineering will take time. The MIT NANDA findings didn't say AI is failing. They said most organizations aren't yet building the learning systems and workflow integration that make AI succeed.
Both are describing the same gap between hype and execution. Both are pointing to the same solution: architecture, not magic.
The agents that work today are the ones built with memory, verification, scope constraints, and tight integration into actual workflows. The ones that fail are the ones that skip those steps because they seem boring compared to the vision of fully autonomous AI assistants doing everything.
Turns out boring architecture is how you get to the exciting capabilities. Always has been.
The decade of agents isn't cancelled. It's just starting. And it's going to look a lot more like careful engineering than the demos suggested. Which is probably how it should have looked all along.
FAQ
- Did the MIT NANDA study actually show that 95% of AI projects fail?
- No. The 95% figure measures projects that did not achieve measurable ROI within six months of pilot deployment, and the authors explicitly note this window may understate longer implementations. It is an execution-gap number, not a verdict on the technology.
- Did Karpathy say AI agents do not work?
- No. He called this "the decade of agents" and argued the industry is overshooting the tooling versus present capability. His critique is that agents today are still missing a whole bundle of capabilities, continual learning and reliability among them, and that working through that broader bundle will take roughly a decade of engineering. External memory, the stored context an agent retrieves across sessions, is a separate and more tractable problem that is already shipping.
- Why did Klarna partly reverse its AI customer-support rollout?
- Klarna deployed aggressively without sufficient escalation paths, quality monitoring, and human fallback. When edge cases accumulated they reassigned engineers and marketers back into support. Lyft ran the same use case successfully by designing tight scope and human handoff from day one.
- If every step in an agent run is 99% reliable, is that good enough?
- Not for long pipelines. At a 1% per-step error rate, a 100-step process has about a 63% chance of failing somewhere. Reliability at scale requires per-step verification, rollback paths, and graceful recovery, not just a better base model.
- If AI projects fail twice as often as traditional IT projects, where does the failure come from?
- Mostly leadership, not the technology. In RAND interviews with 65 practitioners, 84% cited leadership-driven causes as a primary reason: vague objectives, wrong metrics, shifting priorities, and applying ML where simple rules would do.
If your team is ready to move from trial-and-error to systematic agent orchestration, take a look at the Coding Agents meistern workshop: 3 days of building the architecture and mental models that make agents actually work.
