Large language models continue to struggle with obfuscation, which poses a major hurdle for real-world enterprise applications. Minimizing these errors is a messy business, forcing model developers to navigate a tight tradeoff where eliminating factual errors often suppresses valid answers.
In a new paper, Google researchers introduce the concept of "faithful uncertainty," A metacognitive technique that aligns a model’s responses with its internal confidence. This allows the alignment model to offer appropriately hedged hypotheses, such as "My best guess is," Instead of defaulting to something useless "reply or abstain" Binary.
In real-world agentic AI applications, this metacognitive awareness serves as an essential control layer. This empowers autonomous systems to accurately determine when their internal knowledge is sufficient and when they need to dynamically trigger external tools or discovery APIs to resolve deficits.
Utilization of current mitigation strategies
Understanding why LLMs hallucinate depends on separating two abilities: a model knowing the facts versus knowing what is known. Historically, most factuality gains in AI have come from expanding the knowledge boundary, meaning developers pack more facts into the model’s parameters through larger and more training data.
However, expanding a model’s knowledge does not automatically improve its boundary awareness, which is its ability to distinguish known from unknown and recognize its limitations.
“There are broadly two ways to improve LLM factuality,” Gal Yonah, a research scientist at Google and co-author of the paper, told VentureBeat. The first continues to teach more facts to the model. But, says Jonah, “model capacity is limited, and the long tail of knowledge is effectively infinite.”
Once models reach this limit, the hope is that they will learn what they don’t know and simply avoid answering. However, this is naturally difficult for an LLM.
“This is why most practical efforts to reduce hallucinations through various interventions have not really been implemented,” Yona explains. “They reduce hallucinations, but they also harm utility, because the model refuses to answer questions it actually knows.”
This inability to distinguish between known and unknown leads the authors of the paper to so-called "Utility tax." Applying the zero-hallucination standard requires the model to discard enormous amounts of perfectly valid information whenever it is even slightly uncertain. For example, the authors demonstrate that reducing the underlying 25% error rate to a strict 5% target forces developers to discard 52% of the model’s correct answers.
Treating all errors as hallucinations forces enterprise systems to choose between trustworthiness and helpfulness. Application developers are generally unwilling to pay this heavy utility tax and render their models unusable.
As a result, they adapt the system to prioritize coverage, forcing the models to work in a situation where they continue to generate confident hallucinations.
Redefining hallucinations as reliable errors.
To move beyond the utility tax, the researchers propose to stop treating any factual errors as hallucinations. Instead, they repeat the hallucinations as "assured errors": False information given officially without proper qualification.
This subtle reframing breaks down the rigid "reply or abstain" Bifurcation and allows the model to express its uncertainty.
In this new framework, if a model makes a factual mistake but appropriately reserves its response (for example, by saying, "I’m not completely sure, but I think…"), this is not a hallucination. This is merely a hypothesis that is presented to the user for consideration. By expressing uncertainty, AI retains its usefulness without violating the user’s trust – sharing whatever partial or probable knowledge it has.
However, if an AI assistant hides all its responses with disclaimers, the user is forced to double-check everything, completely defeating the purpose of the tool.
The solution that researchers have proposed "Faithful uncertainty." This approach requires a model’s linguistic uncertainty, or the words used to express doubt, to align with its internal uncertainty, which is its actual, internal statistical confidence in that specific answer. This ensures that the model will hedge only when its internal state actually reflects contradictory or low probability information.
Faithful uncertainty forms a core component of “metacognition”, the ability of an AI to be aware of its own uncertainty and act on it. To understand this practically, consider the intuitive example of consulting a doctor. We don’t trust doctors because they are omniscient. We trust them because they reliably distinguish between different diagnoses ("you have a fracture") and an educated hypothesis ("It could be a sprain, but let’s do some testing").
Practical implications for enterprise AI
Under the new framework, errors where a model is genuinely confident but is factually incorrect are classified as “honest mistakes”. This presents knowledge expansion (training the model on more data) and faithful uncertainty reduction as perfectly complementary efforts. Expanding knowledge pushes the absolute knowledge boundary outward to minimize honest mistakes, while faithful uncertainty communicates honestly wherever that boundary currently lies.
This new framing has important implications for agentic applications. The shift to agentic AI may make it seem that it is unnecessary to know what the model does not know, as models can simply search external databases. However, access to external devices actually increases the need for faithful uncertainty. In agentic systems, metacognition becomes the central control layer that controls the entire system.
External tools solve the storage problem because the model no longer needs to encode every fact into its parameters. However, this introduces a new control problem: knowing when to obtain information, verifying the facts, and organizing these external devices. Without credible uncertainty, an agent is essentially going blind and must rely on external, static estimates or over-engineered scaffolds.
Jonah said, “The model may discover something it already confidently knows – wasting latency and cost for no benefit. Or the opposite: it confidently extracts the answer from memory when it should have discovered, producing a plausible but incorrect output.” Today’s agent harnesses try to solve this externally with query classifiers or always-search rules, but Jonah notes that these are "Stable and brittle." By using its internal uncertainty to regulate its own behavior, the agent dynamically adapts its tool usage, choosing to apply a search tool only when its internal confidence is really low.
In addition to deciding when to search, credible uncertainty is important for evaluating the results of a search. If a device returns low-quality or unexpected information, a metacognitive agent does not blindly accept whatever it sees in its context window. Instead, it uses its uncertainty awareness to weigh the retrieved external signals against its internal precursors. This prevents sycophantic behavior where the system might otherwise rely on external sources that conflict with its actual known knowledge.
The Bootstrapping Paradox: The Grip of Teaching Uncertainty
For enterprise manufacturers, achieving this credible uncertainty is more difficult than it seems. This requires teaching the uncertainty syntax to models through supervised fine-tuning (SFT). Since pre-trained models are given mostly authoritative text, they must be taught to say things explicitly, "I’m not entirely sure, but I think VentureBeat was founded…"
But SFT introduces a "The bootstrapping paradox." Unlike the standard training dataset where "correct answer" Regardless of the model, the ground truth of uncertainty is the model’s own dynamic knowledge base.
“Here’s the problem: the ‘correct’ expression of uncertainty is inherently dynamic, because it depends on what this particular model knows or doesn’t know at this particular point in training,” Yonah said. “If you train on a label that says ‘I don’t know
The path to self-aware AI
For enterprises looking to implement these capabilities without costly retraining, prompting serves as the most accessible entry point. “Prompt engineering is already something that most engineers do today, it offers the lowest-friction path to improving metacognitive behavior today,” Jonah said. Enterprise developers can explore frameworks like MetaFaith, an open-source project first co-authored by Jonah, to begin implementing metacognitive prompting in off-the-shelf models.
However, Jonah cautions "There is still considerable scope that signaling alone will not solve," Which means the industry will eventually need to rely on advanced reinforcement learning (RL) to incorporate metacognition deeper into model training.
Ultimately, as enterprises transition from isolated chat applications to complex, multi-agent workflows, self-awareness will become a defining prerequisite for reliable autonomy. But evaluating whether a model actually has this awareness remains a deep technical challenge.
“How do you actually evaluate whether a model can understand its own internal state?” Jonah asks. “Even in humans, it is difficult to define or distinguish ‘true’ self-monitoring capabilities from competent reliance on proxies. We face exactly the same challenges with LLMs: a model can learn to mimic the style of uncertainty without actually realizing its internal state. Developing evaluation frameworks that can tell the difference is one of the most important open problems in this area.”
<a href