Enterprises building and deploying agents have a problem: It’s taking too long for their engineers to figure out that an agent has made a mistake, and the loop is persistent, especially without a human at every step.
Longchain’s monitoring and evaluation platform Langsmith has launched a new capability in public beta that may make that issue more manageable. The LangSmith engine automates the entire chain by detecting production failures, diagnosing root causes against a live codebase, drafting fixes, and preventing regressions. It does this in a single automated pass.
The Langsmith Engine gives AI engineers a faster path to triage, but it launches in a crowded field: Anthropic, OpenAI, and Google are all pulling observation and assessment into their platforms.
Langsmith looks at engine failures
Langchen said in a blog post that the typical agent development cycle starts with tracing the agent to understand what it is doing, followed by identifying gaps, making changes to signals and tools, and building a ground truth dataset. Developers then run experiments and check regressions before shipping the agent.
The problem that customers often face is when defect patterns are not revealed in trace review, error recurrence becomes difficult to see, and there is no targeted assessor to catch the same problem when repeated in production.
According to the blog post, the LangSmith engine works by monitoring production traces for multiple signal types, including “obvious errors, online evaluator failures, trace inconsistencies, negative user feedback, and users asking questions the agent was not designed to answer.”
The engine will then read the live codebase, find the culprit and draft a pull request before proposing a custom evaluator for that specific failure pattern. The human comes into the approval stage.
It is built on top of Langsmith’s existing tracing and assessment infrastructure and also works with an enterprise’s assessor results.
Unlike observational tools like Weights & Biases, Aries Phoenix, and Honeyhive, the LangSmith Engine takes over the entire chain automatically – detecting the failure, diagnosing the root cause, drafting a fix – and brings the human in only to the approval stage.
Model providers are bringing evaluators onto the platform
While Langsmith identified this assessment loop as a need for many enterprises, the engine comes at a time when larger providers are beginning to offer observability tools within their platforms. This means enterprises can choose to use the end-to-end platform rather than adding the LangSmith engine to their existing workflow.
Anthropic’s Cloud Managed Agents brings together agent deployment, evaluation, and orchestration in a single suite. OpenAI’s Frontier offers a similar end-to-end platform for building, operating, and evaluating enterprise agents – though both have faced questions from enterprises wary of committing to a single vendor.
However, practitioners say that not everyone wants to bring assessment and observation completely into one platform.
Leigh Cooney, founder and principal consultant at WorkWise Solutions, told VentureBeat that third-party oversight is the default for many enterprises.
“The fund I work with runs the cloud for analytics and GPT for a separate workflow. If the overview lives inside each provider’s tooling, you now have two systems that can’t talk to each other. Your compliance team can’t produce a unified audit trail,” he said. “So third-party observability is alive because multi-model is already the default in the enterprise, and someone has to sit between the providers.”
Jessica Arredondo Murphy, CEO and co-founder of True Fit, said independent platforms like Langsmith have to prove to enterprises that they can "Answer the long-term question of whether they become the cross-model operating layer for quality and reliability.
“Enterprises are not consolidating on first-party model provider tooling as quickly as model providers would have liked. What I see is a practical split: Teams will use first-party tooling for rapid onboarding and early-stage debugging, but as they care about production reliability, governance, and long-term resiliency, they introduce a more neutral layer for observation and evaluation,” she said.
The Langsmith Engine is now available in public beta. Teams can connect a tracing project, optionally connect their repo, and the engine will automatically start surfacing issues from the production trace.
<a href