As LLMs continue to improve, there has been some discussion in the industry about the continued need for standalone data labeling tools, as LLMs are becoming increasingly capable of working with all types of data. humansignal, The major commercial vendor behind the open-source Label Studio program has a different approach. Rather than less demand for data labeling, the company is seeing more.
Earlier this month, HumanSignal acquired Erude AI and launched its Physical Frontier Data Labs for new data collection. But generating data is only half the challenge. Today, the company is tackling what’s next: proving that AI systems trained on that data actually work. New multi-modal agent evaluation capabilities let enterprises validate complex AI agents creating applications, images, code, and video.
"If you focus on the enterprise segment, all the AI solutions they are building still need to be evaluated, which is another term for data labeling, by humans and even by experts," Michael Malyuk, co-founder and CEO of HumanSignal, told VentureBeat in an exclusive interview.
The intersection of data labeling and agentic AI evaluation
Having accurate data is great, but it is not the ultimate goal for an enterprise. Where modern data labeling is headed is evaluation.
This is a fundamental shift that enterprises need to verify: not whether their model correctly classified an image, but whether their AI agent made good decisions in a complex, multi-step task involving reasoning, tool usage, and code generation.
If the evaluation is simply data labeling for AI outputs, then the shift from models to agents represents a step change in what needs to be labeled. Where traditional data labeling might involve labeling images or classifying text, agent evaluation requires multi-step logic chains, tool selection decisions, and multi-modal outputs – all assessed within a single interaction.
"Now there is a dire need of not only a human being in the loop, but also an expert in the loop." Malyuk said. He pointed to high-risk applications like health care and legal advice as examples where the costs of errors remain prohibitively high.
The relationship between data labeling and AI evaluation goes deeper than semantics. Both activities require the same basic abilities:
-
Structured Interface for Human Decision Making: Whether reviewers are labeling images for training data or assessing whether an agent has correctly orchestrated multiple devices, they need purpose-built interfaces to systematically capture their assessments.
-
multi-reviewer consensus: A high quality training dataset requires multiple labelers who resolve disagreements. High-quality evaluation requires the same – multiple experts assessing the output and resolving differences in judgment.
-
large-scale domain expertise: Modern AI systems require subject matter experts to train, not just crowd workers pushing buttons. Evaluating production AI outputs requires a similar depth of expertise.
-
Feedback gets looped into AI systems: Labeled training data feeds model development. Evaluation data provides continuous improvement, fine-tuning, and benchmarking.
Evaluating the complete agent trace
The challenge with evaluation agents is not just the amount of data, but also the complexity of what is to be evaluated. Agents do not generate simple text output; They generate logic chains, make tool selections, and create artifacts in multiple modalities.
New capabilities in Label Studio Enterprise Address Agent Verification Requirements:
-
Multi-Modal Trace Inspection: The platform provides a unified interface for reviewing the entire agent execution trace—logic steps, tool calls, and outputs across modalities. This addresses a common pain point where teams must parse different log streams.
-
Interactive Multi-Turn Assessment: Evaluators assess the flow of interactions where agents maintain state over multiple turns, validating context tracking and interpretation of intent throughout the interaction sequence.
-
agent arena: Comparative evaluation framework for testing different agent configurations (base models, prompt templates, guardrail implementations) under similar conditions.
-
Flexible Assessment Rubrics: Teams define domain-specific evaluation criteria programmatically, rather than using pre-defined metrics, that support requirements such as accuracy of understanding, response appropriateness, or output quality for specific use cases.
Agent evaluation data labeling is the new battlefield for vendors
HumanSignal is not alone in believing that agent evaluation represents the next phase of the data labeling market. Competitors are making similar pivots as the industry reacts to both technological change and market disruption.
labelbox Launched its assessment studio in August 2025, focusing on rubric-based assessment. Like HumanSignal, the company is expanding beyond traditional data labeling into production AI validation.
The overall competitive landscape for data labeling changed dramatically in June when Meta invested $14.3 billion for a 49% stake in Scale AI, the previous market leader. The deal triggered an exodus of some of Scales’s biggest customers. HumanSignal took advantage of the disruption, with Malyuk claiming his company was able to win several times as many competitive deals last quarter. Malyuk cites platform maturity, configuration flexibility, and customer support as differentiators, although competitors make similar claims.
What does this mean for AI builders
For enterprises building production AI systems, the convergence of data labeling and evaluation infrastructure has several strategic implications:
Start with ground truth. Investing in creating high-quality labeled datasets with multiple expert reviewers who resolve disagreements pays dividends throughout the AI development lifecycle – from initial training to continuous production improvement.
Observability proves necessary but insufficient. While it is important to monitor what AI systems do, observation tools measure activity, not quality. Enterprises require dedicated evaluation infrastructure to assess outputs and drive improvements. These are different problems that require different abilities.
The training data infrastructure doubles as the evaluation infrastructure. Organizations that have invested in a data labeling platform for model development can extend the same infrastructure to production evaluation. These are not different problems that require different tools – these are the same fundamental workflows implemented at different lifecycle stages.
For enterprises deploying AI at scale, the bottleneck has shifted from building models to validating them. Organizations that recognize this shift stand to gain early benefits from shipping production AI systems.
The key question for enterprises has evolved: not whether AI systems are sophisticated enough, but whether organizations can systematically prove that they meet the quality requirements of specific high-stakes domains.