A core element of any data recovery operation is the use of a component known as a retriever. Its job is to retrieve relevant content for a given query.
In the AI era, retrievers have been used as part of RAG pipelines. The approach is straightforward: Retrieve relevant documents, feed them into an LLM, and let the model generate answers based on that context.
Although retrieval may seem like a solved problem, it was not actually solved for modern agentic AI workflows.
In Research Published this week, Databricks introduced Instructed Retriever, a new architecture that the company claims delivers up to 70% improvement compared to traditional RAG on complex, instruction-heavy enterprise question-answer tasks. The difference depends on how the system understands and uses the metadata.
"A lot of the systems that were built for retrieval before the era of large language models were actually built for use by humans, not for use by agents," Michael Bendersky, director of research at Databricks, told VentureBeat. "We found that in many cases, the errors that are coming from the agent are not because the agent is not able to reason about the data. This is because the agent is not able to retrieve the correct data the first time."
What are traditional RAG retrievers missing?
The main problem is how traditional RAG handles Bendersky calls "System-level specifications." These include complete references to user instructions, metadata schema, and examples that define what a successful recovery should look like.
In a typical RAG pipeline, a user query is converted into an embedding, similar documents are retrieved from a vector database, and they are fed into a language model for result generation. The system may include basic filtering, but it basically treats each query as a separate text-matching exercise.
This approach breaks with real enterprise data. Enterprise documents often include rich metadata such as timestamps, author information, product ratings, document types, and domain-specific attributes. When a user asks a query that requires logic on these metadata fields, traditional RAG struggles.
Consider this example: "Show me five-star product reviews from the past six months, but remove anything from Brand X." Traditional RAG cannot reliably translate that natural language constraint into appropriate database filters and structured queries.
"If you just use the traditional RAG system, there’s no way to use all these different signals about the data contained in the metadata," Bendersky said. "They need to be handed over to the agent itself to do the right thing in recovery."
The problem becomes even worse as enterprises move beyond simple document searches toward agentic workflows. A human using the search system can reword queries and manually apply filters when initial results miss the target. An AI agent working autonomously needs a self-recovery system to understand and execute complex, multidimensional instructions.
How does an Instructed Retriever work?
Databricks’ approach fundamentally redesigns the recovery pipeline. The system propagates complete system specifications through each step of both recovery and generation. These specifications include user instructions, labeled examples, and index schema.
The architecture adds three key capabilities:
query decomposition: The system breaks complex, multi-part requests into a search plan that contains multiple keyword search and filter instructions. a request for "Recent FooBrand products except Lite models" Decomposes into structured queries with appropriate metadata filters. Traditional systems will attempt a single semantic search.
metadata logic: Natural language instructions get translated into database filters. "from last year" The date filter becomes, "five star reviews" The rating filter is created. The system understands what metadata is available and how to match it to user intent.
contextual relevance: The reranking phase uses the full context of user instructions to promote documents matching the intent, even if the keywords are weak matches. The system can prioritize recency or specific document types based on specifications rather than just text similarity.
"The magic is in how we frame the questions," Bendersky said. "We try to use the device as an agent, not as a human being. It has all the intricacies of the API and utilizes them to the best possible extent."
episodic memory vs retrieval architecture
In the late 2025s, there was a shift in the industry away from RAG to agentic AI memory, sometimes called episodic memory. approach including massa And a member emerged offering the promise of a RAG-free future.
Bendersky argues that episodic memory and sophisticated retrieval serve different purposes. Both are essential for enterprise AI systems.
"There is no way you can keep everything in your enterprise in your episodic memory," Bendersky noted. "You need both. You need contextual memory to provide specifications, to provide schema, but you still need access to data, which may be distributed across multiple tables and documents."
Episodic memory excels at maintaining task specifications, user preferences, and metadata schema within a session. it keeps "rules of the game" Easily available. But the real enterprise data corpus exists outside this context window. Most enterprises have data volumes that exceed even generous reference windows by orders of magnitude.
Instructed retrievers take advantage of contextual memory for system-level specifications when using retrieval to access broader data assets. Specifications in the context describe how the retriever constructs queries and interprets the results. The retrieval system then pulls specific documents from potentially billions of candidates.
This division of labor matters for practical deployment. Loading millions of documents into context is neither feasible nor efficient. Metadata alone may be sufficient when dealing with heterogeneous systems in an enterprise. Instructed Retriever solves this by making metadata immediately usable without needing to fit into context.
Availability and practical considerations
Instructed Retriever is now available as part databricks agent bricksIt is built into the Knowledge Assistant product. Enterprises using Knowledge Assistant to build question-answer systems on their documents automatically take advantage of the Instructed Retriever architecture without building custom RAG pipelines.
The system is not available as open source, although Bendersky indicated that Databricks is considering wider availability. For now, the company’s strategy is to release benchmarks like Stark-Instruct to the research community while keeping the implementation of its enterprise products proprietary.
The technology shows particular promise for enterprises with complex, highly structured data that includes rich metadata. Bendersky cited use cases in finance, e-commerce and healthcare. Essentially any domain where documents have meaningful attributes beyond raw text can take advantage.
"We have seen in some cases that it unlocks things that the customer could not do without it," Bendersky said.
He explained that without the Instructed Retriever, users would have to perform more data management tasks to put content into the correct structure and tables to properly retrieve the right information for the LLM.
“Here you can create an index with the right metadata, point your retriever at it, and it will work out of the box,” he said.
What this means for enterprise AI strategy
For enterprises building RAG-based systems today, the research brings up an important question: Is your recovery pipeline really capable of the instruction-following and metadata logic required in your use case?
The 70% improvement demonstrated by Databricks is not achievable through incremental optimization. This represents an architectural difference in the flow of system specifications through the retrieval and production process. Organizations that have invested in carefully structuring their data with detailed metadata may find that traditional RAG is leaving much of the value of that structure on the table.
For enterprises that want to implement AI systems that can reliably follow complex, multi-part instructions on heterogeneous data sources, research indicates that retrieval architecture can be a key differentiator.
Those who still rely on basic RAG for production use cases involving rich metadata should evaluate whether their current approach can fundamentally meet their needs. The Databricks performance difference demonstrates that a more sophisticated recovery architecture is now table stakes for enterprises with complex data estates.
<a href