There is a lot of enterprise data trapped in PDF documents. Sure, general AI tools are capable of ingesting and analyzing PDFs, but the accuracy, time, and cost are less than ideal. New technology from Databricks could change this.
The company gave its details this week "ai_parse_document" The technology is now integrated with Databricks’ Agent Bricks platform. The technology addresses a key barrier to enterprise AI adoption: approximately 80% of enterprise knowledge remains locked in PDFs, reports, and diagrams that AI systems struggle to accurately process and understand.
"It is a common belief that parsing a PDF is a solved problem, but in reality it is not," Erich Elsen, principal research scientist at Databricks, told VentureBeat. "The challenge is not just that the documents are unstructured; It’s that enterprise PDFs are inherently complex. They mix digital-native content with tables, charts, and photographs of scanned pages and physical documents with irregular layouts, and most existing tools fail to accurately capture that information."
The hidden complexity behind document parsing
While optical character recognition (OCR) has been in existence for decades, Elson argues that extracting usable, structured data from real-world enterprise documents remains fundamentally unsolved.
Key elements such as tables with merged cells, image captions, and spatial relationships between document elements are routinely omitted or misread by existing tools, rendering downstream AI applications, retrieval-augmented generation (RAG) systems or business intelligence dashboards unreliable.
The typical enterprise workaround is to put together several incomplete tools: one service for layout detection, another for OCR, a third for table extraction, plus additional APIs for data analysis. This approach requires months of custom data engineering and ongoing maintenance as the document format evolves.
"To compensate, teams had to assemble many incomplete tools or build extensive custom pipelines, spending months on data engineering instead of innovation," Elsen said. "ai_parse_document solves this by extracting full, structured data from real-world documents – so organizations can finally trust and query unstructured data directly within Databricks."
Technical Perspective: End-to-End Training vs. Pipeline Stacking
There are several services on the market today for parsing PDFs, including AWS TextRect, Google Document AI, and Azure Document Intelligence. Elsen argued that instead of simply reading text, the tool uses a system of modern AI components trained from end to end to extract structured context with state-of-the-art quality.
The function goes beyond basic extraction to capture:
Tables are preserved exactly as they appear, including merged cells and nested structures
Figures and diagrams with AI-generated captions and descriptions
Spatial metadata and bounding boxes for precise element location
Alternative image output for multimodal search applications
All results are stored directly in the Databricks Unity Catalog as delta tables, meaning that parsed documents become queryable structured data without leaving the Databricks environment. This is an important difference from cloud services that require exporting data for processing.
"Through data-centric training and optimized inference, we have achieved 3-5x lower costs, matching or surpassing leading systems like Textract, Document AI, and Azure Document Intelligence." Elsen said.
Early Enterprise Adoption in Manufacturing and Industrial Sectors
Many leading enterprises have already deployed ai_parse_document in production with use cases of data science workflow optimization, democratization of document processing, and RAG application development.
For example, Elsen notes that Rockwell Automation uses ai_parse_document to reduce configuration overhead for its data scientists.
"The significant setup that was once required to support complex solutions has now been streamlined, allowing their teams to spend more time innovating and less time managing infrastructure." He said.
TE Connectivity, meanwhile, is using ai_parse_document to democratize unstructured data processing.
"Previously, extracting tables, text, and metadata from documents required complex, code-heavy workflows," Elsen said. "With Databricks, they’ve condensed them all into a single SQL function, making advanced document processing accessible not just to data scientists, but to every data team."
Emerson Electric is another early adopter. the company is using ai_parse_document for the RAG use case. Elsen explained that by enabling parallel document parsing directly within Delta tables, Emerson has made RAG applications both faster and simpler, within its existing Databricks environment.
platform integration game
While Databricks has a long history with open source, the ai_parse_document technology is a proprietary component of the Databricks platform.
Unlike the standalone Document Intelligence API, ai_parse_document is deeply integrated with Databricks’ Agent Bricks platform, a collection of AI functions and orchestration capabilities for building production AI agents.
This function works with Databricks’ extensive data infrastructure, including:
Spark Declarative Pipeline: Provide automated incremental processing, meaning new documents arriving in SharePoint, S3 or Azure Data Lake storage are automatically parsed without manual orchestration.
Unity List: Controls permissions, audit trails, and data lineage for parsed content just as it does for structured data.
Vector Search: The index parses document elements including text, tables, and figures with captions for multimodal RAG applications.
AI Function Chaining: Allows developers to pipe ai_parse_document output directly into ai_extract (entity extraction), ai_classify (document classification) and ai_summarize (content summarization) within a single SQL query.
Multi-Agent Supervisor: Coordinates document-processing agents with other specialized agents for complex workflows.
"Parsing is only the beginning and is rarely the end," Elsen said. "The goal is to allow customers to chain our ai_functions, such as ai_extract and ai_classify, with ai_parse_document to transform their documents into actionable data and insights. Our goal is also to make it effortless to transform a collection of documents into a knowledge database for use in RAG or other information retrieval agents."
What this means for enterprise AI strategy
For enterprises building AI agent systems, it is important to understand how PDF documents are actually used and understood by the system.
The Databricks approach sheds new light on an issue that many may have considered a solved problem. It challenges existing expectations with a new architecture that can benefit many types of workflows. However, this is a platform-specific capability that requires careful evaluation for organizations that are not already using Databricks.
For technology decision makers evaluating AI agent platforms, the key takeaway is that document intelligence is moving from a specialized external service to an integrated platform capability.