When initially experimenting with LLM and agentic AI, software engineers perception ai Advanced code generation, complex schema and heavy instructions implemented.
However, soon, trial and error taught the team that it could be done. Get rid of that complex data modelingNotion’s AI engineering lead Ryan Nystrom and his team focused on simple symbols, human-readable representations, minimal abstraction, and familiar Markdown formats, As a result model performance improved dramatically,
Applying this re-wired approach, the AI-native company released V3 of its productivity software in September. Its notable feature: Cutomizable AI Agent – which has quickly become Notion’s most successful AI tool to date. Nystrom calls this a “step function improvement”, based on usage patterns compared to previous versions.
“It’s that feeling when the product is being pulled out of you instead of you trying to push it,” explains Nystrom. VB Beyond the Pilot Podcast“From that moment on, we really already knew we had something, Now it’s, ‘How can I ever use Notion without this feature?'”
‘Rewiring’ for the AI agent era
As a traditional software engineer, Nystrom was accustomed to “extremely deterministic” experiences. But a shocking moment came when a colleague advised him to describe his AI prompts the same way he would for a human, rather than codifying rules for how agents should behave in different scenarios. Reasoning: LLMs are designed to understand, “see” and reason about content in the same way humans can.
“Now, whenever I’m working with AI, I’ll re-read the prompts and tool descriptions [ask myself] Is this something I can give to someone without any context and they can understand what’s going on?” Nystrom said on the podcast. “If not, it will work poorly.”
Stepping back from “too complex rendering” of data within Notion (such as JSON or XML), Nystrom and his team rendered Notion pages as Markdown, the popular device-agnostic markup language that defines structure and meaning using plain text without the need for HTML tags or formal editors. It allows models to interact with text files, reading, searching, and making changes.
Ultimately, this required Notion to restructure its systems, with Nystrom’s team focusing largely on the middleware transition layer.
He also quickly recognized the importance of exercising restraint when it came to context. It’s tempting to load as much information as possible into a model, but this can slow things down and confuse the model. For Notion, Nystrom described the 100,000 to 150,000 token range as the “sweet spot.”
“There are cases where you can load a lot of content into your context window and the model will struggle,” he said. “The more you put into the context window, the more you see degradation in performance, latency, and accuracy.”
A conservative approach to tooling is also important; Nystrom advised that this could help teams avoid the “slippery slope” of endless features. Notion focuses on a “curated menu” of tools rather than a giant cheesecake factory-like menu that creates a paradox of choice for users.
“When people ask for new features, we can add a tool to the model or agent,” he said. But, “the more tools we add, the more decisions the model has to make.”
Bottom Line: Channel the model. Use APIs the way they were meant to be used. Don’t try to be pretentious, don’t try to make it overly complicated. Use plain English.
Listen to the full podcast to hear about it:
- Why AI is still in the pre-BlackBerry, pre-iPhone era;
-
importance of "dog food" In product development;
-
Why you shouldn’t worry about how cost-effective your AI feature is in the early stages – which can be optimized later;
-
How can engineering teams keep equipment to a minimum in the age of MCP;
-
Notion’s evolution from wikis to full-fledged AI assistants.
Subscribe to Beyond the Pilot apple podcasts, spotifyAnd youtube,
<a href