Two years ago, Yuhong and I were facing the same problem again and again. We were in growing teams and it was ridiculously hard to get the right information into our Docs, Slack, meeting notes, etc. Existing solutions required sending our company’s data, lacked customization, and frankly didn’t work well. So, we started Danswer, an open-source enterprise search project that’s self-hosted and easily customized.
As the project grew, we started to see an interesting trend – even though we were clearly a search app, people just wanted to use Danceware to chat with LLMs. We’ve heard, “Connectors, indexing, and search are great, but I’m going to start by connecting GPT-4o, Cloud Sonnet 4, and Queue to provide my team a secure way to use them”.
Many users would later add RAG, agents, and custom tools, but most usage remained ‘Basic Chat’. We wondered: “Why would people adopt enterprise search when other AI chat solutions exist?”
As we continued talking to users, we realized two main things:
(1) Providing a company with secure access to LLM with great UI and simple tools is a big part of the value addition of AI
(2) To provide Well It’s harder than you can imagine and the bar is incredibly high
Consumer products like ChatGPT and Cloud already provide a great experience – and chat with AI for work is (ideally) something everyone in the company uses 10+ times per day. People expect the same fast, simple and intuitive UX with a full feature set. Getting hundreds of little details to take the experience from “it works” to “it feels magical” isn’t easy, and nothing else in the field has managed to do it.
So ~3 months ago we turned to the open-source chat UI Onyx:
– (Truly) world-class chat UX. Usable for both a new college graduate who grew up with AI and an industry veteran using an AI tool for the first time.
– Support for all common add-ons: RAG, Connectors, Web Search, Custom Tools, MCP, Assistant, Deep Research.
– RBAC, SSO, permissions syncing, easy on-premises hosting to work with for large enterprises.
Through extensive research, working between model providers, and building features like code interpreters, we’ve learned a lot of non-obvious things about engineering LLM that have been important in getting Onyx to work. I’d like to share two that were particularly interesting (happy to discuss more in the comments).
First of all, context management is one of the hardest and most important things to get right. We’ve found that LLMs really struggle to remember both system prompts and previous user messages in long conversations. Even simple instructions like “Ignore sources of type X” in the system prompt are often ignored. This is compounded by multiple tool calls, which can often feed huge amounts of context. We solved this problem with the “Reminder” prompt – a short 1-3 sentence blurb injected at the end of the user message that describes the non-negotiables that the LLM must adhere to. Empirically, LLMs attend most toward the very end of the context window, so this placement gives the highest probability of adherence.
Second, we need to understand the “natural tendencies” of certain models when using tools and build around them. For example, the GPT family of models is fine-tuned to use a Python code interpreter that operates in Jupyter notebooks. Even if stated explicitly, this rules out adding `print()` around the last line, because in Jupyter, this last line is automatically written to stdout. Other models do not have this strong priority, so we have to design our model-agnostic code interpreter to automatically `print()` the last bare line.
So far, we have a Fortune 100 team fork Onyx and provide over 10 thousand employees access to every model within a single interface, and thousands of use-cases creating specific assistants for each department, using the best model for each task. We have seen Onyx fully airgapped with locally hosted LLM to provide co-pilot to teams working in sensitive industries that would not have been possible otherwise.
If you want to try Onyx, follow https://docs.onyx.app/deployment/getting_started/quickstart to get set up with Docker locally in <15 minutes. For our cloud: https://www.onyx.app/. If there's anything you'd like to see to make it easier to convert your ChatGPT Enterprise/Cloud Enterprise subscription, we'd love to hear it!
<a href