Eight months ago I left my job at Stripe to build my knowledge base. By then I had suppressed the desire to do so for years. There have been many attempts to create a faster alternative, including some of my own. I don’t think any of them would have come close to accomplishing it.
What does it mean to build a better knowledge base? By the time I left my job, I had worked on several teams, built a few knowledge systems, maintained a few others, and read a lot of them. Stripe was the first company I worked for that figured this out; They formed a team to create their own internal knowledge base, allowing search to bridge the gap with other services.
As it turns out, all you need is speed and ingenuity. Teams had one or more locations, and you didn’t have to remember what was where, because you could find it with a quick search. A task is assigned to your team whenever a document becomes outdated.
The timing couldn’t be more right. Linear had already figured this out for product management. Many documentation tools gave up on building a good product and focused their attention on competing with chat interfaces. Atlassian is ending their Data Center offering, a once-in-a-lifetime event that lets you reach out and sign up for their biggest customers. The rules on data residency are getting stricter in Europe, and it helps to be an Irish company.
How do you make a simple product? Counterintuitively, you start out by building something too complex. It’s easy to wrap a database in a theme and call it quits for a week, but you’ll run into the same scaling issues as the next tool.
This is not where your favorite language comes from. I started by wrapping the database into a theme in a week. I used Go and it’s a nice little language. But within a few weeks, I was writing more code to generate code than actual application code. There are many parts to a knowledge base, and no one has time to write out all the boilerplate. If you are building an increasingly complex system, it would be better if you had some automated way of knowing when services are no longer compatible. At this point, I decided that users needed to be able to collaborate in real time, and I wasn’t satisfied with search latency, and querying the database to check who could do the work was becoming tedious. This is where your preferred language comes in.
const (
CreateSpaceMethod = "POST"
CreateSpacePath = "/v1/spaces"
)
type CreateSpaceRequest struct {
Name string `json:"name" binding:"required"`
}
type CreateSpaceResponse = query.Space
func (r *Router) CreateSpace(c *gin.Context, req CreateSpaceRequest) (int, CreateSpaceResponse, error) {
// [...]
}It didn’t take long to rewrite everything in Rust. Somehow I ended up with fewer lines of code because I replaced my handwritten bits of code with macro crates. What used to look like text soup is now more readable utoipa call out. The ecosystem isn’t the biggest out there, but it has some great crates that make your life so much easier.
#[utoipa::path(get, path = "/v1/spaces", responses((status = OK, body = Vec::Space>)))]
pub async fn list_spaces() -> Result<(StatusCode, Json<Vec::Space>>)> {
// [...]
}a little zanzibar
With Rust in my toolbox, or rather being my toolbox, there were a lot of pieces I could now at least dream of making, and stay up all night to make a reality. Some time ago, Google wrote about its Zanzibar authorization system. I was getting fed up of having to query the database on every call to make sure your guy can actually access the page they’re requesting. Some open source implementations inspired by Google’s architecture already existed, however, they felt a bit clumsy. Launching a new Docker container is straightforward, but you’ve now taken on the burden of maintaining and debugging a service that may not be as documented or supported as you might need.
The concepts behind Zanzibar are compelling; Separate your authorization system from your normal database queries and application code. I chose to make a smaller version of this. It is not decentralized, it is persisted on Postgres, but loaded into memory on startup. It has no complex configuration language; I already know my organizations. The relation is defined in a csv File right next to the code.
Scope, Role, Action, Object
space, owner|admin, read|create|update|list|delete, space
page, viewer, read|list, page|commentPermissions are inherited. Being a member of a team with access to a certain location gives you instant access to that same location. Services can implement the authorization system with a single macro call.
grant!(user_id, Editor, Page(page_id));
must!(user_id, Page(page_id), Update);Without much optimization, it takes nanoseconds to check whether the user has the correct permissions to access the resource. It takes milliseconds to list every resource a user or team has access to.
a more elastic search
An encyclopedia is only as good as its search engine. I was following the success of tantivy For a while. Its demos were convincing. There are easy ways to do this, and all your favorite tools do it the same way; But they may have been very slow. Is your path really better than a hundred others if you’re making the same choices as everyone else? My search engine was ready. Shortly after, I added language detection, partially using whatlangand multilingual tokenization.
I wasn’t expecting much from it, but now I could see the results coming in without any delay as I typed my queries. With authorization also under my control, I went one step further and integrated it with search. At the core of the engine, only the resources you can access are considered. They are kept in sync seamlessly.
It may have taken longer to create these two integrated systems than it should have, but after reading the experiences of other companies, I think it saved me years of headaches.
oxidation prose
i tampered with it prosemirror Earlier and this time too it was the obvious choice. There are many collaboration plugins to use with it. At this point I started to think that my choice of Rust might eventually come back to haunt me. The project itself comes with well-tested collaboration primitives, which I couldn’t use, because they were in JavaScript. I knew of some alternatives, but they felt bloated and became slow with larger documents and more users. It would have been foolish to rewrite such a large project in Rust.
User edits are recorded as a list of steps, which is fed prosemirror and compared to the original document for conflicts. An updated document with the new version hash is generated, and persisted. To overcome the problem, I implemented the steps on the client-side, sending back the entire document with the steps, which I forwarded to other connected users. It worked fine. With everything happening in real time and having fast enough internet, no one will notice. But this meant that anyone could send anything and alter the entire document. It also introduces unnecessary latency and potential lag if you’ve ever suffered through slow internet. With steps, and therefore, new documents being created every few key presses, this was not ideal. After building such a great authority and search system, it felt like a step down.
Is it stupid to write again? prosemirror In war? Around this time I realized that someone had already done this for Go. Ugh. I spent the rest of the week setting up the setup quickjs Or v8 To process the steps on the backend. They were fine, albeit buggy, but I didn’t realize the importance of the added complexity. It’s time to work. I spent the next week porting prosemirror To Rust with all its tests and thousands of compatibility snapshots. I think I barely left the house at that time.
#[test]
fn slice_can_cut_half_a_paragraph() {
let original = doc!(p!("hello world"));
let expected = doc!(p!("hello"));
let result = original.slice(0, Some(6), None).unwrap();
assert_eq!(result.content, *expected.content());
assert_eq!(result.open_start, 0);
assert_eq!(result.open_end, 1);
}It’s hard to know how great something is until you do it. In this case, applying document edits now takes only microseconds.
I didn’t think much of it at the time, but after a while I realized the potential applications of doing this. It’s very easy to extract text content from documents before inserting them into search engines, easy to remove links or mentions, and easy to add tab completions. You can’t rely on a large language model to help you edit a structured document, but if you check for any conflicts and resolve them immediately, it doesn’t matter if half of its suggestions break the structure. Adding tab completion or structured suggestions isn’t my top priority at the moment, but it’s at least possible now.
safe socket
That little crab is quite trustworthy. I think it matters most when you put thousands of lines of depth into your work. It’s better to be told when something is broken than to find out at 2 in the morning a few months later. thanks for doing utoipaAgain, real-time messages are synced to the frontend, so TypeScript can continue to maintain trust.
fn create_docs() -> OpenApi {
OpenApiBuilder::new()
.info(Info::new("live", "1.0.0"))
.components(Some(
ComponentsBuilder::new()
.schema_from::<Command>()
.schema_from::<CommandReply>()
.schema_from::<MsgContent>()
.build(),
))
.build()
}beyond prose
An editor is more than just text. This is its own little app. It is well known by now react Does not match well with prosemirrorYou can hack your way around it, or you can once again choose to do something different, i am quite fond of solidUnfortunately, the ecosystem is still too small to build a complete user interface with it, But it’s solid enough to integrate prosemirror And handle its rendering cycle. By getting the signs out of the way, one can make a lot of sense. Nothing but time is stopping me from adding advanced diagrams, plots, macros, variables, canvases and everything that no other editor has ever dared to even dream of. The knowledge base can be more than just prose; Some threads, some slides, even a spreadsheet.
Tools are about workflow. Linear had figured it out. And workflows are about structure. The document should be finished. They should be prepared for dead links and mistakes. Your knowledge tools should sync with your work management tools. These are some of the hanging pieces below. We live in a time where we can transcend the limits of structure. A link may not go away, but it may have lost any semantic relevance by this point. Recent code changes may have made your diagrams out of date. Language models won’t write your prose, but they can speed up the workflow.
What will happen next?
There is still a lot more work to be done. I’ve created a marketing page with some demos that you can check out. You can join the waiting list. I’m aiming to launch within the next six months. Each seat will cost approximately €/$10. If you see yourself using this product, consider sponsoring Outcrop today for €/$100; You’ll get €/$200 in credit at launch.
If you have any questions or ideas, you can contact me imed under outcrop.app,
Thank you!
Imed
<a href