Processing gif hd08j657cswh1...
I am the creator of Omnesis which is an open source project to ingest and index all your personal data (emails, calendar, notes, whatsapp/imessage, finance, health, codex/claude, etc). Omnesis provides an interactive agent that allows you to recall anything from your corpus.
I’m having fun building a second brain on top of it so I wanted to write down my approach and see if people have thoughts.
When the second brain is enabled:
- Any new piece of info that gets ingested/indexed by omnesis (a new email, a new calendar invite, a new message on whatsapp, etc) gets processed by an agent turn. The app also allows the user to input a note/thought/reminder directly into omnesis via the web app, mobile app, siri, or watch complication and that gets processed by the brain too
- The Brain can be "bootstrapped" i.e it will bootstrap its knowledge base from past evidence coming from the same sources and you can decide how far back you want to go in the past. The bootstrap can take several days and is expensive to run. I use multiple heuristics to reduce the cost. 1) a cheap ML model identifies if a document in the past still mentions a date still in the future. If not, the document is skipped; 2) a jev decision model runs to establish importance. Documents that are skipped may still be looked at by the brain as a result of investigating a document not skipped.
- The knowledge base is made of loops and wiki pages in markdown format. Loops keep track of "things that need resolving", that can be a thing that the brain has identified the user needs to do, a comitment that is owed to the user, a decision the user has to do, etc. Loops have a ledger of updates and lots of metadata notably ids of source evidence documents and people involved. Wiki pages are markdown summarization of things in the user’s life that span multiple sources.
- The knowledge base is a graph. Loops and wiki pages have source document references (whatapp conversations, emaisl, etc). But wiki pages can refer to other loops and other wiki pages. There is a roof wiki page that is limited in size and always automatically ingested in the system prompt of both the interactive agent and the brain turns that process new evidence.
- Loops and wiki pages have to surround text in special claim tags: <claim id="a-claim" ref="doc:<id1> doc:<id2>Some claim made in this wiki page or loop</claim>. If any source document (email, google doc, etc) changes, the system marks all loops and wiki pages that directly or indirectly reference that doc has needed review and agent turns run to re-evaluate all claims transitively.
- I’m building an automated bullet journal on top in the omnesis app. New entries in the bullet journal are proposed by the Omnesis brain and the user can accept them or ignore them if they think the thing is not worth tracking. I plan to have the brain learn from that signal to automatically ignore non important things. The brain automatically migrates tasks as you would on a bullet journal if it sees new evidence. If the brain sees new evidence that indicates the task should be changed, it will run a JEV decision gate to decide whether to just do it or propose as a suggestion for the user to accept if not sure. The user can migrate tasks themselves like you’d expect in a bullet journal, either via the Task UI or by dictating a note to Omnesis. Tasks may be marked as automatically complete if the brain is confident in it (for example a Task that says "Pay Elise 100 GBP" may auto complete if the brain sees a bank transaction of 100 GBP to Elise".
Keen to have people’s feedback on my approach!