Brokentooth.io / 06 Writing / I Got Tired of Being My AI's Memory
I Got Tired of Being My AI's Memory
Why I started building SessionMunch, and why another clever answer was less useful than remembering the last one.
In September, I asked an agent for a recap of work we'd just done. It couldn't reliably give me one. The particularly stupid part was that the previous session had been about getting the memory system to work.
I'd managed to create a memory problem about the memory problem. There should probably be a certificate for that. I would have to explain where we left it.
That is a better explanation of SessionMunch than a paragraph about persistent intelligence. I wanted to finish something, come back later, and keep going without becoming the person responsible for reconstructing every conversation.
SessionMunch is an open-source project in development. This series describes the public v0.1.0 candidate source as inspected on 27 October 2026. It is not an announcement that every integration or release package is finished.
The handoff became another job
I move between agents. Sometimes a different tool is better suited to the next piece of work. Sometimes a usage window makes the decision for me. Sometimes I just want another opinion before an enthusiastic machine makes a very confident mess.
The work belongs to the project, but the explanation keeps getting trapped in whichever conversation happened to produce it.
A fresh agent can read the repository. That helps enormously. It can see the code and the documentation we've kept current. What it may not see is why we rejected the obvious approach, which test actually ran, whether the last fix was deployed, or that the very sensible suggestion it has just made cost us an entire evening last time.
So I write a handoff. Then I explain the handoff. Then something changes and the handoff becomes a historical document wearing a current date.
There is a point where the human in an agent workflow starts doing rather a lot of work for the agents.
I want to spend my attention deciding what to build. Reintroducing the project should not be a recurring occupation.
What I actually wanted
I didn't need an agent to remember everything all at once. I needed it to find the relevant part when it mattered.
Those are different jobs. A context window is the material a model can work with during a particular interaction. Project history can be much larger, spread across different sessions, machines and tools. Shoving all of it into every new prompt would be expensive, noisy, and a wonderful way to bury the one useful sentence.
My practical requirement was smaller: keep useful evidence, organize it so we can search it, and let the next agent bring back enough to continue intelligently.
That includes uncertainty. “A fix was proposed” and “the fix passed a test” and “the fix is running in production” are three different statements. A memory system that quietly merges them is providing excellent customer service to the wrong department.
The question that sharpened it
By October, I was thinking about the first person who downloads the project and imports their history. They don't want to wait indefinitely while a model summarizes everything before they can find anything.
I asked how much useful information we could get out of those sessions without intelligence. Could ordinary Rust code identify requests, commands, checks and files? Could every supported imported session at least get a useful page, with links back to the evidence?
That question changed the order of operations. There should be a dependable floor underneath the clever stuff.
The current candidate has an event-based card builder and a separate optional AI-summary layer. The deterministic part reports what it can extract from the visible record; the model can add an interpretation. Neither gets to pretend an incomplete transcript is complete. The next article explains that separation.
I didn't start from an empty folder
SessionMunch builds on the open-source ai-memory project. That lineage matters. There was already a foundation to work with: Markdown knowledge, a search index, and ways to connect agents to memory. My project is an evolution of that work around the continuity problems I keep running into.
I also like the instinct behind tools such as jCodeMunch and jDocMunch: retrieve the useful piece instead of making the model swallow the whole filing cabinet. They address code and documentation. SessionMunch is concerned with the work history around them. That is a useful comparison, not a claim that these projects are secretly one integrated system.
I don't need to invent an entire new agent platform to keep a decision from disappearing between Tuesday's Claude session and Wednesday's Codex session.
The bar is embarrassingly practical
In October I was still asking whether the project was actually ready to ship, and whether the repository told the truth about it. That question matters more to me than finding a grander name for memory.
Can a person understand what gets captured? Can they see the difference between a record and a summary? Can they retrieve something useful without learning the internals first? Can we say exactly where support stops?
Those are the questions this series is built around. There are technical articles here because the machinery matters. There are human ones because nobody starts building this sort of thing out of an urgent desire to administer another SQLite database.
Mostly, I'd like the next session to begin a little further forward.
Explore SessionMunch · Read the repository and project lineage
Editorial note: the personal episodes were recovered from my preserved September and October conversations. They are a few documented moments, not a complete six-month chronology.