I manage my business with a small fleet of AI agents now, and the hardest problem I've hit is not getting them to reason well. It's getting them to remember the right things, for the right amount of time, and to forget the rest. I learned this the slow way, by building my own memory system twice.
I Built This Twice, and the First Version Failed for a Boring Reason
The first version was what I'd now call an agent-brain: a folder of notes the agents could read and write, organized by project (my take on the Karpathy LLM Wiki). It worked for about a month. Then it started contradicting itself. A decision I'd reversed in August was still sitting next to the reversal in September, and the agent had no way to know which one was current. Nothing was wrong with the notes. What was missing was a layer above them that tracked state: what's an action in progress, what's a decision that's final, what's a fact that just needs distilling later.
That gap is why I built a second layer, a control plane, specifically to hold the temporal stuff, the actions and events and approvals, separate from the distilled business memory. Two stores, two jobs. The vault holds "what we believe is true"; the control plane holds "what just happened and what's still open." Once I split them, the contradictions mostly stopped, because each store only had to answer one kind of question.
I bring this up not because my setup is special. It's a smaller version of a problem every enterprise IT shop is about to hit at a much larger scale, and the trade press has started writing about it as if it's new. It isn't new. It's the same memory-hierarchy problem we've had since the first data warehouse, wearing an agent label.
What "Memory" Actually Is, Mechanically
It helps to say what agent memory usually is under the hood, because the phrase makes people picture something more exotic than it is. Most of it is just files: typically plain text or markdown, organized into folders, that an agent reads at the start of a task and writes to when something durable happens. No database required, no vector store required. A handful of well-named files, structured so a quick keyword search or a simple directory listing surfaces the right one, gets you most of the value. As the human, you can feed this by using an inbox which gets processed into distilled memories and those distilled files can be edited and require curation.
The step up from there is a light database, often something as plain as SQLite, for the stuff that needs structure: timestamps, status fields, who approved what. Either way, the design goal is the same. Information an agent, or a person, can read in a fraction of a second without invoking a tool, a search index, or a retrieval pipeline. The heavier retrieval systems, the vector stores and semantic search layers, earn their place once you have thousands of documents and genuinely need to find a needle in them. Most individuals, and most teams, never get there. They just need a handful of files that don't contradict each other.
The Enterprise Version Is Already Showing Up in the Research
A recent TechTarget piece on persistent AI memory frames this well: giving agents memory adds a new layer of enterprise data to govern, on top of the documents and systems of record that already exist. The rules for that layer are still being written in production, not in a standards committee.
And the stakes are not theoretical. Gartner is on record saying that applying one uniform governance model to every AI agent, regardless of its autonomy or scope, is itself a path to failure, and projects that 40% of enterprises will have demoted or decommissioned an autonomous agent by 2027 because governance gaps surfaced only after something went wrong in production. Deloitte's 2026 State of AI in the Enterprise survey found only 21% of organizations say they have appropriate systems for agent governance in place. That's not a vendor scare statistic. That's most of the market running agents with memory and no real plan for what the memory is allowed to hold.
The privacy press has been calling out this problem from the consumer side. Researchers have shown that chatbot memory can be poisoned through prompt injection or an agent jumping to a conclusion that it later corrected, where content an agent merely reads gets planted into its permanent memory as if the user had said it. Deleting the conversation that caused it doesn't delete the memory it wrote. If that's true for a consumer assistant, it's a much bigger problem for an agent with a company's CRM and HR data in scope.
Segmentation is Key to the Game
For a business past the size of one person, the design question is always the same one I had to answer for myself, just with more tiers: what gets shared, and with whom. I'd break it into four scopes:
- Personal. An individual's working notes, drafts, and preferences. Useful to that person's agent, no business reading this one by default.
- Team. Conventions, shared shortcuts, "here's how we do this" context that should propagate to every agent on the team the moment one member's agent learns it.
- Department. Policy and process memory, HR rules, finance close procedures, the things a function needs consistently applied regardless of who's asking.
- Company-wide. Facts and decisions that are genuinely organizational: the org chart, the brand voice, the approved vendor list.
This is not a new idea dressed up in agent language. It's the same identity-and-access model we've used for file shares and wikis for twenty years. What's new is that it has to be enforced at write time, not just read time, because an agent's memory is something it writes to on its own, continuously, usually without anyone reviewing the entry before it lands. It should also be revisited after the fact for accuracy. Asana's approach to this is a reasonable real-world example: company-wide shared memory by design, with access controls that stop confidential work from leaking to people who shouldn't see it. That's the right instinct. Shared by default, fenced where it has to be, not the reverse. As humans, we have these very same gaps where our memories aren't really reflective of reality, but the tolerance for these same gaps is considered unacceptable in machines.
Latest Versus Past Is the Problem Nobody's Actually Solved
Here's the part that bit me personally and will bite every enterprise doing this at scale: memory has to know which version of a decision is current. A plain document store doesn't handle this well, because the old version and the new version just both exist, and whichever one the retrieval pulls first wins. I don't think there's a clean industry answer yet. The closest thing I've seen in the emerging agent-memory tooling is scoping memory with explicit identifiers, a user, an agent, a session, an org, so a system can at least tell you where a piece of memory came from and when. That's necessary. It isn't sufficient. Knowing the provenance of a fact doesn't tell you it has been superseded. Somebody still has to mark a decision as reversed, the same way you'd expect a change log entry, not just leave the reversal floating nearby and hope the retrieval ranks it higher. In my architecture, the control plan, acts as my temporal layer and helps bridge the gap with the agent-brain becoming additional reference material.
My practical rule, for what it's worth: treat distilled, standing facts as a file the system itself is not allowed to silently overwrite. If a decision changes, that's a deliberate edit with a record of the change, not an automatic memory update written by an agent mid-conversation. The moment memory updates happen the same way a chat response does, quietly and continuously, you've lost the ability to audit what your systems believe and why.
What Belongs in Memory, and What Should Stay Where It Already Lives
The other mistake I seen in the market is trying to make the memory layer the source of truth for everything. It shouldn't be. Your CRM is still the system of record for customer status. Your ERP is still the system of record for the ledger. Memory's job is to hold the things that don't have a natural home anywhere else: the "why," the preference, the standing instruction, the fact that's true across many interactions but too small and too contextual to belong in a formal record. If a fact already lives cleanly in a source system, memory should hold a pointer to it, not a copy of it. Copies drift. Pointers don't.
This is where I think a lot of enterprise AI memory projects will overbuild. The instinct is to have the agent remember everything it sees, because storage is cheap and retrieval is good at finding the needle. But every fact you let into memory is a fact somebody now has to govern: who can read it, how long you keep it, whether it has to be erasable on request. With the EU AI Act's high-risk obligations already phasing in and more regulation coming, an enterprise memory layer with no retention policy is a liability sitting quietly in production, not a convenience.
What I'd Tell a CIO
Don't start by picking a memory product. Start by drawing the four-tier access map I described above and deciding, scope by scope, what's allowed to be written, by whom, and for how long. Decide, in writing, which facts are canonical and immutable until someone deliberately changes them, versus which are disposable context an agent can refresh on its own. Then and only then pick tooling, because the tooling question is now well served: plenty of vendors handle the scoping and the retrieval. Almost none of them will make the governance decisions for you, and that's the part that actually determines whether this helps your business or quietly creates a compliance problem with your name on it.
I didn't get this right the first time in my small business and I continue to make improvements. I don't expect most enterprises will get it right the first time either. The difference between the ones that recover quickly and the ones that end up in Gartner's 40% is whether they treated memory as infrastructure that needs a design, rather than a feature they turned on.
If you're standing up agent memory in your own environment and want a second opinion on where the lines should sit, that's a conversation I'm happy to have.
Comments
Post a Comment