Forgetting in General-Purpose AI Is Rational, But Not Rational for Every Domain

The day my book’s table of contents shrank from 38 chapters to 25, the AI I was working with still clung to the old 38-chapter outline. The book’s positioning had shifted from an academic text to a popular science book, a new structure of 7 parts and 25 chapters was decided, and an official outline with one-line summaries was established. Yet, the AI couldn’t accurately track which chapters had moved, which ones had merged, or which new sections had been created.

I explicitly asked the AI to store the new outline in its memory, and it did. But the AI’s response at that moment made me pause. It told me that the old 38-chapter outline was not directly stored in its explicit memory. That information was in the automatically generated summary memory from accumulated conversations, and this automatic memory couldn’t be directly overwritten. Over time, it would gradually be updated by new conversation content. The AI explained this quite candidly.

Memory Operates on Two Layers

The first realization was that memory operates on two layers.

One layer is information that the user explicitly “pins” by saying, “Remember this.” This layer doesn’t change unless the user deletes or modifies it directly. The other layer is a summary automatically extracted from accumulated conversations. This layer works differently. Information that is frequently referenced in new conversations remains vivid, but if it isn’t called upon over time, it gradually fades.

This is a rational design. A general-purpose chatbot handles endless conversations with countless users. If all information were permanently pinned, the context would explode in size. Preserving frequently referenced information while gradually weakening less-used data is the most natural solution within resource constraints. Human memory is known to work in a similar way.

But the problem lay elsewhere.

The Asymmetry of Forgetting

Everything except information I explicitly pinned is subject to automatic forgetting. Yet, users don’t precisely remember what they have pinned and what they haven’t.

When I changed the book’s outline, I never consciously decided, “I should explicitly save this.” I simply organized the new outline in conversation, and it naturally accumulated. I had no way of knowing whether that accumulation made it into automatic memory, whether it was stored as a strong enough signal, or whether it would be vivid enough to be recalled in the next session.

The risk of this asymmetry is different in nature from the risk of hallucination. Hallucination is an answer that is factually incorrect. If you notice it, you can correct it. But when the AI confidently answers with outdated information, it’s different. It’s not that the answer is factually wrong—it used to be true, but is no longer true. If the user forgets that they updated the information, there’s not even a clue to notice the error.

For knowledge workers, this is not a trivial issue. An author writing a book over 18 months, a doctor seeing the same patient for 10 years, a lawyer handling a case for 3 years, a founder building a product for 5 years—for them, information updates like “it was true yesterday, but not today” are routine.

Revisiting Design Decisions from the User’s Perspective

Over the past year, I’ve been building memory infrastructure. The engine inside is based on a few design principles, one of which is simple: Once information is stored, it is never deleted, never reordered, and no arbitrary weighting is applied except for similarity signals. This principle was reached after repeatedly seeing what undermines memory reliability in long-context recall benchmarks.

But this incident made me revisit that design decision from the user’s perspective. What I knew in theory was confirmed again through the subtle shock of real work. Working in a system where memory quietly fades over time is like building a house on sand that slowly sinks beneath your feet. If the sinking is slow enough, you might not notice until it collapses. But over the course of writing an 18-month book, it will definitely sink.

Defining the Right Place

To avoid misunderstanding, let me be clear: I’m not saying the forgetting design of general-purpose AI is wrong.

It’s unreasonable to demand that a general-purpose chatbot permanently pin all user information. Within resource constraints, prioritizing the most frequently referenced information is the right decision. In that context, the design is correct.

But that decision isn’t right for every domain. In domains where long-term accumulation is essential, forgetting isn’t just rational resource-saving—it becomes a variable that undermines the work itself. Memory for such domains needs a different model: one that never deletes, maintains all information with equal weight until the user explicitly updates it, and, when updates occur, handles them in a way the user can control rather than absorbing them automatically. That’s the kind of infrastructure we’re building.

This isn’t a criticism of general-purpose AI, but a matter of defining the right place for each system. The two systems solve different problems.

What Remains in the End

As AI takes on more and more tasks, there’s one ability users are likely to lose: the ability to know the state of their own information—what’s stored and what’s being forgotten, which information is outdated and which is the latest, what should be pinned and what can be allowed to fade.

You could call this memory governance. As tools become more sophisticated and more of our thinking is delegated to them, memory governance remains with the human. Perhaps this is one of the last places where meta-cognition stays on the human side in the age of AI.

I plan to revisit this topic in a chapter of the book I’m currently writing. This recent experience gave me another chance to confirm the issue from the user’s perspective.