AI Memory Systems: Short-Term vs Long-Term Memory

Here's the sentence that reorganized how I build AI products: the model remembers nothing. Not "little." Nothing. Every single API call is a complete stranger reading your prompt for the first time in its existence. Send "my name is Zeeshan" and then, in a fresh call, ask "what's my name?" — you'll get a polite shrug from a machine that has never met you.

So when an AI app feels like it remembers — your preferences, last week's conversation, that bug you reported in March — none of that is the model. All of it is engineering: something deciding what to store, what to retrieve, and what to paste into the prompt so the stranger can fake familiarity convincingly. After building this several times, here's my map of AI memory systems — short-term, long-term, and the part everyone skips: forgetting.

Short-term memory: the context window is all there is

The model's only "memory" is whatever sits in the current prompt — the context window. Conversation memory, at its simplest, is you re-sending the chat history with every request:

messages: [
  { role: 'user', content: 'My name is Zeeshan' },
  { role: 'assistant', content: 'Nice to meet you, Zeeshan!' },
  { role: 'user', content: "What's my name?" }   // now it "remembers"
]

It remembers because you reminded it. That's the entire mechanism, and it comes with two taxes that grow every turn. The money tax: you re-pay for the whole history on every message (prompt caching softens this considerably, but output and fresh tokens still bill). And the attention tax: models genuinely get worse at using information buried in the middle of huge contexts — stuffing the window is not the same as being remembered.

The standard fix, which I run in production on my support agent: keep the last ~10 messages verbatim, and fold everything older into a rolling summary that a cheap model updates as the conversation grows. Recent turns stay sharp; the past becomes a paragraph. Users never notice the seam.

There's also a sibling worth naming — working memory: the scratch state an agent keeps during a multi-step task ("plan: 4 steps; step 2 done; API returned X"). In my multi-agent system this lives in a database row, not in the conversation, precisely so it survives model calls without bloating them.

Long-term memory: two very different filing systems

Everything beyond one conversation has to live outside the model, and after several builds I've landed on the same two stores every time — they map loosely onto how psychologists split human memory, which is a coincidence I enjoy.

Semantic memory: facts. Durable, structured statements about the user — plan, stack, preferences, recurring issues — extracted after conversations and stored in a plain Postgres table. The extraction is one cheap model call ("pull out stable, useful facts; ignore pleasantries; return JSON"), and injection is three lines at the top of the next system prompt: "Known about this user: Pro plan, uses webhooks, timezone PKT." Small, cheap, and responsible for most of the perceived intelligence. When users say the agent "knows them," they mean this table.

Episodic memory: history. What actually happened — past conversations, past incidents. You can't paste months of transcripts into a prompt, so the trick is retrieval: store summaries of past episodes with embeddings in pgvector, and when a new message arrives, fetch only the two or three past episodes semantically related to it. "It's happening again" pulls up March's timeout saga specifically — not because anything remembered it, but because "it's happening again" landed near that summary in vector space at exactly the right moment.

The one-line summary of the whole architecture: semantic memory answers "who is this user?", episodic memory answers "have we been here before?" — and both are just rows, retrieved on demand and pasted into the prompt for a stranger to read.

The step everyone skips: forgetting

My first long-term memory implementation had a flaw I didn't see for a month: it only ever added. Then a user downgraded their plan, and the facts table confidently contained both "on the Pro plan" and "on the Free plan." The agent, given contradictory memory, alternated between them like a coin flip with customer-facing consequences.

Memory without curation isn't memory — it's hoarding. Three practices fixed it:

Update, don't append. Fact extraction now upserts by category: a new plan fact replaces the old plan fact. Contradiction resolved at write time, when it's cheap, instead of at answer time, when it's embarrassing.

Weight recency. Episodic retrieval blends similarity with freshness, so last week's incident outranks an equally-similar one from last year. Old context should fade, not compete.

Store facts, not transcripts. Raw conversation logs kept forever are noise for retrieval and a privacy liability you have to answer for. Extract what's durable, summarize what's episodic, and let the verbatim past expire on a schedule. The best memory systems remember less, better.

The blueprint, assembled

Every layer in this article maps to a plain component you can build in an afternoon each:

That table is also, not accidentally, the architecture of every "AI with memory" product you've used. There is no memory module inside the model, no secret. There's a stateless genius with total amnesia, and around it, a well-organized filing clerk — the clerk is the product, and the clerk is yours to build.

The stranger never learns your name. Build the system that whispers it at the right moment, and nobody will ever be able to tell the difference.