Skip to content

Building Agent Memory That Doesn't Lie: A Production Guide to Context Management

For AI platform engineers running LLM agents in production. You get a four-way memory taxonomy, a storage decision table, three working Python modules, and five incidents I hit on the job — so your agent remembers what matters and forgets what doesn't.

1. The day my agent forgot its own constraints

I run a fleet of coding and ops agents. One was supposed to connect over HTTPS, always. After a long multi-session run it started issuing plain HTTP calls. Nothing in the prompt said otherwise — because a summary compression pass had silently dropped the "always use HTTPS" constraint from a 90-turn conversation. The model did what the remaining context allowed. That was incident number five, and it is the one that taught me context management is a memory problem, not a prompt problem.

This article is the lessons from that and four other production incidents, plus the system I built to stop them. It assumes you already run agents and have been bitten by context overflow or hallucinated "memories."

2. The four-way memory taxonomy

I split agent memory into four kinds. They live in different stores, expire differently, and fail differently.

  • Working memory — the live dialogue turns in the context window. Short-lived, per-session, dies with the process.
  • Episodic memory — specific events: "user asked to refactor the auth module at 14:32." Timestamped, ordered, useful for recency.
  • Semantic memory — facts and world knowledge: "the API rate limit is 60 req/min." Long-lived, needs freshness checks.
  • Procedural memory — skills and tool-usage patterns: "how to call the deploy tool." Durable, versioned, rarely expires.

The failure mode I see most is treating all four as one blob. When everything is a "memory," everything is retrievable, and retrieval becomes noise.

3. Storage selection table

Memory type Store Why it fits
Working In-memory deque Sub-millisecond, per-session, dies on crash
Episodic Redis TTL, time-ordered, shared across replicas
Semantic Vector DB Similarity search over large fact sets
Procedural SQLite Durable, structured, easy to version and diff

Rule of thumb: in-memory for what must be fast, Redis for what must be shared, vector for what must be searched, SQLite for what must never be lost.

4. Context window management strategies

Five strategies, ordered from cheapest to most aggressive.

  1. Sliding-window truncation — keep the last N turns. Cheap, but drops the middle of the story.
  2. Summary compression — collapse old turns into a summary. Good, but lossy (see incident #5).
  3. RAG externalization — push facts to a vector store, pull only relevant ones back. Best for semantic facts.
  4. Hierarchical rolling summary — nested summaries: per-5-turn, per-30-turn, full-run. Preserves shape without blowing tokens.
  5. Token budget control — hard cap with a reserve. The guardrail that makes 1-4 safe.

I run all five together. The budget is the arbiter; the others are how I stay under it.

5. Code: a memory manager

import time
from dataclasses import dataclass, field

@dataclass
class MemoryItem:
    kind: str                # working | episodic | semantic | procedural
    content: str
    created: float = field(default_factory=time.time)
    last_access: float = field(default_factory=time.time)
    ttl: float | None = 86400.0
    hits: int = 0
    version: int = 1

class MemoryManager:
    def __init__(self, ttl_default: float = 86400.0, max_items: int = 2000):
        self.store: dict[str, MemoryItem] = {}
        self.ttl_default = ttl_default
        self.max_items = max_items

    def write(self, kind: str, content: str, ttl: float | None = None) -> str:
        key = f"{kind}:{int(time.time() * 1000)}:{len(content)}"
        self.store[key] = MemoryItem(kind, content, ttl=ttl or self.ttl_default)
        self._evict_if_full()
        return key

    def recall(self, kind: str, limit: int = 5,
               recency_window: float | None = None) -> list[str]:
        now = time.time()
        pool = [it for it in self.store.values() if it.kind == kind]
        if recency_window:
            pool = [it for it in pool if now - it.created <= recency_window]
        pool.sort(key=lambda it: it.last_access, reverse=True)
        for it in pool[:limit]:
            it.last_access = now
            it.hits += 1
        return [it.content for it in pool[:limit]]

    def invalidate(self, content_substr: str) -> int:
        # bump version instead of deleting: recall can still see it, but knows it is stale
        n = 0
        for it in self.store.values():
            if content_substr in it.content:
                it.version += 1
                n += 1
        return n

    def _evict_if_full(self) -> None:
        if len(self.store) <= self.max_items:
            return
        now = time.time()
        dead = [k for k, it in self.store.items()
                if it.ttl and now - it.created > it.ttl]
        for k in dead:
            del self.store[k]
        while len(self.store) > self.max_items:
            oldest = min(self.store, key=lambda k: self.store[k].last_access)
            del self.store[oldest]

Two methods matter most. invalidate is how I mark a fact dead without losing the record of it — incident #2 only got fixed once recall could distinguish "old fact" from "known-false fact." _evict_if_full is how I stop incident #1, unbounded growth, before it reaches the context window.

6. Code: context budget controller

from collections import deque

class ContextBudget:
    def __init__(self, max_tokens: int = 128_000, reserve: int = 8_000):
        self.max_tokens = max_tokens
        self.hard_limit = max_tokens - reserve   # reserve for output + tool results
        self.window: deque[int] = deque()
        self.sum = 0

    def push(self, turn_tokens: int) -> int:
        self.window.append(turn_tokens)
        self.sum += turn_tokens
        while self.sum > self.hard_limit and len(self.window) > 1:
            self.sum -= self.window.popleft()    # sliding-window truncation
        return self.sum

    def headroom(self) -> int:
        return self.hard_limit - self.sum

    def should_compact(self) -> bool:
        return self.headroom() < int(0.15 * self.hard_limit)

The reserve is non-negotiable in my setup. If I let the window consume the full model limit, the agent has no room for a tool result, so it truncates its own output to fit — which is how a constraint can silently vanish (incident #5).

7. Code: recall with recency filter

import math, time

def _tokens(text: str) -> dict[str, int]:
    freq: dict[str, int] = {}
    for w in text.lower().split():
        freq[w] = freq.get(w, 0) + 1
    return freq

def _cosine(a: dict, b: dict) -> float:
    common = set(a) & set(b)
    num = sum(a[t] * b[t] for t in common)
    da = math.sqrt(sum(v * v for v in a.values()))
    db = math.sqrt(sum(v * v for v in b.values()))
    return num / (da * db) if da and db else 0.0

def recall_scored(items: list[MemoryItem], query: str, top_k: int = 3,
                  decay: float = 0.01) -> list[str]:
    """Score = 0.7*cosine(query, item) + 0.3*recency; skip stale versions."""
    qv = _tokens(query)
    now = time.time()
    scored = []
    for it in items:
        if it.version > 1:               # stale fact: deprioritize, never delete blindly
            continue
        sim = _cosine(qv, _tokens(it.content))
        rec = math.exp(-decay * (now - it.created) / 3600.0)
        scored.append((0.7 * sim + 0.3 * rec, it))
    scored.sort(key=lambda x: x[0], reverse=True)
    return [it.content for _, it in scored[:top_k]]

In production this _tokens is replaced by a real embedding call, but the blend is the point: pure cosine retrieval returns a fact that matches textually and is three weeks old. The recency term is what keeps a revoked API key from resurfacing.

8. Five incidents, root causes, fixes

# Symptom Root cause Fix
1 Context limit blowout at turn 61 ttl=None on everything, no eviction TTL defaults + _evict_if_full budget guard
2 Agent "remembered" a revoked API key Vector recall returned a stale fact, no version check Bump version on invalidate; skip version > 1
3 Session resume produced a confused agent Raw turns persisted without a session id Namespace all writes by session_id
4 Two agents overwrote each other's plan Last-write-wins on a shared key Per-agent namespaces, or Redis INCR/merge
5 HTTPS constraint vanished after compression Naive summarizer dropped an instruction Protect instructions in a non-compressible slot; constraint ledger

The pattern across all five: memory without provenance is a liability, not an asset. Every fact needs a timestamp, a version, and a namespace.

9. Evaluating memory: three numbers

  • Hit rate = recalls that changed the output / total recalls, tracked per kind. A low semantic hit rate means your embedding is wrong or your facts are stale.
  • Contamination rate = wrong or stale items recalled / total recalled. This is the killer. If it climbs, tighten recency and add provenance checks.
  • Forgetting curve = probability of recall success vs. time since write. Fit an exponential P(t) = e^(-t/τ). Use τ to set your TTL: too high means stale recall, too low means wasted writes.

I log all three per agent per day. Hit rate up and contamination rate down is the only signal I trust.

10. FAQ

Q: When should I externalize to RAG instead of keeping content in context? A: Keep high-frequency working tokens in context; push a fact to the vector store once its recency or frequency falls below threshold, then recall it on demand.

Q: How do I stop stale vector facts from causing hallucinations? A: Version every fact, tag it with write time and confidence, filter by recency, and treat recall as a hint — never as ground truth.

Q: What is the cheapest fix I can ship first? A: A TTL plus a sliding-window budget in-process, about 50 lines. It prevents the worst blowups before you build anything fancier.

Q: How do I know memory is actually helping? A: Track hit rate and contamination rate per kind. If contamination climbs, tighten recency and require provenance; if hit rate is flat, your retrieval is the bottleneck.

Q: Which store fits which memory type? A: In-memory for working, Redis for episodic, vector DB for semantic, SQLite for procedural — per the table in section 3.