Preventing PII Leakage in AI Systems — An Ops Field Guide
Use Case: You run LLM, RAG, or agent applications that touch business data containing personal information, and you need an auditable line between "using AI" and " leaking data".
1. Why AI blows up the PII attack surface
In a classic web app the data boundary is enumerable: what sits in the database, what an endpoint returns. You can draw the box. AI applications shred that box. PII gets chopped, vectorized, stuffed into prompts, and streamed into agent tool calls, then scatters across vector stores, session caches, third-party model APIs, and log files.
The part that surprised me early on: the model itself becomes a storage device. Once a record enters fine-tuning data or a RAG document, the PII lives in weights or a retrieval index, and deleting it is far harder than a DELETE row. And the output is a brand-new leak point — an answer generated for user A can surface to user B, or the model simply recites a private record it retrieved.
So the exposure is no longer one place. It is four layers plus a tail (logs) and a memory (the model). This guide works that model layer by layer.
2. The four-layer model
I teach this as a pipeline, because that is how you actually defend it:
- Input layer — PII arriving in the user prompt or uploaded document.
- Retrieval layer — PII pulled back from RAG, vector DBs, or knowledge bases.
- Output layer — PII the model emits, including echoes of retrieved private data.
- Log layer — PII written to access logs, debug dumps, and audit trails.
Each layer needs its own control. A control that only covers the input is worthless if the retrieval layer is wide open. The mistake I see most often is teams redacting the prompt and forgetting that the vector store already holds the plaintext.
3. Classify before you do anything else
You cannot redact what you have not categorized. I classify every field into three buckets:
| Class | Rule | Example |
|---|---|---|
| Prohibited | Never enters the model or store | Raw national IDs, plaintext card numbers |
| Redact-then-use | Replaced with a type token before use | Phone, email, address, name |
| Allowed | Safe to process | Product SKU, public config, anonymized metrics |
This step sounds bureaucratic until the first audit. The prohibited list is what lets you say "no" to a feature request that would pipe raw IDs into a prompt. Write it down, version it, and make it reviewable.
4. Input redaction that survives contact with reality
Regex blacklists fail in three ways: they miss formats you never enumerated, they corrupt valid data, and they fail silently. I use an entity recognizer and keep the entity type in the token so downstream logic still works:
from presidio_analyzer import AnalyzerEngine
from presidio_anonymizer import AnonymizerEngine
analyzer = AnalyzerEngine()
anonymizer = AnonymizerEngine()
def redact(text: str) -> str:
results = analyzer.analyze(text=text, language="en")
# replace each entity with a type-preserving token, e.g. <PHONE>
return anonymizer.anonymize(text=text, analyzer_results=results).text
user_prompt = "Call Jane at 415-555-0199 about account A-10293"
print(redact(user_prompt))
# -> "Call <PERSON> at <PHONE> about account <ACCOUNT>"
The trick that saved me in production: preserve the type, not just a generic mask. <PHONE> lets a later step know it can restore a phone but not a name, and it keeps format-validation logic intact. If Presidio is unavailable, I fall back to a small, audited regex set — but only as a safety net, never the primary path.
5. RAG and vector stores: the invisible residue
This is the layer teams underestimate. The right move is to redact before embedding:
def ingest(document: dict, doc_id: str, batch_id: str, ttl_days: int):
clean = redact(document["body"])
vec = embed(clean)
store.upsert(
id=doc_id,
vector=vec,
metadata={"batch_id": batch_id, "ttl": ttl_days, "redacted": True},
)
Two disciplines matter here. First, tag every collection with a batch_id and ttl so you can physically delete by batch when a leak is found — otherwise "delete the PII" means "rebuild the whole index." Second, run a reverse-search plus entity scan monthly: take known PII samples, query the store, and confirm none return in plaintext. I treat any hit as a P0.
6. Output filtering: stop the model from echoing PII
Even with clean inputs, the model can reconstruct or recite PII. I add a post-generation filter:
def safe_output(model_text: str) -> str:
results = analyzer.analyze(text=model_text, language="en")
if results:
# block, or redact, depending on policy
return anonymizer.anonymize(text=model_text, analyzer_results=results).text
return model_text
The harder case is cross-user leakage: user A's context bleeding into user B's answer. That is a retrieval-scoping problem, not a redaction one — scope every query to the caller's tenant and treat "shared global knowledge" as a distinct, pre-vetted corpus.
7. Logs and audit trails: never let PII sleep in your log store
Logs are where PII goes to die slowly. My rule: log only a salted hash or the redacted form, plus metadata that is useful without the value:
import hashlib, secrets
def log_call(caller: str, raw_pii: str, model: str, tokens: int):
salted = hashlib.sha256((secrets.token_hex(8) + raw_pii).encode()).hexdigest()[:16]
audit.info("model_call", caller=caller, pii_hash=salted,
model=model, tokens=tokens)
If a real value must be retained for dispute resolution, it goes to encrypted storage behind approval, with a retention window that physically expires. The log line itself never holds the value. I export "log PII-residue rate" as a metric and alert at >0.
8. Tool calls: the new blind spot
Agents calling APIs are where PII leaks that no prompt filter catches. A tool returns {"customer": "...", "ssn": "..."} and the agenthelpfully forwards it to the next step or into a summary. I enforce two rules:
- Redact at the tool boundary — wrap every tool response through
redact()before it re-enters the context. - Whitelist outbound fields — an agent may call an API, but only explicitly allowed fields may travel onward.
This caught a bug where a support agent was pasting full customer records into a summarization prompt "for context." The whitelist killed it.
9. An incident I actually handled
A payments team fed support transcripts into a RAG bot. Redaction was on the prompt but not on the ingestion path, so the vector store held plaintext phone numbers. Three months later a compliance scan found them via the reverse-search I described in section 5.
What I did: froze ingestion, ran the batch delete using batch_id, redacted the historical corpus, and added the residue scan to the weekly job. The lesson that stuck: redaction is a pipeline property, not a single function call. One redact point is one point of failure.
10. Post-deploy inspection discipline
After any PII-control change I run a short checklist, not a prayer:
- Keep a labeled sample set; run entity-recall regression weekly, fail the build if coverage drops.
- Export redaction coverage and log residue rate into monitoring; page on P0 if residue > 0.
- Quarterly full compliance re-audit of the classification list and prohibited fields.
- Verify the batch-delete path works on a test collection before you need it for real.
I treat the residue scan like a backup restore test — useless the first time you actually need it and find it broken.