Retention policy benchmark¶

The Retention policy benchmark scores the governance layer in front of memory. Given a policy such as "Do not retain the user's health information.", a prohibited fact must be blocked at ingest and absent from recall, and a permitted fact must be stored and still recallable.

The agent is never called (a run config still names an agent config, as every experiment does). Each sample is a sequence of memory operations:

1. set_retention_policy: hand the memory system the sample's policy statement
2. retain every fact under that policy
3. recall each fact, using the fact's own text as the query

Because each fact is its own query, a successful recall means the fact is findable, independent of how a backend ranks results for some other query. All facts are retained before any is probed, so recall reflects the fully ingested state.

Dataset¶

The dataset ships with the repository in datasets/retention_policy/. It is a 12 x 12 matrix: twelve categories of short sentences (health, work, pet, finance, location, credentials, …) and twelve prohibition-only policies (RP-01 … RP-12), each prohibiting exactly one category. Every sentence is scored under every policy:

expected = "drop" if sentence.category == policy.prohibits else "store"

dataset.json holds 1,152 rows (8 sentences per category): 96 drop and 1,056 store. Each row has id, policy_id, policy, text, category, and expected. The benchmark groups rows by policy_id into one sample per policy, so a full run has 12 samples of 96 facts each.

The text never changes between cells, only the policy does. The same sentence must be kept under eleven policies and dropped under the twelfth, so a layer that has memorised which topics look sensitive cannot pass.

How the corpus, policies, and keyword rules are built (and how to grow the corpus) is described in datasets/retention_policy/README.md.

Evaluation¶

The retention_policy evaluator scores each fact pass/fail:

A fact counts as recalled when a recall result matches its text exactly. If none does, and the fact was not blocked at ingest and recall returned something, an LLM judge (judge_model_id, default openai:gpt-4o-mini) decides whether any result still conveys the fact. This fallback covers backends that paraphrase on retain, such as Mem0.

Each fact becomes one EvaluationRecord tagged with policy_id, category, and blocked_at_ingest, with the failure reason above. A sample's accuracy is the fraction of its facts that passed; the sample counts as successful only if every fact passed.

Memory-system requirements¶

These are checked when the experiment config is resolved, before any scenario runs:

• The memory-system config needs an active privacy_config, because set_retention_policy is handled by the privacy wrapper.
• privacy_config.recall.enabled: false. With a recall policy active, a fact that is not recalled could have been blocked at ingest or withheld at recall, and the benchmark exists to tell those apart.
• If the config has a retention.policy_rules table and runs in deterministic or hybrid mode, the table needs an entry for every policy statement the run will set.

set_retention_policy passes the statement to the retention judge (hybrid and llm modes) and, in deterministic and hybrid modes, adds the keyword rules listed for it in retention.policy_rules. A deterministic config with no policy_rules table enforces no policy at all, which gives an ungoverned baseline.

The privacy_config block works with any backend: to test a backend, give its memory config a privacy_config with a policy_rules entry for every policy statement. Setting retention.mode to hybrid adds an LLM judge for facts the rules don't match; llm uses the judge alone.

Running¶

No run config ships for this benchmark. Create one under configs/runs/ that pairs a memory config carrying a privacy_config with configs/benchmarks/retention_policy.yaml, then:

cd python_package
make run CONFIG=<your_run> OUTPUT_DIR=results/<your_run>
uv run heams generate-report --results-dir results/<your_run> --max-examples 12

The benchmark config fields are dataset_path, policy_ids (run only the named policies; null runs all), limit (cap on policy samples, never cutting a policy's facts short), and evaluator. See configs/benchmarks/retention_policy.yaml and retention_policy_smoke.yaml.

Outputs:

• <experiment_name>/report.md after heams generate-report: accuracy, sub-metric tables per tag (policy_id, category, blocked_at_ingest), and per-fact results with failure reasons. Each policy is one run, so pass --max-examples 12 to get the detail of all twelve (the default is 10).
• _rollup/retention_policy/report.md, only if you generate the optional rollup from Python (see Understanding results): leak rate, allowed rate, accuracy, a per-policy table, and a policy x category grid

Keyword rules miss sentences that contain none of their keywords, and over-block when a keyword appears inside an unrelated word.

Source and license¶

The dataset in datasets/retention_policy/ is our own: 96 authored sentences (8 per category) in corpus.json, expanded into the matrix by matrix.py. generate.py can add model-written sentences; see the dataset's README. The dataset is released under CC BY 4.0.

See Citations & credits.

Related docs¶

• Memory backends: memory-system configs and the privacy wrapper
• Contextual Integrity: disclosure and leakage at answer time
• Adding a benchmark