Skip to content

HEAMS: Holistic Evaluation of Agent Memory Systems

This package contains the HEAMS benchmarking framework.

Overview

The goal of this project is to build an extensible benchmarking library for the holistic evaluation of agent memory systems. The benchmarks evaluate agent memory systems across dimensions of performance, privacy, user sovereignty and security.

What you can do here

  • Install the package and run experiments with heams run
  • Compare memory systems by declaring multiple experiments in a run config
  • Read results and per-experiment reports; add new benchmarks and memory systems

Quick start

From the project root:

  1. Install dependencies:

    make setup

  2. Create your environment file from the template:

    cp .env_template .env

  3. Open .env and set the keys the smoke run needs:

    OPENAI_API_KEY=sk-...
HF_TOKEN=hf_...

    OPENAI_API_KEY is required because the bundled agents and LLM-as-judge evaluators use OpenAI models. The Evo-Memory benchmark in the smoke run needs Hugging Face access to the gated GPQA dataset: either set HF_TOKEN, or leave it empty if you are already logged in with hf auth login. Either way, the account must have accepted the GPQA terms.

  4. Move into python_package/. The remaining steps run from there:

    cd python_package

  5. List the memory systems the framework has discovered, both built-in ones and installed plugins. Each name is a type you can use in a memory-system config:

    make memory-list

  6. Run the benchmark smoke tests (one small scenario per benchmark, with in-context memory). The target also writes a report.md for each experiment under python_package/results/smoke_test_benchmark/:

    make run-smoke-benchmark

  7. Explore the results in the local results browser:

    make browse RESULTS_DIR=results/smoke_test_benchmark

    It opens http://127.0.0.1:8765/ in your browser, where you can drill down from per-benchmark scores to each sample's transcript and judge output. Stop it with Ctrl+C.

See Getting started for more detailed setup instructions, CLI for more information about the CLI commands that the framework supports, and Understanding results for details about the result artefacts that are created by experiments.

Guides

Topic Page
System overview Architecture
Memory systems as component YAML Memory backends
heams run / heams generate-report / heams browse and config fields CLI reference
Reading experiment dirs and reports Understanding results
Reviewing results in a local web GUI Results browser
Slurm campaign tooling Cluster
Papers, datasets, licenses, and BibTeX Citations & credits

Benchmarks

Benchmark Status Description
LoCoMo Runnable via heams run Long-context conversational QA with LLM-as-judge grading
BEAM Runnable via heams run Long-context memory abilities at 100K–10M tokens with nugget scoring
MPBench Runnable via heams run Memory-poisoning RSR via primed payload + LLM judge
LongMemEval Runnable via heams run Timestamped chat-history recall with LLM-as-judge grading
Evo-Memory (single-turn) Runnable via heams run Ordered memory stream over upstream single-turn datasets
Transparency Runnable via heams run Memory audit scenarios (BEAM fixtures)
Contextual Integrity Runnable via heams run Disclosure / leakage scenarios with CI LLM judge
STALE Runnable via heams run Memory-staleness detection across three probing dimensions with LLM-as-judge grading
Erasability & Editability Runnable via heams run Explicit-deletion and cascade-update tasks (MEME corpus) with LLM-as-judge grading
Retention policy Runnable via heams run Whether a privacy layer blocks prohibited facts at ingest and keeps permitted ones recallable

Contributing

To add a new factory-backed benchmark, see Adding a benchmark. Agents (Cursor, Claude Code, Copilot) should also load .agents/skills/implement-benchmark/.

To add a memory system as a plugin, see Adding a memory system.