Getting started¶

Setup¶

This package uses uv for dependency management. From python_package/, install the dependencies and all bundled memory plugins with:

uv sync --dev --extra all-memory-plugins --python 3.12

The make setup target will also run this command for you. From the project root:

make setup

Environment variables¶

Create a .env file at the project root. The repository includes a template you can copy and fill in:

cp .env_template .env

Running experiments¶

Run the benchmark smoke tests¶

The benchmark smoke tests run one small scenario of every benchmark (except Retention policy) with in-context memory. They check that your setup, keys and datasets work, and give you a first set of results to look at:

cd python_package
make run-smoke-benchmark

The target runs two heams commands:

uv run heams run --config configs/runs/smoke_test_benchmark.yaml --output-dir results/smoke_test_benchmark
uv run heams generate-report --results-dir results/smoke_test_benchmark

heams run executes every experiment in the run config and writes the raw results; heams generate-report turns them into a readable report.md per experiment. Datasets are downloaded on first use. Running the target again resumes the existing run instead of starting over; to start fresh, use make run-overwrite CONFIG=smoke_test_benchmark OUTPUT_DIR=results/smoke_test_benchmark.

How the configs fit together¶

A run config under configs/runs/ lists experiments. Each experiment pairs an agent, a memory system and a benchmark, each described by its own YAML file:

# configs/runs/smoke_test_benchmark.yaml (excerpt)
run_name: smoke_test_benchmark
experiments:
  - name: gpt56_luna_in_context_locomo
    agent_config_file: campaigns/locomo.yaml
    memory_system_config_file: campaigns/in_context.yaml
    benchmark_config_file: smoke_campaign/locomo.yaml

The file names are relative to one directory per component:

The smoke benchmark configs in configs/benchmarks/smoke_campaign/ set limit: 1, which keeps the run small. To run your own experiments, write a run config that points at the component files you want and run it with make run CONFIG=<name> OUTPUT_DIR=<dir>. See CLI for the run config format, Memory backends for memory-system configs, and the benchmark pages for each benchmark's options.

Running campaigns on a Slurm cluster¶

If you have access to a Slurm cluster, you can launch campaigns: sets of runs that execute in parallel as separate Slurm jobs. See Cluster for the campaign tooling and how to set it up.

Exploring results¶

Results browser¶

Open the results in a local, read-only web dashboard:

make browse RESULTS_DIR=results/smoke_test_benchmark

It prints a local URL (http://127.0.0.1:8765/ by default) and opens it in your browser; stop it with Ctrl+C. The dashboard has three levels:

• Criterion table: memory systems as columns, and rows grouped by dimension (Performance, Privacy, Efficiency), benchmark and criterion. Each cell shows the score and the number of samples behind it.
• Experiment: status, agent model, accuracy, a pass-rate breakdown per tag (question type, attack class, ...) and a searchable list of samples.
• Sample: the judge's records and reasoning, the full transcript with every tool call, efficiency metrics and the memory ledger.

RESULTS_DIR can be one run, a campaign, or the whole results/ folder; the browser searches it recursively and caches what it reads in _results_browser_cache/. See Results browser for how it picks runs and computes each cell.

Artefacts on disk¶

Each run writes one directory, with one subdirectory per experiment:

results/smoke_test_benchmark/
├── run_config.<run_id>.yaml          # snapshot of the resolved run
├── gpt56_luna_in_context_locomo/
│   ├── experiment_config.<run_id>.yaml  # resolved agent, memory and benchmark configs
│   ├── runs.jsonl                     # one result per completed sample
│   ├── execution_traces.jsonl         # ordered trace of memory ops, LLM and tool calls
│   ├── run_status.json                # success flag and counts
│   ├── report.md                      # written by heams generate-report
│   └── workspace/                     # memory files used during the run
└── gpt56_luna_in_context_beam_100k/
    └── ...

report.md is the quickest read: metrics, a summary, and the detail of up to 10 samples (change that with --max-examples). runs.jsonl is the machine-readable source for everything else. See Understanding results for every file and field.

To evaluate your own memory system, see Adding a memory system.

Makefile entry points¶

You can do everything with the heams CLI directly (uv run heams ... from python_package/; see the CLI reference). For common tasks we have also defined Makefile targets that wrap those commands. Run make help in either directory to list them.

Runs and results¶

Run these from python_package/:

Development¶

Run these from the project root or python_package/:

Documentation (local)¶

Preview the docs site (works from project root or python_package/):

make docs-serve

Open http://127.0.0.1:8000. The site is built with Zensical using the config at the project root (zensical.yml). Use make docs to build static output into site/, or make docs-strict to fail on warnings (same as CI).

Tests¶

make test runs the full suite, including integration tests. In python_package/ it runs uv run --python "${PYTHON_VERSION:-3.12}" pytest -s tests/; the root target runs that and also the Mem0 and HippoRAG plugin tests. make test-ci skips integration and Ollama tests, as CI does.

From the project root (recommended before sharing changes):

make test

For the full suite, including integration tests:

cd python_package && make test

Run the repository-wide pre-commit checks from either directory:

make check

Run TinyAgent integration tests only (requires ollama serve and at least one local model):

cd python_package
uv run --python "${PYTHON_VERSION:-3.12}" pytest -m "integration and requires_ollama" tests/agents/

The tests use ollama:qwen2.5:7b; to use another model, edit MODEL_IDS in tests/agents/conftest.py.

Next steps¶

• Architecture — config-driven runs, experiments, and factories
• CLI — heams run and declarative YAML configs
• Memory backends — compare backends via experiment matrices
• Benchmarks — per-benchmark behavior and migration status