Architecture¶

The harness is organized so campaigns are declared in YAML and each experiment binds one agent, one memory system, and one benchmark before execution.

flowchart TD
    heams["heams run --config --output-dir"] --> resolve["prepare_run"]
    resolve --> runOrch["execute_run"]
    runOrch --> expOrch["execute_experiment"]
    expOrch --> factory["create_benchmark"]
    factory --> samples["BenchmarkSample iterable"]
    samples --> bind["bind agent + memory"]
    bind --> orch["Orchestrator"]
    orch --> agent["TinyAgent"]
    orch --> memory["MemorySystem"]
    orch --> evaluators["Evaluators"]
    evaluators --> artefacts["runs.jsonl"]

Components¶

heams CLI (heams.cli.cli)¶

Entry point for campaigns. Responsibilities:

• Parse the small subcommand surface (run, memory list / memory test, generate-report, browse)
• Load a run config from configs/runs/
• Hand a typed ResolvedRunConfig to the run orchestrator

No CLI overrides for model, memory, limits, or benchmark selection.

Run and experiment orchestration (heams.orchestration)¶

Each experiment is one agent/memory/benchmark tuple. Comparing memory backends means listing multiple experiments in the run YAML, not sweeping inside one experiment.

Benchmark factories (heams.benchmarks.factory)¶

New-path benchmarks are typed configs with a type discriminator:

BenchmarkConfig (discriminated by `type`: locomo, beam, mpbench, long_mem_eval,
                 evomemory_single_turn, transparency, ci, stale, ee, retention_policy)
    → BaseBenchmark → BenchmarkSample

New-path workloads inherit BaseBenchmark: their validated configuration defines workload policy, they eagerly materialize ordered BenchmarkSample objects, orchestration binds resolved agent and memory components, and runtime executes the resulting Scenario objects. Each sample maps mechanically to a scenario via BenchmarkSample.to_scenario().

See Adding a benchmark. Agents implementing a new suite should load .agents/skills/implement-benchmark/ (also linked under .cursor/skills/ and .claude/skills/).

Config ownership¶

Benchmarks must not silently overwrite agent or memory policy at runtime. If a setting is required for compatibility, validate it; keep the serialized experiment config identical to what executes.

Memory strategies¶

Agent memory access is selected by memory_strategy:

Orchestrator execution¶

Every run follows one synchronous setup → execute → audit → teardown path. setup_scenario_runtime builds the optional environment, memory system, tool-bound TinyAgent, and evaluator. execute_scenario then interprets every Scenario.execution_sequence serially in declaration order.

UserStimulus calls stateful agent.run, while EnvironmentManipulation dispatches to the active environment. MemoryManipulation calls the memory system directly with retain, recall, or set_retention_policy (Retention policy uses it), and AgentManipulation supports reset_state (Evo-Memory uses it); other commands raise ValueError.

flowchart TD
    subgraph setup["setup_scenario_runtime"]
        S1["Scenario + ScenarioPriming"]
        S1 --> E1["BaseEnvironment.reset()"]
        S1 --> M1["BaseMemorySystem prime"]
        M1 --> T1["build_tools(env + memory retain/recall/…)"]
        T1 --> A1["TinyAgent.run()"]
        S1 --> EV1["BaseEvaluator"]
    end

    subgraph loop["execute_scenario (ordered, synchronous)"]
        direction TB
        L1["UserStimulus"] -->|"agent.run(prompt, run_context)"| A1
        A1 -->|"tool calls"| E1
        A1 -->|"tool calls"| M1
        A1 --> R1["AgentResponse → transcript"]
        L2["EnvironmentManipulation"] -->|"dispatch_command()"| E1
        L3["MemoryManipulation"] -->|"retain / recall / set_retention_policy"| M1
        L4["AgentManipulation"] -->|"reset_state()"| A1
    end

    setup --> loop
    loop --> AUD["EvaluatorConfig.to_ground_truth()"]
    AUD --> AUD2["BaseEvaluator.audit(responses, ground_truth)"]
    AUD2 --> RR["RunResult"]

Reporting¶

After a config-driven run:

• Per-experiment runs.jsonl and run_status.json
• Resolved config snapshots at run and experiment scope
• heams generate-report then writes a report.md per experiment; per-benchmark rollups are optional and generated from Python

How to read these: Understanding results.

Repository layout¶