Operations & recipes

Recipes for running an instance day to day.

Changing config on a running instance

The broker reads lever.yaml once, at its own startup — and a re-run of lever up/apply deliberately keeps a healthy broker process running. So editing the config (new worker, new tool, changed grants) and re-applying silently changes nothing at the broker layer: the symptom is a correct-looking config plus unexplained denials (“unknown recipient”, 403s for a grant you can see in the file).

Use lever reload:

lever reload   # stops the broker, re-runs apply on the current config, spawns a fresh broker

It restarts the broker onto the edited config, re-reads the worker declarations, and re-applies egress — but leaves the running manager container alone (apply’s start-manager sees it already running), so the manager’s conversation is preserved and your TTY is not taken. It’s the broker half of a lever stop + lever up without the VM power cycle or the re-attach. CA, keys, and enrolments all persist. (A full lever stop && lever up still works and additionally power-cycles the VM if you want that.)

reload validates the edited config before it stops the broker, so a config typo fails with the old broker still serving. If a later step fails (backend, image load), the broker can be briefly down until you re-run lever up — the same recoverable window as a stop+up that fails midway. The instance’s single scion project registration (register-project) is idempotent, so reload’s re-run of apply is a no-op there — it’s the broker restart, not a re-registration, that picks up the edited worker list.

reload adds and changes; it does not remove. reload re-reads the config so the fresh broker knows the current worker list, but it does not tear down a worker you deleted from the config. That worker’s container (if one was dispatched) keeps running with its already-minted capability tokens (valid until broker.grant_ttl, default 24h) even though the new broker no longer knows it — new mints and messaging to it are denied, but held tokens still work. To actually cut off a removed worker, stop it and revoke it before or after the reload: `lever-manager agent stop

` (from the manager) and `lever revoke `, or `lever broker bump-epoch` to kill every outstanding token at once.

Adding a worker to a running instance

1. Create the directory under your tree: mkdir -p workspace/workers/newworker (the dir must exist; lever won’t create it for you — the broker Stats it before dispatch).
2. Declare it in lever.yaml: ```yaml workers:
   • name: newworker dir: workers/newworker ``` Add obtain: grants if it needs brokered tools (see capabilities).
3. lever init — scaffolds the worker’s lever-agent skill into the new dir.
4. lever reload — the broker must restart to learn the new worker (see above).
5. Verify: lever doctor (skills check covers the new dir), then dispatch it from the manager (lever-manager agent start newworker --task "…") or message it from the host (lever msg send "…" --to newworker).

Where the logs are

All host-side state lives in .lever-state/ at the instance root:

Agent-side, lever attach [name] shows the full scrollback including every incoming message and tool call. Run lever doctor first whenever anything looks wrong; every check prints a fix hint.

Log rotation

None of the files above rotate on their own — a long-running instance’s .lever-state/ grows without bound. Rotate them with a weekly logrotate drop-in (adjust the path to your instance):

/path/to/instance/.lever-state/*.log /path/to/instance/.lever-state/tool-logs/*.log {
    weekly
    rotate 8
    copytruncate
    compress
    missingok
    notifempty
}

copytruncate avoids restarting the broker/tools on rotation — they keep writing to the same (now-truncated) file descriptor. Give broker.log (the audit trail) a longer retention than the others if you split it into its own stanza.

Troubleshooting quick table

Upgrading lever

1. Pull and rebuild: cd lever_to && make all (host binary), and if the agent-side binaries changed, make lever-image-bins + rebuild your agent image, then lever apply to load it.
2. lever init — refreshes the scaffolded skills (your edited and adopted copies are left alone; --check to preview). If you’ve customized scaffolds and doctor nags about them, accept them once with lever init --adopt.
3. lever reload (or lever stop && lever up to also power-cycle the VM) — restart onto the new binaries/config.
4. lever doctor — every check green means the upgrade landed.