Lever

The jail

Part of the security model. Sections keep their original § numbers.

2. The jail

The entire runtime, Scion server, broker, the container runtime, and every agent, runs inside one isolated machine (an OrbStack isolated Linux guest that shares no host files and has its own network namespace by default). The lever operator binary runs on the host and drives in. The jail provides two properties:

2.1 Filesystem: the secrets are not in the environment

An isolated machine shares none of the host’s files by default. Lever opts only the project tree in (a single bind mount). Therefore:

  • The broker’s “bind-mount any host path” power is bounded to the project tree, the only directory the jail can see. ~/.ssh, cloud credentials, the runtime’s own host-side state, and the rest of the disk are not in the daemon’s filesystem at all, so a malicious mount of them fails or comes up empty.
  • This is removing the secret from the environment, not fencing an attacker away from a secret that is present. No mount-allowlist patch to the runtime is required; the path simply does not exist to mount.

The tree is mounted in the jail at /lever (writable). Each agent container then gets only part of it: the manager the whole tree, a worker its own dir (§4.1). A worker also gets its enrolment-ticket directory read-only at /run/lever, from the guest’s run user tmpfs, not from the tree. The manager also gets the manager.read_only mounts (§5.1.1) and, under nested_virt, /dev/kvm (§2.4).

2.2 Network: the LAN is unreachable; egress is allowlisted

Inside the jail, containers that use host networking share the jail’s network namespace, not the host’s. Validated behaviour and its limits:

  • The LAN is unreachable. From inside the jail, other devices on the local network (router, NAS, other hosts) cannot be reached; only the host itself is, via a runtime alias. This rests on OrbStack’s default routing for an isolated machine (it is not the --isolate-network mode, which would also cut the host alias we need for tools), backed by explicit firewall drops so it does not depend on OrbStack routing alone: the allowlist drops the private/special-use ranges unconditionally, 10/8, 172.16/12, 192.168/16, 169.254/16 (link-local/metadata), 100.64/10 (CGNAT/Tailscale), and IPv6 fe80::/10 + fc00::/7 (internal/egress/allowlist.go). One address in them is refused out loud instead of dropped: the cloud metadata address 169.254.169.254/32 is REJECTed (icmp-host-prohibited) just before the 169.254/16 DROP, so the scion hub’s GCP project lookup at each start fails at once instead of timing out for about 13 s. Its ICMP answer goes from and to the guest’s own address over lo, so one narrow ACCEPT (-o lo, ICMP host-prohibited, conntrack RELATED) lets it through the private range DROPs. It is the only REJECT in the chain, and lever acceptance checks that the address answers “no route to host” while a neighbouring link-local address is still dropped. LAN-unreachability must still be re-checked on OrbStack upgrades.
  • Host loopback services are reachable via the alias (host.orb.internal), which forwards to the host’s 127.0.0.1 over both IPv4 and IPv6. This is how the manager reaches local tool servers (e.g. the broker, an MCP server) without exposing them on the LAN. The alias exposes all host loopback, so it must be clamped.

Network egress: two postures. All lever rules live in a dedicated LEVER_EGRESS iptables/ ip6tables chain that OUTPUT jumps to (never direct OUTPUT rules), so re-apply replaces only that chain (idempotent, and it never touches non-lever rules). The jump must be OUTPUT’s first rule in both families: a rule ahead of it (an ACCEPT, a RETURN, a goto) would decide packets before the lever chain saw them. Whenever a bring-up does not verify the egress state unchanged (below), it makes the jump first and unique before anything else, deleting any other copy of it and inserting it at position 1 in one iptables-restore commit per family, so OUTPUT is never without it. Other OUTPUT rules are left where they are: behind the jump they cannot weaken it. The replacement is atomic per address family: one iptables-restore --noflush commit per family empties the chain and loads the whole new ruleset in a single transaction, so within a family the old rules stay in force until the new ones are, and a failed commit leaves that family’s old chain in place. The two families are committed one after the other, IPv6 first and IPv4 last: if the IPv6 commit fails, nothing changes; if the IPv4 commit fails (or the apply is interrupted between them), IPv6 has the new rules and IPv4 the old, and the error says so. The re-apply skip for a live closed instance (below) requires both families to carry the closed catch-all DROP, so a half-applied closed posture is always rebuilt, never skipped.

Every bring-up verifies the chains; none assumes them. On an apply or up of a jail that is already provisioned (a root-owned provisioning stamp in the guest names the exact provisioning scripts and the current boot), lever reads both families’ LEVER_EGRESS and OUTPUT listings in one guest call and compares them, rule for rule and in order, with the ruleset it would commit (internal/egress/listing.go, guest.Snapshot.VerifyEgress). It leaves them alone only when they match exactly and the jump is OUTPUT’s first rule, and its only copy, in both families; a chain that is missing, has a rule added, removed or changed, names another alias or other ports, or is half-applied is rebuilt by the atomic commit above. The stamp only lets lever skip re-installing packages and runtimes; it never stands in for the firewall. In the open posture the alias is resolved again for the comparison; in the closed posture it is read from the live chains, as below. The host alias is resolved under the old chain; when that lookup fails and the live IPv4 chain is closed (the guest resolves over IPv4, and that DROP blocks DNS by design), the aliases are read back from the live rules, each from its own family, with a warning naming them. Under an open or absent chain a failed lookup fails the apply. The rules are not persisted in the guest: after a guest reboot there is no chain until the next lever apply/up, which applies egress before it starts any agent.

What that verification can and cannot vouch for:

  • It reads the chains as iptables -S prints them. On an nf_tables guest, a rule written into the lever chain with nft itself, in a form iptables cannot print back, may not show in that listing, and is then not noticed. Before the verification, every apply re-declared the chain in its iptables-restore commit, which emptied it and so removed such a rule as a side effect; a verified bring-up no longer rewrites a matching chain. Writing such a rule takes root in the guest.

  • It is only as trustworthy as root in the guest. On Lima, lever’s root commands (the verification’s read included) run as limactl shell <vm> sudo ... in the run user’s session, and the run user has passwordless sudo: a process running as the run user in the VM is root there, and can make any root command lever runs, the verification and the full rebuild alike, report or do what it likes. This was so before the verification existed. On Lima the boundary against a compromised agent is the rootless container it runs in (and the VM around it), not the guest’s run user. On OrbStack root commands run as orb -u root, without the run user. The posture is chosen by the explicit, jail-wide egress: knob, independent of llm_auth:

  • egress: open (default): OUTPUT default-ACCEPT; LEVER_EGRESS ACCEPTs the allowlisted host host:ports on the alias, DROPs the rest of the host alias and the private ranges above. Public internet stays open (for the model API and package installs). See the exfiltration caveat in §8. On lima the open posture also ACCEPTs the guest resolver’s DNAT targets on the alias (the LIMADNS chain rewrites 192.168.5.3:53 to host.lima.internal:<port> before the filter chain sees it), read from the live chain at apply time — the only way DNS reaches the host agent’s resolver, and scoped to those exact proto:port pairs. Never emitted in the closed posture. See backends.
  • egress: closed: additionally a catch-all DROP for both families at the end of the chain, with loopback (-o lo) ACCEPTed first so the in-machine scion hub (127.0.0.1:8080) and host-alias tools keep working. The jail can then reach only the already-ACCEPTed broker port; arbitrary public-internet egress is closed. closed is valid only for a uniformly api-key instance (a subscription agent needs direct internet to reach Anthropic), enforced at config load.
  • Reachability under closed egress. With the catch-all DROP, DNS/53 is dropped too, so an agent cannot resolve host.orb.internal. Agents therefore dial the broker by its resolved alias IP (already allowlisted), and the broker mints its server cert with that IP as a SAN so TLS still validates (internal/cap/ca/rotate.go NewServerCertSource, which mints via IssueServerCertSANs; internal/brokerctl/serve.go passes the IP from $LEVER_HOST_ALIAS_IP, which internal/cli/host/apply.go sets on the broker child). Re-applying a live closed instance detects the active catch-all DROP in both the IPv4 and the IPv6 chain and leaves them untouched when they are exactly the closed ruleset for the aliases they name and the configured ports; otherwise (a port added to manager.allow_ports, a rule changed by hand) it rebuilds them atomically, taking the alias from the live chains when the closed chain blocks the lookup.

Enforcement lives in the jail’s network namespace, for both postures. A non-privileged agent container is in a separate namespace from the rules and cannot flush them; reaching the rules would require a container/namespace escape (which reduces to the kernel/runtime caveats in §8).

2.3 Rootless podman (required)

Scion runs the agents in rootless podman (it auto-prefers podman over Docker, and lever installs no Docker daemon in the guest). Inside an isolated machine, rootful containers do not work and a rootless runtime is required. An isolated machine applies a seccomp filter that blocks the bpf() syscall, which a rootful OCI runtime (runc/crun) needs to program the cgroup-v2 device controller; containers fail with bpf_prog_query(BPF_CGROUP_DEVICE) failed: operation not permitted. This is the isolation hardening itself, not a missing capability (the machine has full capabilities; a non-isolated machine runs rootful nested containers fine). A rootless runtime does not manage device cgroups, so it never calls bpf() and containers run normally. Rootless is also a security bonus: an extra user-namespace layer around every agent. On the OrbStack VM’s modern kernel the rootless runtime uses native overlayfs (not the slow userspace fuse-overlayfs), so the performance cost is small.

2.4 Substrates: the jail is a contract, not a product

Everything above describes OrbStack, but the jail is a contract. A containment backend must provide a hypervisor boundary between the agent workload and the host kernel (guarantee 0), no host filesystem beyond the project tree (§2.1), a network namespace Lever controls with egress enforced in it (§2.2), and a host-reachable broker endpoint. The full contract, the per-backend guarantee matrix, the lima template mechanism are on the containment backends page. Config validation rejects any backend name that is not implemented rather than falling back to OrbStack, so a containment posture is never silently substituted.

Lima operational notes, from the T13 security review:

  • nested_virt: true widens the host kernel’s exposure. The host’s nested SVM/VMX emulation is reachable from the guest (it has had CVEs, e.g. CVE-2021-29657). It is off by default; use it for dev instances only, and it needs an x86_64 Linux host (the qemu driver; config load refuses it on macOS). Only the manager container gets /dev/kvm (a bind-mount volume on its scion record), so only the manager can reach the guest kernel’s KVM ioctl surface. The volume is create-time only: a manager gains or loses the device only at a fresh create (lever up --fresh, which discards its conversation); lever stop + lever up keeps the old devices. A worker has no device node and cannot make one in its user namespace. The guest kernel is the boundary between agents, so a KVM bug would let the manager break it; the VM boundary to the host still holds. Builds before this change used a podman drop-in that gave every container the device, workers included; apply now removes it.
  • Lima’s in-guest kernel attack surface is intentionally widened for rootless runtimes. Provisioning re-enables the unprivileged user-namespace knob (kernel.apparmor_restrict_unprivileged_userns=0) that Ubuntu ≥ 23.10 disables by default, a prerequisite for rootless podman and pasta. This widens attack surface inside the guest kernel only, in exchange for the rootless containers the whole containment model depends on; the boundary that actually matters — the hypervisor (guarantee 0) — is untouched. An escalation via this surface reaches VM root, not the host, consistent with the §8 “containing runtime authority inside the jail” stance (in-jail privilege escalation is accepted; the jail’s own bound is not).
  • The jail VM survives host reboots, and also lever stop. It is destroyed only by lever destroy (limactl delete --force; lever down is a deprecated alias); there is no reboot-triggered teardown. lever stop (limactl stop) merely powers the VM off — its disk is preserved, and the next lever up resumes it. “Throwaway guest” therefore means per-lever destroy, not per-boot and not per-lever stop — a long-lived instance keeps the same guest state (and any in-guest compromise) across host restarts and stop/resume cycles until explicitly destroyed.
  • The egress allowlist (§2.2) depends on the VM/rootless boundary above it. Every no-reopen / allowlist property in this document assumes the agent lacks CAP_NET_ADMIN in the VM’s init network namespace — enforcement lives in that namespace, not the agent’s container namespace (§2.2). A container→VM-root escape would let the agent rewrite LEVER_EGRESS directly, bypassing the allowlist rather than merely being contained by it; that escape reduces to the kernel/runtime caveats in §8.
  • A global lima config can widen the containment surface beyond what the lever template requests. ~/.lima/_config/{default,override}.yaml, if present on the host, is merged into every lima instance’s realized config, including this one — an operator’s own global lima settings could add a mount or port-forward the lever-rendered template (template.go) never asked for. This is a host-operator supply-chain concern (an attacker would need to already control the operator’s ~/.lima/_config, not just the guest), not a guest-exploitable path. The realized-config drift check (Lima.verifyRealizedConfig, internal/backend/lima/lima.go) is the backstop: lever up fails closed if the VM’s live merged config (mounts/port-forwards/containerd) doesn’t match the template’s intent, whether adopting a pre-existing VM or verifying a freshly created one.
  • Open-posture IPv6 caveat. The alias-scoped ACCEPT/DROP rules (§2.2) are only emitted for a family whose alias actually resolved; host.lima.internal is typically IPv4-only today, so under the OPEN posture no v6-specific alias DROP is emitted (there is no resolved v6 alias to drop traffic to). Safe today — the CLOSED posture’s catch-all DROP covers v6 regardless of alias resolution — but if a future lima version or host configuration ever resolves host.lima.internal to a global-scope IPv6 address, the OPEN posture’s protection depends on that address actually being picked up as the resolved aliasV6: the unconditional private-range drops (fe80::/10, fc00::/7) don’t cover a global-scope address. make test-lima-e2e could assert getent ahosts host.lima.internal returns no global-scope v6 today, to catch this drifting silently in the future.

The orbstack trade is a single shared kernel across the manager and all workers (§8); lima carries the same trade one level up (its own kernel is separate from the host, but still one kernel shared within the jail). That trade is a property of the backend, not of Lever; run lever backends for the live matrix.