Automate the installation #1

Merged
troed merged 27 commits from install-cgroup into main 2026-10-02 13:11:02 +02:00
Owner

Previous installation instructions weren't clear. Have made install.sh into a full blown installation script that will add cgroup delegation (requires sudo) and also implement the correct deny and allow settings as well as the MCP into an opencode configuration file. If you're not using Opencode you'll still need to do those manual steps.

Previous installation instructions weren't clear. Have made install.sh into a full blown installation script that will add cgroup delegation (requires sudo) and also implement the correct deny and allow settings as well as the MCP into an opencode configuration file. If you're not using Opencode you'll still need to do those manual steps.
Two silent failures recorded in the agents' quirk log.

- write_file/edit_file run an unguarded read-modify-write, and the control
  plane threads every connection, so two sessions sharing it (the normal
  steady state) could interleave and lose one of the two writes outright.
  One lock per resolved path now serializes them; distinct paths still run
  concurrently. The lock wait is bounded, so a wedged exec reports cleanly
  instead of parking the request forever.
- egress_env put the hosts of [[egress.tcp]] rules into NO_PROXY, so a
  proxy-aware client (curl, git over https) dialed such a host directly
  into the default-drop ruleset: a host with only a raw tcp rule then
  failed on an opaque connection error instead of getting the gateway's
  explicit verdict. NO_PROXY is loopback-only again; raw-protocol clients
  never read it and still go direct.
Three friction points from the quirk log, all in the base posture.

- ripgrep: the workspace AGENTS.md tells agents to search with `rg`, which
  the image did not ship (opencode's own copy is host-side, off PATH).
- /tmp/opencode: the harness tells agents to prefer it, but the path did
  not exist in-sandbox. A tmpfs, so it is wiped on recreate exactly like
  /tmp - durable scratch still belongs in the workspace.
- $HOME/.cache/opencode (read-only): the skill helper scripts
  (sdd-workspace, task-start, review-package) live there, and plan-driven
  work was hand-reimplemented whenever the mount was missing.

test_config gains a check on the shipped base config itself, which nothing
previously parsed.
A stall ("commands stop coming back, a restart is the only way out") has
three candidate mechanisms and no way to tell them apart after the fact:
the cgroup counters reset when the container is recreated, so the evidence
has to be captured while it is still stuck.

Three probes, chosen because each mechanism leaves a distinct signature:

- CPU starvation: cpu.stat nr_throttled/nr_periods, cpu.pressure, and a
  5s probe-time delta. Lifetime counters cannot see an hour-healthy box
  that stalled a minute ago, so the delta is what separates them.
- OOM kill: memory.events oom_kill plus the host kernel/journal logs.
- Watchdog recreate loop: run.py retries recreate() forever and does not
  log it, so this is only visible host-side as concurrent `podman run/rm`
  or a cycling podman pid set.

Read-only: it never restarts or mutates anything, and every probe degrades
to a note when a tool or a mount is unavailable (the in-sandbox half has no
podman/journalctl). Run it from the host; it reaches into each container
itself via `podman exec --in-sandbox`, bounded by a 25s timeout whose
expiry is itself evidence.
Three bugs, all found by the first real run - and the worst one was that
the script blamed the sandbox for its own failure.

- The self path was built as "$(pwd)/$(basename BASH_SOURCE)", which
  dropped the bin/ component: the host half tried to redirect a nonexistent
  "<cwd>/umwelt-stall-diagnose" into `podman exec bash -s`, so every
  container reported rc=1. Now resolved to an absolute, symlink-followed
  SELF_PATH at startup and used everywhere.
- `bash -s -- <stamp> --in-sandbox < script` passes the mode flag as $2,
  not $1, so the dispatch on "$1" fell through to host_main: the host half
  ran inside the container, tried to mkdir on a read-only path, and its
  failure was reported as "the container is not accepting an exec".
  Dispatch now scans "$@" for the mode; the first non-flag arg is the
  target workspace.
- podman's rc=125 (Exited/Paused) was lumped in with generic failures. It
  is now its own case: a non-Running container is not a CPU stall, it
  points at the recreate loop or an OOM kill. 126/127 are separated as
  probe-setup errors rather than sandbox evidence.

Gateway sidecars (umwelt-egress-gw-*) are excluded from the sweep, and a
host-side mkdir failure now aborts with the reason instead of half-running.
Three reporting defects from the first clean host run, all of which made
the script harder to trust rather than wrong about the sandbox:

- podman refuses to exec into an Exited container with rc=255 ("can only
  create exec sessions on running containers: container state improper"),
  not the 125 I assumed from the shell convention. Match 255 as well, and
  the message too, so classification does not hinge on podman's numbering.
- Non-running containers are now filtered out before probing (via podman
  inspect), and their state is reported as the finding it is: an Exited
  sandbox points at mechanism C, not at CPU starvation. Filtering also
  removes the spurious VERDICT paragraph per dead container.
- REPORT.txt had per-container notes and the A/B/C summary sharing one
  flat list, so four stray "VERDICT:" paragraphs appeared ahead of the
  summary. Split into PER-CONTAINER PROBES and MECHANISM SUMMARY.

Also: target_container_names runs in a command substitution, so its
mechanism-C finding was being set in a subshell and silently discarded
(the exited container never reached the summary). It now hands the finding
back through a file. Its progress lines went to stdout, which is the
return value there, so they leaked into the container list and produced a
bogus "inside-(skipping ...).txt" artifact; they go to stderr now.
The watchdog wrapped everything in a bare `except Exception: continue`, so a
recreate loop left no trace anywhere: a sandbox that stopped answering had no
line, no reason and no timestamp, which is what made the stall investigation
unattributable. It now writes one line per observation to stderr - which the
MCP server redirects to <socket>.log - covering start, recreate (with
duration), failed recreate (with the exception and an attempt count), and a
failure to determine liveness.

Writing that logging exposed a second bug worth more than the logging. The
liveness check ran through container_running(), which catches OSError and
returns False - so "podman cannot be consulted" was indistinguishable from
"the container is dead", and an unreachable podman sent the watchdog into an
endless recreate loop, every 5s, forever. That is mechanism C of the stall
diagnosis, triggered by podman being gone rather than the container dying.
container_is_running() is added for the watchdog: it raises, the failure is
logged as "cannot determine", and no recreate is attempted.
container_running() keeps its bool contract for its three other callers.

log_line() flushes explicitly: stderr redirected to a file is block-buffered,
so a line written just before a wedge would sit in the buffer - the exact
case the logging exists to explain.

Tests cover start/recreate/failed-recreate/cannot-determine and the
distinction itself; each fails against the previous silent version.
The script looked for a log under the umwelt state dir and reported "no log
file found" - it had missed that mcp_server.ensure_sandbox redirects run.py's
stdout and stderr to "<socket>.log", i.e.
$XDG_RUNTIME_DIR/umwelt-<sid>.sock.log. That is the only place the control
plane's lines land, so it is now the primary candidate (narrowed to the
target workspace's session when one was named), with the old paths kept as
fallbacks for a hand-started run.py. The watchdog section now greps for the
lines the watchdog actually writes.
The previous version picked the most recently modified *.sock.log, which on
a host running several sandboxes belongs to an unrelated workspace. A live
run showed it reading umwelt-a9a14f16.sock.log - a session matching no
container on the host - so the tail and the watchdog section came out empty.
An empty watchdog section is exactly what a healthy sandbox looks like, so
reading the wrong file was indistinguishable from good news. That is the
same failure mode as the earlier self-path bug: a confident negative from
the wrong input.

The container name *is* the session id, so the log is now derived from it:
umwelt-<sid> -> $XDG_RUNTIME_DIR/umwelt-<sid>.sock.log. Recency is never
consulted. Each probed sandbox gets its own probe_log call and its own
control-plane-log-<name>.txt artifact, instead of one arbitrary file.

Also: a zero-watchdog-line result now says so, and distinguishes "no
container death happened" from "this log predates watchdog logging" - the
two cases that an empty section used to conflate.

probe_inside_containers returns the names it probed on stdout for the log
lookup, so its console output moved to stderr (the same subshell/return-value
trap that made a container list absorb a status line earlier). Fixed a
`grep -c` exit-1-on-no-match that made the zero case fail its numeric test.
- status: gate on $BASE_IMAGE and $INGRESS_GW_IMAGE, not just the gateway
- install: refuse to run as root (targets uid 0 artifacts otherwise)
- cgroup install: temp-file trap; cgroup_supported honors UMWELT_CGROUP_ROOT
- remove: base-image rmi hint; warn when the cgroup service is not removed
- docs: umbrella header, CONFIGURATION checklist, README v1->v2 migration,
  spec install-time trust assumption
- tests: missing-base-image status, verify-gate abort, root guard
merge_mcp_server upserts/strips/checks mcp.servers.umwelt (JSONC-aware,
comment- and sibling-preserving, reusing the array-merge scanner);
managed_mcp_json emits the canonical entry with the installer's checkout
path. register_server_config wires it in, status_server_plugin reports it.
README/spec restore that the installer owns the MCP block.
merge_mcp_server no longer emits invalid JSON:
- a top-level trailing separator (common .jsonc) is normalized and not
  doubled when inserting the mcp block; an empty {} body is handled too.
- a non-object mcp.servers (array) has its whole value replaced with the
  canonical {umwelt: ...} object instead of splicing a key into the array.
- strip swallows the whitespace-only line an entry leaves behind.

Adds covering tests for all three inputs.
merge_mcp_server's normalize_dangling_comma and the mcp-absent insert
path used an inconsistent depth convention, so a trailing separator was
only normalized in the simplest shape: a top-level dangling comma in the
replace path or one followed by a comment (and the insert path's own
separator choice) could write invalid JSON into a live opencode config.

Replace both with one rule: locate the target object's body span, walk it
with depth 0 = a direct member, and treat a comma at depth 0 as dangling
iff everything from it to the closing brace is whitespace or comments.
normalize_final applies the rule to the top-level object, mcp, and
mcp.servers on every insert/replace/strip write; the insert path's
separator now depends only on whether the object already has a member.

A nested object's own trailing comma (depth >= 1) is never a candidate and
is preserved; no whole-file JSON round-trip is used on write, so comments
survive. Adds regression tests for the five reported shapes.
Whole-branch review fixes (2 Important + minors):

- README.md: state that install.sh owns mcp.servers.umwelt, the
  permissions array, and the plugin entry with replace semantics; the
  old "merge, don't replace" text contradicted the behavior.
- CONFIGURATION.md: post-install checklist now names all three merged
  blocks, not just the permissions array.
- install.sh: add a single fmt_member helper used by every write site so
  the servers-as-array upsert renders exactly like the replace path;
  route it through normalize_final. Two consecutive upserts are now
  byte-identical.
- install.sh: merge_mcp_server check returns 1 when the resolved config
  path is empty (contract: exit 0 only on deep-equal).
- tests: byte-idempotence test for the array branch.
troed merged commit dc732ea35b into main 2026-10-02 13:11:02 +02:00
troed deleted branch install-cgroup 2026-10-02 13:11:06 +02:00
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
troed/umwelt!1
No description provided.