The interesting part isn’t the feature list. It’s that four of the bugs we found don’t exist in any single branch — they only appeared when the branches were combined. That’s the honest argument for integrating deliberately instead of merging fifteen green PRs and hoping.

What shipped

• Durable audit trail. The remediation record (proposal → decision → command → result) now lives in append-only Postgres, not memory. “Auditable” stops being a slide when the record survives a control-plane restart. (#143)
• An in-cluster Kubernetes executor. The pod-restart templates finally have something that runs them: a component that pulls signed commands, re-verifies the Ed25519 signature in-cluster, and executes typed capabilities (delete_pod, restart_deployment, pod_state) under least-privilege, namespaced RBAC. (#146)
• Signing-key rotation with versioned keys and an overlap window — rotate the command-signing key with no fleet re-enrollment. (#150)
• Self-observability. Prometheus /metrics and a Ravn event when the policy circuit breaker trips. The monitoring tool now monitors itself. (#149)
• Air-gapped install, a non-NixOS host quickstart, a Helm chart, a real ravn-eval benchmark harness, and an MCP server — the install story is now first-class on host, Kubernetes, and fully offline.

Through all of it, the founding rule is unchanged: the LLM is never in the detection or action path. It writes the explanation; deterministic code and signed templates decide and act.

The bugs that hid

These are the useful part — each is invisible until two branches meet.

• The guard that rejected its own executor. The in-cluster executor runs commands through the shared handle_command. One branch added a guard there that rejects Kubernetes capabilities — so the executor would have rejected every command it exists to run. Its own unit tests passed, because they test the guard and the executor separately, never composed. We moved the guard to the host actuator’s socket boundary, where it belongs.

• Two designs for one enum. Two branches independently invented the K8s capability type — different variant names, different fields. Both compiled. Reconciling them to one canonical design was a deliberate call, not a textual merge.

• A typed-path check that caught a real template typo. New startup validation of template condition paths immediately rejected a shipped K8s template that used reason instead of payload.reason — a template that, before the check, would have silently never matched.

• A SIGKILL that produced no proposal. This one only surfaced in the NixOS VM end-to-end test. The test kills a unit with SIGKILL; systemd reports the result as signal. But the generic restart template had been narrowed to match only result = "exit-code". So the unit failed, and Ravn proposed nothing — the worst kind of failure, the silent one. The fix was to match any failed unit regardless of result, and to lock it in with a test that fails the moment the catch-all gets narrowed again.

Every one of those passed cargo test on its own branch. The VM test — boot a real NixOS machine, kill a unit, assert a signed heal record appears — is the only thing that catches the last one. Slow tests earn their keep.

A smaller, sharper one: “is it broken, or just deploying?”

A question we kept getting: how does Ravn avoid “healing” something that’s just being restarted or redeployed? Most of the answer is state semantics — systemd’s failed state and Kubernetes’ failure reasons don’t fire for clean restarts or rollouts. But a unit can briefly enter failed before systemd’s own Restart= recovers it. So the failed-unit tap now has a grace period: a unit must stay failed for a configurable window (default 15s) before it emits anything. Brief blip, recovered in time → no proposal, no noise. (#164)

What’s next

The code is on main and green; the cross-target end-to-end proofs (k3d heal, air-gapped VM, live-Postgres restart) are running their way through CI now, and that’s the bar for calling the epic done — not “it compiles.” There’s also a copy-paste remediation template library now, so adding a heal for your service is a TOML file, not a code change.

As always, built in the open. The bugs above are in the git history, not hidden in a changelog — that’s the whole point.

scripts/demo-up.sh

That brings up a systemd “NixOS host” (running the agent + the privileged actuator + a deliberately flaky unit), a k3d cluster (the Ravn controller plus a node-agent and a spread of healthy and failing pods), a self-contained Ollama for the explanations, and the portal — all on one shared network. Then scripts/demo-remediate.sh --auto fails a unit and heals it end to end while you watch.

It works now. Getting there surfaced a string of small, honest problems that are worth writing down — they’re the useful part.

Seeing the fleet

The topology view now groups the fleet by any label you like and draws an icon per kind of thing, so a Kubernetes cluster and a bare host read as two different animals at a glance:

That’s the ☸ k3d-cluster group (the controller agent, currently red because its pods are crashlooping) next to the ❄ nixos group (the host, green and healthy after its last heal). The portal auto-selects the most useful grouping dimension on load, so you land on this instead of one undifferentiated blob.

The last mile: plain-language translations

Deterministic tooling decides whether something is wrong — failed-unit state, CrashLoopBackOff, OOMKilled, ImagePullBackOff. The model’s only job is the last mile: turning that flagged event into a sentence you can act on. Two paths feed it, both pointed at the same local Ollama:

• Host events (failed units, journald) are explained by the agent, inline, before the event is even published.
• Kubernetes events are explained by the control plane, asynchronously, so ingestion never waits on the model.

Click any event and you get the translation plus one concrete check to run:

“The flaky.service failed due to an infinite loop, as it was terminated by a signal but didn’t exit, leading to resource contention…” — with a suggested journalctl -u flaky.service, generated locally by Qwen3 1.7B.

If the model is slow or down, you still get the alarm — you just get a worse explanation. Safe failure, as always.

Supervised self-healing

When a detected fault matches a curated template, the control plane proposes a fix. You approve it; the control plane signs an Ed25519 command; the agent pulls it, the privileged actuator runs it, and the result is recorded at-most-once. Here’s the Remediations page after a few cycles — nothing pending, a clean history of failed-unit-restart actions that each went approved → succeeded (active):

The whole loop — kill a unit, watch the proposal appear, approve, watch it come back — runs in a few seconds.

The problems we hit (the useful part)

Almost none of these were in the “product.” They were in making a distributed system pretend to be one laptop.

• The overlay footgun. A bare docker compose up silently dropped the demo overlay, which disabled remediation templates — so the agent detected the fault, proposed nothing, and everything looked fine. Fix: demo-up.sh always passes every overlay, and the README warns in bold.
• Ollama on 127.0.0.1. No container — control plane, host agent, or k3d — could reach a host Ollama bound to loopback. That’s why explanations never appeared at first. Fix: a self-contained Ollama sidecar on the shared network; nothing on your host required.
• CPU contention starved the explanations. On CPU, one inference stream couldn’t keep up with the deliberately chatty demo, and the most useful event — the failed unit — kept timing out. Putting Ollama on the AMD GPU via ROCm (a 7900 XT, ollama ps → 100% GPU) took it to instant and 100% coverage. The demo auto-detects ROCm / NVIDIA / CPU; the README documents all three plus other GPUs and reusing an external Ollama.
• A kernel-trace flood. Because the host container and the k3d cluster share the laptop’s kernel, the cluster’s OOM stack-traces landed in the host’s journald and buried the real service events. New RAVN_JOURNALD_SKIP_KERNEL option drops raw kernel ring-buffer noise. (On real, separate hosts this never arises.)
• Duplicate topology nodes. Every agent restart minted a fresh ID, leaving offline ghosts in the topology. Fix: a stable RAVN_AGENT_ID (the host agent’s counterpart to the controller’s fixed ID), so redeploys reuse the same node and keep their labels.
• nginx OOMing on a big box. The “healthy” baseline pods crashlooped because nginx spins up one worker per core — dozens on a Threadripper — and blew past a small memory limit. Capped the workers; baseline is green again.

Try it

scripts/demo-up.sh                 # control plane + host + k3d + Ollama (auto GPU)
scripts/demo-remediate.sh --auto   # fail a unit, approve the fix, watch it heal

Then open the portal at http://localhost:8088/topology. Full instructions, including GPU setup for AMD/NVIDIA/CPU, are in demo/README.md.

As always: the model never decides whether something is wrong. It just tells you, in plain words, what already got flagged — and now it does it fast, on a GPU, while the fleet heals itself in front of you.

The short answer is yes, with one important twist. The model should not be the thing that decides whether something is wrong. Deterministic tooling already does that job well — journald, failed-unit state, config diffs, auth events. What’s missing is the last mile: turning a flagged event into a clear sentence a human can act on. That’s the job we’re handing to a small model (think Qwen3 1.7B), and it’s a job a CPU handles comfortably. If the model is slow or wrong, you still get the alarm — you just get a worse explanation. Safe failure.

From there it grew. One agent became a fleet. A fleet needs somewhere to report to, so there’s a control plane. And a fleet you can’t see isn’t much use, so there’s a portal: inventory, a live message feed, and a topology view where you group your agents into whatever categories make sense to you — environment, role, business unit, location.

We’re building this in the open, MIT-licensed, because it’s the kind of tool that gets better with other people’s weird hardware, odd log formats, and hard-won operational scars. The roadmap lays out what, how and when, from a walking skeleton (M0) through to alert routing and an eval harness (M5).

This blog is the devlog. Expect progress, design decisions, and — honestly — the struggles too. The dead ends are usually the useful part. Comments are open on every post (they’re backed by GitHub Discussions), so if you’ve solved something we’re stuck on, or just want to argue about transport protocols, jump in.

If you want to get involved, the issues are organised by epic with good first issue labels, and Discussions is open for ideas.

More soon. 🐦‍⬛