My Safety System Wrote Its First Emergency Plan. It Blamed the Wrong Hazard.

At the end of Part 2, I left you with a loaded sentence: there are tens of thousands of alarm events sitting in a Redis Stream, and nothing is reading them.

This is the part where something starts reading them. And then something starts thinking about them: an async Python worker cluster that folds raw violations into incidents, assembles hazard context from the plant's own topology, and hands it to a language model running entirely on the mini PC under my desk, with one job. Plan the isolation.

Spoiler from the title: the model's first plan was syntactically perfect and diagnostically wrong. That failure turned out to be the most useful artifact of the whole phase. But let's earn it.

Something Is Finally Reading the Alarms

The Rust gateway from Part 2 writes every violation into a Redis Stream, because a stream is durable and pub/sub is a megaphone. Phase 3 opens by attaching a consumer group to that stream, and consumer groups come with a reliability contract worth spelling out: every entry is delivered to a named consumer, sits in a pending list until explicitly acknowledged, and can be reclaimed if its consumer dies.

I tested that contract the way this project tests everything. I gave a worker a chaos flag that makes it die after processing fifty entries without acknowledging any of them, pointed it at a hundred fresh alarms, and pulled the trigger. One hundred entries stranded in a dead consumer's name. Then I started a healthy worker and watched the clock: sixty-three seconds after the crash, one log line. adopted orphaned entries, count: 100. Not sixty-three seconds after the rescuer started. Sixty-three seconds after the crash, because the reclaim threshold protects slow workers from being robbed while abandoning dead ones on schedule.

The healthy worker then processed all one hundred, including the fifty the corpse had already handled. Duplicates, on purpose. At-least-once delivery means redelivery is not an edge case, it is the recovery mechanism working, and it sets the rule every downstream stage must obey: be idempotent, or be wrong.

From Noise to Incidents

Here's the thing about threshold violations: a gas leak is not four thousand events. It is one thing happening, reported four thousand times. Storing it as four thousand rows is technically true and operationally useless.

So the worker's first real job is a state machine. The first violation from a sensor opens an episode. Further violations fold into it: peak value updated, counter bumped. A warning that crosses into critical territory escalates the same episode. And when a sensor goes quiet long enough, a sweeper clears it, because the gateway only speaks when thresholds are crossed, which means "the danger has passed" is something you can only infer from silence.

The concurrency story is my favorite part, because it is one line of SQL doing the work of a distributed systems paper: a partial unique index guarantees at most one live episode per sensor, database-enforced. Every transition is a single atomic upsert. Two workers processing the same sensor cannot double-open an incident; the loser's insert quietly becomes an update. Workers keep no state worth losing, which is why crashing them is boring, and boring is the goal.

One promise from Part 1 also came due here. The database runs with synchronous commit off, because bulk telemetry tolerates sub-second loss. Episodes do not. Every episode write flips synchronous commit back on for just that transaction, and the worker only acknowledges the Redis entry after the commit is on disk. Durable first, then ack. Never the reverse.

Teaching the Machine Where It Is

An isolation plan that only knows the burning zone is a plan to trap people. So before any model gets involved, a context builder assembles what a planner actually needs, straight from the plant topology seeded back in Part 2: the alarm and its severity, the zone it lives in, every adjacent zone with the kind of link between them, and every asset in the zone with its shutdown steps.

The link kinds are the point. A neighbor connected by a duct means airflow coupling: gas travels. A conveyor neighbor means material feed: stop the upstream. A passage neighbor is how people get out. The document hands the planner all three, labeled.

And the document is deliberately boring: fixed field order, sorted lists, intensity expressed as a band instead of a live number. Boring means byte-stable, and byte-stable means the SHA-256 of the document works as a cache key. Identical hazards hash identically. Hold that thought.

The First Plan

The generator picks up pending plan requests and asks three questions in order. Have I seen this exact hazard before? Serve the cached plan, zero milliseconds. No? Ask the LLM, one generation at a time behind a semaphore, because on a 10GB VM the model is the bottleneck and queueing should happen where you can see it. LLM down? A circuit breaker trips after repeated failures and a deterministic rulebook takes over, parsing the same context document: stop every asset, seal the duct links, evacuate via the passage. Clearly labeled as a fallback. A mediocre plan now beats a brilliant plan that never arrives, and a safety system must never return nothing.

Then came the night of the first real generation. Phi-3-mini, 2.2GB, chewed on a methane alarm for twenty-six seconds and produced a beautifully structured plan to shut down a ball mill.

Its stated reason: "to prevent overheating."

The alarm was a gas leak. The model had the zone right, the asset right, the JSON flawless (after one earlier attempt died on a trailing comma at character 384, cured permanently by Ollama's constrained decoding, which makes invalid JSON grammatically impossible). And it had invented a completely different emergency to solve. Everything around the model behaved perfectly. The model just failed the comprehension exam.

I did what you do with a failing exam: I turned it into a metric.

The Bake-Off

The benchmark harness runs the identical hazard context through multiple models and scores four things: JSON validity, latency, plan length against the rulebook's twelve-step baseline, and a metric that exists purely because of that first plan, hazard_named: does the plan actually mention the gas?

Both models went three for three on validity. Phi-3 answered in twenty seconds with two-step plans that never once named the hazard. Llama-3-8B took twice as long and wrote nine-step plans, and its reasoning had texture the small model never showed. In one oxygen-sensor plan it noted that the vent fans should be checked but not immediately stopped, because the oxygen level is below warning. That is an actual hazard-specific judgment call. The rulebook can't do that. Phi-3 didn't.

The verdict: llama3:8b, twice the wait for four times the plan. The latency is fine precisely because of the architecture: the rulebook covers the reflex window, so the LLM is the considered second opinion, not the first responder. And the honest footnote: even the winner named the hazard explicitly only once in three trials, which smells like a prompt deficiency rather than a model one. That's next on the tuning list, and knowing which layer to blame is half of working with these things.

The Final Audit

Then the full-system demo: the Part 2 simulator ignites a methane plume, thirty-five sensors in one zone ramp past critical, and this time nobody is just recording it. Something is thinking.

The audit, every number from the live run: 4,548 violations entered the stream, delta verified exact against the gateway's counters, the Part 2 invariant still holding. The state machine folded them into 450 episodes: 35 criticals from the plume, 415 warning-level watchlist items from ambient sensor jitter, none of which wasted a single planning cycle. The 35 criticals produced 35 plan requests. And because thirty-three of them described the same hazard in the same zone, they hashed identically and were served from cache at zero milliseconds. Total LLM generations for the entire emergency: two.

Two, because there were exactly two different hazards: the methane plume, and one oxygen sensor in a conveyor gallery that picked that evening to have its own genuine crisis. The hash noticed. Poison-entry queue: zero. Plans stuck in flight: zero. Alarms answered with silence: zero.

Four and a half thousand events, four hundred and fifty incidents, two thoughts. That funnel is the whole phase in one picture.

What Phase 3 Taught Me

Put correctness in the database and let workers be cattle. Unique indexes and atomic upserts made crash-anywhere safe, and every chaos test got boring, which is the point. Duplicates are noise, drops are negligence. At-least-once with idempotent handlers beats exactly-once mythology, and knowing precisely which fields tolerate replay is the mature version of that answer. And judge models like you judge interns: on the work, with a rubric. A model can be fluent, fast, structurally perfect, and wrong about what's actually happening. The fix isn't trust or distrust. It's a benchmark, a fallback, and a cache that means you rarely have to ask twice.

Next Time: A Window Into the Plant

Everything so far happens in journals and SQL queries. The next phase builds the control room: a live dashboard riding the pub/sub channel the gateway has been faithfully publishing to since Part 2, alarms cascading across a zone map in real time, plans appearing beside them seconds later.

The plant can sense, remember, and think. Next, you get to watch it.

The mini PC under my desk remains unbothered.