HazShield AI
Distributed industrial safety monitoring built around controlled degradation — safety alarms are never dropped, even with the database down.
The problem
Industrial safety monitoring has to keep working exactly when conditions are worst. Most telemetry pipelines treat every reading the same, so when a broker stalls or storage falls behind, the readings that matter most — hazardous threshold violations — get dropped alongside routine noise.
The goal was a system where safety evaluation never degrades, whatever else is failing. Storage fidelity was allowed to suffer under pressure; hazard detection was not.
What I built
The system runs on a self-hosted OpenStack cluster and is organised around controlled degradation. Readings split across three priority lanes: a hot lane for safety violations that is never dropped and spills to disk if the broker is unreachable, a warm lane for bulk telemetry that degrades to sampling under pressure, and a cold lane for AI planning.
A Rust/Tokio ingest gateway handles per-sensor rate limiting, backpressure and disk-spill recovery. Async Python workers consume Redis Streams, recovering from crashes by adopting orphaned messages and routing malformed data to a dead-letter queue.
An alarm episode state machine (open → escalate → clear) folds raw violations into trackable incidents, with correctness enforced by database constraints rather than in-process state.
A locally-hosted LLM (Ollama) generates equipment isolation plans from plant topology, protected by a content-hash cache, a concurrency semaphore and a circuit breaker with rule-based fallback — so a plan always arrives even when the model is unavailable. I benchmarked two local models on identical hazard contexts before choosing one for production.
A React control room streams live updates over Server-Sent Events, illuminating zones in real time during an incident. Prometheus, Grafana and OpenTelemetry tracing follow a single violation from ingest through to its generated plan.
Outcome
Validated by killing the database mid-load-test: zero alarms lost, zero request errors, and storage degrading exactly as designed — the failure mode the architecture was built for, demonstrated rather than assumed.
In a live test the episode state machine compressed 4,548 raw threshold violations into 450 trackable incidents, keeping the control room actionable instead of flooding operators with noise.
Stack
Rust (Tokio) · Python (async) · Redis Streams · PostgreSQL · Ollama · React · Server-Sent Events · OpenStack · Prometheus · Grafana · OpenTelemetry
The build log
- 9,964 Readings a Second. Then I Pulled the Plug. HazShield AI — Part 2
Ten thousand sensor readings a second into a Rust gateway capped at 512MB — then I killed Postgres mid-cascade during a simulated gas leak. Zero alarms lost, counted rather than assumed.
- I Built a Private Cloud That Runs an AI Safety System Under My Desk
A distributed industrial safety system running on one mini PC under my desk — OpenStack squeezed into 28GB of RAM, five public domains with zero open ports, and a database built to survive the firehose.