A lightweight middleware that ingests readings over HTTP and validates and cleans them before anything downstream trusts them, with checks for range, type, stuck sensors, noise and dropout. It then raises threshold alerts with hysteresis, and every rejected reading carries a reason. You connect a source and see live, clean data in minutes, with no broker, no database and no heavyweight platform to stand up. This page covers how it works and where the hard parts are.
A smart-facilities estate runs hundreds of sensors across dozens of sites, covering power, indoor-air quality, occupancy and water, over a mix of LoRaWAN, Modbus and MQTT. Two problems compound. First, the incumbent stack is a building-management-system monolith plus a heavyweight platform: slow to deploy, hard to scale across sites, and you can't simply point at a new sensor and watch the data. Second, and worse, raw sensor data is dirty: a meter spikes to ten times its range, a sensor sticks on one value for hours, a gateway drops out, a noisy signal jitters. Pipe that straight into dashboards and analytics and you get wrong averages, false alarms, alert fatigue, and a control room that stops trusting the screens. The need is not a bigger platform. It is a lightweight layer that makes the data trustworthy at the point it arrives.
Two instincts, two failures:
So the real job is a thin layer that validates and cleans every reading deterministically, explains every rejection, and alerts with hysteresis, and that you can run in one process next to the data.
A sensor reading arrives as a tiny JSON object on an HTTP endpoint (any LoRaWAN, Modbus or MQTT bridge can post one). Before it is stored or alerted on, it runs a deterministic pipeline: it must be the right type and a known metric, sit inside the sensor's physical range, not be a stuck repeat, and carry enough change to matter. What survives is stored, and what doesn't is logged with a reason. Only then do thresholds fire, with hysteresis, so a signal hovering at a limit doesn't flap.
The gate is five explicit checks, small enough to read and strict enough to matter:
Pointed at a simulated estate of six devices, the middleware ingested 466 readings in about a minute. It accepted the clean ones, deduped the noise, and rejected the rest, each with a reason. The most useful catch was device WCP-WTR-02, which got stuck on 44 L/min. After the sixth identical reading it was flagged stuck (flatline) and every further repeat was rejected. That is a fault a raw dashboard would have shown as a healthy flat line. Meanwhile CPFT-IAQ-03 drifted past the 1000 ppm CO₂ threshold, raised an alert, and cleared it on the way back down.
| Reading | Check | Outcome |
|---|---|---|
| WCP-WTR-02 = 44.0 (×68) | flatline (6+ identical) | rejected · stuck sensor |
| CPFT-IAQ-03 = 9000 ppm | range [350, 5000] | rejected · out of range |
| MSP-PWR-01 = "ERR" | numeric | rejected · non-numeric |
| MSP-PWR-01 = 620.01 | deadband (Δ < 0.2) | cleaned · deduped |
| CPFT-IAQ-03 = 1008 ppm | threshold > 1000 | alert raised → cleared |
Where these come from, honestly. The sensors are a simulator, so no real site data is used. The mix (power, IAQ, occupancy, water) and the dirty-data faults (range spikes, garbage values, a stuck sensor, a dropout, noise) mirror what a real smart-FM estate produces. What the figures prove is the mechanism, that the gate catches each fault and explains it, rather than a benchmark against a specific deployment. The accept rate looks low precisely because one stuck sensor floods the reject count. Remove it and the rest is almost all clean.
The trade-offs you tune. Three dials set the behaviour, openly. The range per metric: tighter rejects more borderline spikes. The deadband: wider dedups more noise but can mask slow drift. And the flatline count and stale interval: shorter flags faults faster but risks a false "stuck" or "offline" on slow or bursty sensors. The lightweight stance also has a real limit. This is a single-node middleware for connect-and-validate, not a clustered platform, and at estate scale you'd shard ingest and move the store to a time-series database. The point is to make the first 90% trivial to deploy, not to rebuild the heavy platform.
The control room's trust depends on the layer being honest about what it filtered. Every rejected or cleaned reading is kept with its device, metric, raw value and the rule that caught it. Every alert is logged with the value, the threshold, and whether it was raised or cleared. So you can always answer "what did the middleware do with this reading, and why". That auditability is what separates a filter you trust from one you don't.
device metric raw rule outcome WCP-WTR-02 water_lpm 44.0 flatline (68×) reject · stuck sensor CPFT-IAQ-03 co2_ppm 9000 range[350,5000] reject · out of range MSP-PWR-01 power_kw "ERR" numeric reject · non-numeric MSP-PWR-01 power_kw 620.01 deadband(<0.2) clean · deduped CPFT-IAQ-03 co2_ppm 1008 threshold>1000 alert · raised → cleared
Faced with an outdated, unscalable stack, my instinct is not to reach for a bigger platform. It is to build the thin layer that does the load-bearing 90%: validate the data at the moment it arrives, clean it deterministically, explain every rejection, and alert without crying wolf. I draw the trust boundary at ingest, so an untrusted reading earns its place in the store by passing a gate you can read. And I am honest about the limit, the point where a single node should give way to a sharded, time-series platform. The instinct most people have when a stack is failing is to buy a bigger one. Mine is to find the thin layer that does the load-bearing 90%, and to say plainly where it should hand off to something heavier. Knowing where it stops is as much the work as building it.
The figures above are from a live run. The console lets you start a sensor source, watch readings stream in, see exactly what the cleaning layer caught and why, set a threshold, and watch an alert raise and clear. It runs as a working application you connect to in seconds.