Waterschap Brabantse Delta · R&D lab · research artifact
main, generalFunctions 1.13.2), in which every node writes to FROST itself.
change-heartbeat, sends every changed value and repeats the last value every 60 s. Nothing is dropped within a tolerance.generalFunctions/src/frost/reducer.js lets a new compression register itself as a reducer. This is where the PCHIP compression of this research goes.The flow has two paths: the write path runs for every measurement, the read path only when someone opens or queries the data. Use the toggle to show where the planned compression components go.
1 · write path · for every measurement
2 · read path · on request
Each component in one sentence, in the order a measurement passes them.[2][4]
Many different types of sensors deliver measurements, some at a fixed interval and some irregularly. In the development environment a digital twin generates the measurements inside the measurement nodes.
Filters noise and unwanted values locally, at the sensor itself, so only cleaned data goes online.
The measurement node converts to its unit, optionally smooths and clamps (or passes 1-to-1 in passthrough mode) and passes every accepted sample in full precision on. Every node (for example a pump) keeps its measurements in its own MeasurementContainer and controls with them.
Taps the node's database output (Port 1) behind the device gate, turns every numeric field into a point per series and handles the FROST metadata chain, buffer, backoff and batches. One client per FROST server is shared by the whole runtime.
One per series (Datastream); decides which values go to FROST. Today change-heartbeat: every change plus a heartbeat every 60 s. It is the plug-in point for a compression.
Implements the OGC SensorThings API (Things, Datastreams, Observations); data is written with HTTP POST and retrieved with HTTP GET.[1]
Stores the Observations FROST receives: today one row per changed value plus heartbeats.
Where the R&D lab views and analyses the data, partly with AI. Between two stored values it draws a straight line.
The steps in the edge layer change the data before it is stored. These facts set the starting point for the compression: what "the original data" is, where the compression plugs in, and what it has to beat.[4]
| Step | What happens | Consequence for compression | Source |
|---|---|---|---|
| Precision | The measurement node no longer rounds: every accepted sample reaches the MeasurementContainer at its source time and in full precision. A passthrough mode passes values 1-to-1. | Values are full float64; a PCHIP curve will practically never hit them exactly. The compression is therefore bounded by the sensor's repeatability, not exact. | measurement/src/channel.js · _writeOutput |
| Snapshot | On every output the node hands its FROST writer a full snapshot of its database fields (Port 1). Only finite numbers become points; a few text fields become Thing properties. | The compression works on this copy, never on the MeasurementContainer the node controls with (US-19). | nodered/BaseNodeAdapter.js · frost/nodeWriter.js |
| Timestamp | Set to the writer's clock when the node sends its output (milliseconds), not the sensor's own measurement time. | "Original timestamp" means this processing timestamp. | frost/nodeWriter.js · frostFormatter.js |
| Missing values | Non-finite values are skipped. When a series disappears from the node's output, the reducer gets absent() and the heartbeat stops; no gap is recorded in FROST. | A gap is only visible as missing heartbeats; the compression has to mark gaps itself (US-8, US-11). | frost/nodeWriter.js · frost/reducer.js |
| Reduction | change-heartbeat sends a value when it differs from the last sent value, and repeats the last value every 60 s so standstill shows as a flat line. There is no tolerance band. | Storage is not compressed: this is the new baseline. The old CoreSync route (angle mode, no fixed error bound) is the second reference; SDT with ε = repeatability is the fair competitor. | frost/reducer.js |
| Plug-in point | A reducer is an object with offer(point), due(now), absent() and flush(); a factory gets the first point of the series (unit, repeatability, accuracy) and is registered with registerReducer(). The code names "the rewritten SDT" as the next reducer. | The PCHIP compression becomes a reducer; buffer, backoff and metadata stay untouched. | frost/reducer.js · frost/domain.js |
| One client per server | The first node that connects to a FROST server sets the client options (heartbeat, reducer) for the whole runtime. | The compression cannot yet be chosen per node; this has to be agreed with the R&D lab. | frost/client.js · getClient |
| Observation parameters | Each Observation carries parameters: the reduction reason, evolvFieldKey, evolvStreamKey, sourceMeasurement and, when relevant, the curve origin. The value itself is 8 bytes. | Metadata still dominates storage per point; the size per Observation has to be measured again for the zero measurement. | frost/frostRequests.js · observationParameters |
| FROST unavailable | Up to 200 observations per series are buffered in memory (keep-first + rolling-latest); a failed observation goes back into the buffer. When full, points after the first are dropped. A restart empties the buffer. | Data can still be lost on long outages and restarts (relevant for US-5). | frost/domain.js |
| RTDP block | Component | Why this block |
|---|---|---|
| data generation | Sensors | Produce the measurements (or the digital twin simulates them). |
| processing | Cleaning module | Filters the data before it is collected. |
| processing | EVOLV node | Converts, filters and uses the measurements for control. |
| processing | Reducer | Decides which values are kept; the place where the compression runs. |
| ingestion | FROST writer | Collects the node's output and posts it to FROST, with buffer and retries. |
| storage · access | FROST server | Standard API in front of the database: receives observations (POST) and serves them (GET). |
| storage | Database | Persists the observations. |
| visualisation | Dashboard | Dashboarding and analysis. |
msg.frost.direct), so knots are not reduced twice. Which reducer runs is chosen per FROST
server for the whole runtime, not per node. The protocol between sensor and edge layer, the database behind FROST and
where the decoder will run are not yet recorded; the database is therefore shown generically.
main: generalFunctions 2df4980 (1.13.2, src/frost/, src/nodered/BaseNodeAdapter.js), measurement eee2c6a, coresync 075a7d7 (0.2.5). consulted 2026-10-12