Data compression — requirements (MoSCoW) for edge-side PCHIP knot compression within the sensor repeatability

Sensor data arrives at the edge layer between the sensor and storage, where every EVOLV node writes its own data to FROST. Depending on the sensor, measurements come in continuously at a fixed interval or irregularly. Instead of storing every measurement, the system keeps a selection of measured points (knots) and later reconstructs the curve between them with a monotone piecewise-cubic method (PCHIP). A point may only be dropped if the reconstruction stays within the repeatability of the sensor: the compression is lossy, but with a fixed bound that matches the sensor's own measurement uncertainty. This artifact lists the requirements for that system as user stories with acceptance criteria, prioritised with the MoSCoW method.^[1]^[2]

20 user stories — 10 Must, 6 Should, 2 Could and 2 Won't have (this phase).
Core promise — compact storage of real measured points only, where every original measurement comes back within the repeatability of the sensor, and no peaks, dips or data in gaps are invented.
Built into EVOLV — a general function in generalFunctions: a reducer in the FROST writer that every node already has, working on a copy of the data so the node's control data is never touched.
Edge constraint — limited memory: a small active window per series, and only segments that can no longer change are written out.
Out of scope — multi-series compression and alarming from the edge device.

Requirements (MoSCoW)

Each requirement is a user story with a description and acceptance criteria. The stakeholder in every story is the R&D lab. Changed in version 1.5: US-1, US-2, US-3, US-12 and US-13; US-16 raised from Could to Must. New: US-19 and US-20.

Must have

Should have

Could have

Won't have (this phase)

Show

All user stories with priority, description and acceptance criteria. Use the filter above to show one MoSCoW category.

ID	User story	Priority	Description	Acceptance criteria
US-1	As the R&D lab I want measurement data to be stored compactly, so that less storage and data traffic is needed.	MUST	The system stores only the important measured points (knots). The course between them is reconstructed later.	Per knot only timestamp and value are stored. Knots are always real measured values. The number of stored points is demonstrably smaller than the number of raw measurements, and smaller than with the current reducer (change-heartbeat).
US-2	As the R&D lab I want the reconstructed series never to deviate more than the repeatability of the sensor, so that the deviation stays within the measurement uncertainty.	MUST	The compression is lossy with a fixed bound: a point is only dropped when the reconstruction stays within ε of it.	At every original measurement timestamp \|reconstruction − measurement\| ≤ ε. ε is determined by the rule in the definitions and can be found by the decoder. Demonstrated with a test set of digital-twin data and real sensor data.
US-3	As the R&D lab I want to retrieve a sensor's series over any period, so that I can work with it as with the measured series.	MUST	A continuous line is reconstructed from the stored knots.	The series can be retrieved per series and period. At every timestamp within the period the decoder gives a value within ε, except in gaps. Open point: whether the timestamps of dropped measurements are kept. Retrieving the same period twice gives exactly the same result.
US-4	As the R&D lab I want the reconstructed series to contain no peaks or dips that were not measured, so that I don't analyse fake data.	MUST	The series is reconstructed with a monotone method (PCHIP) that does not overshoot between two knots.	Between two knots the reconstructed value stays between the values of those two knots. A test with a sudden step (0, 0, 0, 10, 10, 10) gives no values below 0 or above 10.
US-5	As the R&D lab I want no knots to be lost when the edge device restarts or the connection to storage drops, so that the series is always complete.	MUST	The system holds on to knots during an outage and sends them as soon as it can. The existing FROST writer buffers in memory; a restart empties that buffer.	After a restart no knots are missing. When storage is unavailable, data is buffered and sent later.
US-6	As the R&D lab I want the compression to work with a limited amount of memory, so that it can run on the edge device between sensor and storage.	MUST	The system works with a small active window per series and writes out segments that no longer change immediately.	The active window per series has a fixed maximum size. Only segments that can no longer change are written out.
US-7	As the R&D lab I want consecutive stored blocks to join seamlessly, so that no jumps appear in the series.	MUST	Where one stored block ends, the next one begins without interruption.	The last knot of a block is also the first knot of the next block. At block boundaries there is no jump or kink in the reconstructed series.
US-8	As the R&D lab I want to see when a sensor did not measure, so that I don't mistake missing data for real measurements.	MUST	A period without measurements is not filled with invented values.	A gap longer than a configured maximum time is not bridged. The last point before and the first point after a gap are always knots. The reconstructed series shows "no data" inside the gap.
US-19	As the R&D lab I want the compression to work on a copy of the data, so that the control in the nodes (for example pumps) keeps using the full, original data.	MUST	New in 1.5. The compression only sees the points of the FROST writer, never the node's MeasurementContainer.	The compression does not change the points it receives. After compression, values and timestamps in the MeasurementContainer are unchanged (test). An error in the compression does not crash the node and does not affect its control.
US-16	As the R&D lab I want to retrieve values at a time step of my choice, so that I can analyse the series and easily compare series from different sensors.	MUST	Raised from Could to Must in 1.5: because only knots are stored, the reconstructed series has to be evaluated at any desired timestamp to be usable for analysis.	Values can be retrieved at any chosen timestamp and at a chosen interval (e.g. every 5 minutes). Every retrieved value lies within ε of the measured series; inside a gap the result is "no data". Retrieving the same timestamps twice gives exactly the same result.
US-9	As the R&D lab I want the highest and lowest values to be kept exactly, so that I can check afterwards whether a limit was exceeded.	SHOULD	Extreme values, such as a peak water level, are always stored as a knot.	The maximum and minimum per block are always knots. An exceedance in the real measurement is also visible in the stored data.
US-10	As the R&D lab I want recent measurements to be available in storage quickly, so that I can follow the current situation.	SHOULD	The system sometimes waits briefly to decide which points matter, but a maximum delay applies.	A measurement is in storage within an agreed maximum time.
US-11	As the R&D lab I want missing or faulty values to be handled by fixed rules, so that the compression doesn't derail on bad data.	SHOULD	Values such as NaN or sensor error codes are recognised and processed predictably.	Missing and faulty values are treated as a gap. The handling is documented and tested.
US-12	As the R&D lab I want to see per series how much storage is saved and how large the deviation is, so that I know whether the settings are well chosen.	SHOULD	During the test phase the system tracks the saving and the largest measured deviation per series.	Per series the compression factor can be retrieved. The largest deviation per series is always ≤ ε.
US-13	As the R&D lab I want each stored block to record how it was compressed, so that the data can always be reconstructed correctly later.	SHOULD	Each block or Datastream contains a small piece of descriptive information (metadata).	Series ID, algorithm version and ε can be found per block or per Datastream.
US-20	As the R&D lab I want the compression as a general function in generalFunctions, so that every node with a FROST writer can use it.	SHOULD	New in 1.5. The compression is a reducer in the plug-in point of the FROST writer.	Registered as a reducer and selectable with the client option `reducer`. A node needs no compression logic of its own.
US-14	As the R&D lab I want to store timestamps as the difference from the previous knot, so that storage becomes even smaller.	COULD	Instead of a full timestamp only the time difference is stored.	Timestamps can be reconstructed exactly. Storage per knot is smaller than with full timestamps. Only the full timestamp of the first knot is stored.
US-15	As the R&D lab I want to temporarily keep the raw data next to the compressed data in the initial phase, so that I can verify how it works.	COULD	Raw data is kept for a configurable period for validation.	Raw and reconstructed data can be compared side by side.
US-17	As the R&D lab I want to compress multiple series together.	WON'T	Out of scope: too complex for this phase; each series is compressed separately.	n/a in this phase.
US-18	As the R&D lab I want to receive an alarm from the edge device when a threshold is exceeded.	WON'T	Out of scope: alarming belongs to a different system.	n/a in this phase.

assumption Status. Version 1.5 is a proposal of 2026-10-12 and still has to be approved by the product owner. Compared with 1.4 (2026-09-29) the compression is no longer exact: after consultation with the R&D lab the deviation may be as large as the repeatability of the sensor, retrieving values at any chosen time step (US-16) became a Must, and the compression becomes a general function on a copy of the data instead of a separate node. The source document is in Dutch; the stories are translated to English here following the HELIX artifact language rule.

Data compression: requirements for edge-side knot compression with PCHIP reconstruction

Concept

Definitions

Requirements (MoSCoW)

Sources