EVOLV/CORESYNC_FROST_INTERVIEW_HANDOFF.md

CoreSync FROST Interview Handoff

Date: 2026-05-19

Continue Here First

Resume the interview at Question 20. The last open design topic was the reducer comparison method:

Q20. Should slope change be compared by angle in degrees or by relative slope delta?

Recommended direction before pausing:

• Support both eventually.
• Default to angle comparison with normalized time/value axes.
• Compute dx = deltaTimeMs / timeScaleMs.
• Compute dy = deltaValue / valueScale.
• Compare atan2(dy, dx) direction changes against angleToleranceDeg.

Agreed Decisions

• Use FROST/SensorThings instead of direct InfluxDB for the new CoreSync path.
• Keep EVOLV standard outputs:
  • process
  • dbase
  • parent
• Add a dbase output format option for frost.
• dbase = frost emits FROST-ready HTTP request messages.
• The CoreSync node does not post directly to FROST in the first version.
• A normal Node-RED HTTP request node sends the FROST messages.
• HTTP responses feed back into the same CoreSync input with msg.topic = "frost.response".
• All FROST metadata lookup/create/patch requests leave on dbase, not process.
• process is reserved for functional process data and optional functional state.
• The resolver is lazy: streams are resolved only when telemetry arrives.
• Pending queue policy for unresolved/FROST-down streams is keep first + latest, drop middle.
• Observation writes use nested Datastream endpoints:
  • POST /v1.1/Datastreams({datastreamId})/Observations
• Preserve provenance in Observation parameters.
• On angle/slope change, emit the previous point as the knot.
• Do not forward-fill delta-compressed fields.
• Latest values are queried per Datastream:
  • /Datastreams(id)/Observations?$orderby=phenomenonTime desc&$top=1

SensorThings Mapping

• EVOLV asset/apparatus/node -> FROST Thing
• EVOLV field type -> FROST ObservedProperty
• EVOLV variant (measured, predicted, setpoint) -> FROST Sensor
• EVOLV position -> stable FROST FeatureOfInterest
• EVOLV numeric field -> one FROST Datastream
• One reducer-kept knot -> one FROST Observation

Stable FOI convention:

{thingId}:upstream
{thingId}:atEquipment
{thingId}:downstream

Also copy position into Datastream.properties.position for filtering.

Units

Use EVOLV canonical ingest units. UI conversion happens client-side.

• pressure: Pa
• flow: m3/s
• power: W
• temperature: K
• density: kg/m3
• level: m
• volume: m3
• control / percentage / efficiency: normalized ratio 1

No leading zeros in engineering tags:

P-1
PT-1
FT-9999999

Never:

P-001
PT-0001

Identity And Registry

• Node-RED is not the source of truth for asset identity.
• Future central asset registry owns tag allocation and duplicate detection.
• Use one central counter per tag prefix:
  • P
  • PT
  • FT
  • TT
  • etc.
• The central registry, not local Node-RED, performs atomic +1.
• For now, assume the central registry is future work.
• First implementation derives identity when possible and allows overrides.
• Keep a boundary like resolveIdentity(input) so future registry integration is straightforward.

First-version identity behavior:

Thing tag: configured/derived, e.g. P-1
Sensor tag: configured/derived, e.g. PT-1, MODEL-P-1, CTRL-P-1
Stream key: thingTag:type:variant:position:sensorTag

Shared Collector Model

Use one shared CoreSync per FROST target/stack level.

Many EVOLV nodes can connect their dbase output to the CoreSync input, assuming payloads are structured as:

{
  measurement: "P-1",
  fields: {
    "pressure.measured.upstream.PT-1": 12345
  },
  tags: {
    tagcode: "P-1"
  },
  timestamp: Date
}

Also accept arrays of such payloads.

Internal stream key:

thingTag:type:variant:position:sensorTag

Per-stream state:

• FROST id cache
• latest FROST phenomenonTime
• reducer anchor point
• reducer previous point
• pending latest point
• bounded pending queue

FROST Request Message Shape

Outgoing request messages should preserve correlation metadata:

{
  topic: "frost.metadata.lookup",
  requestId: "thing:P-1:lookup",
  _coreSync: {
    kind: "thing",
    action: "lookup",
    externalKey: "thing:P-1",
    streamKey: "P-1:pressure:measured:upstream:PT-1"
  },
  method: "GET",
  url: "...",
  payload: null
}

FROST response feedback:

{
  topic: "frost.response",
  requestId: "...",
  statusCode: 200,
  payload: {},
  _coreSync: {}
}

Observation write target:

POST /v1.1/Datastreams({datastreamId})/Observations

Observation payload:

{
  "phenomenonTime": "2026-05-19T10:15:30.000Z",
  "result": 123.4,
  "FeatureOfInterest": {
    "@iot.id": 7
  },
  "parameters": {
    "reduction": "knot",
    "reductionReason": "first|angle-change|max-gap|flush",
    "evolvFieldKey": "pressure.measured.upstream.PT-1",
    "evolvStreamKey": "P-1:pressure:measured:upstream:PT-1",
    "sourceMeasurement": "Pump A"
  }
}

Reducer Decisions So Far

• Reducer runs independently per Datastream.
• 2D vector means time on X and numeric field value on Y.
• On direction change, emit the previous point.
• First point of a stream is kept.
• Previous point is kept on angle change.
• Pending latest point is emitted on explicit flush, max gap, or close.
• No forward-fill.

Pending queue during unresolved metadata/FROST downtime:

queue empty -> store observation
queue has 1 -> keep first, append latest
queue has 2 -> keep first, replace second with latest

Open Interview Questions

Implementation Progress 2026-05-21

First coding pass added:

• New Node-RED node: nodes/coresync/coresync.js / coresync.html.
• New frost dbase formatter in generalFunctions.
• Root Node-RED registration: package.json -> coresync.
• Focused tests: nodes/coresync/test/basic/coresync.basic.test.js.
• FROST request builder for lazy lookup/create and nested Observation writes.
• Per-stream normalized-angle reducer, defaulting to:
  • angleToleranceDeg = 5
  • timeScaleMs = 60000
  • maxGapMs = 300000
  • keep first + latest pending queue
• Minimal response state machine:
  • GET lookup
  • POST create if missing
  • cache returned @iot.id
  • drain pending Observations once Datastream and FOI ids are known

Validation run:

npx jest nodes/coresync/test/basic/coresync.basic.test.js --runInBand
PASS, 4 tests

npx eslint nodes/coresync/**/*.js nodes/generalFunctions/src/helper/formatters/frostFormatter.js nodes/generalFunctions/src/helper/formatters/index.js
PASS

Full npm run lint still fails on pre-existing unrelated repo issues, mostly browser globals in editor scripts and older lint findings.

Q20 decision implemented: default to normalized angle comparison. Relative slope mode is present as an advanced option in the reducer and editor config.

Q21 decision implemented: first defaults are the candidate defaults from this document, with per-type value-scale defaults in the CoreSync domain and fallback scale 1.

Q22 decision implemented: explicit msg.topic = "coresync.flush", maxGapMs, and close flush are supported. No periodic flush timer was added.

Q23 decision implemented: lazy resolver order is:

Thing
ObservedProperty
Sensor
FeatureOfInterest
Datastream
Observation

Each metadata entity uses lookup/create only. PATCH drift correction is not in this pass.

Q24 decision implemented in editor defaults:

frostBaseUrl
serviceVersion
assetTagOverride
sensorTagOverride
comparisonMode
angleToleranceDeg
timeScaleMs
maxGapMs
minDeltaTimeMs
minDeltaValue
maxQueuedObservationsPerStream
diagnosticsEnabled

Q25 decision implemented: emitted request messages are plain Node-RED HTTP-compatible messages and preserve requestId / _coreSync correlation fields.

Q26 decision implemented: id cache is runtime-only.

Q27 partial: failed metadata responses emit a process diagnostic and clear in-flight metadata. Backoff timing is not implemented yet.

Q28 implemented scope: skeleton, normalizer, reducer, FROST request builder, and minimal response state machine.

Q20. Reducer comparison method

Should slope change be compared by:

• angle in degrees, using normalized axes, or
• relative slope delta?

Recommended: default to normalized angle comparison and keep relative slope as an optional advanced mode.

Q21. Reducer defaults

What should the first defaults be?

Candidate defaults:

angleToleranceDeg = 5
timeScaleMs = 60000
valueScaleMode = auto
minDeltaTimeMs = 0
minDeltaValue = 0
maxGapMs = 300000

Need decide whether valueScale is:

• configured per observed property/unit,
• auto-learned per stream,
• fixed to 1.

Recommended: configured defaults by type, with auto fallback.

Q22. Flush behavior

When should pending latest points flush?

Options:

• on node close only,
• on explicit msg.topic = "coresync.flush",
• on maxGapMs,
• on periodic flush timer.

Recommended: support explicit flush and maxGapMs; avoid periodic flush unless needed.

Q23. Metadata bootstrap order

What exact lazy resolver chain should the first implementation use?

Candidate:

Thing
ObservedProperty
Sensor
FeatureOfInterest
Datastream
Observation

Need decide whether each entity is GET lookup -> POST create if missing, and whether PATCH metadata drift is included in v1.

Recommended v1: lookup/create only; no PATCH drift correction yet.

Q24. FROST base URL config

What config fields belong on the CoreSync node?

Candidate:

frostBaseUrl
serviceVersion = v1.1
dbaseFormat = frost
assetTagOverride
sensorTagOverride
angleToleranceDeg
timeScaleMs
maxGapMs
maxQueuedObservationsPerStream
diagnosticsEnabled

Need decide which are required for v1 editor UI.

Q25. HTTP node compatibility

Do we require a wrapper function around Node-RED HTTP request to preserve _coreSync and requestId, or should CoreSync emit messages exactly in the shape the HTTP node preserves by default?

Recommended: design emitted messages to survive the standard HTTP node, then add a helper/example flow if needed.

Q26. Local id cache persistence

Should resolved FROST ids be runtime-only, or persisted in Node-RED context?

Recommended v1: runtime-only cache, because metadata lookup is lazy and deterministic. Add persistent context later if lookups become expensive.

Q27. Error handling policy

For failed FROST responses, should the stream:

• retry immediately,
• back off,
• mark unresolved and keep first/latest pending,
• drop until manual reset?

Recommended: exponential-ish backoff per stream plus keep first/latest pending.

Q28. First implementation scope

Should the first coding pass create only the node skeleton plus reducer tests, or include the lazy FROST resolver end-to-end?

Recommended: implement skeleton, normalizer, reducer, and FROST request builder together; keep HTTP response state machine minimal but functional.