# dashboardAPI v2 — graph-aware Grafana dashboard generator

_Date: 2026-05-26 · Owner: R&D · Predecessors: `/grill-me` (in-conversation), [`docs/research/dashboardapi-graph-aware-grafana-generator.md`](../research/dashboardapi-graph-aware-grafana-generator.md)_

One `dashboardAPI` node in a Node-RED flow auto-generates one Grafana dashboard by walking its child-registration graph, composing per-node-type panel templates, and pushing the result to Grafana via HTTP on every Node-RED deploy.

## Problem
Every EVOLV example flow today carries a hand-authored Node-RED Dashboard tab — the active `pumpingstation-complete-example` flow has 73 `ui-*` nodes (charts, gauges, text widgets, fan-out function nodes) consuming roughly a third of the flow. Every new example replicates this work, and each one diverges in axis ranges, chart configs, and fan-out logic — so the output side is inconsistent across the 10+ example flows we maintain. The same telemetry already lands in InfluxDB via Port 1 of every node, so Grafana could render it natively, but today each Grafana dashboard is hand-authored JSON (`docker/grafana/provisioning/dashboards/pumping-station.json` is the only one that exists, frozen at one node type). Result: R&D spends disproportionate time on dashboard plumbing, examples drift, and Grafana — the better readout — is underused.

## Goals
1. Dropping a `dashboardAPI` node into a flow and deploying produces a complete Grafana dashboard with no hand-authored JSON.
2. Adding a new EVOLV node *instance* (e.g. a new measurement child) to a flow adds its panels on the next deploy with zero Grafana edits.
3. Adding a new EVOLV node *type* requires only a panel template fragment under `nodes/dashboardAPI/src/templates/<softwareType>.json` — no changes to the layout engine.
4. Cross-example consistency: every example flow's Grafana dashboard uses the same panel set, axis conventions, and dashed-bounds rendering for the same node type.
5. Node-RED Dashboard tab in example flows shrinks to control-only widgets (mode select, operator demand, calibration, signal injection). Target: ≤15 `ui-*` nodes per example flow.

## Non-goals
- Sub-second feedback latency from operator action → Grafana visible state. End-to-end ≤15s is acceptable; faster is not pursued.
- Preserving manual Grafana edits across regenerations. Dashboards are single-source-of-truth from dashboardAPI; manual edits are clobbered on next deploy.
- Per-instance dashboard customization through the Grafana UI. Templates are centralized and code-owned.
- Supporting non-EVOLV (third-party) Node-RED node types as panel sources.
- Live runtime regeneration (no deploy). Regen fires on Node-RED deploy events only.
- Operator (plant-staff) UX. Sole user is R&D until further notice.
- Replacing the InfluxDB write path. dashboardAPI v2 reuses the existing `outputUtils.formatForInflux` + `influxdbFormatter` plumbing unchanged.

## Users & scenarios
Sole user: EVOLV R&D team (Rene, Pim, Janneke, Sjoerd, Dieke, Pieter).

1. **New example flow from scratch.** When R&D builds a new example for `rotatingMachine-complete`, they assemble the node graph (pumpingStation + 3 pumps + measurements), drop in one dashboardAPI, connect each top-level parent to it, and deploy. A Grafana dashboard at the dashboardAPI's UID appears within seconds, with rows per parent and panels per child following the centralized templates.
2. **Adding a measurement to an existing flow.** When R&D wires a new measurement node as a child of an existing pumpingStation in `pumpingstation-complete-example` and redeploys, the corresponding pump panel gains a `measured` series next to its `predicted` series. No Grafana edit.
3. **Adding a new EVOLV node type.** When R&D ships a new node type `mixer`, they author `nodes/dashboardAPI/src/templates/mixer.json` (Grafana panel fragment with `${nodeName}` substitution tokens) and bump dashboardAPI's package version. Existing dashboardAPI instances pick up mixer-typed children on next deploy.

## Requirements

### Functional
1. **F-1.** `dashboardAPI` shall subscribe to `RED.events.on('flows:started')` and, on each event, inspect `payload.diff` to determine whether any of its own subtree (the dashboardAPI node, its registered children, their registered grandchildren) was affected. If yes, regenerate the dashboard. If no, no-op.
2. **F-2.** On regenerate, `dashboardAPI` shall walk its registered children via `ChildRegistrationUtils.getAllChildren()`, recurse one level per registered child to discover grandchildren, and produce an ordered list `[{softwareType, nodeName, position, children: [...]}, ...]`.
3. **F-3.** For each node in the graph, `dashboardAPI` shall load the matching template at `nodes/dashboardAPI/src/templates/${softwareType}.json` and substitute the placeholders `${nodeName}`, `${nodeId}`, `${parentName}`, `${dashboardUid}` and any child-list placeholders into the panel JSON.
4. **F-4.** The layout engine shall compose templates into a single Grafana dashboard JSON with: one row per top-level child of dashboardAPI; nested rows for grandchildren; sequential `gridPos.y` offsets so panels don't overlap.
5. **F-5.** Parent panels shall **not** repeat metrics that any of their children's templates already emit. The template format declares each panel's `emittedFields` so the composer can filter duplicates from the parent's panel set.
6. **F-6.** For each child node of type `rotatingMachine`, the panel set shall include: `%control`, `flow`, `delta P`, any registered measurement child's measured values, and `efficiency`. Where the node config exposes operating bounds (e.g. min/max flow), those bounds shall be rendered as dashed reference lines (`fieldConfig.custom.lineStyle = {fill: "dash", dash: [10,10]}` via a `byName` override) on the same panel as the act value.
7. **F-7.** For each child of type `measurement` registered to a parent that also emits a `predicted` series for the same quantity, the dashboard shall render two panels side by side (predicted left, measured right). If only `predicted` exists, render the predicted panel only. If only `measured` exists, render the measured panel only.
8. **F-8.** `dashboardAPI` shall POST the assembled dashboard to `POST {grafanaUrl}/api/dashboards/db` with body `{dashboard: <json>, overwrite: true, folderUid: <configured>}`, using the configured bearer token in `Authorization: Bearer <token>`. The `dashboard.uid` shall be deterministic from the dashboardAPI node's Node-RED id.
9. **F-9.** On a successful upsert (HTTP 200), `dashboardAPI` shall log the dashboard URL at info level. On failure (non-2xx, timeout, network error), it shall log at error level with the response body and shall **not** retry; the next deploy is the retry mechanism.
10. **F-10.** Each node emitting a value with operating bounds shall write the bounds as additional Influx fields named `<field>.min` and `<field>.max` alongside `<field>` itself. The dashed-line override matches these by suffix.
11. **F-11.** The bearer token shall be stored as a Node-RED encrypted credential, not as a plain `defaults` field. On node startup, if the legacy plain field exists, it is migrated to the credential store and the plain field is cleared, with one info-level log line per migrated instance.
12. **F-12.** `dashboardAPI` shall expose `msg.topic == "regenerate-dashboard"` as a manual trigger that bypasses the diff check and forces a regenerate.

### Non-functional
- **N-1. Performance.** Dashboard composition (graph walk + template merge + JSON build, excluding HTTP roundtrip) shall complete in <500ms for a flow with up to 50 registered children.
- **N-2. Idempotency.** Running the regenerate path twice in a row with no intervening graph change produces a byte-identical dashboard JSON.
- **N-3. Security.** The bearer token shall never appear in any log line, status update, debug output, or admin endpoint response. Token-bearing HTTP requests shall set TLS verification on when the configured Grafana URL is `https://`.
- **N-4. Observability.** Every regenerate emits a structured log line via the `logger` shared utility with fields: `dashboardUid`, `childCount`, `grandchildCount`, `compositionDurationMs`, `httpStatus`, `outcome ∈ {success, http-error, network-error, no-diff}`.
- **N-5. Backward compatibility.** Existing dashboardAPI instances continue to write to InfluxDB exactly as before. The Grafana-push path is additive and disabled if no `grafanaUrl` is configured.

## Constraints & dependencies
- **Grafana version pinned.** `docker-compose.yml` shall pin to `grafana/grafana:11.3.0` (or whatever specific minor exists at first-issue time) instead of `latest`. The legacy `POST /api/dashboards/db` endpoint is the target; the Grafana 12 Kubernetes-style API is out of scope. This resolves research **O-3**.
- **Node-RED runtime events.** Depends on `RED.events.on('flows:started')` firing with a `payload.diff` shape (added/changed/removed arrays) — undocumented but stable in current Node-RED versions. Verified by prototype before first issue ships.
- **InfluxDB write path unchanged.** Reuses existing `outputUtils.formatForInflux` + `influxdbFormatter`. No schema migration to existing telemetry.
- **Tag schema.** Every Influx field used by a panel must be in the existing emission convention (`_measurement = nodeName`, `_field = type.variant.position.childId`).
- **Scaffolding to reuse:** `ChildRegistrationUtils.getAllChildren()` (`nodes/generalFunctions/src/helper/childRegistrationUtils.js:104-106`), `extractChildren()` (`nodes/dashboardAPI/src/specificClass.js:151-163`), `grafanaUpsertUrl()` (`:107-110`, URL builder exists, HTTP send missing), `BaseNodeAdapter` lifecycle pattern.
- **No new npm dependencies** for the HTTP path. Use Node's built-in `https`/`http` modules.

## Success metrics
1. **Hand-authored Grafana JSON in repo = 0.** Measured by counting JSON files in `docker/grafana/provisioning/dashboards/` minus the dynamically-uploaded ones. Current: 2 (pumping-station.json, coresync-frost-demo.json). Target after rollout: 0 file-based, N dynamic.
2. **`ui-*` node count per example flow ≤ 15** (down from 73 in the current `pumpingstation-complete-example`). Measured by grepping `examples/*.flow.json` after migration.
3. **Time-to-first-dashboard for a new example flow ≤ 1 minute of human work** (drop in dashboardAPI, configure URL + token, deploy). Measured by stopwatch on the next example flow that gets built.
4. **Regression coverage:** every example flow's dashboard URL returns HTTP 200 and renders without panel errors. Measured by an integration test that hits the Grafana API after deploying each example.

## Open questions
- **O-1. `flows:started` + `diff` reliability across deploy modes.** Source-readable but needs a spike to confirm `diff` cleanly distinguishes "this dashboardAPI's subtree changed" from "an unrelated flow changed", across `full` / `nodes` / `flows` deploy types. → Resolved by `/prototype` before issue I-3 (the lifecycle hook issue) starts.
- **O-2. Dashed-line `custom.lineStyle` rendering against real Influx series.** Open Grafana bugs [#75259](https://github.com/grafana/grafana/issues/75259) and [#86546](https://github.com/grafana/grafana/issues/86546) may affect us. → Resolved by `/prototype` before issue I-5 (rotatingMachine template) starts.
- **O-5 (new).** Folder UID handling — does dashboardAPI assume a single Grafana folder for all generated dashboards (configured per-instance), or create per-flow folders? Default: per-instance configured folder UID, optional. If empty, dashboards land in the General folder. → Owner: R&D, deadline: before I-4.

## Out of scope (v2 candidates)
- Per-instance panel customization through the Grafana UI with merge-on-regen.
- Operator-facing UX (Grafana role/permission management, embedded dashboards in Node-RED).
- Auto-discovery of measurement units / axis ranges from node config schemas.
- Multi-Grafana-instance fanout (push the same dashboard to staging + prod).
- Grafana alerts / notification policies generated from EVOLV alarm definitions.
- Dashboard versioning / rollback inside Grafana.
- Template fragments living next to their owning node (decentralized template discovery).