infra/docs/architecture.md

Architecture

R&D infrastructure for Waterschap Brabantse Delta. Hub-and-spoke topology:

• Cloud layer — central services, one deployment, internet-facing.
• Edge layer — per-plant, plant-LAN-facing, tunneled to cloud via WireGuard.
• OT layer — per-plant, behind edge. Managed outside this repo.

                Internet
                    │
        ┌───────────┴───────────┐
        │ tcp/80, 443, 8883     │
        │ udp/51820             │
        ▼                       │
┌────────────────────────────────────┐
│ Cloud (central, one)               │
│   nginx + certbot ◀── 80/443/8883  │
│   wireguard-server ◀── 51820/udp   │
│   gitea, jenkins, keycloak, ...    │
│   influxdb, grafana, node-red      │
│   rabbitmq, postfix, portainer     │
│   sql (postgres, single config)    │
│   mlflow, jupyterhub               │
│   frost (SensorThings API)         │
└───────────────┬────────────────────┘
                │ WireGuard tunnels
        ┌───────┼────────┬───────────┐
        ▼       ▼        ▼           ▼
    ┌───────┐ ┌───────┐ ┌───────┐ ┌───────┐
    │ Edge: │ │ Edge: │ │ Edge: │ │  ...  │
    │gemaal1│ │ ...   │ │ ...   │ │       │
    └───┬───┘ └───────┘ └───────┘ └───────┘
        │ TLS
        ▼
    ┌──────────┐
    │ OT       │  ← out of scope of this repo
    │ OPCUA    │
    │ PLC      │
    └──────────┘

Network topology (per layer)

Each layer uses four internal Docker networks:

Network	Purpose	Notes
edge	Outermost. Cloud: internet-facing. Edge: plant-LAN-facing.	Only port-publishers join.
app	Application / automation tier.	Default landing for app services.
data	Databases (influxdb, sql).	internal: true — no internet egress.
mgmt	Identity, control plane (portainer, keycloak admin, wireguard mgmt).	Restricted.

Cloud attachments

edge   : nginx, wireguard-server
app    : nginx, rabbitmq, postfix, node-red, grafana,
         jenkins, gitea, keycloak, mlflow, jupyterhub,
         portainer, frost-http, frost-mqtt
data   : influxdb, sql, grafana, mlflow
mgmt   : portainer, keycloak, wireguard-server, jupyterhub
frost-internal (private to frost stack) :
         frost-db (postgis), frost-mosquitto, frost-http, frost-mqtt

Edge attachments

edge   : nginx                                  ← plant-LAN-facing
app    : nginx, rabbitmq, postfix, node-red,
         grafana, keycloak, wireguard-client
data   : influxdb, grafana
mgmt   : portainer, keycloak, wireguard-client

Ingress (the only ports facing outside)

Cloud (the central host)

Port	Container	Notes
tcp/80	nginx	HTTP → 301 to 443; also serves /.well-known/acme-challenge/ for certbot
tcp/443	nginx	All HTTPS UIs; TLS termination
tcp/8883	nginx	MQTT-TLS via stream {} block; SNI route to rabbitmq:1883
udp/51820	wireguard-server	VPN tunnel ingress

Two containers publish a total of four ports. Everything else is invisible from outside the host.

Edge (per-plant gateway)

Port	Container	Bound to
tcp/80	nginx	Plant-LAN interface only
tcp/443	nginx	Plant-LAN interface only

The edge wireguard-client initiates outbound to the cloud — it publishes no port.

TLS strategy

Stock nginx + certbot sidecar (Let's Encrypt, HTTP-01 webroot).

• Stock nginx:1.27-alpine — required because we use the stream {} context for MQTT-TLS. nginxproxy/nginx-proxy (the jwilder image) is HTTP/HTTPS-only and can't expose stream cleanly.
• certbot/certbot sidecar runs certbot renew every 12h. Shared nginx-certs + nginx-acme-challenge volumes coordinate cert + challenge state between the two containers.
• Initial issuance is manual (one-time docker compose run --rm certbot certonly --webroot …). Renewal is automatic.

For cloud-internal hostnames not reachable via Let's Encrypt HTTP-01, the longer-term plan is a small internal PKI (step-ca or similar) backed by sql. Out of scope for first deploy.

Why segment

• Blast radius: a compromised node-red on app cannot reach influxdb on data unless an explicit attachment is declared. Each service's reachability is auditable from networks: alone.
• Defense in depth: only nginx and wireguard-server bind host ports. No accidental 0.0.0.0 exposures.
• NIS2 / utility audit: WBD is in scope as water-sector. Compose networks evidence segmentation at runtime and on paper.

Special cases

Postfix (cloud + edge)

Postfix is outbound-only. It initiates SMTP to internet MX servers but accepts no inbound. Zero ingress, no published port, no listener facing internet. Just needs egress (every container has it via host NAT).

MQTT — RabbitMQ for public traffic, dedicated mosquitto inside FROST

• RabbitMQ is the only public MQTT broker. SCADA / IoT / edge clients connect to mqtt.wbd-rd.nl:8883 (TLS, via nginx stream {} block proxying to rabbitmq:1883). Authentication uses the standard RABBITMQ_USER/PASS.
• frost-mosquitto lives inside the frost stack on the private frost-internal docker network — it is purely the message bus between frost-http and frost-mqtt. It is not reachable from anywhere outside the frost stack.
• SensorThings-protocol MQTT (the FROST native MQTT API) is exposed to clients via frost-mqtt's WebSocket port, proxied as https://sta.wbd-rd.nl/mqtt.

If FROST consumers also need to see SCADA traffic on RabbitMQ, add a RabbitMQ shovel plugin pointing into the frost stack. Not wired up by default.

Gitea — HTTPS only

No SSH ingress. GITEA__server__DISABLE_SSH=true. All clones over HTTPS via nginx-proxy. Re-evaluate only if Gitea Actions runners require SSH push.

WireGuard server (cloud)

WireGuard is connectionless UDP with crypto-routed packets. Proxying through nginx-stream breaks NAT/MTU and adds no security benefit. The server publishes udp/51820 directly — the only non-nginx public ingress on cloud.

Stacks

The repo defines 15 stacks under stacks/:

• Cloud + edge: nginx-proxy, node-red, influxdb, grafana, keycloak, portainer, rabbitmq, postfix
• Cloud-only: wireguard-server, gitea (HTTPS), jenkins, sql (postgres), mlflow, jupyterhub, frost (SensorThings, dedicated postgis + internal bus)
• Edge-only: wireguard-client

Sites

Site	Status
gemaal1	Scaffolded; awaiting hardware provisioning (PLANT_LAN_IP, WG peer key, OPCUA endpoint)

Additional plants follow the same pattern under sites/<plant>/.

Conventions

• Folder names: kebab-case (node-red, nginx-proxy, wireguard-server).
• Compose filename: compose.yml (official Compose Spec).
• Composition: cloud/site composes pull stacks via include:. Stacks remain runnable standalone for testing.
• Secrets: .env (gitignored) + .env.example (committed with placeholders).
• Per-stack contents: compose.yml, .env.example, README.md, optional config/.
• OT layer: out of scope; PLC + OPCUA managed in a separate process.

Open decisions

Tracked here so we don't forget. Each lands when we harden the relevant stack.

• MinIO / artifact store — MLflow uses local volume for now; switch to S3-compatible MinIO sidecar when artifacts grow.
• JupyterHub auth — target Keycloak OIDC via oauthenticator.generic.GenericOAuthenticator.
• WG client routing — split-tunnel vs full; per-peer AllowedIPs policy.
• FROST auth — currently BasicAuthProvider against the USERS table in frost-db; swap to Keycloak OIDC via the FROST OIDC plugin when SSO is rolled out.
• MQTT cross-broker shovel — only if FROST consumers must see RabbitMQ traffic or vice versa.
• Internal PKI — for cloud-internal hostnames not eligible for Let's Encrypt HTTP-01.
• Backup strategy — for sql (postgres), influxdb, gitea-data, jenkins-home, mlflow-artifacts.
• Provision Gemaal1 — fill in PLANT_LAN_IP, WG peer key, OPCUA endpoint, deploy first stacks.