RnD/infra
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: SQL=postgres, nginx+certbot, MQTT split, ML stacks, gitea HTTPS-only, gemaal1 site

Browse Source 
Round-2 changes locking in scaffold-phase decisions and adding ML/notebook stacks.

Locked decisions
- sql: postgres 16-alpine (was TBD); init.d/ mount for per-app DB provisioning
- nginx-proxy: stock nginx + certbot sidecar (was nginx:alpine TODO).
  Chose stock over nginxproxy/nginx-proxy because stream{} is required for
  MQTT-TLS reverse-proxy on tcp/8883 to rabbitmq:1883.
- gitea: HTTPS-only (DISABLE_SSH=true). No SSH port published.

MQTT split
- Remove stacks/mqtt placeholder.
- Add stacks/rabbitmq — general-purpose broker (AMQP + MQTT plugin),
  used at both cloud and edge. External MQTT clients reach cloud broker
  via nginx stream-proxy on 8883.
- Add stacks/mosquitto — reserved for the FROST (SensorThings) stack
  only. Cloud-only. Internal to its own stack; no external ingress.

ML / notebooks (cloud-only)
- stacks/mlflow — experiment tracking + model registry. Postgres backend
  on sql stack; local volume for artifacts (S3/MinIO is a TODO).
- stacks/jupyterhub — multi-user notebook server. DockerSpawner via
  mounted docker.sock; users spawn into cloud-app network so they can
  reach mlflow, influxdb (via grafana), rabbitmq.

Sites
- sites/gemaal1 — first edge deployment scaffold. Site-local override
  template for binding nginx to PLANT_LAN_IP.

Docs
- README + docs/architecture.md updated: stacks table now lists 15 stacks,
  ingress + attachment tables reflect mlflow/jupyterhub, TLS strategy
  section locked, MQTT-split section added, Gitea HTTPS-only noted.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
znetsixe
2026-05-21 13:22:46 +02:00
parent 8ab9061983
commit 2f5e3b4183
30 changed files with 492 additions and 116 deletions
Download Patch File Download Diff File Expand all files Collapse all files

111
docs/architecture.md
View File

@@ -13,21 +13,23 @@ R&D infrastructure for Waterschap Brabantse Delta. Hub-and-spoke topology:
│ tcp/80, 443, 8883 │
│ udp/51820 │
▼ │
┌───────────────────────────────────┐
│ Cloud (central, one) │
│ nginx-proxy ◀── 80/443/8883 │
│ wireguard-server ◀── 51820/udp │
│ gitea, jenkins, keycloak, ... │
│ influxdb, grafana, node-red │
│ mqtt, postfix, portainer │
│ sql (single point of config) │
└───────────────┬───────────────────┘
┌────────────────────────────────────┐
│ Cloud (central, one) │
│ nginx + certbot ◀── 80/443/8883 │
│ wireguard-server ◀── 51820/udp │
│ gitea, jenkins, keycloak, ... │
│ influxdb, grafana, node-red │
│ rabbitmq, postfix, portainer │
│ sql (postgres, single config) │
│ mlflow, jupyterhub │
│ mosquitto (FROST stack only) │
└───────────────┬────────────────────┘
│ WireGuard tunnels
┌───────┼────────┬───────────┐
▼ ▼ ▼ ▼
┌───────┐ ┌───────┐ ┌───────┐ ┌───────┐
│ Edge: │ │ Edge: │ │ Edge: │ │ ... │
│plant1 │ │plant2 │ │plant3 │ │ │
│gemaal1│ │ ... │ │ ... │ │ │
└───┬───┘ └───────┘ └───────┘ └───────┘
│ TLS
▼
@@ -45,25 +47,27 @@ Each layer uses **four internal Docker networks**:
| Network | Purpose | Notes |
|---|---|---|
| `edge` | Outermost. Cloud: internet-facing. Edge: plant-LAN-facing. | Only port-publishers join. |
| `app` | Application / automation tier (node-red, grafana, jenkins, gitea, …). | Default landing for app services. |
| `app` | Application / automation tier. | Default landing for app services. |
| `data` | Databases (influxdb, sql). | `internal: true` — no internet egress. |
| `mgmt` | Identity, control plane (portainer, keycloak admin, wireguard mgmt). | Restricted. |
### Cloud attachments
```
edge : nginx-proxy, wireguard-server
app : nginx-proxy, mqtt, postfix, node-red, grafana,
jenkins, gitea, keycloak
data : influxdb, sql, grafana
mgmt : portainer, keycloak, wireguard-server
edge : nginx, wireguard-server
app : nginx, rabbitmq, postfix, node-red, grafana,
jenkins, gitea, keycloak, mlflow, jupyterhub
data : influxdb, sql, grafana, mlflow
mgmt : portainer, keycloak, wireguard-server, jupyterhub
```
(`mosquitto` joins `app` only when the FROST stack is deployed.)
### Edge attachments
```
edge : nginx-proxy ← plant-LAN-facing
app : nginx-proxy, mqtt, postfix, node-red,
edge : nginx ← plant-LAN-facing
app : nginx, rabbitmq, postfix, node-red,
grafana, keycloak, wireguard-client
data : influxdb, grafana
mgmt : portainer, keycloak, wireguard-client
@@ -75,9 +79,9 @@ mgmt : portainer, keycloak, wireguard-client
| Port | Container | Notes |
|---|---|---|
| `tcp/80` | nginx-proxy | HTTP → 301 to 443 |
| `tcp/443` | nginx-proxy | All HTTPS UIs; TLS termination |
| `tcp/8883` | nginx-proxy | MQTT-TLS via `stream {}` block, SNI route to broker |
| `tcp/80` | nginx | HTTP → 301 to 443; also serves `/.well-known/acme-challenge/` for certbot |
| `tcp/443` | nginx | All HTTPS UIs; TLS termination |
| `tcp/8883` | nginx | MQTT-TLS via `stream {}` block; SNI route to `rabbitmq:1883` |
| `udp/51820` | wireguard-server | VPN tunnel ingress |
Two containers publish a total of four ports. **Everything else is invisible** from outside the host.
@@ -86,34 +90,63 @@ Two containers publish a total of four ports. **Everything else is invisible** f
| Port | Container | Bound to |
|---|---|---|
| `tcp/80` | nginx-proxy | Plant-LAN interface only |
| `tcp/443` | nginx-proxy | Plant-LAN interface only |
| `tcp/80` | nginx | Plant-LAN interface only |
| `tcp/443` | nginx | Plant-LAN interface only |
The edge `wireguard-client` initiates outbound to the cloud — it publishes **no port**. On-site operators reach SCADA on the plant LAN; remote ops reach the same nginx via the WG tunnel.
The edge `wireguard-client` initiates outbound to the cloud — it publishes **no port**.
## TLS strategy
**Stock nginx + certbot sidecar** (Let's Encrypt, HTTP-01 webroot).
- Stock `nginx:1.27-alpine` — required because we use the `stream {}` context for MQTT-TLS. `nginxproxy/nginx-proxy` (the jwilder image) is HTTP/HTTPS-only and can't expose stream cleanly.
- `certbot/certbot` sidecar runs `certbot renew` every 12h. Shared `nginx-certs` + `nginx-acme-challenge` volumes coordinate cert + challenge state between the two containers.
- Initial issuance is **manual** (one-time `docker compose run --rm certbot certonly --webroot …`). Renewal is automatic.
For cloud-internal hostnames not reachable via Let's Encrypt HTTP-01, the longer-term plan is a small internal PKI (step-ca or similar) backed by `sql`. Out of scope for first deploy.
## Why segment
- **Blast radius**: a compromised node-red on `app` cannot reach influxdb on `data` unless an explicit attachment is declared. Each service's reachability is auditable from `networks:` alone.
- **Defense in depth**: only nginx-proxy and wireguard-server bind host ports. No accidental `0.0.0.0` exposures.
- **NIS2 / utility audit**: WBD is in scope as water-sector. Compose networks are a cheap way to evidence segmentation at runtime and on paper.
- **Defense in depth**: only nginx and wireguard-server bind host ports. No accidental `0.0.0.0` exposures.
- **NIS2 / utility audit**: WBD is in scope as water-sector. Compose networks evidence segmentation at runtime and on paper.
## Special cases
### Postfix (cloud + edge)
The diagram labels it "OUT ONLY". Postfix initiates outbound SMTP to internet MX servers but accepts **no inbound** mail. So Postfix has zero ingress, no published port, no listener facing the internet. It just needs egress (which every container has via host NAT).
Postfix is **outbound-only**. It initiates SMTP to internet MX servers but accepts no inbound. Zero ingress, no published port, no listener facing internet. Just needs egress (every container has it via host NAT).
### MQTT (cloud)
### MQTT — two brokers
nginx-proxy `stream {}` block reverse-proxies `tcp/8883` to the internal broker via SNI. The broker has **no published port**. Cleanest "everything through nginx" model.
- **RabbitMQ** is the **general-purpose** broker. Runs at both cloud and edge. MQTT plugin enabled. Cloud-side reachable externally via nginx stream proxy on `tcp/8883`. Edge-side fully internal.
- **Mosquitto** is reserved for the **FROST (SensorThings API) stack** only — cloud-only. Internal to its own stack — no external ingress unless FROST publishers need to push from outside (in which case use a separate stream block on a different port).
### MQTT (edge)
If FROST needs cross-broker forwarding, add a RabbitMQ `shovel` plugin pointing at `mosquitto`. Not wired up by default.
Broker is **fully internal** to the edge stack — no plant-LAN ingress. Node-RED on edge bridges OPCUA → broker, then broker → cloud broker over the WG tunnel. Field devices that need MQTT publish to the cloud broker via WG, not to the edge broker directly.
### Gitea — HTTPS only
No SSH ingress. `GITEA__server__DISABLE_SSH=true`. All clones over HTTPS via nginx-proxy. Re-evaluate only if Gitea Actions runners require SSH push.
### WireGuard server (cloud)
WireGuard is connectionless UDP with crypto-routed packets. It cannot be sensibly reverse-proxied (NAT/MTU break, no security benefit). The server publishes `udp/51820` directly — the **only** non-nginx public ingress on cloud.
WireGuard is connectionless UDP with crypto-routed packets. Proxying through nginx-stream breaks NAT/MTU and adds no security benefit. The server publishes `udp/51820` directly — the **only** non-nginx public ingress on cloud.
## Stacks
The repo defines **15 stacks** under `stacks/`:
- **Cloud + edge**: `nginx-proxy`, `node-red`, `influxdb`, `grafana`, `keycloak`, `portainer`, `rabbitmq`, `postfix`
- **Cloud-only**: `wireguard-server`, `gitea` (HTTPS), `jenkins`, `sql` (postgres), `mlflow`, `jupyterhub`, `mosquitto` (FROST)
- **Edge-only**: `wireguard-client`
## Sites
| Site | Status |
|---|---|
| `gemaal1` | Scaffolded; awaiting hardware provisioning (PLANT_LAN_IP, WG peer key, OPCUA endpoint) |
Additional plants follow the same pattern under `sites/<plant>/`.
## Conventions
@@ -126,10 +159,12 @@ WireGuard is connectionless UDP with crypto-routed packets. It cannot be sensibl
## Open decisions
These are deferred until we build the respective stack. Tracked here so we don't forget.
Tracked here so we don't forget. Each lands when we harden the relevant stack.
- **SQL flavor**: postgres / mariadb / mysql? Leaning postgres for the "single point of config" use case.
- **SSL strategy**: certbot inside nginx-proxy, acme-companion sidecar, or step-ca-driven internal PKI? Probably acme-companion against Let's Encrypt for external endpoints + internal PKI for service-to-service.
- **Keycloak storage**: bundled H2 (dev only) vs external SQL backend (probably the same `sql` stack).
- **Backup strategy** for `data` (influxdb, sql) and `mgmt` (gitea, jenkins workspaces).
- **First site**: which plant gets `sites/<plant>/` scaffolded first?
- **MinIO / artifact store** — MLflow uses local volume for now; switch to S3-compatible MinIO sidecar when artifacts grow.
- **JupyterHub auth** — target Keycloak OIDC via `oauthenticator.generic.GenericOAuthenticator`.
- **WG client routing** — split-tunnel vs full; per-peer `AllowedIPs` policy.
- **MQTT cross-broker shovel** — only if FROST consumers must see RabbitMQ traffic or vice versa.
- **Internal PKI** — for cloud-internal hostnames not eligible for Let's Encrypt HTTP-01.
- **Backup strategy** — for `sql` (postgres), `influxdb`, `gitea-data`, `jenkins-home`, `mlflow-artifacts`.
- **Provision Gemaal1** — fill in `PLANT_LAN_IP`, WG peer key, OPCUA endpoint, deploy first stacks.