feat(cloud): single-shot deploy.sh + FROST stack + healthchecks

Stage 5 — make the cloud composition spin up in one command and add
the SensorThings (FROST) stack as a fully segregated tenant.

cloud/deploy.sh — idempotent, 7-step bring-up:
  preflight → validate → up + wait → cert state → issue/renew →
  service status → endpoint smoke test. Reissues LE cert only when
  current issuer no longer matches ACME_CA_URI. Move-aside-then-
  restore-on-failure so the bootstrap cert survives a failed certbot.

stacks/frost — new stack, segregated from shared sql/rabbitmq:
  - dedicated postgis container (frost-db)
  - dedicated internal mosquitto bus (frost-mosquitto)
  - frost-http + frost-mqtt on a private frost-internal network,
    joined to cloud-app only for nginx ingress at frost.wbd-rd.nl
  - shared mosquitto stack deleted; rabbitmq remains the only public
    MQTT broker (mqtt.wbd-rd.nl:8883 via stream proxy)

stacks/sql — pg_isready healthcheck so keycloak/gitea/mlflow can gate
on service_healthy via cloud-level depends_on overrides.

stacks/nginx-proxy:
  - nginx-init service generates a self-signed bootstrap cert on
    fresh deploy so nginx starts before certbot has issued a real one
  - frost.wbd-rd.nl vhost (/FROST-Server → frost-http:8080,
    /mqtt → frost-mqtt:9876 WebSocket)

stacks/mlflow — custom Dockerfile (upstream + psycopg2-binary) so the
official image can speak to the shared sql backend.

stacks/jupyterhub — DummyAuthenticator stub gated by
JUPYTERHUB_ADMIN_PASSWORD; TODO comments point at OIDC + DockerSpawner.

stacks/rabbitmq — config/{enabled_plugins,rabbitmq.conf} stubs
(management + mqtt plugins, MQTT auth required).

stacks/portainer — ports unpublished; nginx now the only ingress.

stacks/node-red — pin to 4.1 (the floating "4" tag does not exist).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
znetsixe
2026-05-21 16:37:58 +02:00
parent 035ac757ae
commit 4117ec6063
25 changed files with 660 additions and 95 deletions

View File

@@ -22,7 +22,7 @@ R&D infrastructure for Waterschap Brabantse Delta. Hub-and-spoke topology:
│ rabbitmq, postfix, portainer │
│ sql (postgres, single config) │
│ mlflow, jupyterhub │
mosquitto (FROST stack only)
frost (SensorThings API)
└───────────────┬────────────────────┘
│ WireGuard tunnels
┌───────┼────────┬───────────┐
@@ -56,13 +56,14 @@ Each layer uses **four internal Docker networks**:
```
edge : nginx, wireguard-server
app : nginx, rabbitmq, postfix, node-red, grafana,
jenkins, gitea, keycloak, mlflow, jupyterhub
jenkins, gitea, keycloak, mlflow, jupyterhub,
portainer, frost-http, frost-mqtt
data : influxdb, sql, grafana, mlflow
mgmt : portainer, keycloak, wireguard-server, jupyterhub
frost-internal (private to frost stack) :
frost-db (postgis), frost-mosquitto, frost-http, frost-mqtt
```
(`mosquitto` joins `app` only when the FROST stack is deployed.)
### Edge attachments
```
@@ -117,12 +118,13 @@ For cloud-internal hostnames not reachable via Let's Encrypt HTTP-01, the longer
Postfix is **outbound-only**. It initiates SMTP to internet MX servers but accepts no inbound. Zero ingress, no published port, no listener facing internet. Just needs egress (every container has it via host NAT).
### MQTT — two brokers
### MQTT — RabbitMQ for public traffic, dedicated mosquitto inside FROST
- **RabbitMQ** is the **general-purpose** broker. Runs at both cloud and edge. MQTT plugin enabled. Cloud-side reachable externally via nginx stream proxy on `tcp/8883`. Edge-side fully internal.
- **Mosquitto** is reserved for the **FROST (SensorThings API) stack** only — cloud-only. Internal to its own stack — no external ingress unless FROST publishers need to push from outside (in which case use a separate stream block on a different port).
- **RabbitMQ** is the **only public MQTT broker**. SCADA / IoT / edge clients connect to `mqtt.wbd-rd.nl:8883` (TLS, via nginx `stream {}` block proxying to `rabbitmq:1883`). Authentication uses the standard RABBITMQ_USER/PASS.
- **frost-mosquitto** lives **inside the frost stack** on the private `frost-internal` docker network — it is purely the message bus between `frost-http` and `frost-mqtt`. It is not reachable from anywhere outside the frost stack.
- SensorThings-protocol MQTT (the FROST native MQTT API) is exposed to clients via `frost-mqtt`'s WebSocket port, proxied as `https://frost.wbd-rd.nl/mqtt`.
If FROST needs cross-broker forwarding, add a RabbitMQ `shovel` plugin pointing at `mosquitto`. Not wired up by default.
If FROST consumers also need to see SCADA traffic on RabbitMQ, add a RabbitMQ `shovel` plugin pointing into the frost stack. Not wired up by default.
### Gitea — HTTPS only
@@ -137,7 +139,7 @@ WireGuard is connectionless UDP with crypto-routed packets. Proxying through ngi
The repo defines **15 stacks** under `stacks/`:
- **Cloud + edge**: `nginx-proxy`, `node-red`, `influxdb`, `grafana`, `keycloak`, `portainer`, `rabbitmq`, `postfix`
- **Cloud-only**: `wireguard-server`, `gitea` (HTTPS), `jenkins`, `sql` (postgres), `mlflow`, `jupyterhub`, `mosquitto` (FROST)
- **Cloud-only**: `wireguard-server`, `gitea` (HTTPS), `jenkins`, `sql` (postgres), `mlflow`, `jupyterhub`, `frost` (SensorThings, dedicated postgis + internal bus)
- **Edge-only**: `wireguard-client`
## Sites
@@ -164,6 +166,7 @@ Tracked here so we don't forget. Each lands when we harden the relevant stack.
- **MinIO / artifact store** — MLflow uses local volume for now; switch to S3-compatible MinIO sidecar when artifacts grow.
- **JupyterHub auth** — target Keycloak OIDC via `oauthenticator.generic.GenericOAuthenticator`.
- **WG client routing** — split-tunnel vs full; per-peer `AllowedIPs` policy.
- **FROST auth** — currently `BasicAuthProvider` against the USERS table in `frost-db`; swap to Keycloak OIDC via the FROST OIDC plugin when SSO is rolled out.
- **MQTT cross-broker shovel** — only if FROST consumers must see RabbitMQ traffic or vice versa.
- **Internal PKI** — for cloud-internal hostnames not eligible for Let's Encrypt HTTP-01.
- **Backup strategy** — for `sql` (postgres), `influxdb`, `gitea-data`, `jenkins-home`, `mlflow-artifacts`.