feat(cloud): harden nginx-proxy + sql foundation; HTTP-01 interim cert plan

Wire up the three foundation stacks (nginx-proxy, sql, portainer) in cloud/compose.yml and add real configs for the first two. nginx-proxy - Base nginx.conf with http + stream contexts, modern TLS profile, client_max_body_size baseline for gitea LFS / mlflow artifacts. - Vhosts under conf.d/: grafana, gitea, keycloak, nodered, mlflow, jupyter, portainer (HTTPS upstream), rabbitmq, jenkins. WebSocket upgrade headers where needed (grafana live, node-red editor, jupyterhub kernels, jenkins agents). - conf.d/00-default.conf serves /.well-known/acme-challenge/ on :80 and 301-redirects everything else. - stream.d/mqtt.conf terminates MQTT-TLS at 8883, proxies to rabbitmq:1883 internally. - All vhosts reference /etc/letsencrypt/live/infra/* — a stable path via certbot --cert-name infra, so the wildcard migration changes nothing in the vhost files. - README documents: HTTP-01 SAN interim during Versio period → DNS-01 wildcard via certbot-dns-transip after migration; bootstrap procedure (self-signed fallback → real cert issuance → reload). sql - config/init.d/01-databases.sh provisions gitea/keycloak/mlflow databases + roles on first start. Idempotent only via fresh data volume — change the script after first run requires manual psql or a volume wipe. - compose env extended with GITEA_DB_PASSWORD, KEYCLOAK_DB_PASSWORD, MLFLOW_DB_PASSWORD. cloud - include: now wires nginx-proxy + sql + portainer. Other stacks stay commented for future rounds. - .env.example adds KEYCLOAK_DB_PASSWORD and sensible defaults (LETSENCRYPT_EMAIL, GRAFANA_ROOT_URL, KEYCLOAK_HOSTNAME, GITEA_ROOT_URL, POSTFIX_FROM_DOMAIN all pointing at wbd-rd.nl). - Operator note inline: bring portainer's standalone instance down before deploying via cloud compose; comment its ports: block. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-21 13:43:35 +02:00
parent 67b37b9b2a
commit 5d95f8bfcc
19 changed files with 426 additions and 41 deletions
--- a/stacks/nginx-proxy/README.md
+++ b/stacks/nginx-proxy/README.md
@@ -1,47 +1,100 @@
 # nginx-proxy

-The single web ingress for cloud + edge. Reverse-proxies HTTPS UIs and stream-proxies MQTT-TLS to RabbitMQ. TLS certificates managed by a certbot sidecar (Let's Encrypt, HTTP-01 webroot challenge).
+The single web ingress for cloud + edge. Reverse-proxies HTTPS UIs and stream-proxies MQTT-TLS to RabbitMQ. TLS certs managed by a certbot sidecar (Let's Encrypt).

- **Image**: stock `nginx:1.27-alpine` (we don't use `nginxproxy/nginx-proxy` because we need the `stream {}` context for MQTT-TLS, which that image doesn't expose cleanly)
- **Sidecar**: `certbot/certbot:latest` — renews every 12h, shared `nginx-certs` + `nginx-acme-challenge` volumes
+- **Image**: stock `nginx:1.27-alpine` (we don't use `nginxproxy/nginx-proxy` because we need the `stream {}` context for MQTT-TLS)
+- **Sidecar**: `certbot/certbot:latest` — renews every 12h via HTTP-01 webroot challenges
 - **Networks**: `edge` (the only port-publisher) + `app` (talks to upstream services)
 - **Host ports**: `tcp/80`, `tcp/443`, `tcp/8883`

+## Cert strategy
+
+**Interim (Versio DNS)**: HTTP-01 SAN cert covering all subdomains, issued via `--webroot`. Requires:
+- Public DNS A records for each subdomain pointing at the cloud host
+- `tcp/80` reachable from the internet
+
+**After TransIP migration**: switch to DNS-01 wildcard (`*.wbd-rd.nl`). Swap the `certbot/certbot` image for a build that includes `certbot-dns-transip` and reissue with `--cert-name infra` so the cert path stays stable — **no vhost config changes needed**.
+
 ## Config layout

 ```
 config/
-├── nginx.conf            # base config — must include `stream {}` directive
-├── conf.d/               # HTTP vhosts (one per upstream UI)
-│   ├── grafana.conf
-│   ├── node-red.conf
-│   ├── gitea.conf
-│   └── ...
+├── nginx.conf                    # base — http + stream contexts
+├── conf.d/
+│   ├── 00-default.conf           # port 80: ACME challenge + HTTPS redirect
+│   ├── grafana.conf              # grafana.wbd-rd.nl
+│   ├── gitea.conf                # gitea.wbd-rd.nl
+│   ├── keycloak.conf             # keycloak.wbd-rd.nl
+│   ├── nodered.conf              # nodered.wbd-rd.nl
+│   ├── mlflow.conf               # mlflow.wbd-rd.nl
+│   ├── jupyter.conf              # jupyter.wbd-rd.nl
+│   ├── portainer.conf            # portainer.wbd-rd.nl (HTTPS upstream)
+│   ├── rabbitmq.conf             # rabbitmq.wbd-rd.nl (mgmt UI)
+│   └── jenkins.conf              # jenkins.wbd-rd.nl
 └── stream.d/
-    └── mqtt.conf         # MQTT-TLS stream block, SNI route to rabbitmq:1883
+    └── mqtt.conf                 # mqtt.wbd-rd.nl:8883 → rabbitmq:1883
 ```

 Volumes:
- `nginx-certs` — Let's Encrypt cert chains (`/etc/letsencrypt`), read-only mounted into nginx, writable from certbot
- `nginx-acme-challenge` — webroot for HTTP-01 challenges (`/var/www/certbot`)
+- `nginx-certs` — Let's Encrypt cert chains at `/etc/letsencrypt/`; read-only into nginx, writable from certbot
+- `nginx-acme-challenge` — webroot for HTTP-01 challenges at `/var/www/certbot/`

-## Initial cert issuance
+All vhosts reference `/etc/letsencrypt/live/infra/fullchain.pem` and `privkey.pem` — a stable path independent of the issuance method.

-1. Start with HTTP-only nginx config (serving `/.well-known/acme-challenge/`).
-2. Issue:
-   ```bash
-   docker compose run --rm certbot certonly \
-     --webroot -w /var/www/certbot \
-     --email "$LETSENCRYPT_EMAIL" --agree-tos --no-eff-email \
-     -d gitea.example.com -d grafana.example.com -d nodered.example.com
-   ```
-3. Drop HTTPS vhost configs into `config/conf.d/` and reload nginx.
+## First-run bootstrap

-The sidecar then renews automatically.
+The HTTPS server blocks won't load without a cert at `/etc/letsencrypt/live/infra/`. Bootstrap procedure (one-time):
+
+```bash
+cd stacks/nginx-proxy
+
+# 1. Self-signed fallback so nginx starts and serves /.well-known/acme-challenge/
+docker compose run --rm --entrypoint=/bin/sh nginx -c \
+  "mkdir -p /etc/letsencrypt/live/infra && \
+   openssl req -x509 -nodes -days 1 -newkey rsa:2048 \
+     -keyout /etc/letsencrypt/live/infra/privkey.pem \
+     -out   /etc/letsencrypt/live/infra/fullchain.pem \
+     -subj  '/CN=bootstrap-infra'"
+
+# 2. Start nginx (HTTPS blocks load with the dummy cert)
+docker compose up -d nginx
+
+# 3. Issue the real cert via HTTP-01
+docker compose run --rm certbot certonly \
+  --webroot -w /var/www/certbot \
+  --email "$LETSENCRYPT_EMAIL" --agree-tos --no-eff-email \
+  --cert-name infra \
+  -d grafana.wbd-rd.nl  -d gitea.wbd-rd.nl     -d keycloak.wbd-rd.nl \
+  -d nodered.wbd-rd.nl  -d mlflow.wbd-rd.nl    -d jupyter.wbd-rd.nl \
+  -d portainer.wbd-rd.nl -d rabbitmq.wbd-rd.nl -d jenkins.wbd-rd.nl \
+  -d mqtt.wbd-rd.nl
+
+# 4. Reload nginx to pick up the real cert
+docker compose exec nginx nginx -s reload
+```
+
+The certbot sidecar then renews every 12h automatically.
+
+## DNS prereqs (HTTP-01)
+
+Before bootstrap, ensure A records exist in Versio for:
+
+```
+grafana.wbd-rd.nl     A  <cloud-public-ip>
+gitea.wbd-rd.nl       A  <cloud-public-ip>
+keycloak.wbd-rd.nl    A  <cloud-public-ip>
+nodered.wbd-rd.nl     A  <cloud-public-ip>
+mlflow.wbd-rd.nl      A  <cloud-public-ip>
+jupyter.wbd-rd.nl     A  <cloud-public-ip>
+portainer.wbd-rd.nl   A  <cloud-public-ip>
+rabbitmq.wbd-rd.nl    A  <cloud-public-ip>
+jenkins.wbd-rd.nl     A  <cloud-public-ip>
+mqtt.wbd-rd.nl        A  <cloud-public-ip>
+```

 ## TODO

- Write base `config/nginx.conf` (`http` + `stream` contexts)
- Per-upstream vhost templates with OIDC `auth_request` to Keycloak
- Decide internal PKI vs Let's Encrypt for cloud-internal hostnames not reachable from the public internet
- Edge-side variant: bind to plant-LAN IP only, internal CA for plant.local hostnames
+- Wildcard cert via `certbot-dns-transip` (post TransIP migration)
+- OIDC `auth_request` to Keycloak in front of services without native OIDC (mlflow, portainer-CE)
+- Edge-side variant: bind to plant-LAN IP, internal CA for `*.local` hostnames
+- HSTS + security headers (`add_header Strict-Transport-Security ...`)
--- a/stacks/nginx-proxy/config/conf.d/00-default.conf
+++ b/stacks/nginx-proxy/config/conf.d/00-default.conf
@@ -0,0 +1,15 @@
+# Port 80 — ACME HTTP-01 challenges + force HTTPS redirect for everything else.
+
+server {
+    listen 80 default_server;
+    server_name _;
+
+    location /.well-known/acme-challenge/ {
+        root /var/www/certbot;
+        default_type "text/plain";
+    }
+
+    location / {
+        return 301 https://$host$request_uri;
+    }
+}
--- a/stacks/nginx-proxy/config/conf.d/gitea.conf
+++ b/stacks/nginx-proxy/config/conf.d/gitea.conf
@@ -0,0 +1,19 @@
+server {
+    listen 443 ssl;
+    http2 on;
+    server_name gitea.wbd-rd.nl;
+
+    ssl_certificate     /etc/letsencrypt/live/infra/fullchain.pem;
+    ssl_certificate_key /etc/letsencrypt/live/infra/privkey.pem;
+
+    # Allow large LFS uploads + raw blob downloads
+    client_max_body_size 1G;
+
+    location / {
+        proxy_pass http://gitea:3000;
+        proxy_set_header Host              $host;
+        proxy_set_header X-Real-IP         $remote_addr;
+        proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
+        proxy_set_header X-Forwarded-Proto $scheme;
+    }
+}
--- a/stacks/nginx-proxy/config/conf.d/grafana.conf
+++ b/stacks/nginx-proxy/config/conf.d/grafana.conf
@@ -0,0 +1,21 @@
+server {
+    listen 443 ssl;
+    http2 on;
+    server_name grafana.wbd-rd.nl;
+
+    ssl_certificate     /etc/letsencrypt/live/infra/fullchain.pem;
+    ssl_certificate_key /etc/letsencrypt/live/infra/privkey.pem;
+
+    location / {
+        proxy_pass http://grafana:3000;
+        proxy_set_header Host              $host;
+        proxy_set_header X-Real-IP         $remote_addr;
+        proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
+        proxy_set_header X-Forwarded-Proto $scheme;
+
+        # Grafana Live + alerting use WebSockets
+        proxy_http_version 1.1;
+        proxy_set_header   Upgrade    $http_upgrade;
+        proxy_set_header   Connection "upgrade";
+    }
+}
--- a/stacks/nginx-proxy/config/conf.d/jenkins.conf
+++ b/stacks/nginx-proxy/config/conf.d/jenkins.conf
@@ -0,0 +1,23 @@
+server {
+    listen 443 ssl;
+    http2 on;
+    server_name jenkins.wbd-rd.nl;
+
+    ssl_certificate     /etc/letsencrypt/live/infra/fullchain.pem;
+    ssl_certificate_key /etc/letsencrypt/live/infra/privkey.pem;
+
+    location / {
+        proxy_pass http://jenkins:8080;
+        proxy_set_header Host              $host;
+        proxy_set_header X-Real-IP         $remote_addr;
+        proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
+        proxy_set_header X-Forwarded-Proto $scheme;
+
+        proxy_redirect http:// https://;
+
+        # Jenkins agent + plugin upgrades use WebSockets
+        proxy_http_version 1.1;
+        proxy_set_header   Upgrade    $http_upgrade;
+        proxy_set_header   Connection "upgrade";
+    }
+}
--- a/stacks/nginx-proxy/config/conf.d/jupyter.conf
+++ b/stacks/nginx-proxy/config/conf.d/jupyter.conf
@@ -0,0 +1,24 @@
+server {
+    listen 443 ssl;
+    http2 on;
+    server_name jupyter.wbd-rd.nl;
+
+    ssl_certificate     /etc/letsencrypt/live/infra/fullchain.pem;
+    ssl_certificate_key /etc/letsencrypt/live/infra/privkey.pem;
+
+    location / {
+        proxy_pass http://jupyterhub:8000;
+        proxy_set_header Host              $host;
+        proxy_set_header X-Real-IP         $remote_addr;
+        proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
+        proxy_set_header X-Forwarded-Proto $scheme;
+
+        # JupyterHub uses WebSockets for terminals + notebook kernels
+        proxy_http_version 1.1;
+        proxy_set_header   Upgrade    $http_upgrade;
+        proxy_set_header   Connection "upgrade";
+
+        # Long-running notebooks
+        proxy_read_timeout 24h;
+    }
+}
--- a/stacks/nginx-proxy/config/conf.d/keycloak.conf
+++ b/stacks/nginx-proxy/config/conf.d/keycloak.conf
@@ -0,0 +1,17 @@
+server {
+    listen 443 ssl;
+    http2 on;
+    server_name keycloak.wbd-rd.nl;
+
+    ssl_certificate     /etc/letsencrypt/live/infra/fullchain.pem;
+    ssl_certificate_key /etc/letsencrypt/live/infra/privkey.pem;
+
+    location / {
+        proxy_pass http://keycloak:8080;
+        proxy_set_header Host              $host;
+        proxy_set_header X-Real-IP         $remote_addr;
+        proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
+        proxy_set_header X-Forwarded-Proto $scheme;
+        proxy_set_header X-Forwarded-Host  $host;
+    }
+}
--- a/stacks/nginx-proxy/config/conf.d/mlflow.conf
+++ b/stacks/nginx-proxy/config/conf.d/mlflow.conf
@@ -0,0 +1,22 @@
+server {
+    listen 443 ssl;
+    http2 on;
+    server_name mlflow.wbd-rd.nl;
+
+    ssl_certificate     /etc/letsencrypt/live/infra/fullchain.pem;
+    ssl_certificate_key /etc/letsencrypt/live/infra/privkey.pem;
+
+    # Large model artifact uploads
+    client_max_body_size 5G;
+
+    location / {
+        proxy_pass http://mlflow:5000;
+        proxy_set_header Host              $host;
+        proxy_set_header X-Real-IP         $remote_addr;
+        proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
+        proxy_set_header X-Forwarded-Proto $scheme;
+
+        proxy_read_timeout 10m;
+        proxy_send_timeout 10m;
+    }
+}
--- a/stacks/nginx-proxy/config/conf.d/nodered.conf
+++ b/stacks/nginx-proxy/config/conf.d/nodered.conf
@@ -0,0 +1,23 @@
+server {
+    listen 443 ssl;
+    http2 on;
+    server_name nodered.wbd-rd.nl;
+
+    ssl_certificate     /etc/letsencrypt/live/infra/fullchain.pem;
+    ssl_certificate_key /etc/letsencrypt/live/infra/privkey.pem;
+
+    location / {
+        proxy_pass http://node-red:1880;
+        proxy_set_header Host              $host;
+        proxy_set_header X-Real-IP         $remote_addr;
+        proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
+        proxy_set_header X-Forwarded-Proto $scheme;
+
+        # Node-RED editor + deploy API use WebSockets
+        proxy_http_version 1.1;
+        proxy_set_header   Upgrade    $http_upgrade;
+        proxy_set_header   Connection "upgrade";
+
+        proxy_read_timeout 1h;
+    }
+}
--- a/stacks/nginx-proxy/config/conf.d/portainer.conf
+++ b/stacks/nginx-proxy/config/conf.d/portainer.conf
@@ -0,0 +1,23 @@
+server {
+    listen 443 ssl;
+    http2 on;
+    server_name portainer.wbd-rd.nl;
+
+    ssl_certificate     /etc/letsencrypt/live/infra/fullchain.pem;
+    ssl_certificate_key /etc/letsencrypt/live/infra/privkey.pem;
+
+    location / {
+        # Portainer CE 2.x exposes HTTPS only (self-signed cert inside the container)
+        proxy_pass https://portainer:9443;
+        proxy_ssl_verify off;
+
+        proxy_set_header Host              $host;
+        proxy_set_header X-Real-IP         $remote_addr;
+        proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
+        proxy_set_header X-Forwarded-Proto $scheme;
+
+        proxy_http_version 1.1;
+        proxy_set_header   Upgrade    $http_upgrade;
+        proxy_set_header   Connection "upgrade";
+    }
+}
--- a/stacks/nginx-proxy/config/conf.d/rabbitmq.conf
+++ b/stacks/nginx-proxy/config/conf.d/rabbitmq.conf
@@ -0,0 +1,16 @@
+server {
+    listen 443 ssl;
+    http2 on;
+    server_name rabbitmq.wbd-rd.nl;
+
+    ssl_certificate     /etc/letsencrypt/live/infra/fullchain.pem;
+    ssl_certificate_key /etc/letsencrypt/live/infra/privkey.pem;
+
+    location / {
+        proxy_pass http://rabbitmq:15672;
+        proxy_set_header Host              $host;
+        proxy_set_header X-Real-IP         $remote_addr;
+        proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
+        proxy_set_header X-Forwarded-Proto $scheme;
+    }
+}
--- a/stacks/nginx-proxy/config/nginx.conf
+++ b/stacks/nginx-proxy/config/nginx.conf
@@ -0,0 +1,41 @@
+# nginx.conf — base config (http + stream contexts)
+
+user  nginx;
+worker_processes  auto;
+
+error_log  /var/log/nginx/error.log notice;
+pid        /var/run/nginx.pid;
+
+events {
+    worker_connections  1024;
+}
+
+http {
+    include       /etc/nginx/mime.types;
+    default_type  application/octet-stream;
+
+    log_format main '$remote_addr - $remote_user [$time_local] "$request" '
+                    '$status $body_bytes_sent "$http_referer" '
+                    '"$http_user_agent" "$http_x_forwarded_for"';
+    access_log /var/log/nginx/access.log main;
+
+    sendfile        on;
+    tcp_nopush      on;
+    tcp_nodelay     on;
+    keepalive_timeout  65;
+    server_tokens   off;
+    client_max_body_size 200M;
+
+    ssl_protocols TLSv1.2 TLSv1.3;
+    ssl_ciphers HIGH:!aNULL:!MD5;
+    ssl_prefer_server_ciphers off;
+    ssl_session_cache shared:SSL:10m;
+    ssl_session_timeout 1d;
+    ssl_session_tickets off;
+
+    include /etc/nginx/conf.d/*.conf;
+}
+
+stream {
+    include /etc/nginx/stream.d/*.conf;
+}
--- a/stacks/nginx-proxy/config/stream.d/mqtt.conf
+++ b/stacks/nginx-proxy/config/stream.d/mqtt.conf
@@ -0,0 +1,16 @@
+# MQTT-TLS reverse proxy.
+# Public:   mqtt.wbd-rd.nl:8883 (TLS, terminated here)
+# Upstream: rabbitmq:1883 (plaintext on internal `app` network)
+
+server {
+    listen 8883 ssl;
+
+    ssl_certificate     /etc/letsencrypt/live/infra/fullchain.pem;
+    ssl_certificate_key /etc/letsencrypt/live/infra/privkey.pem;
+    ssl_protocols TLSv1.2 TLSv1.3;
+    ssl_ciphers HIGH:!aNULL:!MD5;
+
+    proxy_pass rabbitmq:1883;
+    proxy_timeout 10m;
+    proxy_connect_timeout 5s;
+}