RnD/infra
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
main
infra/stacks/nginx-proxy
History
R de Ren 33a794e35d feat(sso): wire Keycloak SSO end-to-end across all apps 
New stack:
- stacks/oauth2-proxy/ — per-app sidecars (mlflow, portainer, rabbitmq)
  that gate vhosts via nginx auth_request against Keycloak's wbd realm.

Native OIDC wired into:
- grafana       (generic_oauth, role-attribute-path → Admin/Editor/Viewer)
- jupyterhub    (oauthenticator.GenericOAuthenticator)
- node-red      (passport-openidconnect; in-memory state store + users()
                 resolver because adminAuth doesn't expose req.session)
- jenkins       (oic-auth plugin via JCasC; matrix-auth for authz; setup
                 wizard suppressed; custom image with plugins.txt)

Infra fixes uncovered while bringing the above online:
- nginx-proxy: bump proxy_buffer_size to 16k so oauth2-proxy callbacks
  don't 502 on the JWT-bearing Set-Cookie header.
- nginx-proxy: add `resolver 127.0.0.11 valid=30s` so service names
  re-resolve after sidecar recreates (was cross-wiring oauth2-proxy
  upstreams after restart).
- jupyterhub: pass --allow-root to the singleuser spawner (hub runs as
  root inside its container; jupyter-server refused root without flag).
- jupyterhub Dockerfile: install jupyterlab + notebook so
  SimpleLocalProcessSpawner has something to launch.
- node-red Dockerfile: install passport-openidconnect into the image
  so settings.js can require() it.
- portainer: pre-seed local admin via --admin-password=<bcrypt-hash>
  so the 5-minute "no admin → lockout" timer can never trigger.
- deploy.sh: restore executable bit (was 644 in repo).

Admin/viewer policy:
- Created realm role `app-admin` in keycloak wbd realm.
- Grafana maps app-admin → Admin (default Viewer).
- Jenkins matrix-auth grants r.de.ren Overall/Administer, authenticated
  users get Overall/Read + Job/Read + View/Read.
- Node-RED: NODERED_ADMIN_USERS env list → permissions "*", others
  ["read"]. (TODO: switch to app-admin realm role.)
- JupyterHub: JUPYTERHUB_ADMIN_USERS env list. (Same TODO.)
- Gitea: r.de.ren pre-created as local admin; OIDC auto-links via email.

Docs:
- README, cloud/README, stacks/oauth2-proxy/README, and per-stack
  READMEs updated to reflect the new state and remove resolved TODOs.
- cloud/.env.example gains all the new OIDC client + cookie-secret keys.
- cloud/README documents the full kcadm realm bootstrap, including the
  hardcoded-audience mapper and post-logout redirect URIs that are
  non-obvious gotchas.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-21 18:34:37 +00:00
..
config
feat(sso): wire Keycloak SSO end-to-end across all apps
2026-05-21 18:34:37 +00:00
.env.example
feat: SQL=postgres, nginx+certbot, MQTT split, ML stacks, gitea HTTPS-only, gemaal1 site
2026-05-21 13:22:46 +02:00
compose.yml
feat(cloud): single-shot deploy.sh + FROST stack + healthchecks
2026-05-21 16:37:58 +02:00
README.md
refactor(dns): rename frost.wbd-rd.nl → sta.wbd-rd.nl; drop redundant portainer.wbd-rd.nl
2026-05-21 16:46:32 +02:00

README.md

nginx-proxy

The single web ingress for cloud + edge. Reverse-proxies HTTPS UIs and stream-proxies MQTT-TLS to RabbitMQ. TLS certs managed by a certbot sidecar (Let's Encrypt).

  • Image: stock nginx:1.27-alpine (we don't use nginxproxy/nginx-proxy because we need the stream {} context for MQTT-TLS)
  • Sidecar: certbot/certbot:latest — renews every 12h via HTTP-01 webroot challenges
  • Networks: edge (the only port-publisher) + app (talks to upstream services)
  • Host ports: tcp/80, tcp/443, tcp/8883

Cert strategy

Interim (Versio DNS): HTTP-01 SAN cert covering all subdomains, issued via --webroot. Requires:

  • Public DNS A records for each subdomain pointing at the cloud host
  • tcp/80 reachable from the internet

After TransIP migration: switch to DNS-01 wildcard (*.wbd-rd.nl). Swap the certbot/certbot image for a build that includes certbot-dns-transip and reissue with --cert-name infra so the cert path stays stable — no vhost config changes needed.

Config layout

config/
├── nginx.conf                    # base — http + stream contexts
├── conf.d/
│   ├── 00-default.conf           # port 80: ACME challenge + HTTPS redirect
│   ├── grafana.conf              # dash.wbd-rd.nl
│   ├── gitea.conf                # git.wbd-rd.nl
│   ├── keycloak.conf             # auth.wbd-rd.nl
│   ├── nodered.conf              # flow.wbd-rd.nl
│   ├── mlflow.conf               # ml.wbd-rd.nl
│   ├── jupyter.conf              # hub.wbd-rd.nl
│   ├── portainer.conf            # ops.wbd-rd.nl
│   ├── rabbitmq.conf             # mq.wbd-rd.nl (mgmt UI)
│   ├── jenkins.conf              # ci.wbd-rd.nl
│   └── frost.conf                # sta.wbd-rd.nl (FROST / SensorThings)
└── stream.d/
    └── mqtt.conf                 # mqtt.wbd-rd.nl:8883 → rabbitmq:1883

Volumes:

  • nginx-certs — Let's Encrypt cert chains at /etc/letsencrypt/; read-only into nginx, writable from certbot
  • nginx-acme-challenge — webroot for HTTP-01 challenges at /var/www/certbot/

All vhosts reference /etc/letsencrypt/live/infra/fullchain.pem and privkey.pem — a stable path independent of the issuance method.

First-run bootstrap

The HTTPS server blocks won't load without a cert at /etc/letsencrypt/live/infra/. Bootstrap procedure (one-time):

cd stacks/nginx-proxy

# 1. Self-signed fallback so nginx starts and serves /.well-known/acme-challenge/
docker compose run --rm --entrypoint=/bin/sh nginx -c \
  "mkdir -p /etc/letsencrypt/live/infra && \
   openssl req -x509 -nodes -days 1 -newkey rsa:2048 \
     -keyout /etc/letsencrypt/live/infra/privkey.pem \
     -out   /etc/letsencrypt/live/infra/fullchain.pem \
     -subj  '/CN=bootstrap-infra'"

# 2. Start nginx (HTTPS blocks load with the dummy cert)
docker compose up -d nginx

# 3. Issue the real cert via HTTP-01
docker compose run --rm certbot certonly \
  --webroot -w /var/www/certbot \
  --email "$LETSENCRYPT_EMAIL" --agree-tos --no-eff-email \
  --cert-name infra \
  -d git.wbd-rd.nl   -d auth.wbd-rd.nl  -d dash.wbd-rd.nl \
  -d flow.wbd-rd.nl  -d ml.wbd-rd.nl    -d hub.wbd-rd.nl \
  -d ops.wbd-rd.nl   -d mq.wbd-rd.nl    -d ci.wbd-rd.nl \
  -d mqtt.wbd-rd.nl  -d sta.wbd-rd.nl

# Easier: from the cloud directory just run ./deploy.sh — it handles steps 1-4.

# 4. Reload nginx to pick up the real cert
docker compose exec nginx nginx -s reload

The certbot sidecar then renews every 12h automatically.

DNS prereqs (HTTP-01)

Before bootstrap, ensure A records exist in Versio for the 11 short functional subdomains (the canonical tool-named ones — gitea.wbd-rd.nl, grafana.wbd-rd.nl, etc. — stay pointed at the existing Versio stack during the transition):

git.wbd-rd.nl    A  <cloud-public-ip>      # gitea (new)
auth.wbd-rd.nl   A  <cloud-public-ip>      # keycloak
dash.wbd-rd.nl   A  <cloud-public-ip>      # grafana (new)
flow.wbd-rd.nl   A  <cloud-public-ip>      # node-red (new)
ml.wbd-rd.nl     A  <cloud-public-ip>      # mlflow
hub.wbd-rd.nl    A  <cloud-public-ip>      # jupyterhub
ops.wbd-rd.nl    A  <cloud-public-ip>      # portainer
mq.wbd-rd.nl     A  <cloud-public-ip>      # rabbitmq mgmt UI
ci.wbd-rd.nl     A  <cloud-public-ip>      # jenkins
mqtt.wbd-rd.nl   A  <cloud-public-ip>      # MQTT-TLS broker
sta.wbd-rd.nl    A  <cloud-public-ip>      # FROST / SensorThings API

TODO

  • Wildcard cert via certbot-dns-transip (post TransIP migration)
  • OIDC auth_request to Keycloak in front of services without native OIDC (mlflow, portainer-CE)
  • Edge-side variant: bind to plant-LAN IP, internal CA for *.local hostnames
  • HSTS + security headers (add_header Strict-Transport-Security ...)
Reference in New Issue View Git Blame Copy Permalink