Waterschap Brabantse Delta · R&D-lab · research artifact

Vault on the hub: dynamic credentials without the babysitting

R&D-lab first version 2026-08-07 last updated 2026-08-07 version 1.0 concept — for review
All 24 credentials on the cloud hub live in one plaintext .env file, are substituted into container configuration at deploy time, and have no rotation mechanism at all. This artifact asks whether HashiCorp Vault with event-driven dynamic credentials can fix that without adding recurring administrative work. The short answer: yes for the part that matters, but not by the route the phrase suggests — and the honest gain is smaller and differently shaped than "dynamic credentials everywhere".
24
secrets in one file
16
stacks on the hub
5
Vault can rotate unattended
19
custody & audit only
0
rotation mechanisms today

Where we are

Every secret on the hub is a line in cloud/.env. Docker Compose substitutes it into an environment: block when a container is created, so the value ends up inside the container's stored configuration — visible to anything that can call docker inspect, read /proc/<pid>/environ, or open the Portainer UI.[6] There is no rotation path: changing a value means editing the file and recreating the container.

cloud/.env SQL_PASSWORD=… GITEA_DB_PASSWORD=… KEYCLOAK_DB_PASSWORD=… … 21 more plaintext, on the hub one file · 24 values substituted at up -d Docker Compose interpolation frozen into container config keycloak gitea mlflow frost grafana … 11 more stacks docker inspect /proc/<pid>/environ Portainer UI rotate one value = edit the file, recreate the container, brief outage — which is why it has never been done —
The current path of a secret, from the single .env file to the running container. The red return path is the reason the inventory is static: rotation is a manual, outage-bearing edit, so it does not happen. Source: infra/cloud/.env.example and infra/stacks/*/compose.yml, read 2026-08-07.[6]

The question that sorts everything

Whether a secret can be made dynamic has nothing to do with Vault. It depends on one property of the consumer: when does it read the value? Sort the hub by that question and three groups fall out, each needing a different treatment.

When does the consumer read the secret? at every use psql from a laptop · Jenkins job JupyterHub notebook · HELIX, Eddy, passport, Node-RED flows Dynamic credentials Vault mints a throwaway PostgreSQL or RabbitMQ user on request, with a 1-hour lease, and deletes it when the lease ends. Nothing ever restarts. No shared password exists. once, at container boot keycloak · gitea · mlflow · frost — the DB password is an env var, read when the process starts Vault-owned static roles Vault owns the password and rotates it on a schedule you pick. The Agent rewrites .env and triggers one controlled restart. Rotation becomes scheduled and logged instead of never. only once, ever influx init · rabbitmq default user portainer admin · keycloak bootstrap — plus the 8 OIDC client secrets Custody, versioning and audit only No Vault engine can rotate these. Vault stores them, records who read them and when, and keeps old versions. Several are bootstrap residue that no longer authenticates anything. The treatment is dictated by the consumer, not by Vault — this is why "dynamic credentials everywhere" is not achievable here.
The sorting rule. A secret can only be short-lived if whatever uses it asks for a fresh copy each time. Off-the-shelf server images almost never do; our own code and our people can. That asymmetry, not any Vault setting, decides the shape of the whole design.

assumption The lane-C classification of INFLUX_*, RABBITMQ_DEFAULT_*, PORTAINER_ADMIN_* and KEYCLOAK_ADMIN_PASSWORD as first-boot-only is read off the compose files — DOCKER_INFLUXDB_INIT_MODE: setup, RABBITMQ_DEFAULT_USER/PASS, KC_BOOTSTRAP_ADMIN_PASSWORD are all initialisation-time parameters.[6] I did not verify at runtime that changing them after first boot has no effect. If that holds, part of the .env is dead weight that can be removed rather than migrated. Worth confirming before Phase 0. The exception is INFLUX_ADMIN_TOKEN, which stays a live, all-powerful API token.

The actual inventory of the hub, sorted into the three treatments. Click a group. The counts are exact: 24 credential-bearing variables in cloud/.env.example, counted 2026-08-07.[6] Note that group A contains no existing variables — it is the capability we do not have today, and it is where the benefit lives.

What "event-driven" can actually mean here

Vault does have a real event bus: a WebSocket at sys/events/subscribe/{eventType} that publishes KV and database-engine events. It requires a Vault Enterprise licence.[1] It also gives at-most-once delivery with no persistence, so anything built on it needs a reconciliation path anyway.

The Community-edition mechanism that produces the same outcome is the Vault Agent lease loop. The Agent holds the lease, notices the credential change, re-renders a file from a template and runs a command when the rendered output differs.[2] The trigger is the credential's own lifecycle rather than a message on a bus — for our purposes that is the same event, delivered reliably, with no licence and no extra listener to keep alive.

A · sys/events WebSocket bus Enterprise licence Vault publishes listener you must build acts app Extra moving part to run and monitor. At-most-once delivery, no persistence — a missed event needs a reconcile loop. 3 components · licence cost · new failure mode B · Vault Agent lease loop Community edition Vault lease expires Vault Agent renders + execs app The Agent is the listener; it already holds the lease. Renewal is a guaranteed local event, not a delivered message — nothing to miss. 2 components · no licence · no new failure mode Same outcome — a credential change drives a config rewrite — reached with one fewer component and no licence.
The difference between the two mechanisms is the listener. Option A publishes an event and requires something of ours to be subscribed at that moment; option B collapses publisher and subscriber into the Agent that already holds the lease. For a single hub, B is strictly simpler. Sources: [1], [2].
Recommendation Build on the Vault Agent lease loop. Do not design anything that depends on sys/events — it would tie a small internal platform to an Enterprise licence for a capability we can get for free.

The loop, one rotation at a time

This is the mechanism that makes the whole thing hands-off. Step through it.

Six steps, no human in any of them. Press “Run a rotation”.
Vault on the hub Vault Agent systemd, on host cloud/.env rendered file gitea postgres sql AppRole login new password + lease render exec: docker compose up -d gitea reconnects old credential revoked — it no longer exists anywhere Total human involvement: zero. Total downtime: one container restart, at a time you choose.
One full rotation of a boot-only consumer (group B). The Agent runs on the host under systemd, not in a container: that keeps the Docker socket out of a container that also holds a Vault token, and it lets the exec hook call the same docker compose up -d our deploy already uses.[2]
Why this keeps the deploy flow intact Vault renders the same .env file Compose already reads. Nothing in compose.yml, deploy.sh or the stack layout changes, and if Vault is stopped the last-rendered file is still on disk and the hub still boots. Vault slots in behind the existing deploy rather than replacing it — which is most of the reason this can be done without administrative pain.

The one irreducible chore: unseal

A sealed Vault cannot decrypt its own storage. After every reboot something must supply the unseal key. This is the only genuinely new recurring obligation, and the three ways out differ in exactly one respect: where the key sits.[3]

A · Shamir, by hand the VPS Vault key holder a person, at every boot ✗ unattended reboot = hub down until someone is reachable B · transit-unseal, same host the VPS Vault unsealer Vault key on this disk the key sits inside the boundary it is protecting ~ boots by itself stolen disk still safe; compromised host root is not C · external KMS the VPS Vault unwrap KMS key the key never sits on the machine it unlocks ✓ boots by itself, key off the host needs a KMS we are entitled to use Only option C both removes the human and moves the key off the machine it is protecting. B is not worthless: it still protects stolen disks and backups, just not a compromised host root.
The three seal options, drawn by the property that distinguishes them. Option B is the common self-hosted answer and carries a bootstrap paradox — the unsealer itself has to be unsealed, so its key ends up in a file on the same disk.[7] That is still an improvement over today, but it is worth being clear-eyed that it is convenience, not separation.
Open decision — I need input here Option C is the recommendation if WBD has a KMS we may use. A Dutch water authority with a Microsoft tenancy usually has Azure Key Vault available, which Vault supports as an auto-unseal provider.[3] assumption I have not verified that such a tenancy exists, that we may use it for R&D infrastructure, or that routing an unseal call to it is acceptable under our data-residency rules. If the answer is no, take option B and record the limitation. Do not take option A — an unattended reboot would take the whole hub down until someone is reachable.

The administrative ledger

The fair way to judge "no administrative pain" is to count the chores on both sides, including the ones Vault adds.

Recurring administrative work, today versus after. The bottom block is new work that does not exist today; it is the honest cost of the proposal.
ChoreTodayAfterNet
Rotate a database passwordManual edit + recreate; never doneScheduled, unattendedbetter
Give a colleague database accessShare the static passwordThey mint their own, 1 hbetter
Revoke that accessRotate everything, or nothingExpires by itselfbetter
Give a CI job database accessStatic password in JenkinsJob mints per-run credsbetter
Answer "who used this credential?"Not answerableVault audit logbetter
Someone leaves the teamIn principle rotate 24 valuesRevoke their Vault identitybetter
Rotate an OIDC client secretManual, two-sidedManual, two-sided — but trackedunchanged
Deploy the hub./deploy.sh./deploy.shunchanged
Unseal after reboot—Automatic (option B/C)new
Custody of unseal / recovery keys—Offline, split, documentednew
Back up Vault's own storage—Raft snapshot in the backup runnew
Keep policies and roles current—Reviewed when apps changenew
Vault is now a dependency—Agent renders to disk, so the hub still boots if Vault is downnew

Six chores improve, two are unchanged, five are new. Four of the five new ones are set-up-once obligations rather than repeating work; the genuinely recurring one is key custody discipline. That is the trade being proposed, stated plainly.

A migration that can stop at any phase

Each phase is useful on its own and reversible on its own. If we stop after Phase 1 we have already got most of the benefit.

0 Stand up Vault + auto-unseal. The 24 values move into KV; Agent renders the .envwe already use. no behaviour change 1 Dynamic PostgreSQL + RabbitMQ engines. People, Jenkins, notebooks and our apps stop sharing passwords. most of the benefit 2 Rotate Static roles on the five PostgreSQL passwords, rotated on a schedule you pick. restarts, in a window 3 Automate Agent exec hooks do the restart. Rotation becomes fully hands-off. optional polish
The phases. Phase 0 is deliberately a no-op from the outside: the same file is produced, by a different producer, so it can be rolled back by simply not running the Agent. Phase 1 is where the security benefit actually lands, and it touches no existing service.

Discussion & limitations

This does not make the hub "secretless". Nineteen of twenty-four values remain long-lived strings that a person changes by hand. Vault gives them custody, versioning, an audit trail and one place to look — not automation. Anyone selling this project as "dynamic credentials everywhere" would be overstating it.

Secret zero moves, it does not vanish. The Agent needs credentials to authenticate to Vault (an AppRole role-id and secret-id). That file becomes the new most-sensitive thing on the host. It is a much smaller and more auditable target than 24 live passwords, but it is not nothing.

Licensing is worth a deliberate choice. Vault Community is BUSL-1.1, which permits our internal use but is not an OSI open-source licence. OpenBao is the Linux Foundation fork of the last MPL-2.0 Vault and carries the same database secrets engine.[5] assumption I have not verified OpenBao's Agent/templating parity or its event-notification story; if licence purity matters to WBD procurement, that verification should happen before Phase 0, because the two are hard to swap later.

What I could not check. SSH to the hub is firewalled from my current network, so everything here is derived from the infra repository rather than from the running machine. Three things need a look on the box itself before committing: whether a KMS is reachable and permitted, whether the first-boot-only classification in §2 holds, and how much free disk and memory the hub has for one more always-on service.

Questions for you

These are the decisions I cannot make from the repository. Comment on any of them directly in this document.

  1. 1 Unseal: do we have an external KMS we are allowed to use (Azure Key Vault via a WBD tenancy, or anything equivalent)? If not, is option B — automatic unseal with the key on the same host — acceptable, with the limitation written down?
  2. 2 Scope: is Phase 1 alone enough for now? It delivers per-person and per-job credentials without touching a single running service, and it is where I would stop if the appetite for change is low.
  3. 3 Rotation window: for the five PostgreSQL passwords, what restart window is acceptable — and how often? Monthly, quarterly, or only on demand?
  4. 4 Vault or OpenBao: does the BUSL licence on Vault Community matter for a waterschap R&D platform, enough to justify verifying the fork instead?
  5. 5 Bootstrap residue: can we confirm and then simply delete the first-boot-only values rather than migrating them? That would shrink the inventory before we start.
  6. 6 Ownership: who holds the recovery keys, and who is the second person who can unseal when the first is on holiday? This is the chore that is genuinely new and genuinely recurring.

Sources

  1. HashiCorp, Event Notifications — concepts. States GA from v1.16, "Appropriate Vault Enterprise license required", at-most-once delivery, subscription via WebSocket at /v1/sys/events/subscribe/{eventType}. link ↗ accessed 2026-08-07
  2. HashiCorp, Vault Agent — templates. The exec block runs a command when rendered output changes; no Enterprise marking on the templating feature. link ↗ accessed 2026-08-07
  3. HashiCorp, Seal/Unseal and Auto unseal. Supported auto-unseal providers include AliCloud KMS, AWS KMS, Azure Key Vault, Google Cloud KMS, OCI KMS and Transit. link ↗ accessed 2026-08-07
  4. HashiCorp, Database secrets engine (incl. PostgreSQL plugin and static roles with rotation_period) and the RabbitMQ secrets engine. link ↗ · link ↗ accessed 2026-08-07
  5. OpenBao (Linux Foundation), fork of Vault 1.14 under MPL-2.0; database secrets engine documentation. link ↗ accessed 2026-08-07
  6. WBD R&D, infra repository — cloud/.env.example, cloud/compose.yml, stacks/{sql,gitea,keycloak,mlflow,frost,influxdb,rabbitmq}/compose.yml. 24 credential-bearing variables, 16 stacks included by the cloud composition. read 2026-08-07
  7. hashicorp/vault issue #25380, transit auto-unseal without third-party cloud services — the bootstrap paradox for self-hosted unsealers. link ↗ accessed 2026-08-07
Data vintage infra repo as checked out on 2026-08-07 · HashiCorp and OpenBao documentation consulted 2026-08-07
Method static inventory of the compose files and .env.example; classification of each secret by when its consumer reads it; verification of Vault feature availability and licence gating against vendor documentation
Not verified no access to the running hub (SSH firewalled from the authoring network) — no runtime check of first-boot-only behaviour, KMS availability, or host resources
Tools hand-authored inline SVG (no chart library required for this document)
Code & reproducibility counts reproducible with grep -E '^[A-Z_0-9]+=' cloud/.env.example | grep -E 'PASSWORD|SECRET|TOKEN|HASH'
Contact R&D-lab · lab.wbd-rd.nl