Deploying

The docker-compose.yml in Installation is a reference topology — the service definitions, networks and limits to adapt, not a file guaranteed to run unmodified in your environment.

Images

Four images are published to GHCR (ghcr.io/patrickpingol/zilean-*), tagged vX.Y.Z, vX.Y, vX, and latest on every release — see Installation for the full docker-compose.yml.

Image	Contains	Base
zilean-slim	All five non-script binaries — zilean-core, zilean-ui, zilean-worker, zilean-mcp, zilean-webhook. The Compose command: selects which one runs.	alpine + ca-certificates, tzdata
zilean-scripts-bash	The Bash worker only. Installs bash and coreutils explicitly rather than relying on busybox sh, so scripts behave as they do elsewhere.	alpine (+~2.5 MB over slim)
zilean-scripts-node	The Node/TypeScript worker only, with tsx.	alpine + pinned nodejs (+~110 MB)
zilean-scripts-groovy	The Groovy worker only, with Groovy on PATH.	alpine + pinned openjdk17-jre-headless (+~205 MB)

Apache Groovy and tsx are the externally-fetched dependencies in these images. Each is downloaded from its upstream source during the image build and verified against a pinned SHA-256, and neither is resolved through a package manager at build time — so the exact bytes that ship are fixed by the image tag, not by whatever a dependency range resolves to on the day it is built.

Networks

Two networks, and the separation is the point.

• An application network carrying Postgres and every service. It is deliberately not internal: true, because scripts legitimately need outbound internet access — the isolation being bought here is from neighbouring unrelated services, not from the internet.
• An external network joined by zilean-core, zilean-ui, zilean-mcp, and zilean-webhook, so your reverse proxy can reach them.

The three script workers are attached to the application network alone. Neighbouring services on the external network are therefore unroutable from them by construction, rather than by policy.

Note

Docker networks are isolated by default, and a multi-homed container does not forward traffic between its networks — so attaching core and ui to both does not bridge them.

zilean-core and zilean-mcp sharing the application network is also what lets core live-probe zilean-mcp’s health (GET /healthz) for the MCP Settings page’s “not reachable” warning — see MCP. It defaults to http://zilean-mcp:8092/healthz, matching the reference compose file’s service name and MCP_ADDR; set MCP_HEALTH_URL on zilean-core if you renamed that service or changed MCP_ADDR.

Resource limits

These are ceilings, not reservations. The rationale is specific: on a busy host, an unbounded Go service that leaks memory gets a neighbour OOM-killed, because the kernel picks the victim. pids_limit is the fork-bomb ceiling for a careless — not malicious — script.

Service	mem_limit	cpus	pids_limit
zilean-core	512m	1.0	256
zilean-ui	512m	1.0	256
zilean-worker	512m	2.0	512
zilean-mcp	512m	1.0	256
zilean-webhook	512m	1.0	256
zilean-script-worker-bash	256m	1.0	256
zilean-script-worker-node	512m	1.0	256
zilean-script-worker-groovy	1g	1.0	256

Notes on the numbers that aren’t obvious:

• zilean-worker gets 2 CPUs because pool_count fans one firing into concurrent HTTP calls. Its memory stays modest because response bodies are truncated.
• cpus, mem_limit and WORKER_MAX_CONCURRENCY move together. The variable sets how many HTTP jobs zilean-worker runs at once per replica (20 by default, at most 200); raising it without raising the CPU and memory ceilings trades throughput for contention, and it needs a matching connection pool — see Deploying to AWS for the pool_max_conns and max_connections side.
• zilean-webhook matches core/ui/mcp — it serves one public route, so it is sized like the other small HTTP services rather than the worker.
• Bash keeps full CPU and pids despite the small memory limit — bash is the forkiest runtime (every pipeline stage and $(…) is a process), and five concurrent jobs share one pid ceiling.
• Groovy’s 1g is load-bearing. A JVM sizes its heap as a percentage of the container limit, so this number is what -XX:MaxRAMPercentage is a percentage of. Without a container limit, the JVM sizes its heap from total host memory instead.

Script workers also mount a size-capped /tmp

Each of the three script workers mounts a tmpfs at /tmp — 128m for Bash, 256m for Node and Groovy. tmpfs pages are charged to mem_limit rather than added on top of it, and up to five concurrent jobs share both. If you change a script worker’s memory limit, revisit its tmpfs size in step — see Container confinement settings. Leaving the two out of step means a script that fills /tmp can OOM-kill the container instead of getting “no space left on device”.

Size Groovy for concurrency, not for one JVM

The Groovy worker runs up to 5 concurrent jobs, so five concurrent JVMs can each claim 25% of mem_limit — roughly 1.25 GB of heap against a 1 GB limit, before non-heap overhead. If you enable Groovy, confirm the limit is present and at least as large as the reference value.

Groovy also pays ~1–2 s of JVM startup on every invocation, which makes it a poor fit for high-frequency schedules — an every-30-seconds job would spend ~5% of wall clock just starting JVMs. Prefer Bash or JS/TS for anything frequent or short-looped.

Container confinement settings

The three script workers run inside a container hardened the same way, and can additionally sandbox every script with bwrap (bubblewrap) for filesystem and pid-namespace confinement.

Container hardening

All three script workers carry the same always-on settings in the reference Compose file. They are subtractive — they surrender privileges rather than needing anything from the host — so they apply on every host, including one where bwrap cannot run, and they are independent of SCRIPT_SANDBOX_ENABLED:

Service	read_only	tmpfs at /tmp	cap_drop	security_opt
zilean-script-worker-bash	true	size=128m,noexec,nosuid,nodev	ALL	no-new-privileges:true
zilean-script-worker-node	true	size=256m,noexec,nosuid,nodev	ALL	no-new-privileges:true
zilean-script-worker-groovy	true	size=256m,noexec,nosuid,nodev	ALL	no-new-privileges:true

• read_only: true — the image cannot be modified, and /tmp is the only writable path (plus a script volume, if you mount one). Each run gets its own working directory under it, so relative-path writes work; writes to the container’s own filesystem fail. The container’s root filesystem is read-only; with bwrap sandboxing enabled, a script’s writes outside its per-run directory land in the sandbox’s own temporary root and are discarded when the run ends, never reaching the container.
• tmpfs at /tmp — size-capped at 128m for Bash and 256m for Node and Groovy, noexec, nosuid, nodev. The caps are suggested rather than tuned: each is at most half of the worker’s reference mem_limit (256m / 512m / 1g — see Resource limits), because tmpfs pages are charged to that limit rather than added to it, and the rest of the limit is left for the interpreters of up to five concurrent jobs. Keeping the cap below the limit is what lets a script that fills /tmp get no space left on device instead of OOM-killing the container. Because the tmpfs is noexec, a helper the script writes to scratch cannot be executed as a program — chmod +x helper && ./helper fails with permission denied; run it through its interpreter instead (bash helper.sh, node helper.js). A script volume is not mounted noexec. The interpreters live on the read-only rootfs, so Bash, Node/TypeScript via tsx, Groovy, stdin payloads and environment variables are unaffected.
• cap_drop: ALL — empties the capability set.
• no-new-privileges:true — stops setuid or file-capability privilege gain on exec.

This hardening narrows what a script can do to the container, but it does not give per-job separation: every job on a worker runs as the same UID (65532), so concurrent jobs can still read each other’s scratch files, and the image contents remain readable. That separation is what bwrap adds.

Opt-in: bubblewrap sandboxing

Each of the three script workers can run every script under bwrap (bubblewrap) for filesystem and pid-namespace confinement — but only once you opt in. Set SCRIPT_SANDBOX_ENABLED=1 in .env; unset (the default) means these workers run scripts without per-job confinement, with no warning that they’re doing so, because sandboxing an existing self-hosted deployment must never be something an upgrade turns on for you. The hardening above still applies without it: a script can read the whole container filesystem it runs in, and nothing stops it walking outside its own working directory — its writes stay confined to /tmp (and any mounted script volume), but its reads are not. See Installation’s “Recommended: enable script sandboxing” for why you should set it — the short version: without it, one script’s temp files are visible to every other concurrently-running script job.

Once SCRIPT_SANDBOX_ENABLED=1 is set, these three workers also need the two security_opt relaxations that the rest of the stack does not. no-new-privileges:true is not one of them — it is part of the always-on hardening above:

Service	security_opt	Why
zilean-script-worker-bash	seccomp:unconfined, systempaths:unconfined	Runs every script under bwrap
zilean-script-worker-node	seccomp:unconfined, systempaths:unconfined	Runs every script under bwrap
zilean-script-worker-groovy	seccomp:unconfined, systempaths:unconfined	Runs every script under bwrap
All other services	(none)	Execute no operator-supplied code

• seccomp:unconfined — bwrap calls unshare(CLONE_NEWUSER), which Docker’s default seccomp profile blocks for an unprivileged caller.
• systempaths:unconfined — bwrap mounts a fresh /proc for the sandbox, and the kernel refuses that while Docker’s masked /proc paths are mounted over it.

Both loosen the container’s own confinement, by one syscall family and Docker’s /proc masking respectively. Neither loosens the sandbox: what a script can reach is decided by bwrap’s bind list and namespaces, which these settings do not touch.

Host prerequisite: unprivileged user namespaces

A host that restricts unprivileged user namespaces — Ubuntu 23.10 and newer default kernel.apparmor_restrict_unprivileged_userns=1 — creates the namespace with no capabilities in it, and bwrap fails whatever security_opt says. See Installation for how to check and change it. The always-on hardening above is unaffected: it needs no host prerequisite and stays in place whether or not bwrap can run.

Each worker runs a real bwrap self-test once at startup when SCRIPT_SANDBOX_ENABLED=1 is set — not just a check that the binary exists — because the host prerequisite above is exactly the kind of failure a bare existence check misses. If the self-test fails (that sysctl being the most common reason), the worker logs a clear warning and falls back to running scripts without per-job confinement (the container hardening still applies) instead of failing them outright. Check docker compose logs zilean-script-worker-bash zilean-script-worker-node zilean-script-worker-groovy after a deploy for that warning if scripts should be sandboxed but you’re not sure they are.

Script volumes

Script volumes are opt-in. The reference Compose file mounts no volumes directory and sets no SCRIPT_VOLUMES_ROOT, so nothing changes unless you add them. To enable volumes, mount one host directory at /volumes in each script worker and set SCRIPT_VOLUMES_ROOT to it. Every script volume is a direct child directory of that root, so files a script writes survive container recreation. Add the same two entries to zilean-script-worker-bash, zilean-script-worker-node and zilean-script-worker-groovy:

    environment:
      SCRIPT_VOLUMES_ROOT: /volumes
    volumes:
      - /srv/zilean-volumes:/volumes

The host directory itself must be writable by UID 65532, the user all three script worker images run as. Files a script writes are owned by that UID on the host:

mkdir -p /srv/zilean-volumes
sudo chown -R 65532:65532 /srv/zilean-volumes

Once mounted with the sandbox off (the default), every script job on that worker can read and write every volume, and ro is advisory; enable sandboxing when you use volumes. A volume is persistent and is not mounted noexec, so it is the one place a script’s writes survive and where a written file could be executed.

A read-only (:ro) root mount does not work. A root that is unset, missing, or unwritable is treated as not mounted: the worker still starts and runs every job that declares no volume, but it reports volumes_root_mounted: false (logging a warning when the value is set but unusable), and every job that declares a volume fails until the worker is restarted with a writable root. SCRIPT_VOLUMES_ROOT cannot be /. The mount and the variable are container settings that need a redeploy of the script workers. Back the host directory up like any other host data: it is not in Postgres. See Volumes.

Stop behaviour

All eight app services use stop_grace_period: 30s, set above River’s 25-second soft-stop so a graceful shutdown completes before Docker escalates to SIGKILL.

Startup order

zilean-postgres → zilean-core → everything else. Core is first because it alone applies migrations.

Remember that depends_on waits for a container to start, not for migrations to finish. The other services exiting non-zero and restarting until the schema exists is the expected path, and restart: unless-stopped is what makes it converge.

Reverse proxy routing

Core and ui share one origin split by path prefix — same-origin is required there, since the SPA served by ui calls core’s API same-origin through the proxy. zilean-mcp is a separate upstream on its own path, needed only if you’re using the MCP endpoint — see MCP. zilean-webhook is a separate upstream too, serving the public inbound webhook route for jobs whose schedule_type is webhook.

Path	→ Service	→ Port
/api/*	core	8091 (CORE_ADDR)
/prometheus/metrics	core	8091 (CORE_ADDR)
/riverui/*	ui	8090 (UI_ADDR)
/login, /logout, /admin/assets/*, /admin/accept-invite	ui	8090 (UI_ADDR)
/ (everything else)	ui	8090 (UI_ADDR)
/mcp	mcp	8092 (MCP_ADDR)
/api/webhooks/*	webhook	8093 (WEBHOOK_ADDR)

Why the constraints matter:

• Core registers no login routes. An unauthenticated request to core gets redirected to /login — which ui, not core, serves. Put them on different origins and login redirects break.
• /login, /logout, /admin/assets/ and /admin/accept-invite are registered outside the auth middleware so the login form stays reachable. Everything else on ui sits behind it. /logout accepts POST only; a GET to it does not end the session.
• Auth is already enforced in-app. /prometheus/metrics requires the metrics-read permission; /riverui/ requires a write permission, because River UI can retry, cancel and delete jobs — session validity alone would let a read-only user mutate the queue.
• /api/webhooks/* must stay public. It is served by zilean-webhook, not core, and sits outside admin session auth — the per-job token in the URL is the credential, so anything that can reach the path can fire the job it names. Route it straight through rather than behind the admin auth gate, and rate-limit it at the proxy: because it is public and effectively unauthenticated apart from the token, that limit is the primary control. Point WEBHOOK_TRUST_PROXY_HEADERS at it only if the proxy actually sets and sanitizes X-Forwarded-For (see Environment variables). The webhook limiter keys off the leftmost entry only when it is a valid IP; a blank or non-IP entry, or an absent header, falls back to the connection’s remote address. IPv6 clients are keyed by their /64 prefix.
• POST /login and POST /admin/accept-invite deserve the same proxy limit. They are served outside the auth middleware and both reach a store lookup and bcrypt work before any outcome is known, so rate-limit them at the proxy — that limit is the primary control, exactly as it is for /api/webhooks/*. In-process, each route has its own per-IP limiter — IPv4 per address, IPv6 per /64 prefix — admitting 10 POSTs per minute with a burst of 5 by default, configurable with UI_LOGIN_RATE_PER_MINUTE and UI_LOGIN_RATE_BURST, and caps the request body at 64 KiB before any store lookup or bcrypt work. On /login only a POST carrying a valid CSRF token spends a limiter token, so a cross-site page cannot burn your login budget. GET /login is never counted, so normal page loads and session-expiry redirects are unaffected, and saturating one route’s bucket does not block the other. A limited or rejected submission re-renders the login or accept-invite page with the status and a message: 429 “Too many sign-in attempts. Try again in a minute.” with Retry-After: 60; 413 “Form submission too large.” for an oversized body; or 400 “Could not read form submission.” for a malformed one. Those are the login messages; invite acceptance shows equivalent messages that also tell the user to open their invite link again. ui also bounds each connection itself (10 s for request headers, 30 s to read the request, 120 s idle, 64 KiB of headers, 60 s to write the response), so a stalled client cannot pin a handler. The limiters read only the leftmost X-Forwarded-For entry, and only when UI_TRUST_PROXY_HEADERS is true; a blank or invalid entry — or an absent header — falls back to the connection’s remote address. Behind a reverse proxy with UI_TRUST_PROXY_HEADERS unset, every client keys to the proxy’s address and shares one bucket, so an attacker making about 10 POSTs a minute could lock every admin out of sign-in. Leave it false only if every client reaching ui connects directly; otherwise set UI_TRUST_PROXY_HEADERS=true when the proxy sets and sanitizes X-Forwarded-For (overwriting rather than appending to a client-supplied value), and keep the proxy-level /login limit as the primary control (see Environment variables).
• Keep /prometheus/metrics off any internet-facing hostname. It is authenticated, but there is no reason to expose it. Scrape it on an internal hostname. Its own path is also what keeps it from colliding with the admin SPA’s client-side /metrics route — routing that path to ui instead lets the SPA render its Metrics page on a hard refresh too, not just on in-app navigation.

Express the split as two disjoint matchers

Write the routing as /api/* and not /api/*, rather than a specific rule followed by a catch-all. Declaration order does not reliably determine evaluation order in every proxy, and a catch-all that evaluates first will swallow /api/* — producing an empty job list rather than an error. Making the matchers mutually disjoint removes the dependence on ordering entirely.

Validate the generated proxy config before applying it, and confirm /api/* resolves ahead of the catch-all.

Address core by its service name and literal port (e.g. zilean-core:8091). Proxy templates that auto-discover upstreams typically only resolve the container they are attached to.

Verifying a deploy

In this order:

1. Login works end to end — the site root serves the login page, logging in lands in the admin SPA, logging out returns to the login page.
2. /riverui/ renders. Served by ui, whose River client is deliberately never started — worth an explicit check, since no test can cover it.
3. The admin UI’s job list loads. This is the real test of whether /api/* reaches core. The SPA calls its API same-origin, so a wrong prefix shows as an empty list, not an error.
4. A job runs. Use SEED_JOB=1 for one boot, or trigger an existing job manually and watch it in River UI.
5. Script worker isolation is actually in place, before enabling any script job. See Script jobs for what that does and does not cover.

Probe isolation by IP, never by hostname

A hostname probe from an isolated container fails on DNS whether or not isolation exists, so it proves nothing while reading as a pass. Test with addresses.

Rollback

There is no down migration path — dbmigrate/migrations/ contains only *.up.sql. Schema additions are generally additive and an older binary won’t reference new columns, but that is not a guarantee across every migration: at least one has renamed a column, which requires all binaries to be rebuilt and restarted together rather than rolled out incrementally, and needs the rename reversed by hand to roll back.

Check what a version bump actually contains before assuming a rollback is safe.