Architecture

Zilean is one Go module that builds eight service binaries, all sharing one Postgres database and one River job-queue system split into four disjoint queues. Each binary has a narrow, deliberate job.

The services

Service	Owns	Listens on	Runs jobs from
zilean-core	Admin API (/api/*), /prometheus/metrics, the cron scheduler, the retention sweep, first-boot seeding, and all schema migrations	CORE_ADDR (:8091)	— inserts only, never executes
zilean-ui	Admin SPA, embedded River UI (/riverui/), login/logout/accept-invite	UI_ADDR (:8090)	— never executes
zilean-worker	HTTP job execution	—	the default queue
zilean-script-worker-bash	Bash script execution	—	scripts-bash
zilean-script-worker-node	JavaScript + TypeScript execution	—	scripts-node
zilean-script-worker-groovy	Groovy execution	—	scripts-groovy
zilean-mcp	MCP endpoint (/mcp)	MCP_ADDR (:8092)	— inserts only (run-now), never executes
zilean-webhook	Public inbound webhook endpoint (/api/webhooks/{job_id}/{token})	WEBHOOK_ADDR (:8093)	— inserts only (webhook-triggered), never executes

zilean-core must stay at one replica

Core holds the only cron reconciler, and there is no leader election. Two instances would each fire every schedule and double-insert every job.

This is the one scaling change that silently doubles work rather than failing loudly.

Note the reason is the scheduler, not migrations — golang-migrate takes a Postgres advisory lock, so concurrent migration attempts would be safe on their own.

To add execution capacity, scale the workers, not core.

Neither core nor ui executes jobs

Both hold a River client, and neither starts it:

• core inserts jobs and never works them.
• ui only reads (so River UI can list jobs), and its client is deliberately never started.

Only the four worker services consume the queue.

Queue subscription is the isolation boundary

Each worker subscribes to exactly one queue, and the queue sets are disjoint. A Groovy job can never execute in the Node container, because that container is not subscribed to the queue the job was inserted on.

This is enforced rather than assumed — each binary logs its queue names at startup, and each has a test asserting the queue sets stay disjoint.

Concurrency per service:

• zilean-worker — up to 20 concurrent HTTP jobs per replica by default, set by WORKER_MAX_CONCURRENCY (max 200).
• Each script worker — up to 5 concurrent script jobs, independently. A host running all three therefore permits up to 15 concurrent script jobs and opens six connection pools against Postgres.

The three script images also run as an unprivileged user (65532:65532); no other image sets one.

How a job flows

1. A row exists in scheduled_jobs — created through the admin UI or the JSON API.
2. Core’s reconciler notices it on its next tick and inserts a River job onto the queue matching the job’s type. A row edit takes effect here, on the next tick, with no restart.
3. The matching worker picks the job up, executes it, and applies that job’s own loop and retry rules — re-inserting or rescheduling itself as those rules dictate.
4. Every attempt is recorded — invocation counters on the job row, a bucketed activity series in job_call_stats, and River’s own job row with its state, attempt count and last error.
5. Core’s retention sweep ages out old metrics and audit entries on a daily cycle.

Timeouts, layered

Two independent ceilings bound a job, and they are not the same thing:

• exec_timeout_ms (per job, per attempt) bounds one HTTP call or one script process. Capped at 30 minutes.
• River’s JobTimeout is a safety net set to that 30-minute cap plus 5 minutes — it exists to kill a job that somehow outlives its own timeout, not to enforce per-job limits.

On shutdown, workers get a 25-second soft-stop, deliberately under the Compose stop_grace_period of 30 seconds, so in-flight jobs finish gracefully before Docker sends SIGKILL.

Startup ordering

Core applies migrations, so it must come up first. The other seven will start, fail their first query, exit non-zero, and be restarted by Docker until the schema exists.

Convergence comes from restart: unless-stopped, not from depends_on — which waits for a container to start, not for migrations to finish.

Data stored

Table	Holds
scheduled_jobs	The job registry — every job definition.
river_job, river_queue, …	River’s own queue state. Applied by River’s CLI, not by Zilean.
schema_migrations	Zilean’s migration tracking (golang-migrate).
job_call_stats	Per-job invocation buckets, powering activity charts and heartbeats.
worker_metrics	Per-service CPU and memory samples.
global_variables	Values shared across jobs.
users, roles, sessions, invites	Auth state. Sessions are rows, not signed cookies.
audit_log	Recorded administrative actions.
app_settings	Database-backed settings, e.g. audit retention window.

The admin UI

The admin UI is a React 19 + TypeScript SPA built with Vite, served by zilean-ui from assets embedded in the binary. The embedded River UI is a third-party Go package mounted at /riverui/ — it exposes raw queue internals (job rows, states, attempts, errors) and is not read-only: it can retry, cancel and delete jobs, which is why it is gated behind a write permission rather than mere session validity.