This is the abridged developer documentation for Zilean # Zilean > Add, edit, and disable scheduled jobs by writing a Postgres row — no code deploy required. ## Why Zilean [Section titled “Why Zilean”](#why-zilean) Registry-first Every behavior change is a row write, never a code or deploy change. A job’s URL, auth, retry backoff and looping take effect on its **next attempt** — the worker re-reads the row every time. Loop, retry and pooling stay separate Success-path looping, error-path backoff and parallel fan-out are three independent per-job mechanisms. A job can loop without retrying, retry without looping, or do either across a pool of 20. HTTP and script jobs Call an endpoint on a schedule, or run Bash, JavaScript/TypeScript or Groovy. Each runtime executes in its own worker, on its own queue — a job can never run in the wrong runtime’s container. Visibility over automation Per-attempt call history, heartbeats, per-job activity charts, Prometheus metrics per service, an audit log, and the embedded River UI for raw queue internals. ## Start here [Section titled “Start here”](#start-here) [Introduction](/start/introduction/)What Zilean is, who it's for, and how it differs from cron or a workflow engine. [Installation](/start/installation/)Prerequisites, the two migration systems, and first boot. [Your First Job](/start/your-first-job/)Register a job, watch it run, then edit it live. [Architecture](/concepts/architecture/)The eight services, what each owns, and how a job flows through them. # Architecture > The eight services, what each one owns, and how a job flows through them. Zilean is one Go module that builds eight service binaries, all sharing one Postgres database and one River job-queue system split into four disjoint queues. Each binary has a narrow, deliberate job. ## The services [Section titled “The services”](#the-services) | Service | Owns | Listens on | Runs jobs from | | ----------------------------- | --------------------------------------------------------------------------------------------------------------------------------------- | ------------------------ | -------------------------------------------------- | | `zilean-core` | Admin API (`/api/*`), `/prometheus/metrics`, the cron scheduler, the retention sweep, first-boot seeding, **and all schema migrations** | `CORE_ADDR` (`:8091`) | — inserts only, never executes | | `zilean-ui` | Admin SPA, embedded River UI (`/riverui/`), login/logout/accept-invite | `UI_ADDR` (`:8090`) | — never executes | | `zilean-worker` | HTTP job execution | — | the default queue | | `zilean-script-worker-bash` | Bash script execution | — | `scripts-bash` | | `zilean-script-worker-node` | JavaScript + TypeScript execution | — | `scripts-node` | | `zilean-script-worker-groovy` | Groovy execution | — | `scripts-groovy` | | `zilean-mcp` | MCP endpoint (`/mcp`) | `MCP_ADDR` (`:8092`) | — inserts only (run-now), never executes | | `zilean-webhook` | Public inbound webhook endpoint (`/api/webhooks/{job_id}/{token}`) | `WEBHOOK_ADDR` (`:8093`) | — inserts only (webhook-triggered), never executes | ## `zilean-core` must stay at one replica [Section titled “zilean-core must stay at one replica”](#zilean-core-must-stay-at-one-replica) Core holds the **only cron reconciler**, and there is no leader election. Two instances would each fire every schedule and double-insert every job. This is the one scaling change that silently doubles work rather than failing loudly. Note the reason is the *scheduler*, not migrations — `golang-migrate` takes a Postgres advisory lock, so concurrent migration attempts would be safe on their own. To add execution capacity, scale the **workers**, not core. ## Neither core nor ui executes jobs [Section titled “Neither core nor ui executes jobs”](#neither-core-nor-ui-executes-jobs) Both hold a River client, and neither starts it: * **core** inserts jobs and never works them. * **ui** only reads (so River UI can list jobs), and its client is deliberately never started. Only the four worker services consume the queue. ## Queue subscription is the isolation boundary [Section titled “Queue subscription is the isolation boundary”](#queue-subscription-is-the-isolation-boundary) Each worker subscribes to exactly one queue, and the queue sets are disjoint. A Groovy job can never execute in the Node container, because that container is not subscribed to the queue the job was inserted on. This is enforced rather than assumed — each binary logs its queue names at startup, and each has a test asserting the queue sets stay disjoint. Concurrency per service: * `zilean-worker` — up to **20** concurrent HTTP jobs per replica by default, set by `WORKER_MAX_CONCURRENCY` (max 200). * Each script worker — up to **5** concurrent script jobs, independently. A host running all three therefore permits up to 15 concurrent script jobs and opens six connection pools against Postgres. The three script images also run as an unprivileged user (`65532:65532`); no other image sets one. ## How a job flows [Section titled “How a job flows”](#how-a-job-flows) 1. **A row exists** in `scheduled_jobs` — created through the admin UI or the JSON API. 2. **Core’s reconciler** notices it on its next tick and inserts a River job onto the queue matching the job’s type. A row edit takes effect here, on the next tick, with no restart. 3. **The matching worker** picks the job up, executes it, and applies that job’s own loop and retry rules — re-inserting or rescheduling itself as those rules dictate. 4. **Every attempt is recorded** — invocation counters on the job row, a bucketed activity series in `job_call_stats`, and River’s own job row with its state, attempt count and last error. 5. **Core’s retention sweep** ages out old metrics and audit entries on a daily cycle. ## Timeouts, layered [Section titled “Timeouts, layered”](#timeouts-layered) Two independent ceilings bound a job, and they are not the same thing: * **`exec_timeout_ms`** (per job, per attempt) bounds one HTTP call or one script process. Capped at 30 minutes. * **River’s `JobTimeout`** is a safety net set to that 30-minute cap plus 5 minutes — it exists to kill a job that somehow outlives its own timeout, not to enforce per-job limits. On shutdown, workers get a 25-second soft-stop, deliberately under the Compose `stop_grace_period` of 30 seconds, so in-flight jobs finish gracefully before Docker sends `SIGKILL`. ## Startup ordering [Section titled “Startup ordering”](#startup-ordering) Core applies migrations, so it must come up first. The other seven will start, fail their first query, exit non-zero, and be restarted by Docker until the schema exists. Convergence comes from `restart: unless-stopped`, **not** from `depends_on` — which waits for a container to *start*, not for migrations to finish. ## Data stored [Section titled “Data stored”](#data-stored) | Table | Holds | | --------------------------------------- | -------------------------------------------------------------------- | | `scheduled_jobs` | The job registry — every job definition. | | `river_job`, `river_queue`, … | River’s own queue state. Applied by River’s CLI, not by Zilean. | | `schema_migrations` | Zilean’s migration tracking (`golang-migrate`). | | `job_call_stats` | Per-job invocation buckets, powering activity charts and heartbeats. | | `worker_metrics` | Per-service CPU and memory samples. | | `global_variables` | Values shared across jobs. | | `users`, `roles`, `sessions`, `invites` | Auth state. Sessions are rows, not signed cookies. | | `audit_log` | Recorded administrative actions. | | `app_settings` | Database-backed settings, e.g. audit retention window. | ## The admin UI [Section titled “The admin UI”](#the-admin-ui) The admin UI is a React 19 + TypeScript SPA built with Vite, served by `zilean-ui` from assets embedded in the binary. The embedded River UI is a third-party Go package mounted at `/riverui/` — it exposes raw queue internals (job rows, states, attempts, errors) and is **not** read-only: it can retry, cancel and delete jobs, which is why it is gated behind a write permission rather than mere session validity. # Job Types > HTTP and script jobs, the queue each routes to, and which worker runs it. A job’s `job_type` decides what it does, which queue it lands on, and therefore which service executes it. | `job_type` | Queue | Executed by | Interpreter | | ------------ | ---------------- | ----------------------------- | ----------- | | `http` | `default` | `zilean-worker` | — | | `bash` | `scripts-bash` | `zilean-script-worker-bash` | `bash` | | `javascript` | `scripts-node` | `zilean-script-worker-node` | `node` | | `typescript` | `scripts-node` | `zilean-script-worker-node` | `tsx` | | `groovy` | `scripts-groovy` | `zilean-script-worker-groovy` | `groovy` | Anything that isn’t `http` is a script job. `job_type` is immutable It cannot be changed after creation — an attempt returns **409**. An HTTP job cannot become a script job; create a new one. ## Fields are mutually exclusive [Section titled “Fields are mutually exclusive”](#fields-are-mutually-exclusive) The API enforces the separation rather than ignoring irrelevant fields: * A **script** job must have empty `url`, `query_params`, `headers` and `body`, and `auth_type: "none"`. * An **HTTP** job must have empty `script_body` and `script_args`. ## Unknown types fail closed [Section titled “Unknown types fail closed”](#unknown-types-fail-closed) An unrecognised script type routes to a queue that **nothing subscribes to**. The job sits unworked rather than being executed by whichever worker happened to be available. The same shape applies to a runtime you simply haven’t deployed: a Groovy job with no Groovy worker running queues indefinitely, visible in River UI as stuck in `scheduled`/`available`. Deploy only the runtimes you use, but know that’s the consequence. ## Queue subscription is the security boundary [Section titled “Queue subscription is the security boundary”](#queue-subscription-is-the-security-boundary) Each worker subscribes to exactly one queue, and the sets are disjoint — enforced by a test, and each binary logs its queues at startup so you can verify it. This matters because the HTTP worker and the script workers have deliberately different network reach. A script job can never execute in the HTTP worker’s container, because that container is not subscribed to any script queue. It’s structural, not a policy check that could be bypassed. ## Choosing a runtime [Section titled “Choosing a runtime”](#choosing-a-runtime) * **Bash** is the lightest. It is also the forkiest — every pipeline stage and `$(…)` is a process — so it’s the runtime most likely to hit a process-count ceiling. * **JavaScript / TypeScript** share the Node worker. TypeScript runs through `tsx`, so no separate build step is needed. * **Groovy** pays **1–2 seconds of JVM startup on every invocation**, and enforces a minimum `exec_timeout_ms` of 10 seconds for that reason. Don’t schedule Groovy frequently An `@every 30s` Groovy job spends roughly 5% of wall clock just starting JVMs. Prefer Bash or JS/TS for anything frequent or short-looped. Groovy’s dependency fetching (`@Grab`/Grape) is **disabled at the interpreter**, not merely discouraged — a script cannot pull from a Maven repository on every tick. ## What’s shared [Section titled “What’s shared”](#whats-shared) Both kinds use the same machinery for everything else: schedule types, loop, retry, pooling, overlap policy, global variable interpolation, timeouts, invocation counters and activity history. The differences are only in how the work is performed and how the loop signal is read: | | HTTP | Script | | --------------------- | --------------------------------------------------- | ------------------------------------ | | Success | Status `200–299` | Exit code `0` | | Loop signal read from | Response body | **stdout** | | Pool index surface | `pool_idx` query param (only when `pool_count > 1`) | `ZILEAN_POOL_IDX` env var (always) | | Error snippet kept | 500 chars | 2000 chars (stack traces are longer) | | Extra permission | — | `jobs:script` | See [HTTP jobs](/guides/http-jobs/) and [Script jobs](/guides/script-jobs/). # Loop, Retry & Pooling > Three independent per-job mechanisms — what each does, and how they compose. These are three separate mechanisms with three separate purposes. Zilean deliberately never merges them into one “retry policy.” | | Triggered by | Purpose | | ----------- | ------------ | --------------------------------------------------------------------- | | **Loop** | **Success** | Keep calling while there’s more work — drain a queue, page a backlog. | | **Retry** | **Failure** | Back off and try again. | | **Pooling** | A firing | Do *N* of it at once, in parallel. | ## Loop [Section titled “Loop”](#loop) When `loop_enabled` is true, a **successful** attempt inspects the response and decides whether to run again. ### How the decision is made [Section titled “How the decision is made”](#how-the-decision-is-made) The response body is evaluated with [gjson](https://github.com/tidwall/gjson) at the path in `loop_response_path` (default `result.remaining`): * The loop **continues** only if the path resolves to a **strictly positive number**. * The loop **stops** if the path is absent, or the value is zero or negative. For a script job, the same evaluation is applied to the script’s **stdout**, with no script-specific handling. gjson paths, not JSONPath `loop_response_path` uses gjson syntax — dot notation, array indices, wildcards, `#` queries. `result.remaining`, `data.items.#`, `queue.0.depth`. It is not JSONPath and has no leading `$`. Because the check reads gjson’s numeric field, a JSON `true` or a quoted `"5"` does not continue the loop — return an actual number. ### How the next iteration is scheduled [Section titled “How the next iteration is scheduled”](#how-the-next-iteration-is-scheduled) A continuation is a **fresh queue insert**, scheduled at: ```plaintext delay = loop_delay_ms + random(0, loop_jitter_ms) ``` A flat delay plus jitter — completely independent of the retry backoff settings. Note that `loop_jitter_ms` has no applied default: leaving it at `0` means no jitter at all. The pool index is carried forward unchanged, so a pool member’s loop stays addressed to its own index for the life of that run. ### Hitting the iteration cap [Section titled “Hitting the iteration cap”](#hitting-the-iteration-cap) `loop_max_iterations` (default 50) bounds **one logical run**, not the job’s lifetime — the iteration counter resets to 0 on the next scheduled firing. When the cap is hit: * An error is logged (`loop max iterations hit, stopping`) with the job id, name, pool index and cap. * The overlap lock is released and any one-off completion is applied. * **The queue job is not failed.** It returns cleanly, so there is no retry and no discarded job. There is no alerting on this yet Hitting the cap produces a log line only. No Slack, no email, no notification. If a looping job silently stops making progress, the log is the only signal. ## Retry [Section titled “Retry”](#retry) When `retry_enabled` is true, a **failed** attempt is rescheduled with exponential backoff. Zilean overrides River’s default retry policy entirely. ### The backoff formula [Section titled “The backoff formula”](#the-backoff-formula) ```plaintext delay = min( retry_base_delay_ms × retry_multiplier ^ (attempt − 1), retry_max_delay_ms ) ``` `attempt` is 1-based. **There is no jitter on the retry path** — unlike loop. Retry settings are re-read from the registry on every retry decision, so editing them takes effect on the next backoff calculation. Values that are unusable are coalesced: base `≤ 0` → 1000 ms, multiplier `≤ 1` → 2, max delay `≤ 0` → 5 minutes. ### Attempt budget [Section titled “Attempt budget”](#attempt-budget) `retry_max_attempts` (default 5) becomes the queue job’s `MaxAttempts`: * `retry_enabled: false` → **1** attempt. A failure is cancelled outright rather than handed to any retry schedule. * `retry_max_attempts ≤ 0` → 5. `MaxAttempts` is fixed at insert time Editing `retry_max_attempts` affects jobs inserted **after** the change. Jobs already queued or mid-retry keep the budget they were created with. ### What counts as a failure [Section titled “What counts as a failure”](#what-counts-as-a-failure) For an HTTP job: any status outside `200–299`, plus transport and body-read errors. For a script job: a non-zero exit code, or a timeout. A **configuration** error — an unparseable URL, an unknown auth type, a missing interpreter, an unsupported job type — is cancelled rather than retried. It will never succeed, so it doesn’t burn the retry budget. ## Pooling [Section titled “Pooling”](#pooling) `pool_count` (default 1, max 20) fans **one firing** out into that many parallel jobs, one per pool index `0 … pool_count − 1`. Each pool member is fully independent: its own queue job, its own overlap lock, its own loop, its own retry budget. One member being blocked does not prevent the others from firing. ### How a job knows its index [Section titled “How a job knows its index”](#how-a-job-knows-its-index) | Job type | Surface | | -------- | -------------------------------------------------------------------------------------------------------------------------------------------- | | HTTP | The `pool_idx` query parameter — added **only when `pool_count > 1`**, so a non-pooled job’s request is byte-identical to a pre-pooling one. | | Script | The `ZILEAN_POOL_IDX` environment variable — set **always**, even at `pool_count: 1`. | The index is also carried through every loop continuation unchanged. ### Partial firings are normal [Section titled “Partial firings are normal”](#partial-firings-are-normal) If some indices are already running and others aren’t, the firing proceeds for the ones that are free and logs which were skipped. It only reports “run in progress” (HTTP **409** on a manual trigger) when **every** index was already locked. One-off jobs must have `pool_count: 1`. ## Overlap policy [Section titled “Overlap policy”](#overlap-policy) `overlap_policy` decides what happens when a job is still running as its next firing comes due. | Value | Behavior | | ---------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `skip` (default) | A lock is taken per pool index before inserting. If that index is still running, **that index’s firing is dropped**. The lock is held for the entire run — every loop iteration and every retry — and released only when that member has fully finished. | | `allow` | Fire regardless. No lock is ever taken. | Under `skip`, the lock is a per-index entry in `running_since_by_pool`, acquired with a single guarded statement so two concurrent firings cannot both decide the job is free. Locks are released on every terminal path, including retry exhaustion — which is handled by a dedicated error handler, because when River exhausts attempts it discards the job *without* calling the worker again. Without that handler, the lock would sit until the one-hour stale timeout. ## How they compose [Section titled “How they compose”](#how-they-compose) * **Loop and retry are orthogonal axes.** Each loop iteration is its own queue job with its own attempt budget, so a failing iteration retries *within* that iteration. The loop path never goes through the retry scheduler. * **A live loop holds its overlap lock.** The lock is not released between iterations, and its timestamp is refreshed on every attempt — success or failure — so a genuinely progressing multi-hour loop never trips the stale-lock backstop. * **Pooling multiplies both.** Each of the `pool_count` members loops and retries on its own, and under `skip` holds its own independent lock. * **Every attempt counts separately.** A loop iteration and a retry attempt are each their own invocation, so the count grows with the work a job does rather than with the number of firings. An [invocation alarm](/guides/monitoring/#invocation-alarms) reads that count the same way, whether it watches every attempt or failures only. # Scheduling > Cron, rate, one-off and webhook schedules, and how the reconciler applies row edits. Every job has a `schedule_type` that decides which fields govern when it fires. | `schedule_type` | Fields it uses | | --------------- | ---------------------------------------------------------------------------------------- | | `cron` | `cron_expression` | | `rate` | `cron_expression`, as an `@every ` descriptor | | `one_off` | `run_at`, `run_at_fired_at`, `one_off_after_run` — `cron_expression` is ignored entirely | | `webhook` | None — it holds no schedule fields, and fires only on an inbound `POST` | ## Cron [Section titled “Cron”](#cron) Expressions are parsed by [robfig/cron v3](https://pkg.go.dev/github.com/robfig/cron/v3) in **standard 5-field form**: ```plaintext minute hour day-of-month month day-of-week ``` Seconds are **not** enabled — a 6-field expression is not valid here. Descriptors are supported: `@hourly`, `@daily`, `@weekly`, `@monthly`, `@every `. Expressions are validated when you write the row, so a malformed expression is rejected at the API rather than silently failing later. ### Timezone [Section titled “Timezone”](#timezone) Cron expressions are evaluated in `zilean-core`’s **local time**, as set by the `TZ` environment variable. There is no per-job timezone and no scheduler-wide override. Because this is invisible when wrong, the server reports what it actually resolved: ```bash curl https:///api/meta # {"timezone":"America/Los_Angeles"} ``` ## Rate [Section titled “Rate”](#rate) A rate schedule is written into the same `cron_expression` column as an `@every` descriptor: ```plaintext @every 90s @every 5m @every 1h30m ``` The shape is validated against `@every `, and the duration itself is parsed by the cron library. There is no separate code path — rate rows go through the identical reconciler as cron rows. ## One-off [Section titled “One-off”](#one-off) A one-off job fires exactly once, at `run_at`. * It is **never** cron-scheduled; the reconciler skips one-off rows outright. * It is inserted directly onto the queue with a scheduled delivery time of `run_at`, the moment it becomes eligible — on create, on update, or on being enabled. * `run_at_fired_at` is the “already inserted” guard. It is set at **insert** time, not run time, and is server-managed — ignored on create and update. * The queue insert and the `run_at_fired_at` marker commit together in one transaction. If the insert fails, the job stays unfired and can be retried, and it is never inserted twice. * `run_at` must be in the future when the job has not yet fired, and becomes **immutable** once it has. * Switching `schedule_type` away from `one_off` clears `run_at_fired_at`. * `pool_count` must be `1`. * `POST /jobs/{id}/run` is rejected with **409** for a one-off job — a manual trigger would run it early, which is precisely what a one-off schedule exists to prevent. ### What happens after it runs [Section titled “What happens after it runs”](#what-happens-after-it-runs) `one_off_after_run` decides: | Value | Behavior | | ---------------- | ------------------------------------------------------------------- | | `keep` (default) | The run finishes, the row is disabled, and it stays for inspection. | | `delete` | The row is removed once the run finishes. | This is applied on **every** terminal path, including retry exhaustion, so a failing one-off is still cleaned up according to its setting. Unknown values fall through to `keep`. ## Webhook [Section titled “Webhook”](#webhook) A webhook job fires when something outside Zilean sends it an HTTP `POST`. There is no time or interval involved — the inbound request is the trigger. * It is **never** cron-scheduled; the reconciler skips webhook rows outright, exactly as it skips one-off rows. No cron entry is ever created for it. * `cron_expression` and `run_at` are both meaningless for it. A leftover `cron_expression` is accepted and ignored rather than rejected. * Each webhook job has its own URL, `POST /api/webhooks/{job_id}/{token}`. The token in the path is the credential — the route sits outside admin session auth, so anything holding the URL can fire the job. * The token is generated the moment `schedule_type` is set to `webhook`. It is stored only as a sha256 digest, so the plaintext is shown **exactly once** — in the response to the create or update that set the schedule type, or from a rotate action. Afterwards the job carries an 8-character `webhook_token_hint` so you can tell which token is in use. * **Rotating the token invalidates the old URL immediately**, with no redeploy. Rotation is an explicit action — from the admin UI or `POST /api/jobs/{id}/webhook/rotate` — and a routine edit of the job never rotates the token, so a URL a third party already holds keeps working across unrelated edits. * `POST /jobs/{id}/run` is rejected with **409**, the same as a one-off job: a manual trigger would fire it with an empty payload, and the run’s source should stay unambiguous. * `pool_count` and `overlap_policy` work exactly as for any other schedule type. A webhook job can have `pool_count > 1`, and under the default `overlap_policy: skip`, a call that arrives while a previous loop is still in flight returns **409** — retryable by whatever is sending the webhook. * **Webhook chains are capped at 8 hops.** A webhook job can itself call another webhook job, and every HTTP request a run makes carries `X-Zilean-Webhook-Hops` — `1` when the run was not webhook-triggered, otherwise the inbound hop count plus one. Zilean sets that header itself, overriding any header of the same name configured on the job. The receiving endpoint refuses a request that arrives at **8 or more hops**, so a chain that loops back on itself terminates instead of running forever. Workflow HTTP nodes do not send the header — a workflow cannot be webhook-triggered. * A job cannot target **its own** webhook URL. Saving an existing job — an edit or an import update — whose URL points at its own `/api/webhooks/…` endpoint is rejected with **400**. The check inspects the URL as written, including a templated host such as `{{variable}}` in the host position, but it cannot see a path assembled from variables at call time; the webhook hop limit bounds that case. A create is not checked, because the token does not exist yet at that point. ### Payload [Section titled “Payload”](#payload) The inbound request body and its metadata reach the triggered job: * For an **HTTP** job, they are available as flat substitution variables, the same `{{name}}` mechanism as global variables: `{{webhook_body}}`, `{{webhook_event}}`, `{{webhook_content_type}}`, `{{webhook_request_id}}`, `{{webhook_received_at}}` and `{{webhook_job_id}}`. * For a **script** job, the body arrives on **stdin**, and the metadata as `ZILEAN_WEBHOOK_*` environment variables. The payload never reaches `script_args` or the command line, and is never shell-interpolated — it can change what data the already-chosen script reads, never what the script itself does. * The inbound hop count is among those variables: `ZILEAN_WEBHOOK_HOPS` is set on a webhook-triggered script run and absent otherwise. It is **advisory** — a script that forwards the call to another webhook must send `X-Zilean-Webhook-Hops` itself, incrementing the value it received and treating an absent value as `0`. Both survive a retry and a loop continuation: the payload and metadata are forward-copied into follow-up loop iterations. ### Request handling [Section titled “Request handling”](#request-handling) `zilean-webhook` enforces these itself, in addition to whatever your reverse proxy does: * The request body is capped at **256 KiB**. * The request must carry a **JSON `Content-Type`**; anything else is a **415**. * Every lookup failure returns the **same 404** — a wrong token, an unknown or disabled job, and a disabled group are indistinguishable, so the endpoint never reveals which case applied. * A request whose `X-Zilean-Webhook-Hops` header is **8 or more** is refused with a **508** (`webhook hop limit exceeded`) and the job never fires. An absent or malformed header counts as `0`, and the hop check never reveals whether a job exists — a bad token or unknown job still gets the same **404**. * A call that arrives while a run is already in flight for that pool index is a **409**, retryable by the sender. * A successful call is a **202**. The payload is never logged or echoed — only the job id, byte size, content type and a sha256 of the body. Because this route is public and effectively unauthenticated apart from the per-job token, **the reverse-proxy rate limit is the primary control**, with an in-process per-IP limiter as defense-in-depth. The same layering applies to the admin UI’s unauthenticated login and invite-acceptance POSTs: each is per-IP rate limited (by default 10 POSTs per minute, burst 5, configurable) and body-capped in-process, and the proxy limit in front of them remains the primary control. With `UI_TRUST_PROXY_HEADERS` unset, clients behind a reverse proxy share one bucket. See [Deploying](/guides/deploying/) for the routes and [Environment Variables](/reference/environment-variables/) for `WEBHOOK_TRUST_PROXY_HEADERS` and `UI_TRUST_PROXY_HEADERS`. ## The reconciler [Section titled “The reconciler”](#the-reconciler) `zilean-core` reconciles the registry against its in-memory cron schedule **every 30 seconds**. This interval is not configurable. It also reconciles immediately on startup. What happens each tick: * Enabled jobs are re-read from the registry. * A job whose `cron_expression` is **unchanged** is left alone, so its next-run timer is not reset by an unrelated edit. * A job whose expression **changed** has its entry removed and re-added. * An invalid expression is logged and skipped. * Jobs that are disabled, deleted, or gated behind a disabled group have their entries removed. ### Why most edits take effect immediately [Section titled “Why most edits take effect immediately”](#why-most-edits-take-effect-immediately) Only `cron_expression` is cached by the scheduler. The queued job payload carries just `{job_id, loop_iteration, pool_index}`, and the worker **re-reads the job row on every attempt**. So a change to the URL, method, headers, auth, loop settings, retry settings or script body takes effect on the **very next attempt** — no restart, no reconcile wait. A change to the schedule itself takes effect within 30 seconds. ### Double-fire protection [Section titled “Double-fire protection”](#double-fire-protection) Inserts are made with a uniqueness guard scoped to the reconcile interval, so the same firing cannot be duplicated across two close ticks. A manual trigger deliberately skips this — it should fire now, not deduplicate against a recent cron tick. ## Disabling jobs [Section titled “Disabling jobs”](#disabling-jobs) Disabled state is checked twice, deliberately: 1. The reconciler only schedules enabled jobs (and skips jobs whose group chain is disabled). 2. The worker **re-checks** `enabled` and the group chain at execution time and exits cleanly if either is off. So disabling a job stops work already queued, not just future firings. ## Retention sweeps [Section titled “Retention sweeps”](#retention-sweeps) Core runs four cleanup passes once at startup and then every 24 hours: | Sweep | Retention | | ---------------- | ---------------------------------------------------------------------------------- | | `worker_metrics` | 30 days | | `job_call_stats` | 30 days | | Expired sessions | 24-hour grace | | `audit_log` | Runtime-configurable (see [Users & permissions](/concepts/users-and-permissions/)) | ## Stale lock backstop [Section titled “Stale lock backstop”](#stale-lock-backstop) If a worker process crashes mid-attempt, its overlap lock would otherwise be held forever. A lock older than **one hour** is stolen by the next firing. A legitimately long-running job never trips this: the lock timestamp is refreshed on every attempt, so a multi-hour loop stays fresh. See [Loop, retry & pooling](/concepts/loop-retry-and-pooling/). # Users & Permissions > How authentication, roles and the permission model work. Zilean is multi-user. Accounts, roles and sessions all live in the database; only the single Super Admin is bootstrapped from the environment. ## Authentication [Section titled “Authentication”](#authentication) Login is a form POST to `/login` served by `zilean-ui`. Passwords are hashed with **bcrypt** at its default cost, everywhere — login, invite acceptance, admin resets and self-service changes. An unknown email is compared against a fixed dummy hash so the bcrypt cost is paid either way, and every failure renders the same message. There is no way to tell “no such account” from “wrong password” by response content. ### Sessions are database rows [Section titled “Sessions are database rows”](#sessions-are-database-rows) A session is a row, not a signed cookie: * The token is 32 random bytes. Only its **SHA-256 hash** is stored — the raw token is never persisted. * The cookie is `river_admin_session`, set `HttpOnly`, `Secure`, `SameSite=Lax`, `Path=/`. * **TTL is 24 hours**, refreshed only at login. There is no sliding renewal. * There is no signing secret to configure. Sessions are revoked on logout, when a user is disabled or deleted, on an admin password reset, and by a daily sweep of expired rows. A session whose user is no longer active is rejected by the middleware regardless — disabling a user takes effect immediately, without relying on the revocation call. Unauthenticated API requests get a 302, not a 401 Every authentication failure redirects to `/login`, including requests to `/api/*`. An API client that follows redirects will receive **login-page HTML with a 200 status**, not an error. The admin UI detects exactly this and hard-navigates to the login page. A script you write against the API should check whether it landed on the login page rather than trusting the status code. ### Forced password change [Section titled “Forced password change”](#forced-password-change) A user created by an admin with a temporary password must change it on first login. Until they do, `/api/*` requests return: ```plaintext 403 {"error":"password change required"} ``` Exempt: `GET /api/account/me`, `POST /api/account/change-password`, and static assets — enough for the UI to show the change-password interstitial and submit it. ## The Super Admin [Section titled “The Super Admin”](#the-super-admin) The one Super Admin is seeded from `ADMIN_USERNAME` and `ADMIN_PASSWORD_HASH` on first boot only. * If any row already has the super-admin flag, seeding **returns immediately**. The database is authoritative after first boot; changing the environment variables later has no effect. * `ADMIN_USERNAME` must be an email address or core refuses to start. See [Installation](/start/installation/). Beyond that one-time bootstrap, more than one account can hold the super-admin flag: being assigned the **Super Admin** role sets it — whether by accepting an invite into that role, by being created directly into it, or by a later role change — since that role carries no permission rows of its own (see [Built-in roles](#built-in-roles) below); its grant is computed from the flag, not from role rows, so without this the account would activate with zero effective permissions. Only an existing Super Admin can assign someone the Super Admin role, or any role whose permissions exceed their own — a caller holding only `invites:create` or `users:write` can’t use either to escalate themselves or anyone else past what they already have. Only the bootstrapped account is protected: role changes, status changes and deletion all return **409** against it, as does an admin password reset targeting it. A Super Admin granted the role later (invite, direct creation, or a role change) is a regular user in every other respect — another Super Admin can demote, disable, reset the password of, or delete that account like any other. Internally this is a second flag, set only by the one-time bootstrap seed and never by the invite/create/role-change paths that can grant the super-admin flag itself. Separately, an invite cannot be minted for an already-active user regardless of super-admin status. (That guard closes an account-takeover path available to a caller holding only `invites:create`.) ### How the Super Admin relates to permission checks [Section titled “How the Super Admin relates to permission checks”](#how-the-super-admin-relates-to-permission-checks) Two distinct behaviours: * **On a check**, `IsSuperAdmin` short-circuits to allowed without consulting any permission set. The role isn’t even loaded. * **On enumeration** — `GET /api/account/me` — every permission key is returned, so the UI’s gating works. ## Permissions [Section titled “Permissions”](#permissions) There are 13 permission keys. They live in Go, deliberately **not** in a database table, so adding one automatically grants it to the Super Admin without a migration. See the [permissions reference](/reference/permissions/) for the full table of what each one gates. Enforcement is per route, returning: ```plaintext 403 {"error":"you do not have permission to perform this action"} ``` The one exception is `jobs:script`, which can’t be a route-level check because the job type is only known after the body is decoded. It’s enforced inside the create and update handlers instead. ## Built-in roles [Section titled “Built-in roles”](#built-in-roles) A user can hold more than one role, and always holds at least one — the admin UI’s user form is a checkbox multi-select rather than a single dropdown, and the API rejects a create or update that would leave a user with zero roles. Effective permissions are the **union** of every assigned role’s permissions: holding Read-Only plus a custom `jobs:script`-only role grants both, not whichever role was assigned “last.” Three system roles are seeded and cannot be renamed or deleted. Super Admin’s permissions also can’t be touched — its access is computed from the user’s super-admin flag, not from role rows, so there’s nothing meaningful to edit. Admin and Read-Only can have their permissions edited (by a caller holding `roles:write`) even though they can’t be renamed or deleted; the table below shows each one’s seeded defaults. | Role | Permissions | | --------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Super Admin** | **None stored.** Its behaviour is computed from the user’s super-admin flag, not from role rows. | | **Admin** | `jobs:read`, `jobs:write`, `workflows:read`, `workflows:write`, `globalvars:read`, `globalvars:write`, `metrics:read`, `audit:read` (seeded default; editable) | | **Read-Only** | `jobs:read`, `workflows:read`, `globalvars:read`, `metrics:read`, `users:read`, `roles:read` (seeded default; editable) | Five permissions are in no built-in role `jobs:script`, `users:write`, `roles:write`, `invites:create` and `audit:write` are carried by **no seeded role**. They reach a non-Super-Admin only through a custom role you create — which is the intended design, not an oversight. Custom roles are created under **Settings → Roles**, with a checkbox grid built from the live permission list (`GET /api/roles/permissions`). A `roles:write` holder can only **add** a permission to a role if they hold it themselves; attempting otherwise returns **403**. Permissions already on a role can be left in place, renamed around or removed, so a lower-privileged role editor can still maintain a role a higher-privileged admin built. Super Admins are exempt. ## River UI needs a write permission [Section titled “River UI needs a write permission”](#river-ui-needs-a-write-permission) The embedded River UI at `/riverui/` is gated behind **`jobs:write`**, not mere session validity, because it is not read-only — it can retry, cancel and delete queued jobs. Gating it on session validity alone would let a Read-Only user mutate the queue. ## Audit log [Section titled “Audit log”](#audit-log) Actions are recorded across jobs, workflows, groups, global variables, users, roles, invites, authentication, and settings such as SMTP, MCP tokens, notification targets, job alarms and metrics thresholds. A manual run of a job (**Run now**, `POST /api/jobs/{id}/run`, or the MCP `zilean_run_job` tool) is recorded as `job.run` on the `job` resource type when it succeeds; a rejected run is not recorded. Each entry holds the actor’s id and email, the action, resource type and id, before/after JSON snapshots, the request IP and a timestamp. The actor id deliberately has **no foreign key** to `users`, so entries survive the deletion of the user who made them. Two gaps worth knowing: * **Failed logins are only recorded for emails that resolve to a real user.** An unknown email has no actor to attach, and recording it would make the audit log an email-enumeration oracle. * **Changing the audit retention setting is not itself audited.** Retention defaults to 180 days, seeded once from `DEFAULT_AUDIT_RETENTION_DAYS` into a database setting. After that it’s changed under **Settings → Audit log** by anyone with `audit:write`, accepting 1–3650 days. The daily sweep re-reads the setting each run. Reads are keyset-paginated newest-first, filterable by actor, action, resource type and time range. The Audit Log filter lists every resource type. Page size is fixed at 50 and is not client-tunable. ## What’s managed where [Section titled “What’s managed where”](#whats-managed-where) | Managed in the admin UI | Bootstrapped from the environment | | ----------------------- | ------------------------------------------------ | | Users, roles, invites | The single Super Admin’s email and password hash | | Audit retention window | Its initial default | | Global variables | — | See [Managing users](/guides/managing-users/) for the invite and temp-password flows. # CSV Import > Bulk-create or bulk-update jobs from a spreadsheet instead of one at a time. CSV import lets you create or update many jobs in one pass, from a spreadsheet, instead of going through the job form one job at a time. It’s the same create/update path as the admin UI and the API — every imported row runs through the same validation and defaults, so an imported job can’t bypass a rule a hand-authored one follows. In the admin UI, open the Jobs page and click **Import CSV**, next to **New job**. ## Getting the template [Section titled “Getting the template”](#getting-the-template) ```bash curl https:///api/jobs/import/template -o jobs-template.csv ``` The **Import CSV** modal links to the same download. The template’s header row covers every settable job field — name, scheduling, HTTP or script config, loop, retry, pooling, and so on — plus one filled-in example row to copy from. See [Job Fields](/reference/job-fields/) for what each column means and how it’s validated; the CSV columns map directly to those fields. ## Volumes in a CSV [Section titled “Volumes in a CSV”](#volumes-in-a-csv) The `capabilities_json` column holds a script job’s volume declaration as JSON, for example `{"volumes":[{"name":"reports","mode":"rw"}]}`. Leave it blank for no declaration. An import row runs the same validation as the API: the volumes must exist, the mode cannot exceed a volume’s `max_mode`, and a row for an HTTP job with a non-empty value is reported as a per-row error. See [Volumes](/guides/script-jobs/#volumes). The column is part of the downloaded template. ## Uploading [Section titled “Uploading”](#uploading) ```bash curl -X POST https:///api/jobs/import \ -F file=@jobs.csv \ -F on_collision=merge \ -F merge_priority=source ``` The response streams as newline-delimited JSON — one row result per line, printed as soon as that row finishes — rather than a single JSON body, so `curl` prints each row’s outcome as it happens instead of all at once at the end. A row is matched against existing jobs by **`group_name` + `name`**. No match is a plain create. A match is a **collision**, handled according to `on_collision`: | `on_collision` | Effect | | -------------- | ----------------------------------------------------- | | `overwrite` | Replace the existing job’s fields with the CSV row’s. | | `skip` | Leave the existing job untouched. | | `merge` | Combine the existing job and the CSV row — see below. | ## Merge priority [Section titled “Merge priority”](#merge-priority) `merge` also needs `merge_priority`, `source` (the CSV row wins) or `target` (the existing job wins). Priority only decides the outcome when **both sides have a non-blank value for a field** — if a field is blank on one side, the other side’s value is used regardless of priority. A blank cell never wipes out an existing value in a merge. ## `job_type` can’t change via import [Section titled “job\_type can’t change via import”](#job_type-cant-change-via-import) Just as editing a job in the UI can’t change its `job_type` after creation, an import row that would change an existing job’s type across a collision is not applied — it’s reported as a per-row error instead, and the rest of the batch continues. ## Script jobs still need the script permission [Section titled “Script jobs still need the script permission”](#script-jobs-still-need-the-script-permission) A CSV row with `job_type` set to `bash`, `javascript`, `typescript`, or `groovy` needs the same **`jobs:script`** permission a script job created directly in the UI needs (see [Script Jobs](/guides/script-jobs/)). A script row uploaded without that permission is reported as a per-row error — it doesn’t fail the rest of the import. ## `group_name` limits [Section titled “group\_name limits”](#group_name-limits) A `group_name` is limited to 255 bytes and 16 levels deep — the name is normalized first (each `/`-separated segment is trimmed of whitespace and blank segments are dropped), and the limits apply to the normalized name, so `a/ /b` is 2 levels. A row over either limit is always reported as a per-row error and never skipped, whatever the `on_collision` policy, with the message `group_name must be at most 255 bytes` or `group_name must be at most 16 levels deep`; the rest of the batch continues. An existing job whose `group_name` is over the limits cannot be repaired by import: the row’s own `group_name` is validated before the collision lookup, so an over-limit value is a row error. Fix it through the job’s edit form or a `PUT` update. ## Nothing aborts the batch, and progress is live [Section titled “Nothing aborts the batch, and progress is live”](#nothing-aborts-the-batch-and-progress-is-live) Every row in the file gets its own outcome — created, updated, skipped, or error. The response streams one result per row as it happens, rather than waiting for the whole file to finish, so the admin UI’s import screen fills in its results table live, row by row, while the upload is still running. One malformed or rejected row never blocks the rows around it — it just reports its own error and the rest of the batch keeps going. # Deploying > Docker image targets, Compose topology, resource limits, and reverse-proxy routing. The `docker-compose.yml` in [Installation](/start/installation/) is a **reference topology** — the service definitions, networks and limits to adapt, not a file guaranteed to run unmodified in your environment. ## Images [Section titled “Images”](#images) Four images are published to GHCR (`ghcr.io/patrickpingol/zilean-*`), tagged `vX.Y.Z`, `vX.Y`, `vX`, and `latest` on every release — see [Installation](/start/installation/) for the full `docker-compose.yml`. | Image | Contains | Base | | ----------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------ | | `zilean-slim` | **All five** non-script binaries — `zilean-core`, `zilean-ui`, `zilean-worker`, `zilean-mcp`, `zilean-webhook`. The Compose `command:` selects which one runs. | `alpine` + `ca-certificates`, `tzdata` | | `zilean-scripts-bash` | The Bash worker only. Installs `bash` and `coreutils` explicitly rather than relying on busybox `sh`, so scripts behave as they do elsewhere. | `alpine` (+\~2.5 MB over slim) | | `zilean-scripts-node` | The Node/TypeScript worker only, with `tsx`. | `alpine` + pinned `nodejs` (+\~110 MB) | | `zilean-scripts-groovy` | The Groovy worker only, with Groovy on `PATH`. | `alpine` + pinned `openjdk17-jre-headless` (+\~205 MB) | Apache Groovy and `tsx` are the externally-fetched dependencies in these images. Each is downloaded from its upstream source during the image build and verified against a pinned SHA-256, and neither is resolved through a package manager at build time — so the exact bytes that ship are fixed by the image tag, not by whatever a dependency range resolves to on the day it is built. ## Networks [Section titled “Networks”](#networks) Two networks, and the separation is the point. * **An application network** carrying Postgres and every service. It is deliberately *not* `internal: true`, because scripts legitimately need outbound internet access — the isolation being bought here is from *neighbouring unrelated services*, not from the internet. * **An external network** joined by `zilean-core`, `zilean-ui`, `zilean-mcp`, and `zilean-webhook`, so your reverse proxy can reach them. The three script workers are attached to the application network **alone**. Neighbouring services on the external network are therefore unroutable from them by construction, rather than by policy. `zilean-core` and `zilean-mcp` sharing the application network is also what lets core live-probe `zilean-mcp`’s health (`GET /healthz`) for the MCP Settings page’s “not reachable” warning — see [MCP](/guides/mcp/). It defaults to `http://zilean-mcp:8092/healthz`, matching the reference compose file’s service name and `MCP_ADDR`; set `MCP_HEALTH_URL` on `zilean-core` if you renamed that service or changed `MCP_ADDR`. ## Resource limits [Section titled “Resource limits”](#resource-limits) These are ceilings, not reservations. The rationale is specific: on a busy host, an unbounded Go service that leaks memory gets a *neighbour* OOM-killed, because the kernel picks the victim. `pids_limit` is the fork-bomb ceiling for a careless — not malicious — script. | Service | `mem_limit` | `cpus` | `pids_limit` | | ----------------------------- | ----------- | ------ | ------------ | | `zilean-core` | 512m | 1.0 | 256 | | `zilean-ui` | 512m | 1.0 | 256 | | `zilean-worker` | 512m | 2.0 | 512 | | `zilean-mcp` | 512m | 1.0 | 256 | | `zilean-webhook` | 512m | 1.0 | 256 | | `zilean-script-worker-bash` | 256m | 1.0 | 256 | | `zilean-script-worker-node` | 512m | 1.0 | 256 | | `zilean-script-worker-groovy` | **1g** | 1.0 | 256 | Notes on the numbers that aren’t obvious: * **`zilean-worker` gets 2 CPUs** because `pool_count` fans one firing into concurrent HTTP calls. Its memory stays modest because response bodies are truncated. * **`cpus`, `mem_limit` and `WORKER_MAX_CONCURRENCY` move together.** The variable sets how many HTTP jobs `zilean-worker` runs at once per replica (20 by default, at most 200); raising it without raising the CPU and memory ceilings trades throughput for contention, and it needs a matching connection pool — see [Deploying to AWS](/guides/deploying-to-aws/) for the `pool_max_conns` and `max_connections` side. * **`zilean-webhook` matches core/ui/mcp** — it serves one public route, so it is sized like the other small HTTP services rather than the worker. * **Bash keeps full CPU and pids** despite the small memory limit — bash is the forkiest runtime (every pipeline stage and `$(…)` is a process), and five concurrent jobs share one pid ceiling. * **Groovy’s `1g` is load-bearing.** A JVM sizes its heap as a percentage of the *container* limit, so this number is what `-XX:MaxRAMPercentage` is a percentage of. Without a container limit, the JVM sizes its heap from total host memory instead. Size Groovy for concurrency, not for one JVM The Groovy worker runs up to 5 concurrent jobs, so five concurrent JVMs can each claim 25% of `mem_limit` — roughly 1.25 GB of heap against a 1 GB limit, before non-heap overhead. If you enable Groovy, confirm the limit is present and at least as large as the reference value. Groovy also pays \~1–2 s of JVM startup on **every** invocation, which makes it a poor fit for high-frequency schedules — an every-30-seconds job would spend \~5% of wall clock just starting JVMs. Prefer Bash or JS/TS for anything frequent or short-looped. ## Container confinement settings [Section titled “Container confinement settings”](#container-confinement-settings) The three script workers run inside a container hardened the same way, and can additionally sandbox every script with `bwrap` (bubblewrap) for filesystem and pid-namespace confinement. ### Container hardening [Section titled “Container hardening”](#container-hardening) All three script workers carry the same always-on settings in the reference Compose file. They are subtractive — they surrender privileges rather than needing anything from the host — so they apply on every host, including one where `bwrap` cannot run, and they are independent of `SCRIPT_SANDBOX_ENABLED`: | Service | `read_only` | `tmpfs` at `/tmp` | `cap_drop` | `security_opt` | | ----------------------------- | ----------- | ------------------------------- | ---------- | ------------------------ | | `zilean-script-worker-bash` | `true` | `size=128m,noexec,nosuid,nodev` | `ALL` | `no-new-privileges:true` | | `zilean-script-worker-node` | `true` | `size=256m,noexec,nosuid,nodev` | `ALL` | `no-new-privileges:true` | | `zilean-script-worker-groovy` | `true` | `size=256m,noexec,nosuid,nodev` | `ALL` | `no-new-privileges:true` | * **`read_only: true`** — the image cannot be modified, and `/tmp` is the only writable path (plus a [script volume](#script-volumes), if you mount one). Each run gets its own working directory under it, so relative-path writes work; writes to the container’s own filesystem fail. The container’s root filesystem is read-only; with bwrap sandboxing enabled, a script’s writes outside its per-run directory land in the sandbox’s own temporary root and are discarded when the run ends, never reaching the container. * **`tmpfs` at `/tmp`** — size-capped at 128m for Bash and 256m for Node and Groovy, `noexec`, `nosuid`, `nodev`. The caps are suggested rather than tuned: each is at most half of the worker’s reference `mem_limit` (256m / 512m / 1g — see [Resource limits](#resource-limits)), because tmpfs pages are charged to that limit rather than added to it, and the rest of the limit is left for the interpreters of up to five concurrent jobs. Keeping the cap below the limit is what lets a script that fills `/tmp` get `no space left on device` instead of OOM-killing the container. Because the tmpfs is `noexec`, a helper the script writes to scratch cannot be executed as a program — `chmod +x helper && ./helper` fails with `permission denied`; run it through its interpreter instead (`bash helper.sh`, `node helper.js`). A script volume is not mounted `noexec`. The interpreters live on the read-only rootfs, so Bash, Node/TypeScript via `tsx`, Groovy, stdin payloads and environment variables are unaffected. * **`cap_drop: ALL`** — empties the capability set. * **`no-new-privileges:true`** — stops setuid or file-capability privilege gain on exec. This hardening narrows what a script can do to the container, but it does **not** give per-job separation: every job on a worker runs as the same UID (65532), so concurrent jobs can still read each other’s scratch files, and the image contents remain readable. That separation is what `bwrap` adds. ### Opt-in: bubblewrap sandboxing [Section titled “Opt-in: bubblewrap sandboxing”](#opt-in-bubblewrap-sandboxing) Each of the three script workers can run every script under `bwrap` (bubblewrap) for filesystem and pid-namespace confinement — but only once you opt in. **Set `SCRIPT_SANDBOX_ENABLED=1` in `.env`; unset (the default) means these workers run scripts without per-job confinement**, with no warning that they’re doing so, because sandboxing an existing self-hosted deployment must never be something an upgrade turns on for you. The hardening above still applies without it: a script can read the whole container filesystem it runs in, and nothing stops it walking outside its own working directory — its writes stay confined to `/tmp` (and any mounted script volume), but its reads are not. See [Installation’s “Recommended: enable script sandboxing”](/start/installation/) for why you should set it — the short version: without it, one script’s temp files are visible to every other concurrently-running script job. Once `SCRIPT_SANDBOX_ENABLED=1` is set, these three workers also need the two `security_opt` relaxations that the rest of the stack does not. `no-new-privileges:true` is not one of them — it is part of the always-on hardening above: | Service | `security_opt` | Why | | ----------------------------- | ---------------------------------------------- | --------------------------------- | | `zilean-script-worker-bash` | `seccomp:unconfined`, `systempaths:unconfined` | Runs every script under `bwrap` | | `zilean-script-worker-node` | `seccomp:unconfined`, `systempaths:unconfined` | Runs every script under `bwrap` | | `zilean-script-worker-groovy` | `seccomp:unconfined`, `systempaths:unconfined` | Runs every script under `bwrap` | | All other services | *(none)* | Execute no operator-supplied code | * `seccomp:unconfined` — `bwrap` calls `unshare(CLONE_NEWUSER)`, which Docker’s default seccomp profile blocks for an unprivileged caller. * `systempaths:unconfined` — `bwrap` mounts a fresh `/proc` for the sandbox, and the kernel refuses that while Docker’s masked `/proc` paths are mounted over it. Both loosen the **container’s** own confinement, by one syscall family and Docker’s `/proc` masking respectively. Neither loosens the sandbox: what a script can reach is decided by `bwrap`’s bind list and namespaces, which these settings do not touch. Host prerequisite: unprivileged user namespaces A host that restricts unprivileged user namespaces — Ubuntu 23.10 and newer default `kernel.apparmor_restrict_unprivileged_userns=1` — creates the namespace with no capabilities in it, and `bwrap` fails whatever `security_opt` says. See [Installation](/start/installation/) for how to check and change it. The always-on hardening above is unaffected: it needs no host prerequisite and stays in place whether or not `bwrap` can run. Each worker runs a real `bwrap` self-test once at startup when `SCRIPT_SANDBOX_ENABLED=1` is set — not just a check that the binary exists — because the host prerequisite above is exactly the kind of failure a bare existence check misses. If the self-test fails (that sysctl being the most common reason), the worker logs a clear warning and falls back to running scripts without per-job confinement (the container hardening still applies) instead of failing them outright. Check `docker compose logs zilean-script-worker-bash zilean-script-worker-node zilean-script-worker-groovy` after a deploy for that warning if scripts should be sandboxed but you’re not sure they are. ## Script volumes [Section titled “Script volumes”](#script-volumes) Script volumes are opt-in. The reference Compose file mounts no volumes directory and sets no `SCRIPT_VOLUMES_ROOT`, so nothing changes unless you add them. To enable volumes, mount one host directory at `/volumes` in each script worker and set `SCRIPT_VOLUMES_ROOT` to it. Every [script volume](/guides/script-jobs/#volumes) is a direct child directory of that root, so files a script writes survive container recreation. Add the same two entries to `zilean-script-worker-bash`, `zilean-script-worker-node` and `zilean-script-worker-groovy`: ```yaml environment: SCRIPT_VOLUMES_ROOT: /volumes volumes: - /srv/zilean-volumes:/volumes ``` The host directory itself must be writable by UID `65532`, the user all three script worker images run as. Files a script writes are owned by that UID on the host: ```bash mkdir -p /srv/zilean-volumes sudo chown -R 65532:65532 /srv/zilean-volumes ``` Once mounted with the sandbox off (the default), every script job on that worker can read and write every volume, and `ro` is advisory; enable [sandboxing](#opt-in-bubblewrap-sandboxing) when you use volumes. A volume is persistent and is not mounted `noexec`, so it is the one place a script’s writes survive and where a written file could be executed. A read-only (`:ro`) root mount does not work. A root that is unset, missing, or unwritable is treated as not mounted: the worker still starts and runs every job that declares no volume, but it reports `volumes_root_mounted: false` (logging a warning when the value is set but unusable), and every job that declares a volume fails until the worker is restarted with a writable root. `SCRIPT_VOLUMES_ROOT` cannot be `/`. The mount and the variable are container settings that need a redeploy of the script workers. Back the host directory up like any other host data: it is not in Postgres. See [Volumes](/guides/script-jobs/#volumes). ## Stop behaviour [Section titled “Stop behaviour”](#stop-behaviour) All eight app services use `stop_grace_period: 30s`, set above River’s 25-second soft-stop so a graceful shutdown completes before Docker escalates to `SIGKILL`. ## Startup order [Section titled “Startup order”](#startup-order) `zilean-postgres` → `zilean-core` → everything else. Core is first because it alone applies migrations. Remember that `depends_on` waits for a container to *start*, not for migrations to finish. The other services exiting non-zero and restarting until the schema exists is the expected path, and `restart: unless-stopped` is what makes it converge. ## Reverse proxy routing [Section titled “Reverse proxy routing”](#reverse-proxy-routing) Core and ui share **one origin split by path prefix** — same-origin is required there, since the SPA served by ui calls core’s API same-origin through the proxy. `zilean-mcp` is a separate upstream on its own path, needed only if you’re using the MCP endpoint — see [MCP](/guides/mcp/). `zilean-webhook` is a separate upstream too, serving the public inbound webhook route for jobs whose `schedule_type` is `webhook`. | Path | → Service | → Port | | -------------------------------------------------------------- | --------- | ----------------------- | | `/api/*` | core | `8091` (`CORE_ADDR`) | | `/prometheus/metrics` | core | `8091` (`CORE_ADDR`) | | `/riverui/*` | ui | `8090` (`UI_ADDR`) | | `/login`, `/logout`, `/admin/assets/*`, `/admin/accept-invite` | ui | `8090` (`UI_ADDR`) | | `/` (everything else) | ui | `8090` (`UI_ADDR`) | | `/mcp` | mcp | `8092` (`MCP_ADDR`) | | `/api/webhooks/*` | webhook | `8093` (`WEBHOOK_ADDR`) | Why the constraints matter: * **Core registers no login routes.** An unauthenticated request to core gets redirected to `/login` — which **ui**, not core, serves. Put them on different origins and login redirects break. * **`/login`, `/logout`, `/admin/assets/` and `/admin/accept-invite`** are registered outside the auth middleware so the login form stays reachable. Everything else on ui sits behind it. `/logout` accepts `POST` only; a `GET` to it does not end the session. * **Auth is already enforced in-app.** `/prometheus/metrics` requires the metrics-read permission; `/riverui/` requires a *write* permission, because River UI can retry, cancel and delete jobs — session validity alone would let a read-only user mutate the queue. * **`/api/webhooks/*` must stay public.** It is served by `zilean-webhook`, not core, and sits outside admin session auth — the per-job token in the URL is the credential, so anything that can reach the path can fire the job it names. Route it straight through rather than behind the admin auth gate, and **rate-limit it at the proxy**: because it is public and effectively unauthenticated apart from the token, that limit is the primary control. Point `WEBHOOK_TRUST_PROXY_HEADERS` at it only if the proxy actually sets and sanitizes `X-Forwarded-For` (see [Environment variables](/reference/environment-variables/)). The webhook limiter keys off the leftmost entry only when it is a valid IP; a blank or non-IP entry, or an absent header, falls back to the connection’s remote address. IPv6 clients are keyed by their `/64` prefix. * **`POST /login` and `POST /admin/accept-invite` deserve the same proxy limit.** They are served outside the auth middleware and both reach a store lookup and bcrypt work before any outcome is known, so **rate-limit them at the proxy** — that limit is the primary control, exactly as it is for `/api/webhooks/*`. In-process, each route has its own per-IP limiter — IPv4 per address, IPv6 per `/64` prefix — admitting 10 POSTs per minute with a burst of 5 by default, configurable with `UI_LOGIN_RATE_PER_MINUTE` and `UI_LOGIN_RATE_BURST`, and caps the request body at **64 KiB** before any store lookup or bcrypt work. On `/login` only a POST carrying a valid CSRF token spends a limiter token, so a cross-site page cannot burn your login budget. `GET /login` is never counted, so normal page loads and session-expiry redirects are unaffected, and saturating one route’s bucket does not block the other. A limited or rejected submission re-renders the login or accept-invite page with the status and a message: `429` “Too many sign-in attempts. Try again in a minute.” with `Retry-After: 60`; `413` “Form submission too large.” for an oversized body; or `400` “Could not read form submission.” for a malformed one. Those are the login messages; invite acceptance shows equivalent messages that also tell the user to open their invite link again. ui also bounds each connection itself (10 s for request headers, 30 s to read the request, 120 s idle, 64 KiB of headers, 60 s to write the response), so a stalled client cannot pin a handler. The limiters read only the leftmost `X-Forwarded-For` entry, and only when `UI_TRUST_PROXY_HEADERS` is `true`; a blank or invalid entry — or an absent header — falls back to the connection’s remote address. **Behind a reverse proxy with `UI_TRUST_PROXY_HEADERS` unset, every client keys to the proxy’s address and shares one bucket**, so an attacker making about 10 POSTs a minute could lock every admin out of sign-in. Leave it `false` only if every client reaching ui connects directly; otherwise set `UI_TRUST_PROXY_HEADERS=true` when the proxy sets and sanitizes `X-Forwarded-For` (overwriting rather than appending to a client-supplied value), and keep the proxy-level `/login` limit as the primary control (see [Environment variables](/reference/environment-variables/)). * **Keep `/prometheus/metrics` off any internet-facing hostname.** It is authenticated, but there is no reason to expose it. Scrape it on an internal hostname. Its own path is also what keeps it from colliding with the admin SPA’s client-side `/metrics` route — routing that path to ui instead lets the SPA render its Metrics page on a hard refresh too, not just on in-app navigation. Express the split as two disjoint matchers Write the routing as `/api/*` **and** `not /api/*`, rather than a specific rule followed by a catch-all. Declaration order does not reliably determine evaluation order in every proxy, and a catch-all that evaluates first will swallow `/api/*` — producing an empty job list rather than an error. Making the matchers mutually disjoint removes the dependence on ordering entirely. Validate the generated proxy config before applying it, and confirm `/api/*` resolves ahead of the catch-all. Address core by its service name and literal port (e.g. `zilean-core:8091`). Proxy templates that auto-discover upstreams typically only resolve the container they are attached to. ## Verifying a deploy [Section titled “Verifying a deploy”](#verifying-a-deploy) In this order: 1. **Login works end to end** — the site root serves the login page, logging in lands in the admin SPA, logging out returns to the login page. 2. **`/riverui/` renders.** Served by ui, whose River client is deliberately never started — worth an explicit check, since no test can cover it. 3. **The admin UI’s job list loads.** This is the real test of whether `/api/*` reaches core. The SPA calls its API same-origin, so a wrong prefix shows as an *empty list*, not an error. 4. **A job runs.** Use `SEED_JOB=1` for one boot, or trigger an existing job manually and watch it in River UI. 5. **Script worker isolation is actually in place**, before enabling any script job. See [Script jobs](/guides/script-jobs/) for what that does and does not cover. Probe isolation by IP, never by hostname A hostname probe from an isolated container fails on DNS whether or not isolation exists, so it proves nothing while reading as a pass. Test with addresses. ## Rollback [Section titled “Rollback”](#rollback) There is no `down` migration path — `dbmigrate/migrations/` contains only `*.up.sql`. Schema additions are generally additive and an older binary won’t reference new columns, but that is not a guarantee across every migration: at least one has renamed a column, which requires **all** binaries to be rebuilt and restarted together rather than rolled out incrementally, and needs the rename reversed by hand to roll back. Check what a version bump actually contains before assuming a rollback is safe. # Deploying to AWS > Recommended EC2 sizing and a sample Terraform config for running Zilean on a single instance. Zilean ships no AWS-specific artifacts — it’s the same Docker Compose topology from [Installation](/start/installation/) and [Deploying](/guides/deploying/), run on an EC2 instance instead of bare metal or a VM elsewhere. This page covers instance sizing and a starting Terraform config; the Compose file, `.env`, and reverse-proxy routing are unchanged from the guides above. ## Recommended EC2 settings [Section titled “Recommended EC2 settings”](#recommended-ec2-settings) Sizing is driven by the `mem_limit` ceilings in [Deploying](/guides/deploying/), not by a generic scheduler sizing guide. Three tiers, depending on which script workers you actually run: | | Minimum (core + ui + worker only) | + script workers (Bash / Node) | Full stack, incl. Groovy | | ---------------------- | --------------------------------------------------------- | ------------------------------ | ---------------------------- | | **Container ceilings** | 512m+512m+512m = 1.5 GB | +256m/+512m = \~2.3–2.8 GB | +1024m (Groovy) = \~3.3 GB | | **Instance type** | `t4g.small` (2 vCPU / 2 GiB) or `t4g.medium` for headroom | `t4g.medium` (2 vCPU / 4 GiB) | `t4g.large` (2 vCPU / 8 GiB) | | **AMI** | Amazon Linux 2023, **arm64** | Amazon Linux 2023, **arm64** | Amazon Linux 2023, **arm64** | | **Storage** | 20 GiB gp3 | 30 GiB gp3 | 50 GiB gp3 | Groovy is the one place to size up, not down The Groovy worker’s `1g` container limit is load-bearing — the JVM sizes its heap as a percentage of *that* limit, and up to 5 concurrent script jobs can each claim 25% of it (see [Deploying](/guides/deploying/)), which can overcommit the limit under real concurrency. `t4g.large` covers the full stack’s container ceilings with headroom for that overcommit risk; move to `t4g.xlarge` (4 vCPU / 16 GiB) only if you run Groovy at sustained concurrency and see memory pressure in practice — don’t provision for it up front. Postgres itself is light at Zilean’s scale (a job-scheduling schema, not an application-data warehouse) and needs no separate sizing beyond what’s already budgeted above — but if you expect a large job history or heavy `worker_metrics` retention, move it to RDS rather than growing the instance further. ## Sample Terraform [Section titled “Sample Terraform”](#sample-terraform) A single EC2 instance running the full Compose stack end to end: a security group exposing only what the reverse-proxy routing in [Deploying](/guides/deploying/) requires, and `user_data` that installs Docker, writes `.env` and `docker-compose.yml` from the reference topology in [Installation](/start/installation/), and brings the stack up in the required order (`zilean-postgres` → `zilean-core` → everything else) on first boot. ```hcl terraform { required_providers { aws = { source = "hashicorp/aws" version = "~> 5.0" } } } provider "aws" { region = var.aws_region } variable "aws_region" { default = "us-east-1" } variable "instance_type" { description = "t4g.medium for core+ui+worker (+ Bash/Node); t4g.large once Groovy is enabled" default = "t4g.large" } variable "ssh_allowed_cidr" { description = "CIDR allowed to reach port 22 — never leave this at 0.0.0.0/0" type = string } variable "zilean_version" { description = "Image tag to deploy — pin to vX.Y.Z, never latest" default = "v1.0.0" } variable "db_password" { description = "Password for the zilean Postgres role" type = string sensitive = true } variable "admin_username" { description = "Super Admin login — must be an email address; zilean-core refuses to start otherwise" type = string } variable "admin_password_hash" { description = "bcrypt hash for the Super Admin password — generate with the htpasswd command in Installation; never pass the plaintext password itself" type = string sensitive = true } data "aws_ami" "al2023_arm64" { most_recent = true owners = ["amazon"] filter { name = "name" values = ["al2023-ami-*-arm64"] } filter { name = "architecture" values = ["arm64"] } } resource "aws_security_group" "zilean" { name = "zilean" description = "Zilean host — HTTP(S) from anywhere, SSH restricted" ingress { description = "HTTPS (reverse proxy)" from_port = 443 to_port = 443 protocol = "tcp" cidr_blocks = ["0.0.0.0/0"] } ingress { description = "HTTP (reverse proxy — redirect to HTTPS)" from_port = 80 to_port = 80 protocol = "tcp" cidr_blocks = ["0.0.0.0/0"] } ingress { description = "SSH" from_port = 22 to_port = 22 protocol = "tcp" cidr_blocks = [var.ssh_allowed_cidr] } egress { from_port = 0 to_port = 0 protocol = "-1" cidr_blocks = ["0.0.0.0/0"] } } resource "aws_instance" "zilean" { ami = data.aws_ami.al2023_arm64.id instance_type = var.instance_type vpc_security_group_ids = [aws_security_group.zilean.id] root_block_device { volume_type = "gp3" volume_size = 30 encrypted = true } # Secrets land in .env via user_data below, which IMDSv2 restricts to # processes on the instance itself — not a public artifact, but still # plaintext on disk. See the caution after this block. metadata_options { http_tokens = "required" } user_data = <<-EOF #!/bin/bash set -euo pipefail dnf update -y dnf install -y docker systemctl enable --now docker usermod -aG docker ec2-user mkdir -p /usr/local/lib/docker/cli-plugins curl -SL "https://github.com/docker/compose/releases/latest/download/docker-compose-linux-aarch64" \ -o /usr/local/lib/docker/cli-plugins/docker-compose chmod +x /usr/local/lib/docker/cli-plugins/docker-compose mkdir -p /opt/zilean/postgres/data cat > /opt/zilean/.env <<'ENVFILE' DATABASE_URL=postgres://zilean:${var.db_password}@zilean-postgres:5432/zilean?sslmode=disable ADMIN_USERNAME=${var.admin_username} ADMIN_PASSWORD_HASH=${var.admin_password_hash} SCRIPT_SANDBOX_ENABLED=1 ENVFILE cat > /opt/zilean/docker-compose.yml <<'COMPOSE' services: zilean-postgres: image: postgres:17.10-alpine restart: unless-stopped env_file: [.env] volumes: - ./postgres/data:/var/lib/postgresql/data zilean-core: image: ghcr.io/patrickpingol/zilean-slim:${var.zilean_version} command: ["/usr/local/bin/zilean-core"] restart: unless-stopped env_file: [.env] depends_on: - zilean-postgres zilean-ui: image: ghcr.io/patrickpingol/zilean-slim:${var.zilean_version} command: ["/usr/local/bin/zilean-ui"] restart: unless-stopped env_file: [.env] depends_on: - zilean-core zilean-worker: image: ghcr.io/patrickpingol/zilean-slim:${var.zilean_version} command: ["/usr/local/bin/zilean-worker"] restart: unless-stopped env_file: [.env] depends_on: - zilean-core zilean-mcp: image: ghcr.io/patrickpingol/zilean-slim:${var.zilean_version} command: ["/usr/local/bin/zilean-mcp"] restart: unless-stopped # Zero-key service: a narrowed environment, not env_file. It never # receives SETTINGS_ENCRYPTION_KEY or SCRIPT_SECRETS_KEY and refuses to # start if either is set. environment: DATABASE_URL: $${DATABASE_URL} TZ: $${TZ:-UTC} MCP_ADDR: $${MCP_ADDR:-:8092} depends_on: - zilean-core zilean-webhook: image: ghcr.io/patrickpingol/zilean-slim:${var.zilean_version} command: ["/usr/local/bin/zilean-webhook"] restart: unless-stopped # Zero-key service: a narrowed environment, not env_file. It never # receives SETTINGS_ENCRYPTION_KEY or SCRIPT_SECRETS_KEY and refuses to # start if either is set. environment: DATABASE_URL: $${DATABASE_URL} TZ: $${TZ:-UTC} WEBHOOK_ADDR: $${WEBHOOK_ADDR:-:8093} WEBHOOK_TRUST_PROXY_HEADERS: $${WEBHOOK_TRUST_PROXY_HEADERS:-false} depends_on: - zilean-core # Always-on hardening on all three script workers, plus the two # security_opt relaxations that SCRIPT_SANDBOX_ENABLED=1 above needs — # see Deploying's "Container confinement settings". zilean-script-worker-bash: image: ghcr.io/patrickpingol/zilean-scripts-bash:${var.zilean_version} restart: unless-stopped env_file: [.env] read_only: true tmpfs: - /tmp:size=128m,noexec,nosuid,nodev cap_drop: - ALL security_opt: - seccomp:unconfined - systempaths:unconfined - no-new-privileges:true depends_on: - zilean-core zilean-script-worker-node: image: ghcr.io/patrickpingol/zilean-scripts-node:${var.zilean_version} restart: unless-stopped env_file: [.env] read_only: true tmpfs: - /tmp:size=256m,noexec,nosuid,nodev cap_drop: - ALL security_opt: - seccomp:unconfined - systempaths:unconfined - no-new-privileges:true depends_on: - zilean-core zilean-script-worker-groovy: image: ghcr.io/patrickpingol/zilean-scripts-groovy:${var.zilean_version} restart: unless-stopped env_file: [.env] read_only: true tmpfs: - /tmp:size=256m,noexec,nosuid,nodev cap_drop: - ALL security_opt: - seccomp:unconfined - systempaths:unconfined - no-new-privileges:true depends_on: - zilean-core COMPOSE cd /opt/zilean docker compose pull docker compose up -d zilean-postgres docker compose up -d zilean-core docker compose up -d zilean-ui zilean-worker zilean-mcp zilean-webhook \ zilean-script-worker-bash zilean-script-worker-node zilean-script-worker-groovy EOF tags = { Name = "zilean" } } output "public_ip" { value = aws_instance.zilean.public_ip } ``` The `tmpfs` sizes above are charged against each worker’s memory limit; see [Container hardening](/guides/deploying/#container-hardening) for how to size them against `mem_limit`. Secrets in `user_data` are plaintext on the instance `db_password` and `admin_password_hash` land in `/opt/zilean/.env` verbatim. `metadata_options.http_tokens = "required"` (IMDSv2) keeps that out of the *public* instance metadata surface, but anything with shell access to the box, or read access to the Terraform state file, still sees them in the clear. That’s an acceptable tradeoff for a single self-managed instance with SSH already restricted to a known CIDR — for anything more sensitive, pull the values from AWS Secrets Manager or SSM Parameter Store inside `user_data` instead of passing them as Terraform variables. Either way, keep your state file itself out of anywhere it could leak (a private, encrypted S3 backend — see the AI prompt below — not a laptop or a public bucket). Only the script runtimes you actually plan to use need to stay in the Compose file — drop the `zilean-script-worker-*` blocks (and `SCRIPT_SANDBOX_ENABLED` if you’re running none of them) for tiers that don’t need them, per the sizing table above. This is a starting point, not a hardened baseline No IAM instance profile, no EBS snapshot schedule, no Elastic IP, and the security group assumes a reverse proxy terminating TLS on this same instance. Add those to match your own environment — this config only reproduces the EC2 sizing, networking, and first-boot Compose setup above. Swap `docker-compose-linux-aarch64` for `docker-compose-linux-x86_64` (and the AMI filter’s `arm64` for `x86_64`) if you deploy to a `t3`-class instance instead. ## AI prompt for provisioning this [Section titled “AI prompt for provisioning this”](#ai-prompt-for-provisioning-this) If you’d rather have an AI coding agent write and apply the Terraform for you than copy the snippet above by hand, paste this in as-is — it points the agent back at this page for sizing and asks for a properly structured, production-shaped Terraform layout rather than a single-file starting point: ```text Provision an EC2 host for running Zilean (a self-hosted job scheduler) with Terraform, following the sizing and networking guidance at https://zilean.dev/guides/deploying-to-aws/. Requirements: - Ask me which of that page's three sizing tiers applies (core+ui+worker only; + Bash/Node script workers; full stack including Groovy) before choosing an instance type, rather than assuming one. - Use an Amazon Linux 2023 arm64 AMI (Graviton) unless I say otherwise, resolved via an aws_ami data source — never a hardcoded AMI ID. - Split the config into main.tf, variables.tf, outputs.tf, and versions.tf rather than one file, with a required_providers block pinning both the AWS provider and Terraform's own version constraint. - Configure a remote backend (S3 with native S3 state locking, or S3 + DynamoDB if the Terraform version predates native locking) instead of local state. - Restrict SSH (port 22) to a CIDR I supply as a variable — never default it to 0.0.0.0/0 — and expose only 80/443 publicly. - Use a gp3, encrypted root volume sized to the chosen tier. - Tag every resource consistently: at minimum Name, Environment, and ManagedBy = "terraform". - Never hardcode secrets (DB password, ADMIN_PASSWORD_HASH, etc.) into the Terraform config or user_data. Surface them as sensitive variables, or point me at AWS Secrets Manager / SSM Parameter Store instead. - Run terraform fmt and terraform validate on the result, and show me the plan before applying anything. Scope this to the instance and networking only — provisioning .env and docker-compose.yml on the box is a separate step, covered in the same guide's "After the instance is up" section. ``` ## After the instance is up [Section titled “After the instance is up”](#after-the-instance-is-up) If you used the sample Terraform above, `.env` and `docker-compose.yml` are already on the instance and the stack is already up in the required order — jump straight to [Deploying’s “Verifying a deploy”](/guides/deploying/#verifying-a-deploy). If you used the AI prompt instead (deliberately scoped to the instance and networking only), follow [Installation](/start/installation/) from “Set the required configuration” onward — create `.env` and `docker-compose.yml` on the instance, then bring core up first so it applies migrations before the other services start. # HTTP Jobs > Build the request, authenticate it, and understand what counts as success. An HTTP job calls a URL on a schedule. It is the default `job_type`. ```bash curl -X POST https:///api/jobs \ -H 'Content-Type: application/json' \ -d '{ "name": "nightly-sync", "job_type": "http", "schedule_type": "cron", "cron_expression": "0 2 * * *", "http_method": "POST", "url": "https://example.com/api/sync", "headers": {"X-Request-Source": "zilean"}, "body": {"mode": "incremental"}, "exec_timeout_ms": 60000, "auth_type": "bearer", "auth_config": {"token": "{{SYNC_TOKEN}}"} }' ``` ## How the request is built [Section titled “How the request is built”](#how-the-request-is-built) In order: 1. **Method** — `http_method`, defaulting to `GET`. 2. **Query params** — `query_params`, with `{{variable}}` substitution applied to **both keys and values**. 3. **URL** — `url` with substitution applied, then query params merged in. A param **overrides** a same-named param already present in the raw URL. 4. **Body** — `body` (raw JSON) as a string, with substitution applied, sent verbatim. 5. **Headers** — `headers`, keys and values substituted. 6. **Content-Type** — set to `application/json` only if there’s a body and you didn’t set `Content-Type` yourself. 7. **Auth** — applied **last**, so it overrides a manually-set header of the same name. ## Authentication [Section titled “Authentication”](#authentication) | `auth_type` | `auth_config` | Effect | | ---------------- | ------------------------------------------ | --------------------------------------------------------------------------------------------------- | | `none` (default) | ignored | Nothing. | | `bearer` | `{"token": "..."}` | `Authorization: Bearer `. Errors if `token` is empty. | | `basic` | `{"username": "...", "password": "..."}` | HTTP Basic. Errors if `username` is empty; an **empty password is allowed**. | | `api_key` | `{"api_key": "...", "header_name": "..."}` | Sets `header_name: `. `header_name` defaults to `X-API-Key`. Errors if `api_key` is empty. | Every field of `auth_config` gets `{{variable}}` substitution before use, so credentials can live in [global variables](#global-variables) rather than inline. ## Success and failure [Section titled “Success and failure”](#success-and-failure) **Success is strictly `200 ≤ status < 300`.** Everything else fails: ```plaintext endpoint returned status 404: ``` Also failures: transport errors, and errors reading the response body. A response body larger than **4 MiB** fails the attempt too (and is retried like any other failure) rather than being truncated, since a cut-off payload can’t be trusted for `loop_response_path`. A **configuration** error is different. An unparseable URL, an unknown `auth_type`, or a missing required auth field **cancels** the job rather than retrying it — it can never succeed, so it doesn’t burn the retry budget. A failed attempt keeps its real status code and duration, but response snippets and error text are redacted before they’re recorded, so a body that echoes back a credential reads `REDACTED` in call history, River UI, and failure notifications. Only the recorded copy is affected: the request itself sends the real values, and `loop_response_path` is evaluated against the real response, so looping is unaffected. ## Timeouts [Section titled “Timeouts”](#timeouts) `exec_timeout_ms` bounds one attempt. * Default: **60 000 ms** (60 s) if unset or `0`. * Maximum: **1 800 000 ms** (30 min). Higher is rejected at write time. Above that sits River’s 35-minute safety net, which exists only to kill a job that somehow outlives its own timeout — not to enforce per-job limits. ## Pooling [Section titled “Pooling”](#pooling) With `pool_count > 1`, each parallel call receives a `pool_idx` query parameter (`0 … pool_count − 1`), which overwrites any `pool_idx` you set yourself. At `pool_count: 1` the parameter is **not added at all**, so a non-pooled job’s request is byte-identical to what it would have been before pooling existed. See [Loop, retry & pooling](/concepts/loop-retry-and-pooling/). ## Global variables [Section titled “Global variables”](#global-variables) Any text field supports `{{name}}` placeholders — URL, query params, body, headers and every `auth_config` field. Whitespace inside the braces is tolerated (`{{ name }}`). Names may contain letters, digits and underscores. Variables are loaded fresh on every attempt, so changing a variable’s value takes effect on the next attempt with no job edit. Manage them under **Settings → Global variables**, or via `/api/global-variables`. A variable marked secret is write-only: after saving, the UI shows “Value hidden” and the API returns an empty `value`, so you can replace a secret but not read it back. Secret values are AES-256-GCM-encrypted at rest with `SETTINGS_ENCRYPTION_KEY`, masked in the audit log, and redacted from what Zilean records about a call: the recorded URL, the response snippet, the error text, and the failure notification all show `REDACTED` in place of the value. A job’s resolved [auth credentials](#authentication) are redacted the same way, whether or not they came from a variable. ## Looping through a backlog [Section titled “Looping through a backlog”](#looping-through-a-backlog) Set `loop_enabled` and point `loop_response_path` at a count of remaining work. The job re-calls itself on success while that number stays positive: ```json { "loop_enabled": true, "loop_response_path": "result.remaining", "loop_delay_ms": 2000, "loop_jitter_ms": 300, "loop_max_iterations": 100 } ``` Your endpoint would return something like `{"result": {"remaining": 42}}`, and `0` when done. The path is [gjson](https://github.com/tidwall/gjson) syntax, and only a strictly positive **number** continues the loop. ## Inspecting what happened [Section titled “Inspecting what happened”](#inspecting-what-happened) ```bash curl https:///api/jobs//calls ``` Each entry carries the queue state, attempt number, timestamps, last error, pool index, loop iteration, and a `last_call` object with the method, URL, status code, duration and a 500-character response snippet. See [Monitoring jobs](/guides/monitoring/). # Managing Users > Invites, temporary passwords, roles and resets. Users are managed under **Settings → Users**, which requires `users:read` to view and `users:write` to change. Inviting requires `invites:create`. There are two ways to add someone, and they differ in who chooses the password. ## Option 1 — invite by email [Section titled “Option 1 — invite by email”](#option-1--invite-by-email) ```bash curl -X POST https:///api/invites \ -H 'Content-Type: application/json' \ -d '{"email": "newperson@example.com", "role_ids": [""]}' ``` A user can be assigned more than one role — pass multiple ids in `role_ids`. At least one is required; an empty list is rejected. What happens: 1. A `users` row is created with status `invited` and **no password**, and an invite row is created — in **one transaction**, so a failure can’t leave an orphaned invited user. 2. The token is 32 random bytes; only its SHA-256 hash is stored. 3. The response contains the invite `link`. If `PUBLIC_BASE_URL` is set it’s absolute; otherwise it’s the relative `/admin/accept-invite?token=...`. 4. If SMTP is configured (Settings → SMTP), an email is sent. **Best-effort** — a send failure is logged, never returned as an API error, because the invite already exists and the link is the guaranteed delivery path. **Invites expire after 7 days.** Expiry never deletes the user row; an admin resends or deletes it deliberately. The invitee opens the link, sets a password (**8 characters minimum, 72 bytes maximum**), and their row flips to `active`. The accept page checks the token up front, so an expired or invalid link shows that immediately rather than after a failed submit. Submitting the form is per-IP rate limited (by default 10 POSTs per minute with a burst of 5, configurable with `UI_LOGIN_RATE_PER_MINUTE` and `UI_LOGIN_RATE_BURST`, answering `429` with `Retry-After: 60`) and body-capped at 64 KiB, so a retry loop cannot hammer the password hash. ### Without SMTP configured [Section titled “Without SMTP configured”](#without-smtp-configured) The flow is identical server-side minus the send attempt. The copyable link is always returned and always shown in the UI — so with no SMTP configured on Settings → SMTP, copying the link and sending it yourself is the delivery path. ### Resending [Section titled “Resending”](#resending) ```bash curl -X POST https:///api/invites//resend ``` The path parameter is the invitee’s user id Not the invite row’s id. Passing the invite id fails. A resend creates a **fresh** invite row with a new token and a new 7-day expiry, and returns the new link. It returns **409** unless the target’s status is exactly `invited`. Resend applies the same role-grant rule as creating the invite: you can only resend an invite whose roles you could grant yourself, otherwise it returns **403**. ## Option 2 — create with a temporary password [Section titled “Option 2 — create with a temporary password”](#option-2--create-with-a-temporary-password) ```bash curl -X POST https:///api/users \ -H 'Content-Type: application/json' \ -d '{"email": "newperson@example.com", "role_ids": [""]}' ``` The server generates a 20-character password from an ambiguity-free alphabet (no `i`, `l`, `1`, `O`, `0`), creates the user as `active` with `must_change_password` set, and returns: ```json { "user": { ... }, "temp_password": "..." } ``` The temporary password is returned exactly once It is never stored in plaintext and cannot be retrieved again. Copy it before closing the panel. If you lose it, reset the password — don’t recreate the user. The user must change it at first login; until they do, API calls return `403 password change required`. ## Changing roles [Section titled “Changing roles”](#changing-roles) ```bash curl -X PATCH https:///api/users/ \ -H 'Content-Type: application/json' \ -d '{"role_ids": ["", ""]}' ``` `role_ids` replaces the user’s full set of roles — it isn’t additive. At least one role must remain; a request that would leave the user with none is rejected. ## Disabling and deleting [Section titled “Disabling and deleting”](#disabling-and-deleting) ```bash curl -X PATCH https:///api/users/ -d '{"status": "disabled"}' curl -X PATCH https:///api/users/ -d '{"status": "active"}' curl -X DELETE https:///api/users/ ``` `status` must be exactly `active` or `disabled`. Both disabling and deleting **revoke all of that user’s sessions immediately**, so access ends at once rather than at session expiry. Prefer disabling over deleting when you may need the account back — deletion cascades the user’s sessions and invites, though audit entries survive. ## Resetting a password [Section titled “Resetting a password”](#resetting-a-password) ```bash curl -X POST https:///api/users//reset-password ``` Generates a new temporary password, requires a change at next login, revokes existing sessions, and returns the password once. There is no self-service password reset No “forgot password” email exists — this is a stated non-goal. An admin reset is the only recovery path, which means **at least one other person needs `users:write`**, or a locked-out Super Admin requires a manual database edit. ## Self-service [Section titled “Self-service”](#self-service) Any signed-in user, with no permission required: ```bash curl https:///api/account/me curl -X POST https:///api/account/change-password \ -d '{"current_password": "...", "new_password": "..."}' ``` `/api/account/me` returns the email, super-admin flag, whether a password change is required, and the caller’s full permission list — which is what the UI uses to decide what to show. ## The Super Admin is protected [Section titled “The Super Admin is protected”](#the-super-admin-is-protected) Only the account bootstrapped from `ADMIN_USERNAME`/`ADMIN_PASSWORD_HASH` is protected: role changes, status changes, deletion and admin password resets against it all return **409**. An invite also cannot be minted for an already-active user, regardless of super-admin status. A Super Admin can still change their own password through the self-service route. More than one account can hold Super Admin: assigning a user the **Super Admin** role sets the flag, whether by invite (on accept), by creating the user directly, or by a later role change. Only an existing Super Admin can grant that role, or grant anyone a role with more permissions than their own — a plain `invites:create` or `users:write` holder can’t self-escalate or escalate someone else this way. Unlike the bootstrapped account, one of these later-granted Super Admins is not protected — another Super Admin can demote, disable, reset the password of, or delete it like any other user. ## What gets audited [Section titled “What gets audited”](#what-gets-audited) `user.create`, `user.delete`, `user.role_change`, `user.status_change`, `user.password_reset` (both admin resets and self-service changes), `invite.create`, `invite.resend`, plus `auth.login`, `auth.login_failed` and `auth.logout`. Invite entries use the **invitee’s** user id as the resource id, so a create and its resends correlate. Failed logins are recorded only for emails that resolve to a real user — see [Users & permissions](/concepts/users-and-permissions/). # MCP > Connect an AI agent to Zilean over the Model Context Protocol. Zilean exposes a [Model Context Protocol](https://modelcontextprotocol.io/) endpoint so an AI agent can read your job registry, check on activity and metrics, and enable, disable or run a job — without you building a custom integration on top of the admin API. ## Quick start [Section titled “Quick start”](#quick-start) 1. Turn the endpoint on: **Settings → MCP → Enabled**. 2. Click **Create token**, pick the least-privileged role that covers what the agent needs, and copy the token — it is shown only once. 3. Paste the snippet for your client from [Connecting a client](#connecting-a-client), replacing `` and ``. Then ask the agent something like “list my Zilean jobs” to confirm it works. ## Enabling the endpoint [Section titled “Enabling the endpoint”](#enabling-the-endpoint) The endpoint is off by default. Turn it on under **Settings → MCP**, or with the API: ```bash curl -X PUT https:///api/settings/mcp \ -H 'Content-Type: application/json' \ -d '{"enabled": true}' ``` While disabled, every request to the endpoint gets a **503** — a signal that the surface exists but is turned off, rather than a 404 that would send you hunting for a typo. `cmd/mcp` — the process that actually serves this endpoint — is deployed separately from the admin API/UI (see [Architecture](/concepts/architecture/)), so turning the toggle on doesn’t guarantee it’s running. The Settings page live-probes it (an internal health check against `cmd/mcp`, not the metrics dashboard) on every load and warns — without blocking the toggle or token creation — when it isn’t reachable; if you see that warning, check that binary is actually up before assuming a client-side problem. ## Issuing a token [Section titled “Issuing a token”](#issuing-a-token) Tokens are issued and managed under **Settings → MCP**, alongside the enabled toggle, or: ```bash curl -X POST https:///api/mcp-tokens \ -H 'Content-Type: application/json' \ -d '{"name": "claude-desktop", "role_id": "", "expires_in": "90d"}' ``` `role_id` picks the role the token acts as — this is what decides which tools the token can see, so choose the least-privileged role that covers what the agent needs to do (see [Permissions](/reference/permissions/)). `expires_in` is one of `30d`, `90d`, `365d`, or `never`. The response’s `token` field is the plaintext bearer credential, returned **exactly once**. Copy it before closing the panel — only its hash is stored, and there is no way to recover it afterward. Losing it means issuing a new token. Each token is backed by its own hidden service account, so calls made with it are attributable to that exact token in the audit log rather than to a shared identity. ## Connecting a client [Section titled “Connecting a client”](#connecting-a-client) The endpoint is `https:///mcp`, and it authenticates with a bearer token sent as `Authorization: Bearer `. Every client below is configured with those two values. * Claude Code ```bash claude mcp add --transport http zilean https:///mcp \ --header "Authorization: Bearer " ``` * Cursor Add to `~/.cursor/mcp.json` (or `.cursor/mcp.json` in a project): ```json { "mcpServers": { "zilean": { "url": "https:///mcp", "headers": { "Authorization": "Bearer " } } } } ``` * VS Code Add to `.vscode/mcp.json`: ```json { "servers": { "zilean": { "type": "http", "url": "https:///mcp", "headers": { "Authorization": "Bearer " } } } } ``` * Claude Desktop Claude Desktop’s config file launches local commands rather than connecting to a URL, so bridge to the endpoint with [`mcp-remote`](https://www.npmjs.com/package/mcp-remote) (needs Node.js). Add to `claude_desktop_config.json`: ```json { "mcpServers": { "zilean": { "command": "npx", "args": [ "-y", "mcp-remote", "https:///mcp", "--header", "Authorization:${ZILEAN_AUTH}" ], "env": { "ZILEAN_AUTH": "Bearer " } } } } ``` The header value goes through `env` because some clients mangle spaces inside `args`. * Any other client Any client that supports remote (streamable HTTP) MCP servers works: give it the URL and an `Authorization: Bearer ` header. ### Troubleshooting [Section titled “Troubleshooting”](#troubleshooting) | Symptom | Cause | | -------------------------------- | ------------------------------------------------------------------------------------------------------------------ | | **503** | The endpoint is disabled — enable it under **Settings → MCP**. | | **401** | The token is missing, mistyped, revoked, or expired. There is no session-cookie fallback; issue a new token. | | Connects, but a tool is missing | The token’s role lacks that tool’s permission. `tools/list` only shows what the role allows — see [Tools](#tools). | | Certificate or connection errors | The client must reach `https:///mcp` directly; check your reverse proxy passes `/mcp` through. | ## Tools [Section titled “Tools”](#tools) A token only ever sees the tools its role’s permissions allow — `tools/list` reflects exactly what that token can call, nothing more. | Tool | Permission | Does | | ----------------------- | -------------- | ----------------------------------------------------------------------------------------- | | `zilean_list_jobs` | `jobs:read` | List jobs, with filters. Each entry includes `capabilities`. | | `zilean_get_job` | `jobs:read` | Read one job’s full configuration, including its `capabilities` (declared script volumes) | | `zilean_job_activity` | `jobs:read` | Windowed call volume, latency, and lifetime invocation counters for one job | | `zilean_recent_calls` | `jobs:read` | Recent attempts for one job, from River’s own job rows | | `zilean_worker_metrics` | `metrics:read` | CPU/memory history for a Zilean service | | `zilean_audit_log` | `audit:read` | Query the audit log | | `zilean_enable_job` | `jobs:write` | Enable one job | | `zilean_disable_job` | `jobs:write` | Disable one job | | `zilean_run_job` | `jobs:write` | Trigger one job to run immediately, bypassing its schedule | ## What’s deliberately not here [Section titled “What’s deliberately not here”](#whats-deliberately-not-here) * **No job authoring.** There is no create, edit, or delete tool. A token can look at jobs and flip them on or off, but shaping a job’s request, auth, loop, or retry behavior stays an admin-API/UI operation. * **No volume surface.** A token can read which volumes a job declares, but nothing here creates a volume or changes a declaration. * **No `jobs:script` surface.** Nothing here can create or edit a script job, regardless of the token’s role. * **No bulk operations.** Every write tool acts on exactly one job by id — there’s no enable-all, disable-all, or group enable/disable. One malformed or over-eager agent call should never be able to touch every job at once. These are intentional boundaries, not gaps waiting to be filled in. # Monitoring Jobs > Heartbeats, activity charts, call history, worker metrics and River UI. Zilean surfaces execution detail rather than hiding it. There are six places to look, each answering a different question. | Question | Where | | ---------------------------------------------- | --------------------------------------------------------- | | Has this job been running, and succeeding? | Heartbeat bar on the jobs list | | What did the last few attempts actually do? | **Calls** modal / `/api/jobs/{id}/calls` | | How has this job’s volume and latency trended? | **Activity** modal / `/api/jobs/{id}/activity` | | Are the worker processes healthy? | Metrics page / `/api/metrics/*` and `/prometheus/metrics` | | Are all eight services reachable at a glance? | **Settings → System Health** / `/api/system/health` | | What is the queue actually doing right now? | River UI at `/riverui/` | ## What counts as one invocation [Section titled “What counts as one invocation”](#what-counts-as-one-invocation) **One attempt.** A retry and a loop iteration each count separately — so a single cron firing can produce many invocations. Every terminal outcome is recorded, including transport errors, non-2xx responses, body-read failures and configuration errors. Recording writes to three places: queue job metadata, lifetime counters on the job row, and a per-minute bucket table. A failure to record is logged but never propagated — losing a count must not change a job’s retry or loop decision. ## Call history [Section titled “Call history”](#call-history) ```bash curl 'https:///api/jobs//calls?limit=20' ``` Default limit 20, maximum 100. Returns a bare JSON array (`[]`, never `null`), newest first. Each entry carries the queue state, attempt number, scheduled/attempted/finalized timestamps, the last error, the pool index, the loop iteration, and a `last_call` object: | For HTTP jobs | For script jobs | | --------------------------------------------------------------- | ------------------------------------------------------------------------------------------------ | | `method`, `url`, `status_code`, `response_snippet` (≤500 chars) | `interpreter`, `script_args`, `exit_code`, `stdout_snippet`, `stderr_snippet` (≤2000 chars each) | Both share `job_name`, `timeout_ms`, `duration_ms`, and `error` when one occurred. For HTTP jobs, secret values are redacted from what’s recorded: every secret global variable’s value and the job’s resolved auth credentials are replaced with `REDACTED` in `url`, `response_snippet`, `error` and `last_error`. Script jobs do the same for the secret variables their job was allowed to receive. The endpoint and the script itself still see the real values. `exit_code` is only present when the process actually ran to completion — which is how you distinguish a genuine exit `0` from “never started.” ## Heartbeats and activity [Section titled “Heartbeats and activity”](#heartbeats-and-activity) Heartbeats are fetched for **every job in one request**, because the jobs list polls on a timer and per-job requests would be N requests per tick. A job with no history returns an empty bucket list, which renders as an all-grey “never ran” bar — deliberately distinguishable from a failed fetch. Per-job activity adds lifetime counters (`invocations_total`, `invocations_completed`, `invocations_failed`) alongside the windowed buckets. Those counters cover the job’s **whole life**, not the selected window. ### Bucketing [Section titled “Bucketing”](#bucketing) Calls accumulate into fixed **1-minute** buckets, which keeps row count independent of call volume — roughly 1,440 rows per job per day whether the job runs once an hour or loops constantly. Reads aggregate those into wider display buckets. Every series response includes the window geometry: ```json { "window_start": "...", "bucket_width_seconds": 60, "slot_count": 40 } ``` The UI needs this because the queries are `GROUP BY` aggregates — they emit nothing for an empty window — so gaps are filled client-side. It can’t re-derive the boundaries itself: its clock may differ, and buckets are epoch-aligned rather than now-relative. | Param | Default | Cap | Invalid | | --------- | ------- | --------------------------------------------------------- | ------------------------------------------------- | | `since` | `24h` | `720h` (30 days), **clamped silently** to match retention | `400 since must be a positive duration, e.g. 24h` | | `buckets` | `40` | `500`, clamped silently | `400 buckets must be a positive integer` | Latency is only populated on the per-job route `duration_ms_avg` and `duration_ms_max` are filled in by the per-job activity query only. The heartbeat and worker-wide series leave both at `0`, which is indistinguishable from a genuinely zero-latency bucket. Call history buckets are retained for **30 days**. ## Worker metrics [Section titled “Worker metrics”](#worker-metrics) Each service samples **itself** once a minute — CPU as a percentage of one core, Go heap allocation, bytes obtained from the OS, and goroutine count — and writes a row tagged with its service name. Service labels: `core`, `ui`, `worker`, `mcp`, `webhook`, `script-worker-bash`, `script-worker-node`, `script-worker-groovy`. ```bash curl 'https:///api/metrics/live?service=worker' curl 'https:///api/metrics/history?since=24h&buckets=40&service=all' ``` `live` returns the latest sample, or JSON `null` with a 200 if that service has never sampled. `service=all` returns a map keyed by service instead of one series. An unknown service value is a **400**, not a silent fallback. Retained for **30 days**. The metrics page also charts a worker-wide invocation series summed across all jobs. Memory figures are Go-runtime, not container RSS `mem_sys_bytes` is what the Go runtime obtained from the OS. Summing it across services would not give you “Zilean’s memory” — per-service lines are the honest reading. CPU percentages *do* sum meaningfully, since they’re percent-of-one-core. ## Alerting on worker metrics [Section titled “Alerting on worker metrics”](#alerting-on-worker-metrics) A per-service CPU%/memory threshold is created under **Settings → Notifications** with **Add alarm**: choose the service (`core`, `ui`, `worker`, `mcp`, `webhook`, or one of the three script workers), set a CPU% and/or memory (MB) threshold, and pick which targets to notify. Thresholds accept fractional values such as `0.5` or `0.01`. Leaving both empty shows “Set a CPU or memory threshold.”; a value that is not a number, or is negative, shows “CPU threshold must be 0 or greater (for example 0.5).” for the CPU field and “Memory threshold must be 0 or greater (for example 256).” for the memory field. There is one alarm per service, and each is listed with its current state (OK or Breached). When an enabled alarm’s latest sample is at or above either threshold, Zilean notifies the alarm’s chosen targets once on the way up (state becomes `breached`) and once on the way back down (state returns to `ok`) — not on every sample while still breached, so a sustained outage doesn’t spam the same channel every minute. This reuses the same target registry as a job’s own **Notify** step, but only its Apprise and email targets: the metrics alarms are not event-driven, so webhook targets are not offered for one. Per-job [invocation alarms](#invocation-alarms) are state-change driven and **do** support webhook targets. See [Configuration](/start/configuration/#notifications-is-a-settings-page-feature-too). ## Invocation alarms [Section titled “Invocation alarms”](#invocation-alarms) A per-job alarm that fires when a job’s **invocation count** over a rolling window reaches a threshold. Where a worker-metrics alarm watches a service’s CPU or memory, an invocation alarm watches how often a job runs. Open a job’s **Activity** modal and use its **Invocation alarms** section to add one. Each row shows the metric, threshold and window, and the current state (`OK` or `Breached`). A breached alarm also shows the time it has been breached since. Each row has controls to edit, disable or delete the alarm. | Setting | Meaning | | --------------- | ------------------------------------------------------------------------------------- | | Metric `total` | Every attempt — each loop iteration and each retry counts separately. | | Metric `failed` | Failed attempts only, counted the same way. | | Threshold | Fires when the count reaches **at or above** this value. | | Window | `1 … 1440` minutes, counted in 1-minute buckets including the current partial minute. | An alarm is evaluated every **60 seconds**. The count crossing the threshold moves the alarm to `breached` and sends a notification; falling back below it moves the alarm to `ok` and notifies again. Only those transitions notify — a sustained breach does not re-notify on every evaluation. A breach notification contains: * Title: `Zilean: invocation alarm breached` * Body: ` had invocations in the last minutes (alarm at >= ).` A recovery notification puts `recovered` in the title and keeps the same body shape. Invocation alarms notify the job’s own targets — the ones picked in the job wizard’s **Notify** step — whether they are Apprise, email or webhook targets. That target picker is shared with the job’s success/failure notifications, but `notify_on` does not gate an invocation alarm: a job with `notify_on: none` still delivers to the targets selected for it. A webhook target receives these deliveries only when its own event filter includes **Invocation alarm** (see [Receiving a webhook](/start/configuration/#receiving-a-webhook)). A job with no targets selected and **All targets** off has nowhere to deliver an alarm, so set at least one target on that step. ## Prometheus [Section titled “Prometheus”](#prometheus) `/prometheus/metrics` on `zilean-core` exposes four gauges, each labelled by `service`: * `zilean_process_cpu_percent` * `zilean_process_mem_alloc_bytes` * `zilean_process_mem_sys_bytes` * `zilean_process_goroutines` Core **re-exports the other services’ samples** on its own endpoint, because the workers run no HTTP server and can’t be scrape targets. Values refresh on a one-minute ticker, so a scrape never issues a database query. A service with no samples yet is **skipped, never published as zero** — “not running” must not look like “running at 0% CPU”. `/prometheus/metrics` is authenticated It requires a valid session cookie **and** `metrics:read`. An unauthenticated scrape gets a 302 to the login page, not a 401. Plan your scrape configuration accordingly, and keep the endpoint off any internet-facing hostname. ## Script worker liveness [Section titled “Script worker liveness”](#script-worker-liveness) ```bash curl https:///api/scripts/liveness ``` Reports per script job type whether that runtime’s worker looks alive, based on whether it has reported a metrics sample within the last **2 minutes** (two sampling intervals — tolerating one missed tick while still catching a real outage in about a minute). Gated on `jobs:script` rather than `metrics:read`: the people who need this signal are the ones who can create script jobs. The new-job wizard uses it to warn when the runtime that would execute a job isn’t running. ## System Health [Section titled “System Health”](#system-health) ```bash curl https:///api/system/health ``` Returns one entry per running service — `worker`, `script-worker-bash`, `script-worker-node`, `script-worker-groovy`, `core`, `ui`, `mcp`, `webhook` — each with a `live` boolean and a `last_sample_at` timestamp (`null` if that service has never sampled). A service is live if it has reported a metrics sample within the last **2 minutes**, the same staleness window script worker liveness uses. This is a simplified status roll-up across every service Zilean runs, built from the same stored samples the Worker Metrics chart already has — it adds no new sampling of its own. Where it differs from the other two liveness-shaped signals on this page: Worker Metrics charts CPU/memory **over time** for services you pick one at a time, and Script worker liveness covers only the three script runtimes, gated on `jobs:script`. System Health is a **live/stale/never-seen** status for **all eight** services at once, gated on `metrics:read`, and is what **Settings → System Health** in the admin UI renders as a table. A read failure for one service degrades just that entry to “never seen” rather than failing the whole request. ## River UI [Section titled “River UI”](#river-ui) Mounted at `/riverui/`, this is the third-party River dashboard showing raw queue internals — job rows, states, attempt counts, errors, scheduled times. Use it when you need to know what the queue itself is doing: a job stuck in `scheduled`/`available` (usually a runtime with no deployed worker), retry timing, or the full error on a discarded job. River UI can mutate the queue It offers retry, cancel and delete, which is why it’s gated behind `jobs:write` rather than session validity alone. Finished job rows shown here don’t stick around forever: how long `completed`, `cancelled`, and `discarded` rows are kept is an admin-configurable retention window (Settings → River Jobs, `settings:river_jobs`), swept daily alongside the audit log’s own retention sweep. See [River job retention](/reference/api/#river-job-retention) in the API reference. ## Known gaps [Section titled “Known gaps”](#known-gaps) * **Loop-max-iterations has no notification.** Reaching `loop_max_iterations` logs an error and is otherwise silent — no target is notified. Other terminal job states do notify, through the job’s own Notify step (Apprise, email, or webhook targets; see [Configuration](/start/configuration/#notifications-is-a-settings-page-feature-too)). * **Failed logins for unknown emails aren’t recorded** (deliberately — it would be an enumeration oracle). * **Audit retention changes aren’t audited.** # Script Jobs > Run Bash, JavaScript/TypeScript or Groovy on a schedule — and what that does not contain. A script job runs operator-supplied code as a subprocess on a schedule. ```bash curl -X POST https:///api/jobs \ -H 'Content-Type: application/json' \ -d '{ "name": "prune-exports", "job_type": "bash", "schedule_type": "cron", "cron_expression": "0 4 * * *", "script_body": "set -euo pipefail\nfind /exports -mtime +7 -delete\necho done", "script_args": ["--verbose"], "exec_timeout_ms": 120000 }' ``` ## Permissions [Section titled “Permissions”](#permissions) Creating or editing a script job requires **`jobs:script`** in addition to `jobs:write`. This is checked inside the create/update handler rather than at the route, because the job type is only known after the request body is decoded. Without it you get: ```plaintext 403 {"error":"script jobs require the jobs:script permission"} ``` **No built-in role carries `jobs:script`** — not even Admin. It reaches a non-Super-Admin only through a custom role, which is deliberate: the permission is the ability to run arbitrary code on your infrastructure. Grant it as such. ## How a script runs [Section titled “How a script runs”](#how-a-script-runs) 1. `script_body` and every element of `script_args` are used exactly as written. `{{variable}}` placeholders are **not** substituted in scripts, so editing a global variable can never change the code a script runs. Variables reach a script through its environment instead (see below). 2. The body is written to a temp file (mode `0600`, no execute bit) with the runtime’s extension. 3. The interpreter is invoked with the temp file as an argument, followed by `script_args`. 4. The temp file is removed on **every** path — success, non-zero exit, or timeout. A temp file rather than `bash -c ''` avoids shell-escaping bugs, argument-length limits, and exposing the script body in the process list. The process runs in its own process group, and a timeout kills the **whole group** — so a forked child can’t survive to hold the output pipes open. A further 5-second delay guards against a hung wait. ### Where a script can write [Section titled “Where a script can write”](#where-a-script-can-write) `/tmp` is the only path where a script’s writes persist, unless you have mounted a [script volume](#volumes) and the job declares it; each run gets its own working directory under it, so relative-path writes work (`echo x > out.txt`), and writes to the container’s own filesystem fail. The container’s root filesystem is read-only; with bwrap sandboxing enabled, a script’s writes outside its per-run directory land in the sandbox’s own temporary root and are discarded when the run ends, never reaching the container. That `/tmp` is a size-capped, `noexec` tmpfs shared by every job on the worker — see [Container confinement settings](/guides/deploying/#container-confinement-settings). Because it is `noexec`, a file the script writes to scratch cannot be run as a program directly (a [volume](#volumes) is not mounted `noexec`, so a file written there can be): `chmod +x helper && ./helper` fails with `permission denied`. Run it through its interpreter instead — `bash helper.sh`, `node helper.js`. The interpreters live on the read-only root filesystem, so this does not change how the script itself runs: Bash, Node/TypeScript via `tsx`, Groovy, stdin payloads and environment variables are unchanged. Scratch is not private to a job: all jobs on a worker run as the same UID (65532), so concurrent jobs can read each other’s files under `/tmp`. Per-job separation is what the opt-in `bwrap` sandbox provides — see [Script jobs are not sandboxed](#script-jobs-are-not-sandboxed). ## What the script receives [Section titled “What the script receives”](#what-the-script-receives) The child environment is built **from scratch** — nothing is inherited. A script can rely on: | Variable | Notes | | ----------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `PATH` | Copied from the worker process. | | `ZILEAN_JOB_NAME` | The job’s name. | | `ZILEAN_POOL_IDX` | Pool index. Set **always**, even at `pool_count: 1`. | | `ZILEAN_LOOP_ITERATION` | Current loop iteration, `0` on the first. | | `TZ` | Only if set on the worker process. | | `ZILEAN_VOLUME_` | One per volume the job declares, holding the path where it is mounted. See [Volumes](#volumes). | | `GLOBALVAR_` | One per non-secret global variable, plus any secret variable explicitly opted in. A secret is placed in the environment only when the variable is marked “Allow in scripts” (`allow_in_scripts`) **and** the job’s allowed-secrets list (`script_secret_vars`) names it; otherwise it is never injected. | Groovy additionally gets `JAVA_HOME` and `JAVA_OPTS`. `GLOBALVAR_` names are upper-cased with every run of non-alphanumeric characters replaced by `_`. A variable whose name is empty, separators-only, non-ASCII, or starts with a digit is **skipped rather than mangled**, so it is not exposed to the script. Runner-injected variables cannot be shadowed by a global variable; a collision keeps the injected value and logs a warning. ## Opting a secret variable into a script’s environment [Section titled “Opting a secret variable into a script’s environment”](#opting-a-secret-variable-into-a-scripts-environment) Secret global variables stay out of script environments by default. A deliberate, two-sided opt-in can expose specific ones to specific jobs: 1. On the **Global Variables** settings page, turn on **Allow in scripts** next to a variable’s secret toggle (`allow_in_scripts`). This marks the variable as *eligible*; on its own it does not expose it to any job. The toggle carries a warning that the value becomes visible in the environment and captured output of any script job it is injected into. 2. In the job wizard’s script step, select the variable’s name in the allowed-secrets multi-select (`script_secret_vars`). This lists which secret variable names *this* job may receive. The picker is only shown to a caller who holds `globalvars:write` in addition to `jobs:script`. A value is injected as `GLOBALVAR_` only when **both** sides are set. Setting `allow_in_scripts` requires `globalvars:write`; setting a job’s `script_secret_vars` list requires both `jobs:script` and `globalvars:write`. Both changes are recorded in the audit log. Injected secret values are redacted to `REDACTED` if they appear in the job’s captured output in the Recent Calls / run-log view. That masking is display-only: the running script itself sees the real value, so this is a controlled exposure, not prevention. HTTP jobs get the same masking without the opt-in, because their secrets are resolved by Zilean rather than handed to a script: every secret global variable’s value, plus the job’s resolved auth credentials, is redacted from what a call records. See [Global variables](/guides/http-jobs/#global-variables). ## Success, failure and output [Section titled “Success, failure and output”](#success-failure-and-output) * **Exit `0`** → success. * **Non-zero exit** → failure: `script exited 2: `. * **Timeout** → `script timed out: …`. Checked *before* exit status, because the kill surfaces as a signal-terminated exit. stdout and stderr are each captured up to **64 KiB**; beyond that, output is dropped while the full length is still reported. Error snippets kept in metadata are truncated to 2000 characters — four times the HTTP limit, because script failures are stack traces and compiler diagnostics rather than short error bodies. **stdout is also the loop signal.** If `loop_enabled` is set, stdout is evaluated exactly as an HTTP response body would be — see [Loop, retry & pooling](/concepts/loop-retry-and-pooling/). That’s why the output memory limit is well above the snippet limit. ## Editing a script in the browser [Section titled “Editing a script in the browser”](#editing-a-script-in-the-browser) The script field is a code editor rather than a plain text box. It highlights the selected runtime — Bash, JavaScript or TypeScript; Groovy is highlighted with Java rules, so closures, GString interpolation and `def` highlight approximately. Bracket matching, auto-indent, find/replace and word-based suggestions are available, but there is no language-aware IntelliSense or diagnostics. The editor follows the light/dark theme and needs no internet access. | Key | Effect | | ------------------------------------ | ------------------------------------------------------------------------------------------------------------------------- | | `Tab` | Inserts indentation. | | **Ctrl+M** (`Ctrl+Shift+M` on macOS) | Toggles whether `Tab` indents or moves keyboard focus to the next control, so keyboard users can always leave the editor. | Comments and indentation are kept as you write them; mixed line endings are normalised to a single style. ## Runtime-specific notes [Section titled “Runtime-specific notes”](#runtime-specific-notes) | Runtime | `job_type` | Notes | | ---------- | ------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Bash | `bash` | Real `bash` plus `coreutils` — not busybox `sh` — so scripts behave as they do elsewhere. | | JavaScript | `javascript` | Runs under `node`. | | TypeScript | `typescript` | Runs under `tsx`; no separate build step. | | Groovy | `groovy` | **Minimum `exec_timeout_ms` of 10 000 ms**, enforced at write time, because a fresh JVM costs 1–2 s to start. Grape (`@Grab`) dependency resolution is disabled at the interpreter. | A job whose runtime has no deployed worker queues indefinitely with nothing to pick it up. Check before creating one: ```bash curl https:///api/scripts/liveness ``` It reports `{"live": bool, "last_sample_at": ...}` per script type, based on whether that worker has reported a metrics sample in the last two minutes. The new-job wizard surfaces this as a warning. ## Volumes [Section titled “Volumes”](#volumes) A script can be given persistent storage, and a job declares exactly which storage it may reach. Without a declaration a script sees no volume. Volumes are named directories that live under one host directory you mount into the script workers. Volumes are opt-in. A default deploy mounts no volumes directory and sets no `SCRIPT_VOLUMES_ROOT`, and nothing changes until you add both. Until then every script worker reports `volumes_root_mounted: false` and a job that declares a volume fails. ### Set up the host once [Section titled “Set up the host once”](#set-up-the-host-once) 1. Create one host directory to hold all volumes, for example `/srv/zilean-volumes`. 2. Make **the root directory itself** writable by the user the script worker containers run as — UID `65532`, the user all three script worker images run as. Confirm it with `docker compose exec zilean-script-worker-bash id -u`. Files a script writes are owned by that same ID on the host. Docker creates a missing bind-mount directory owned by `root`, so create and `chown` the root before first start. 3. Mount it into each script worker you use and point `SCRIPT_VOLUMES_ROOT` at the mount: ```yaml zilean-script-worker-bash: # ...existing configuration... environment: SCRIPT_VOLUMES_ROOT: /volumes volumes: - /srv/zilean-volumes:/volumes ``` `SCRIPT_VOLUMES_ROOT` cannot be `/`. At worker startup the root is probed by creating and removing a temporary directory inside it, so the root itself must be writable: writable subdirectories under a root the worker cannot write to do not help, and a root mounted read-only (`:ro`) fails the probe. Repeat for `zilean-script-worker-node` and `zilean-script-worker-groovy` if you run them. Mounting the directory and setting `SCRIPT_VOLUMES_ROOT` are container settings, so they need a redeploy of the script workers. Defining and changing volumes afterwards does not. A worker whose root is unset, missing, not a directory, `/`, or not writable by UID `65532` still starts and still runs every job that declares no volume. It treats the root as **not mounted**: `GET /api/script-workers/status` reports `volumes_root_mounted: false` for that queue, and a job that declares a volume fails on that worker (see [Failure modes](#volume-failure-modes)). The worker also logs a warning naming the path when the value is set but unusable. For an unwritable root the warning reads: ```text script volumes root is not writable by this process; script volumes will fail closed (likely wrong ownership or permissions, e.g. a Docker-created root-owned directory; UID 65532 needs write access) ``` The probe runs once, at worker startup, and the heartbeat reports that result for the life of the process — fixing ownership or permissions changes nothing until the worker is restarted. After a fix, restart it (`docker compose restart zilean-script-worker-bash`) and confirm the status flips to `true`. Each volume is one directory directly under the root, named after the volume, created on first use if it does not exist. Keep it a real directory: a volume directory that is a symbolic link, or that resolves outside the root, is rejected. ### Define a volume [Section titled “Define a volume”](#define-a-volume) Open **Settings → Script volumes** and create a volume. Defining volumes needs `scriptvolumes:write`; viewing them needs `scriptvolumes:read` (see [Permissions](/reference/permissions/)). A volume has: | Field | Notes | | ------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `name` | Lowercase letters, digits and underscores, starting with a letter, at most 63 characters (`^[a-z][a-z0-9_]{0,62}$`). It cannot be renamed. Create a new volume instead. | | `max_mode` | `ro` or `rw`. The most access any job may be granted to this volume. | | `description` | Free text. | Creating a volume is a row write and takes effect on the next run. It does not create the directory. The worker creates it the first time a job mounts it. Deleting a volume removes only the catalog entry. The files on disk are never deleted. A volume that a job or workflow script block still declares cannot be deleted; the request fails with `409` and lists what references it. ### Declare volumes on a job [Section titled “Declare volumes on a job”](#declare-volumes-on-a-job) Pick volumes in the job wizard’s script step, or set `capabilities` in the API: ```json { "capabilities": { "volumes": [ { "name": "reports", "mode": "rw" } ] } } ``` * `mode` is required and is `ro` or `rw`. A volume whose `max_mode` is `ro` cannot be declared `rw`. * Each volume may be declared once. * `{}` and `{"volumes": []}` both mean no capabilities. * Any other top-level key is rejected. `egress` is reserved and is rejected as not yet supported. * Only script jobs and workflow script blocks can declare capabilities. An HTTP job, or any other workflow block, with a non-empty declaration is rejected with `400`. * Editing a declaration needs the same `jobs:script` permission as editing the script. Declaring an existing volume needs no volume permission, so a role holding `jobs:script` alone can use volumes but cannot create storage. ### What the script sees [Section titled “What the script sees”](#what-the-script-sees) For each declared volume the script gets the directory mounted at `/`, read-only for `ro` and writable for `rw`, and an environment variable with its path: ```bash echo "report" >> "$ZILEAN_VOLUME_REPORTS/today.txt" ``` The variable is `ZILEAN_VOLUME_` followed by the upper-cased volume name. A global variable cannot shadow it. Volumes the job did not declare do not exist inside the sandbox, and the volumes root itself is not mounted. The declaration is checked again on every attempt against the current catalog, not only when the job is saved. A volume removed from the catalog, or lowered from `rw` to `ro` after a job declared `rw`, fails the attempt rather than being silently downgraded. ### Enforcement needs the sandbox [Section titled “Enforcement needs the sandbox”](#enforcement-needs-the-sandbox) Volume confinement is carried out by `bwrap`, so it is enforced only on a worker where the sandbox is active (see [Installation](/start/installation/)). When the sandbox is not active, for example `SCRIPT_SANDBOX_ENABLED` is unset or the `bwrap` self-test failed: * The declaration is still validated, so a bad declaration fails the same way on every host. * The script runs unconfined with the same environment variables. Every script job on a worker with a mounted root can read and write every volume, and `ro` is advisory. Enable the sandbox when you use volumes. * Each worker process logs one warning, the first time any job with a declaration runs unenforced. It is not logged again per job. **Settings → Script volumes** shows, per script worker queue, whether the sandbox is enforced, whether the volumes root is mounted, and whether the worker has reported recently. Each worker reports every 60 seconds, and a report older than 3 minutes is shown as stale. The job wizard warns when you declare a volume for a queue that is not enforcing. ### Volume failure modes [Section titled “Volume failure modes”](#volume-failure-modes) * **The volume cannot be used** (no usable root on this worker — unset, missing, or not writable by UID `65532`; volume removed; mode above the ceiling; or a volume directory that is a symlink): the attempt fails immediately and is **not retried**, so it does not spend the retry budget. The error names the volume and the reason (`volume root is not configured` when the root is not mounted), and never contains file contents. Check the queue’s status in **Settings → Script volumes** (a worker without a usable root shows **Root not mounted**), or the worker’s log, to tell which. * **Volumes are persistent and executable.** A volume is not mounted `noexec`, so it is the one place a script’s writes survive and where a file a script wrote could be executed. Grant `rw` sparingly. * **Concurrent writers.** A volume is shared, persistent storage. Pool members, loop iterations and different jobs that write the same `rw` volume at once are your script’s concern. `overlap_policy: skip` serialises a single job. * **Egress is not supported.** A declaration cannot restrict outbound hosts. The Docker network split described below remains the only network boundary. ## Script jobs are not sandboxed [Section titled “Script jobs are not sandboxed”](#script-jobs-are-not-sandboxed) Worth stating exactly, because the partial confinement described below is easy to mistake for a full sandbox. What the design **does** close: * The application’s own credentials are absent from the child environment — it’s built from scratch, not inherited. * Execution in the HTTP worker’s container is impossible, because queue subscriptions are disjoint. This is structural, not a bypassable check. * Services on other Docker bridge networks are unreachable, because Docker doesn’t forward between bridges. * Writes to the container’s own filesystem are blocked. The worker’s root filesystem is read-only, so a script cannot modify the image, and `/tmp` is the only path where its writes persist (other than to a [volume](#volumes) you mounted and the job declares) — a size-capped, non-executable tmpfs with a fresh working directory per run, so relative-path writes work. A helper written to scratch must be run through its interpreter rather than executed directly. See [Where a script can write](#where-a-script-can-write). What remains **open, and is explicitly accepted**: A script can recover `DATABASE_URL`, and therefore global variables and script bodies The script runs as the **same UID** as the script-runtime worker process, so it can read `/proc//environ` and recover that process’s environment — including `DATABASE_URL`. From there it can connect to Postgres as the application. Non-secret global variable values and other jobs’ `script_body` are plaintext and readable this way. `auth_config` secrets (a bearer token, basic password, or API key) and secret global variables are not — they’re AES-256-GCM-encrypted at rest, keyed by `SETTINGS_ENCRYPTION_KEY`. Recovering `DATABASE_URL` gets the ciphertext, not the master key: the script-runtime workers deliberately never hold `SETTINGS_ENCRYPTION_KEY` itself, since every script job has `auth_type: none` by construction and so never needs to decrypt `auth_config`. This is the actual reason for that omission, not an oversight — see [Configuration](/start/configuration/#configuration-is-split-by-trust-not-by-convenience). What a script worker *does* hold is `SCRIPT_SECRETS_KEY` — a sub-key derived from `SETTINGS_ENCRYPTION_KEY` via HKDF-SHA256 (see [Generating `SCRIPT_SECRETS_KEY`](/reference/environment-variables/#generating-script_secrets_key) for the command). It can decrypt only global variables with `allow_in_scripts` set — never `auth_config` or the SMTP password. This is what makes the opt-in mechanism above (see [Opting a secret variable into a script’s environment](#opting-a-secret-variable-into-a-scripts-environment)) work: both gates must be set by someone holding the right permissions before the real value reaches a script, with only the captured-output view redacted afterward. The per-job `script_secret_vars` allowlist is an **authorization** boundary, not a **cryptographic** one: `SCRIPT_SECRETS_KEY` can decrypt *every* variable with `allow_in_scripts` set, not just the ones the currently-running job names. A well-behaved worker only ever fetches the values its own job’s allowlist names, but a compromised script-worker process holding this key could, in principle, decrypt any opted-in secret directly — not just the ones belonging to its own job. Every variable that has not opted in remains ciphertext-only to a script worker, undecryptable by either key it might hold. The curated child environment is **blast-radius reduction, not a boundary**. What actually contains the remaining exposure (global variables, script bodies) is the container’s own narrowed environment plus Postgres authentication. Closing it properly needs a restricted database role; that doesn’t exist today. * **Every script runtime’s filesystem access can be confined** via `bwrap` (bubblewrap): the script sees only its own runtime binaries, a writable temp directory for that run alone, and any [volumes](#volumes) the job declares, and nothing else — **but only when you set `SCRIPT_SANDBOX_ENABLED=1`**. This is opt-in and off by default, so an existing deployment never has script sandboxing silently turned on for it by an upgrade. Without it, one script’s temp files are visible to every other concurrently-running script job, and a script can read the whole container filesystem — the always-on hardening restricts *writes* to `/tmp`, but it is not a per-job boundary — enabling it is recommended; see [Installation](/start/installation/) for how. The `bwrap` wrapper also fails open even when enabled: if the binary is missing, or a real self-test of it fails on this host (most commonly a distro that restricts unprivileged user namespaces — see [Deploying](/guides/deploying/)), the worker logs a warning and scripts run without per-job confinement (the container hardening still applies) rather than failing. * **Outbound internet access is intentional** — the application network is deliberately not `internal`. * **Host-published ports defeat Docker network isolation.** A service published on the host’s address is reachable from a script regardless of which Docker network it’s on. The real control is **host-level egress filtering**, not Docker networking, and it must be in place before you enable script execution. * **Nothing bounds CPU, memory or process count** beyond the per-attempt timeout and whatever container limits you set, and disk writes only as far as the size-capped `/tmp` tmpfs — 128m for Bash, 256m for Node and Groovy, charged to the container’s memory limit (see [Container confinement settings](/guides/deploying/#container-confinement-settings)). Groovy’s `MaxRAMPercentage` is a per-JVM heap ceiling read from the cgroup limit, so `mem_limit` must cover five concurrent JVMs, not one — see [Deploying](/guides/deploying/). ## Before enabling script jobs in production [Section titled “Before enabling script jobs in production”](#before-enabling-script-jobs-in-production) 1. Put host-level egress filtering in place, and verify it **by IP, not hostname** — a hostname probe from an isolated container fails on DNS whether or not isolation exists, so it proves nothing while reading as a pass. 2. Confirm the script workers’ container environments really are narrowed to `DATABASE_URL`, `TZ` and `SCRIPT_SECRETS_KEY`. 3. Confirm `mem_limit` on the Groovy worker if you use Groovy. 4. Grant `jobs:script` to a specific role, deliberately — never alongside `jobs:write` by default. # Workflows > Graph-shaped scheduled work — branching, looping and multi-step pipelines built from blocks. A workflow is a second schedulable entity alongside a job. Where a job performs one HTTP call or one script run, a workflow executes a **graph of blocks**: a sequence of nodes that can branch, loop, and pass data forward from one step to the next. Reach for a workflow when the work is multi-step logic rather than a single call — call an API, parse its response, branch on a field, iterate the result one page at a time, and finish by recording success or failure. A single HTTP or script job cannot express that. A workflow is its own entity with its own schedule and its own permission keys. See [Job Types](/concepts/job-types/) for the job side of the system. ## Block types [Section titled “Block types”](#block-types) A workflow’s graph is built from these node types: | Type | Purpose | | -------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `http` | Make an HTTP call, using the same request-building machinery as an HTTP job — method, URL, query params, headers, body and auth. | | `script` | Run a script under one of the runtimes script jobs use: Bash, JavaScript, TypeScript or Groovy. Uses the same execution machinery as a script job — temp file, process group, timeout, and output capture/truncation. | | `parse` | Parse a prior step’s output as JSON or XML — or auto-detect the format — into structured data other blocks can reference. | | `if` | A branch node with two outgoing edges, `true` and `false`. | | `switch` | A branch node that routes to the first case whose value equals the chosen value, and otherwise to its `default` branch. See [Switch](#switch). | | `wait` | Pauses the run for a number of seconds, minutes or hours, up to 24 hours. The duration can be a `{{template}}` such as a `Retry-After` value. See [Wait](#wait). | | `set_variable` | Stores run-scoped values for later blocks in the run. These are not global variables. See [Set variable](#set-variable). | | `container` | A nested-execution node with two kinds: `for_each` and `while`. See [Branching and containers](#branching-and-containers). | | `break` / `continue` | Loop-control nodes usable inside a container, with the same semantics as `break` and `continue` in a general-purpose language. | | `success` / `fail` | Terminal nodes. | A `script` block’s body is edited in the same code editor as a script job. See [Editing a script in the browser](/guides/script-jobs/#editing-a-script-in-the-browser). A `script` block can also declare [volumes](/guides/script-jobs/#volumes) in its `data.capabilities`, in the same `{"volumes": [{"name": "...", "mode": "ro"|"rw"}]}` shape a script job uses, and picks them in the block’s configuration panel. The block gets the same mounts and `ZILEAN_VOLUME_` variables as a script job. Only `script` blocks accept a declaration: saving a workflow with a non-empty `capabilities` on any other block is rejected with `400`. Declarations are checked again each time the step runs, and a volume that is no longer usable fails the step without retrying. Saving a graph with a script block still needs `jobs:script`. A run ends in exactly one of three terminal statuses — **succeeded**, **failed** or **cancelled**. Cancelled is not reachable from a block: it happens when a workflow is disabled or deleted, or its group is disabled, while a run is mid-flight. A cancelled run records no error step; only a failed run records one. ## Building a workflow [Section titled “Building a workflow”](#building-a-workflow) The workflow editor is a canvas. A new workflow starts with a single **Start** block, which marks where every run begins; it cannot be removed and has nothing to configure. Add blocks from the palette by clicking one — it lands in the middle of the visible canvas — or by dragging it onto the spot you want. The first block you add is connected to Start automatically. Connect the rest by dragging from a block’s bottom handle to the next block’s top handle; an `if` block has two bottom handles, `true` on the left and `false` on the right. Select a block to edit its configuration in the side panel. Every block other than Start needs an incoming connection. Until the graph is complete, the editor names what is missing and **Save** stays disabled. Start itself is an editor marker rather than a block type: the run begins at whichever block Start points to. ### Building a loop body [Section titled “Building a loop body”](#building-a-loop-body) A `for_each` or `while` container appears on the canvas as a resizable box. The blocks inside the box are its body; the blocks outside it are not. To put blocks in a body: * drag a palette entry and release it over the box; * select the box, then click palette entries — each lands below the last block in the body (this is also how to add `break` and `continue`); * drag a block that is already on the canvas into the box. While you drag, the box the block will land in is highlighted and labelled **Drop into …**. Whether a block goes in is decided by where its centre is when you release it, and the box grows to fit a block placed near its edge, so a block is never left half inside. Dragging a block out until the box is no longer highlighted moves it back onto the canvas. Deleting a container deletes its body with it. Connections stay on one level: blocks inside the same container can be wired together, and blocks on the canvas can be wired together, but the editor won’t connect across a container’s border. If you drag a wired block into or out of a box, its old connection is drawn as a red dashed line and **Save** stays disabled until you reconnect it. ## Branching and containers [Section titled “Branching and containers”](#branching-and-containers) `if` gives the graph two branches. The node’s condition is evaluated and the run continues down the `true` edge or the `false` edge, so one workflow can take different paths depending on what an earlier step returned. A **container** nests a set of blocks and executes them repeatedly. There are two kinds: * **`for_each`** — iterate over a list, running the container’s body once per item and collecting one output per pass. * **`while`** — run the container’s body for as long as a condition holds. A container’s body is the set of blocks nested inside it; the editor draws it as a box around them (see [Building a loop body](#building-a-loop-body)). The body has exactly one entry block — the one no other body block connects into — and a run enters the body there without needing a connection from the container itself. `break` and `continue` apply inside a container — `break` leaves the loop, `continue` moves to the next iteration. While a `for_each` loop is open, the run stores a copy of its items. Nested loops each keep their own copy, and the copies held by all open loops together are capped at 2 MiB; a run that exceeds the cap fails with an error stating the size it reached and the limit. ### Switch [Section titled “Switch”](#switch) A `switch` block compares one value against a list of cases and continues down the branch of the first case that matches. * **Switch on** is the value being compared, usually a reference such as `{{fetch.status}}`. A reference that makes up the whole field keeps its type. Write the reference with braces. A bare name such as `fetch.status` is compared as the literal text, and the editor warns about it. See [Conditions](#conditions). * Each **case** has a value, which is fixed text or a reference, and its own connector on the block. A **default** connector always exists. * Cases are checked from top to bottom. The first case whose value equals the chosen value wins, and there is no fallthrough: only that case’s branch runs. * Numbers and true/false values compare by value, so a case of `1` matches a chosen value of `1`, `"1"` or `1.0`. Other values compare as text. * If no case matches, the run continues down `default`. An unresolved `{{...}}` in the chosen value stays as literal text, so it normally takes `default`. * If the chosen connector is not connected to anything, the branch ends there, the same as an unconnected `if` branch: inside a container it finishes that pass, and at the top level it ends the run as succeeded. * To send two values to the same block, add a case for each and connect both connectors to that block. * Two cases with the same fixed value are rejected on save, because the later one could never match. * Removing a case removes the connection on its connector. ### Wait [Section titled “Wait”](#wait) A `wait` block pauses the run, then continues to the next block. It has a **Duration** and a **Unit** of seconds, minutes or hours. * The duration is a positive whole number, or a reference that resolves to one, such as `{{fetch.headers.retry_after}}` (a number or numeric text). * The longest single wait is 24 hours. A fixed duration over the limit is rejected on save. A referenced duration is checked when the block runs. * A referenced duration that does not resolve, is zero, negative, fractional or not a number, or exceeds 24 hours fails the run at that step, with no retry. * A wait has no name and stores nothing. It needs a connection to a next block, unless it is the last block in a container’s body, where the wait comes before the container’s next pass. * Waits work inside `for_each` and `while` containers. A wait in a `while` loop repeats on every pass, so a run can last a very long time. * Nothing is held open during the wait. The run resumes when the time is up. While a run waits, it stays `running`. See [Concurrency and overlap](#concurrency-and-overlap) for how the wait interacts with the overlap lock and with disabling the workflow. ### Set variable [Section titled “Set variable”](#set-variable) A `set_variable` block writes one or more values into the run so later blocks can use them. It needs a name, like most blocks, and holds 1 to 50 **assignments**, each with a name, a value, a type and a mode. * An assignment name is letters, digits and underscores, starts with a letter or underscore, and is unique within the block. * Each assignment’s **Template** / **Expression** toggle picks how its value is worked out. **Template** is the default: the value is text that can contain references, the references are replaced by their values, and text with no reference is a fixed value. * A block named `vars` with an assignment `count` is read as `{{vars.count}}`, the same way every other block’s output is read. There are no bare top-level names. * The values are local to the run. A Set variable block never creates or changes a [global variable](/guides/http-jobs/#global-variables). * Assignments in one block cannot refer to each other, because the values are stored together after all of them are worked out. Read them from a later block instead. * A value derived from a secret variable is stored redacted, so later blocks see it masked. Reference the secret variable directly (for example `{{secret_var}}`) in the block that needs it. * Coercing a secret-derived value to `number`, `boolean` or `json` is rejected: the assignment fails, following the block’s **Continue on error** setting. * **Keep earlier outputs** and **Continue on error** work as they do on `parse`. In **Expression** mode the value is evaluated as an expression: `{{fetch.body.total}} * 2`, `upper(trim({{user.name}}))`, `{{page.next}} ?? 0`. * References are still `{{path}}` placeholders, but each one is bound as a value rather than spliced in as text. An unresolvable reference is nil, so `{{x}}.a` fails when the expression runs; write `{{x}}?.a` for nil-safe access. A reference cannot appear inside a string literal: `"a {{x}}"` is not substituted. Use concatenation instead, for example `"a " + string({{x}})`. A reference also cannot be used as a member name; use index access instead, as in `{{y}}[{{x}}]`. * Operators: `+ - * / % **`, `== != < > <= >=`, `&& || !`, the ternary `a ? b : c`, `??`, `contains`, `startsWith`, `endsWith`, `in`, the range `a..b`, member and index access, `?.`, and slices such as `x[a:b]`. The `matches` operator is not supported. `+` also joins strings. A `/` always yields a float, while `%` requires integers — JSON numbers are floats, so write `int(x) % 2`. Division by zero and a result that is not finite (`Inf`, `NaN`) fail the step. Integer arithmetic that overflows fails with an integer overflow error. * Functions, and only these: `len`, `upper`, `lower`, `trim`, `split`, `join`, `string`, `int`, `float`, `abs`, `round`, `floor`, `ceil`, `min`, `max`. `round` takes one argument — there is no precision argument. `map`, `filter` and the other collection functions are not available, and neither are `$env`, `let`, pipes or the `if … else … end` form. A function name used without parentheses is an error. * Limits: an expression is at most 1024 bytes and 200 nodes. An integer result must be within ±2^53 (the safe integer range). The result is capped at the step output size, 8 KiB of JSON. Evaluation also runs under a memory budget of roughly 1000 elements, so a large range such as `1..1000` fails. String operations are not counted by this budget. Instead, the referenced values together may not exceed 64 KiB, counting each use of a reference, so using one reference several times multiplies its size. Beyond that, the step fails with an error. The cap counts the whole referenced value, so referencing a large object just to read one field fails; reference the field directly instead (for example `{{fetch.body.total}}` rather than `{{fetch.body}}.total`). * `split` returns at most 1000 items, and the result of `join` is limited to 8 KiB. Otherwise the step fails with an error. `split(x, "")` returns one item per character, so it fails for a string longer than 1000 characters. Pass a count as the third argument, as in `split(x, "", 100)`, to get at most that many items, the last one holding the rest. The count must be a whole number. * An expression cannot reference a secret variable: the step fails with `cannot use a secret variable in an expression`. * A syntax error, an unknown function or an unknown name is rejected when the workflow is saved, with an error naming the block and the assignment. An error during evaluation fails the step the same way a failed type conversion does; with **Continue on error** on, it stores `{ok: false, error}` instead. Each assignment’s **type** controls the stored value, after either mode has produced it: | Type | Result | | --------- | ---------------------------------------------------------------------------------------------- | | `auto` | The value as resolved. A value that is a single reference keeps its type. This is the default. | | `string` | Text. Objects and arrays become JSON text. | | `number` | A number, or the step fails. | | `boolean` | `true` or `false`, or the step fails. | | `json` | Text parsed as JSON, or the step fails. A value that is already an object or array is kept. | In Template mode, a value made of a single reference that cannot be resolved stays as literal `{{...}}` text under `auto`, and fails under `number`, `boolean` and `json`. With **Continue on error** on, a failed conversion stores `{ok: false, error}` instead of failing the run. Inside a container’s body a stored value is visible to later blocks in the same pass. The next pass starts empty, and nothing stored in a body is visible after the container finishes. If the Set variable block is the last block in the body that produces output, its values become that pass’s entry in the container’s output array, which is a way to shape each pass’s result. ## Data flow between blocks [Section titled “Data flow between blocks”](#data-flow-between-blocks) A run carries a **context**: a key/value bag that blocks read from and write into. Each block that produces output stores it in the context under the block’s **name**, and any text field in a later block can read it with a `{{...}}` reference — `{{fetch.status}}`, `{{orders.items[0].id}}`. When the run reaches that field, the reference is replaced by the value itself. ### Block names [Section titled “Block names”](#block-names) Every block except `wait`, `break`, `continue`, `success` and `fail` needs a name, and a block’s name is how its output is referenced. A name is letters, digits and underscores, starts with a letter or underscore, and is unique in the workflow; `trigger` is reserved. The editor names each block as it is added — `http_1`, `parse_1`, `if_1`, `switch_1`, `set_variable_1`, `for_each_1` and so on — and you can rename it. The editor also avoids names already used by a global variable. ### What each block stores [Section titled “What each block stores”](#what-each-block-stores) | Source | Reference | What it holds | | ---------------------------- | ------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------- | | The run’s trigger | `{{trigger.source}}`, `{{trigger.at}}`, `{{trigger.input}}` | What started the run (`cron`, `rate`, `one_off`, `manual` or `webhook`), when it started, and the input passed to a manual run. | | `http` block | `{{name.ok}}`, `{{name.status}}`, `{{name.body}}`, `{{name.error}}` | Whether the call succeeded, the HTTP status code, the response body **as text**, and the error message when it failed. | | `script` block | `{{name.ok}}`, `{{name.exit_code}}`, `{{name.stdout}}`, `{{name.stderr}}`, `{{name.error}}` | Whether the script exited 0, its exit code, its output streams as text, and the error message when it failed. | | `parse` block | `{{name}}`, `{{name.field}}`, `{{name.list[0]}}` | The parsed value itself — its shape is whatever the JSON or XML contained. | | `set_variable` block | `{{name.assignment}}` | The value of each assignment, converted to its type. For example, `{{vars.count}}`. | | Container, after it finishes | `{{name}}` | An array with each pass’s last output. | | Inside a container’s body | `{{name.item}}`, `{{name.index}}` | The current item (`for_each` only) and the pass number, counting from 0. | | Global variable | `{{variable}}` | The variable’s value. Only a bare name — a global has no fields. | `if`, `switch`, `wait`, `break`, `continue`, `success` and `fail` store nothing. An `http` block’s body and a `script` block’s output are plain text, so `{{fetch.body.items}}` does not reach into them. Put a `parse` block after the call with `{{fetch.body}}` as its input, then reference the parsed value: `{{parsed.items}}`. ### Which references a block can use [Section titled “Which references a block can use”](#which-references-a-block-can-use) A block can use the outputs of the blocks that run before it on its path from Start, the trigger, and global variables. Inside a container’s body it can also use that container’s `item`/`index`, the outputs of body blocks before it, and everything available to the container itself. Blocks inside a body are not visible outside it — after the loop, use the container’s own output array. A reference is looked up in this order: the enclosing containers’ `item`/`index`, then block outputs, then global variables. A block whose name matches a global variable therefore hides that variable everywhere in the workflow; the editor warns when a name does this. Each block’s config panel has an **Available data** list showing every reference that block can use at its place in the graph; click one to copy it. Typing `{{` in any field offers the same list, and typing `{{fetch.` narrows it to that block’s fields. `http`, `script`, `parse`, `set_variable` and container blocks have a **Keep earlier outputs** setting, on by default. Turning it off makes the block’s output *replace* what is stored at its level instead of joining it, so earlier outputs — and, at the top level, the trigger — are no longer available to the blocks after it. For example, with it off on a `parse` block that reads `{{fetch.body}}`, the blocks after it see only the parsed value, and the large raw response is not carried through the rest of the run. Global variables are unaffected. ### How a reference is filled in [Section titled “How a reference is filled in”](#how-a-reference-is-filled-in) * In a text field — a URL, a header, a body — a number, boolean, object or array is written as JSON text. * In a field that is used as a value — an `if` or `while` condition, a `for_each` source, a `parse` input, a script block’s environment values — a reference that makes up the whole field keeps its type, so `{{parsed.items}}` stays an array and `{{fetch.status}}` stays a number. * A reference that cannot be resolved — a typo, or a block on a branch that did not run — is left as its literal `{{...}}` text rather than becoming empty. In an assignment’s Expression mode it is nil instead. ### Conditions [Section titled “Conditions”](#conditions) An `if` block (or a `while` container) compares a **Field** with a **Value** using an operator such as `==`, `>`, `contains`, `exists` or `is_empty`. Both inputs follow the rules above. Field is almost always a reference — `{{fetch.status}}` — and Value is usually fixed text such as `200`, though it can be a reference too. Field needs the braces. `fetch.status` without them is compared as the literal text “fetch.status”, so `== 200` is never true and `exists` is always true; the editor warns when a condition’s Field, a `switch` block’s **Switch on** value, or a `for_each` source contains no reference. ## Triggers and scheduling [Section titled “Triggers and scheduling”](#triggers-and-scheduling) A workflow supports the same schedule types as a job: | `schedule_type` | Fires | | --------------- | ------------------------------------------------------------------ | | `cron` | On a cron expression. | | `rate` | At a fixed interval. | | `one_off` | Exactly once, at `run_at`. | | `manual` | Never on its own — started only by an explicit **Run now** action. | The cron, rate and one-off mechanics are shared with jobs — expression format, timezone handling, the reconciler, and the fields each type uses. See [Scheduling](/concepts/scheduling/) for the full description rather than a restatement here. `manual` is the workflow-only addition: a workflow with no schedule at all, started only when someone asks for it. ### One-off [Section titled “One-off”](#one-off) A one-off workflow fires exactly once, at `run_at`, using the same `run_at`-based mechanism a one-off job uses. It waits for its scheduled time before it is considered to have started, and it does not fire early. Until that time arrives it is a schedule waiting for its moment — not a stalled or abandoned run. The run and its `scheduled` marker commit together, so a failed insert leaves the workflow unscheduled and it can be retried. After it runs, `one_off_after_run` decides what remains: `keep` (the default) leaves the row disabled for inspection, and `delete` removes it once the run finishes. See [Scheduling](/concepts/scheduling/#what-happens-after-it-runs). ## Concurrency and overlap [Section titled “Concurrency and overlap”](#concurrency-and-overlap) A workflow run is **sequential**. There is no pooling and no equivalent of a job’s `pool_count`: one firing produces one run, and its blocks execute in the order the graph dictates. Overlap is handled at the whole-workflow level with a single lock, using the same skip/allow vocabulary a job’s overlap policy uses. Rather than jobs’ per-pool-slot locking, the choice applies to the workflow as a whole: a new firing either starts a concurrent run or is skipped because one is already in progress. A run that is waiting in a `wait` block is still in progress. With the `skip` policy the workflow stays locked until the wait ends, so scheduled firings during the wait are skipped. With `allow`, no lock is held and new firings start their own runs. A waiting run is not treated as stalled while it waits; if its next step has not started within about an hour of the wait ending, it is failed as stalled. Disabling a workflow, or its group, during a wait does not stop the wait. The run is cancelled when the wait ends and its next step is about to run. Deleting the workflow removes its runs. If the workflow is re-enabled before the wait ends, the run continues normally. For looping, retry and pooling on the job side, see [Loop, retry & pooling](/concepts/loop-retry-and-pooling/). Branching and iteration are expressed as blocks in a workflow’s graph instead, and there is no pooling to configure. ## Run monitoring [Section titled “Run monitoring”](#run-monitoring) Every workflow run is recorded with its status — `running`, `succeeded`, `failed` or `cancelled`. Every block that executes within a run is recorded as its own **step**, with a status, timing and output. In the run detail, each step is labelled with its block’s name, or its block type for an unnamed block such as `wait`, `success` or `fail`, with the block’s internal id shown as secondary text. Labels come from the workflow as it was when the run started, so later edits or renames do not change how an older run reads. When a workflow is deleted, disabled, or its group disabled while a run is in flight, the run ends as `cancelled` at its next step. No error step is added to the step list for that. Failures, such as a step that cannot be dispatched or an unknown block, end the run as `failed` and record an error step. A run that is paused in a `wait` block stays `running` and shows **Waiting until** the time it resumes, so it does not read as stuck. The `wait` step’s output records that resume time. That gives a per-step run history shaped like the graph, rather than a job’s flat attempt sequence. The runs list and the run detail view are where you read it. See [Monitoring jobs](/guides/monitoring/) for the monitoring surfaces available across the system. ## Permissions [Section titled “Permissions”](#permissions) | Action | Permission | | -------------------------------------------------------------------- | ------------------------------------------------- | | List and get workflows, their runs, and their run steps. | `workflows:read` | | Create, update, delete, enable/disable, and manually run a workflow. | `workflows:write` | | Save a workflow whose graph contains any `script` block. | `jobs:script`, *in addition to* `workflows:write` | `workflows:read` and `workflows:write` are distinct from `jobs:read` and `jobs:write` because a workflow is its own schedulable entity, not a `scheduled_jobs` row. Admin carries both keys; Read-Only carries `workflows:read` only — authoring a workflow that could contain a script block is treated at the same tier as script-job authoring, so it stays out of Read-Only by default. See [Permissions](/reference/permissions/) for the full key list and built-in role defaults. The `jobs:script` check happens **after the graph is decoded**, because whether a workflow contains a script block is only known once its body has been parsed. That mirrors the script-job gate, which is enforced inside the create and update handlers for the same reason. ## Opting a secret variable into a script block’s environment [Section titled “Opting a secret variable into a script block’s environment”](#opting-a-secret-variable-into-a-script-blocks-environment) A `script` block receives secret global variables through the identical two-gate mechanism a script job uses: 1. On the **Global Variables** settings page, the variable must have **Allow in scripts** (`allow_in_scripts`) turned on. This requires `globalvars:write`. On its own it exposes the variable to no workflow. 2. The workflow’s own allowed-secrets list must name that variable. This requires both `jobs:script` and `workflows:write`. A secret reaches a script block’s environment only when **both** gates are set. The environment-injection mechanics are identical to a script job’s — the variable name is normalized to `GLOBALVAR_`, and an injected value is redacted from the step’s captured output. See [Opting a secret variable into a script’s environment](/guides/script-jobs/#opting-a-secret-variable-into-a-scripts-environment) for the mechanics in full rather than a duplicate here. A script block runs arbitrary code A `script` block uses the same execution machinery as a script job, so it inherits the same characteristics and the same limits — including exactly what the runtime filesystem sandbox does and does not cover. Read [Script jobs](/guides/script-jobs/) before placing script blocks in a workflow that runs against infrastructure you care about. # API Reference > Every admin API route, its permission, parameters and response shape. All routes are mounted under `/api` on `zilean-core` and require a valid session cookie plus the permission listed. Authentication failures redirect — they don’t 401 An unauthenticated request gets a **302 to `/login`**. A client that follows redirects receives login-page HTML with a 200 status. Check for that rather than trusting the status code. A user who must change their password gets `403 {"error":"password change required"}` on `/api/*` until they do. ## Jobs [Section titled “Jobs”](#jobs) | Method | Path | Permission | | -------- | -------------------- | ---------------------------------------------- | | `GET` | `/jobs` | `jobs:read` | | `POST` | `/jobs` | `jobs:write` (+ `jobs:script` for script jobs) | | `GET` | `/jobs/{id}` | `jobs:read` | | `PUT` | `/jobs/{id}` | `jobs:write` (+ `jobs:script` for script jobs) | | `DELETE` | `/jobs/{id}` | `jobs:write` | | `POST` | `/jobs/{id}/enable` | `jobs:write` | | `POST` | `/jobs/{id}/disable` | `jobs:write` | | `POST` | `/jobs/{id}/run` | `jobs:write` | | `POST` | `/jobs/enable-all` | `jobs:write` | | `POST` | `/jobs/disable-all` | `jobs:write` | A script job’s `capabilities` (declared volumes) is part of the job object on both requests and responses. `POST` and `PUT` return `400` if it is non-empty on an HTTP job or names an unknown volume, or a mode above the volume’s ceiling. Job responses also carry a read-only `capabilities_invalid` boolean. It is `true` when a job’s stored volume declaration cannot be read; `capabilities` then shows `{}`, and the job is cancelled at run time instead of running without its volumes. A `PUT` that includes `capabilities` replaces the unreadable declaration; a `PUT` that omits it leaves it as is. `PUT` takes the **full** job object — there are no partial updates. See [Job fields](/reference/job-fields/). `POST` and `PUT` return `400` when `group_name` is over 255 bytes or 16 levels deep (`group_name must be at most 255 bytes` / `group_name must be at most 16 levels deep`). A job whose stored `group_name` is over a limit still displays and can still be enabled, disabled, run and deleted; saving it through `PUT` must bring the value within the limits in the same save. **`POST /jobs/{id}/run`** triggers immediately, bypassing the reconcile-window uniqueness guard. Notable responses: | Status | Meaning | | ------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `409` | Every pool index is already running, the job is disabled, its group is disabled, or the job is `one_off` (a manual trigger would run it early) or `webhook` (a manual trigger would fire it with an empty payload). | ## Groups [Section titled “Groups”](#groups) | Method | Path | Permission | Body | | ------ | ------------------ | ------------ | ----------------------- | | `GET` | `/groups/disabled` | `jobs:read` | — | | `POST` | `/groups/enable` | `jobs:write` | `{"group_name": "..."}` | | `POST` | `/groups/disable` | `jobs:write` | `{"group_name": "..."}` | A disabled group gates its jobs — checked by the scheduler and again by the worker at execution time. `POST /groups/disable` returns `400` for a group name over the 255-byte or 16-level limit (the same messages as jobs: `group_name must be at most 255 bytes` / `group_name must be at most 16 levels deep`) and rejects a name that is blank after trimming segments — only slashes or whitespace — with `group_name is required`. `POST /groups/enable` accepts any non-empty name regardless of length or depth, so a group disabled before the limits existed can still be re-enabled. ## Job history and activity [Section titled “Job history and activity”](#job-history-and-activity) | Method | Path | Permission | | ------ | --------------------- | ----------- | | `GET` | `/jobs/{id}/calls` | `jobs:read` | | `GET` | `/jobs/heartbeats` | `jobs:read` | | `GET` | `/jobs/{id}/activity` | `jobs:read` | **`/jobs/{id}/calls`** — `?limit=` default `20`, maximum `100`. A non-integer or non-positive value is a `400`. Returns a bare array (`[]`, never `null`), newest first. Each entry: `job_id` (River’s queue job id, **not** the registry UUID), `state`, `attempt`, `attempted_at`, `finalized_at`, `scheduled_at`, `last_error`, `pool_index`, `loop_iteration`, and `last_call`. For HTTP jobs, secret values are replaced with `REDACTED` in `last_call.url`, `last_call.response_snippet` and the error fields. **`/jobs/heartbeats`** returns every job in one response: ```json { "window_start": "...", "bucket_width_seconds": 60, "slot_count": 40, "jobs": [{ "job_id": "...", "buckets": [] }] } ``` A job with no history gets `"buckets": []`. **`/jobs/{id}/activity`** returns the same window geometry plus `buckets` and lifetime `invocations_total` / `invocations_completed` / `invocations_failed` — which cover the job’s whole life, not the window. ### Shared window parameters [Section titled “Shared window parameters”](#shared-window-parameters) | Param | Default | Cap | Invalid | | --------- | ------- | ---------------------------- | ------------------------------------------------- | | `since` | `24h` | `720h`, **clamped silently** | `400 since must be a positive duration, e.g. 24h` | | `buckets` | `40` | `500`, clamped silently | `400 buckets must be a positive integer` | A bucket contains `bucket_start`, `total`, `completed`, `failed`, `duration_ms_avg`, `duration_ms_max`. The duration fields are only populated on the per-job activity route. ## Job invocation alarms [Section titled “Job invocation alarms”](#job-invocation-alarms) | Method | Path | Permission | | -------- | ----------------------------- | -------------- | | `GET` | `/jobs/{id}/alarms` | `jobs:read` | | `POST` | `/jobs/{id}/alarms` | `notify:write` | | `PUT` | `/jobs/{id}/alarms/{alarmID}` | `notify:write` | | `DELETE` | `/jobs/{id}/alarms/{alarmID}` | `notify:write` | A per-job alarm on the job’s invocation count over a rolling window. `GET` returns a bare array (`[]`, never `null`), and an empty array for a job without any. `POST` returns `201` with the created alarm, `PUT` returns `200` with the updated one, and `DELETE` returns `204`. `POST` and `PUT` take the same body: ```json { "enabled": true, "metric": "failed", "threshold": 5, "window_minutes": 30 } ``` | Field | Type | Constraints | | ---------------- | ------ | -------------------------------------------------------------------------------------------------- | | `enabled` | bool | A disabled alarm is kept but not evaluated. | | `metric` | string | `total` (every attempt, including loop iterations and retries) or `failed` (failed attempts only). | | `threshold` | int | `1 … 2147483647`. Fires at or above this count. | | `window_minutes` | int | `1 … 1440`. Counted in 1-minute buckets, including the current partial minute. | An alarm is returned in this shape: ```json { "id": 3, "job_id": "...", "enabled": true, "metric": "failed", "threshold": 5, "window_minutes": 30, "last_state": "breached", "last_changed_at": "2026-09-22T20:01:00Z", "created_at": "2026-09-22T19:00:00Z" } ``` `last_state` is `ok` or `breached`, and `last_changed_at` is `null` until the state first changes. An unknown job id, or an `alarmID` belonging to a different job, is a `404`; an invalid body is a `400`. See [Monitoring jobs](/guides/monitoring/#invocation-alarms) for the evaluation and notification behavior. ## Workflows [Section titled “Workflows”](#workflows) | Method | Path | Permission | | -------- | ------------------------------------ | ----------------- | | `GET` | `/workflows` | `workflows:read` | | `POST` | `/workflows` | `workflows:write` | | `GET` | `/workflows/{id}` | `workflows:read` | | `PUT` | `/workflows/{id}` | `workflows:write` | | `DELETE` | `/workflows/{id}` | `workflows:write` | | `POST` | `/workflows/{id}/enable` | `workflows:write` | | `POST` | `/workflows/{id}/disable` | `workflows:write` | | `POST` | `/workflows/{id}/run` | `workflows:write` | | `GET` | `/workflows/{id}/runs` | `workflows:read` | | `GET` | `/workflows/{id}/runs/{runID}` | `workflows:read` | | `GET` | `/workflows/{id}/runs/{runID}/steps` | `workflows:read` | | `POST` | `/workflows/enable-all` | `workflows:write` | | `POST` | `/workflows/disable-all` | `workflows:write` | `GET /workflows`, `GET /workflows/{id}/runs` and `GET /workflows/{id}/runs/{runID}/steps` each return a bare array (`[]`, never `null`) when there is nothing to list. `POST /workflows` and `PUT /workflows/{id}` apply the same `group_name` limits as jobs — 255 bytes and 16 levels deep; an over-limit value is a `400`. `workflows:read`/`workflows:write` are distinct from `jobs:read`/`jobs:write` — a workflow is its own schedulable entity, not a `scheduled_jobs` row. Saving a workflow whose graph contains any `script` block also requires `jobs:script`, checked after the graph is decoded (the same reason `jobs:script` is enforced in the job handlers rather than at the route). See [Workflows](/guides/workflows/) for the graph model, block types, and run-monitoring surfaces these routes back. ## Metrics [Section titled “Metrics”](#metrics) | Method | Path | Permission | | ------ | ------------------ | -------------- | | `GET` | `/metrics/live` | `metrics:read` | | `GET` | `/metrics/history` | `metrics:read` | `?service=` accepts `core`, `ui`, `worker`, `mcp`, `webhook`, `script-worker-bash`, `script-worker-node`, `script-worker-groovy`. Empty defaults to `worker`; an unknown value is a **400**. `live` returns the latest sample, or JSON `null` with a `200` if that service has never sampled. `history` accepts `since` and `buckets` as above; `service=all` switches the response to `samples_by_service` and includes a worker-wide `invocations` series. The Prometheus endpoint is `/prometheus/metrics` (not under `/api`), also requiring `metrics:read`. ## Scripts [Section titled “Scripts”](#scripts) | Method | Path | Permission | | ------ | ------------------- | ------------- | | `GET` | `/scripts/liveness` | `jobs:script` | Returns, per script job type, `{"live": bool, "last_sample_at": }`. A runtime is live if it has reported a metrics sample within 2 minutes. ## Script volumes [Section titled “Script volumes”](#script-volumes) | Method | Path | Permission | | -------- | ------------------------ | --------------------- | | `GET` | `/script-volumes` | `scriptvolumes:read` | | `POST` | `/script-volumes` | `scriptvolumes:write` | | `PUT` | `/script-volumes/{name}` | `scriptvolumes:write` | | `DELETE` | `/script-volumes/{name}` | `scriptvolumes:write` | | `GET` | `/script-workers/status` | `scriptvolumes:read` | A volume is `{"name", "max_mode", "description", "usage", "created_at", "updated_at"}`. `usage` is the number of jobs plus the number of workflows that declare it; a workflow counts once however many of its script blocks use the volume. `name` must match `^[a-z][a-z0-9_]{0,62}$`, `max_mode` is `ro` or `rw`, and `PUT` changes only `max_mode` and `description`. Volume create, update and delete are recorded in the audit log. `DELETE` removes the catalog entry only and never touches files on disk. If a job or workflow still declares the volume it returns `409`: ```json { "error": "", "jobs": [{ "id": "", "name": "nightly-report" }], "workflows": [{ "id": "", "name": "export-pipeline" }] } ``` `GET /script-workers/status` returns one entry per script worker queue: ```json [ { "queue": "scripts-bash", "sandbox_enforced": true, "volumes_root_mounted": true, "updated_at": "2026-10-06T12:00:00Z", "stale": false }, { "queue": "scripts-node", "sandbox_enforced": true, "volumes_root_mounted": false, "updated_at": "2026-10-06T12:00:00Z", "stale": false } ] ``` `volumes_root_mounted` is `false` when that worker’s volumes root is unset, missing, not a directory, `/`, or not writable by the worker. A false value means the worker treats the root as not mounted and still runs jobs that declare no volume; when the value is set but unusable the worker also logs a warning, and jobs that declare a volume fail. The check runs once, at worker startup, and the heartbeat repeats its result for the process’s lifetime, so restart the worker after fixing ownership or permissions. `stale` is `true` when the worker has not reported for more than 3 minutes. See [Volumes](/guides/script-jobs/#volumes). ## System health [Section titled “System health”](#system-health) | Method | Path | Permission | | ------ | ---------------- | -------------- | | `GET` | `/system/health` | `metrics:read` | Returns a JSON array, one entry per running service (`worker`, `script-worker-bash`, `script-worker-node`, `script-worker-groovy`, `core`, `ui`, `mcp`, `webhook`): ```json [{ "service": "worker", "live": true, "last_sample_at": "2026-09-22T12:00:00Z" }] ``` `live` is true only if that service’s latest metrics sample is newer than 2 minutes; `last_sample_at` is `null` if the service has never sampled. This is a status roll-up derived from the same stored samples `/metrics/live` and `/scripts/liveness` read — it adds no new sampling. Unlike `/scripts/liveness`, a read failure for one service degrades just that entry to “never seen” rather than failing the whole request. See [Monitoring jobs](/guides/monitoring/#system-health). ## Global variables [Section titled “Global variables”](#global-variables) | Method | Path | Permission | | -------- | ------------------------ | ------------------ | | `GET` | `/global-variables` | `globalvars:read` | | `POST` | `/global-variables` | `globalvars:write` | | `PUT` | `/global-variables/{id}` | `globalvars:write` | | `DELETE` | `/global-variables/{id}` | `globalvars:write` | A variable has `name`, `value`, `is_secret` and `allow_in_scripts`. Names may contain letters, digits and underscores, and are referenced from jobs as `{{name}}`. `allow_in_scripts` only matters when `is_secret` is `true` — it marks the variable as eligible to be injected into a script job’s environment; a job still needs the variable named in its own `script_secret_vars` list to actually receive it. See [Script jobs](/guides/script-jobs/#opting-a-secret-variable-into-a-scripts-environment). A secret variable’s value is write-only. Responses return `value: ""` for a secret, plus a `has_value` boolean saying whether one is stored. On `PUT`, an empty `value` for a variable that is secret before and after the update keeps the stored value. Changing a variable from secret to non-secret with an empty `value` clears it rather than revealing it. Audit log entries for a global variable mask a secret’s value the same way. ## Account [Section titled “Account”](#account) | Method | Path | Permission | | ------ | -------------------------- | ---------------------- | | `GET` | `/account/me` | Any authenticated user | | `POST` | `/account/change-password` | Any authenticated user | `/account/me` returns `{email, is_super_admin, must_change_password, permissions[]}`. `change-password` takes `{current_password, new_password}` → `204`. A wrong current password is a `400`, as is a `new_password` shorter than 8 characters or longer than 72 bytes (bcrypt’s input limit). ## Users [Section titled “Users”](#users) | Method | Path | Permission | | -------- | ---------------------------- | ------------- | | `GET` | `/users` | `users:read` | | `POST` | `/users` | `users:write` | | `PATCH` | `/users/{id}` | `users:write` | | `POST` | `/users/{id}/reset-password` | `users:write` | | `DELETE` | `/users/{id}` | `users:write` | `POST /users` takes `{email, role_ids}` and returns `201 {user, temp_password}` — the plaintext password is returned **once** and never stored. `role_ids` is a non-empty array; a user can hold more than one role, and effective permissions are the union of every assigned role’s permissions. `PATCH` accepts `{role_ids?, status?}`; `status` must be exactly `active` or `disabled`. A supplied `role_ids` replaces the full set and must not be empty — a user always has at least one role. Disabling and deleting both revoke that user’s sessions immediately. Super Admin targets return `409` on role change, status change, delete and reset-password. `GET /users` returns a bare array (`[]`, never `null`). ## Roles [Section titled “Roles”](#roles) | Method | Path | Permission | | -------- | -------------------- | ------------- | | `GET` | `/roles` | `roles:read` | | `GET` | `/roles/permissions` | `roles:read` | | `POST` | `/roles` | `roles:write` | | `PATCH` | `/roles/{id}` | `roles:write` | | `DELETE` | `/roles/{id}` | `roles:write` | `/roles/permissions` is the live list of permission keys. The three system roles cannot be renamed or deleted. Super Admin’s permissions can’t be edited either; Admin and Read-Only can have their permissions updated via `PATCH /roles/{id}` (name must be left unchanged). `GET /roles` returns a bare array (`[]`, never `null`). ## Invites [Section titled “Invites”](#invites) | Method | Path | Permission | | ------ | ---------------------- | ---------------- | | `POST` | `/invites` | `invites:create` | | `GET` | `/invites` | `invites:create` | | `POST` | `/invites/{id}/resend` | `invites:create` | `POST /invites` takes `{email, role_ids}` → `201 {invite, link}`. `role_ids` is a non-empty array — an invitee can be assigned more than one role. Tokens last **7 days**; only their hash is stored. `GET /invites` returns a bare array (`[]`, never `null`) of pending invites. Caution `{id}` on resend is the **invitee’s user id**, not the invite row’s id. Resend returns `409` unless the target’s status is exactly `invited`, creates a fresh token, and returns the new link — **it does not send an email**. ## Audit log [Section titled “Audit log”](#audit-log) | Method | Path | Permission | | ------ | ----------------- | ------------- | | `GET` | `/audit-log` | `audit:read` | | `GET` | `/settings/audit` | `audit:read` | | `PUT` | `/settings/audit` | `audit:write` | `/audit-log` filters: `actor_user_id` (uuid), `action`, `resource_type`, `since` / `until` (RFC 3339), `cursor`. Malformed values are a `400`. Returns `{"entries": [...], "next_cursor": "..."}` — newest-first, keyset-paginated. `entries` is an empty array (`[]`, never `null`) when nothing matches. **Page size is fixed at 50** and is not client-tunable. `/settings/audit` reads and writes `{"audit_retention_days": n}`, accepting **1–3650**. Changing it is not itself audited. ## Notifications [Section titled “Notifications”](#notifications) | Method | Path | Permission | | -------- | ------------------------------ | ----------------------- | | `GET` | `/notify/targets` | `notify_settings:read` | | `POST` | `/notify/targets` | `notify_settings:write` | | `PUT` | `/notify/targets/{id}` | `notify_settings:write` | | `DELETE` | `/notify/targets/{id}` | `notify_settings:write` | | `POST` | `/notify/targets/{id}/test` | `notify_settings:write` | | `GET` | `/settings/apprise` | `notify_settings:read` | | `PUT` | `/settings/apprise` | `notify_settings:write` | | `POST` | `/settings/apprise/test` | `notify_settings:write` | | `GET` | `/notify/thresholds` | `notify_settings:read` | | `PUT` | `/notify/thresholds/{service}` | `notify_settings:write` | A notify target has a `kind` of `apprise`, `email` or `webhook` — all three share the same `/notify/targets` route family. A job’s own `notify_on`/`notify_all_targets`/ `notify_targets` fields are read and written through the regular [Jobs](#jobs) routes instead, gated by `notify:read`/`notify:write` rather than `notify_settings:*`. See [Configuration](/start/configuration/#notifications-is-a-settings-page-feature-too) for target kinds and the Apprise server connection, and [Monitoring jobs](/guides/monitoring/#alerting-on-worker-metrics) for metrics alarms (`/notify/thresholds`); per-job invocation alarms use the [alarm routes](#job-invocation-alarms) instead. ## Outbound email (SMTP) [Section titled “Outbound email (SMTP)”](#outbound-email-smtp) | Method | Path | Permission | | ------ | --------------------- | --------------- | | `GET` | `/settings/smtp` | `settings:smtp` | | `PUT` | `/settings/smtp` | `settings:smtp` | | `POST` | `/settings/smtp/test` | `settings:smtp` | `GET` returns `{host, port, username, from, enabled, has_password}` — the password itself is **never** returned, only the boolean `has_password`. It is stored encrypted, and is masked in the audit log. `PUT` takes `{host, port, username, password, from, enabled}`. An **empty `password` means “keep the existing one”**, so the UI can save other fields without re-entering it. `enabled: true` with an empty `host` is a `400`. `POST /settings/smtp/test` sends a test message, taking `{"to": "..."}`. An optional `draft` object of the same shape as the `PUT` body tests **unsaved** edits instead of the stored settings — a blank password keeps the saved one only when `host` and `port` match the saved settings; otherwise the request is a `400` asking for the password. Responses: | Status | Meaning | | ------ | -------------------------------------------------------- | | `200` | `{"ok": true}` — the message was sent. | | `400` | `to` is missing. | | `409` | No `draft` was supplied and SMTP isn’t configured. | | `502` | The send itself failed; the body carries the SMTP error. | ## River job retention [Section titled “River job retention”](#river-job-retention) | Method | Path | Permission | | ------ | ---------------------- | --------------------- | | `GET` | `/settings/river-jobs` | `settings:river_jobs` | | `PUT` | `/settings/river-jobs` | `settings:river_jobs` | `/settings/river-jobs` reads and writes `{"river_job_retention_days": n}`, accepting **1–3650**. Unlike the audit log (a `zilean` table), this governs River’s own `river_job` table — finished (`completed`/`cancelled`/`discarded`) rows older than the window are swept daily by `zilean-core`’s scheduler, the same as `audit_retention_days`. River’s own built-in job cleaner is disabled in every worker binary so the two mechanisms don’t race on the same rows; this setting is now the only thing controlling `river_job` retention. ## MCP tokens [Section titled “MCP tokens”](#mcp-tokens) | Method | Path | Permission | | -------- | ------------------ | -------------- | | `GET` | `/mcp-tokens` | `settings:mcp` | | `POST` | `/mcp-tokens` | `settings:mcp` | | `DELETE` | `/mcp-tokens/{id}` | `settings:mcp` | | `GET` | `/settings/mcp` | `settings:mcp` | | `PUT` | `/settings/mcp` | `settings:mcp` | A token authenticates the MCP endpoint; this API is the admin-facing management surface for it (issuing and revoking tokens, and toggling the endpoint on or off). `POST /mcp-tokens` takes `{"name": "...", "role_id": "...", "expires_in": "30d"}` — `expires_in` is one of `30d`, `90d`, `365d`, or `never`. The response’s `token` field is the plaintext bearer credential and is returned **exactly once**; only its sha256 hash is ever stored. `role_id` must name a role whose permissions don’t exceed the caller’s own, and can never name the Super Admin role (see [Permissions](/reference/permissions/)). Each token is backed by a hidden, non-interactive user account carrying that role, deleted together with the token on `DELETE /mcp-tokens/{id}`. `GET /mcp-tokens` lists name, linked role, creation time, expiry, and last-used time — never the token value itself, which isn’t recoverable after creation. Returns a bare array (`[]`, never `null`) when no tokens exist. `PUT /settings/mcp` takes `{"enabled": bool}`, gating whether the MCP endpoint accepts connections at all. `GET`/`PUT` both respond with that same field plus `live` and `checked_at`: whether a live health probe against the separate `cmd/mcp` process (its own `GET /healthz`, 2s timeout) just succeeded, and when that probe ran. This is informational only — enabling MCP while `cmd/mcp` is unreachable still succeeds; the admin UI just warns, since `cmd/mcp` runs as its own binary/process and isn’t guaranteed to be deployed just because core/ui are (see [Architecture](/concepts/architecture/)). Where that probe points is configurable server-side via `MCP_HEALTH_URL` (see [Deploying](/guides/deploying/)). ## Job webhooks [Section titled “Job webhooks”](#job-webhooks) | Method | Path | Permission | | ------ | --------------------------- | ------------ | | `GET` | `/jobs/{id}/webhook` | `jobs:read` | | `POST` | `/jobs/{id}/webhook/rotate` | `jobs:write` | A job whose `schedule_type` is `webhook` has a per-job token that fires it. This API is the admin-facing management surface for that token — reading its identification hint and rotating it. Chains of webhook-triggered jobs are bounded by the loop guard described in [Scheduling](/concepts/scheduling/). `GET /jobs/{id}/webhook` returns the token’s 8-character `webhook_token_hint` and when it was issued, never the token itself: only a sha256 digest is stored, so the plaintext is **not recoverable** after it is first shown. `POST /jobs/{id}/webhook/rotate` issues a fresh token, returns the plaintext **exactly once**, and invalidates the previous URL immediately — no redeploy is needed. A routine edit of the job never rotates the token. ## Meta [Section titled “Meta”](#meta) | Method | Path | Permission | | ------ | ------- | ---------- | | `GET` | `/meta` | None | Returns `{"timezone": "..."}` — the zone the scheduler evaluates cron expressions in. The only ungated API route: harmless server info the UI shell needs before any permission check. ## Non-API routes [Section titled “Non-API routes”](#non-api-routes) Served by `zilean-ui`: | Path | Notes | | -------------------------------------------------------------- | ------------------------------------------------------------------------------------ | | `/` | Admin SPA (client-side routes: `/jobs`, `/metrics`, `/settings`, `/account`). | | `/riverui/*` | Embedded River UI. Requires **`jobs:write`** — it can retry, cancel and delete jobs. | | `/login`, `/logout`, `/admin/assets/*`, `/admin/accept-invite` | Registered outside the auth middleware. `/logout` accepts `POST` only. | # Environment Variables > Every environment variable Zilean reads, which service reads it, and its default. ## Required [Section titled “Required”](#required) | Variable | Read by | Description | | ------------------------- | ---------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `DATABASE_URL` | **all eight services** | Postgres DSN. Startup is **fatal** without it (`DATABASE_URL is required`). The migration runner rewrites a `postgres://`/`postgresql://` scheme to `pgx5://` internally. | | `ADMIN_USERNAME` | core | Login email of the bootstrap Super Admin seeded on first boot. **Must contain `@`** or core refuses to start. | | `ADMIN_PASSWORD_HASH` | core | bcrypt hash for that Super Admin. Generate with `docker run --rm httpd:2.4-alpine htpasswd -bnBC 10 "" '...' \| cut -d: -f2`. | | `SETTINGS_ENCRYPTION_KEY` | core, worker | 32-byte key (base64) encrypting secrets at rest: the SMTP password (see [Configuration](/start/configuration/#email-is-a-settings-page-feature)), a job’s `auth_config` secret (bearer token / basic password / API key), a webhook notification target’s signing secret, and the value of any global variable marked secret. `worker` needs it too, to apply `auth_config` and secret variables to real outgoing HTTP calls — the three script workers, `zilean-mcp` and `zilean-webhook` deliberately do not get it, and the last two refuse to start if it is set (see [Which service needs what](#which-service-needs-what) below). Generate one the same way you generate other secrets for this deploy; rotating it without re-saving the SMTP settings (and any job with an encrypted auth secret) afterward makes the stored values undecryptable. | | `SCRIPT_SECRETS_KEY` | script workers | 32-byte key (base64), same encoding as `SETTINGS_ENCRYPTION_KEY` but not a fresh random value — it’s a sub-key **derived from** `SETTINGS_ENCRYPTION_KEY` (HKDF-SHA256). Generate it with the command in [Generating `SCRIPT_SECRETS_KEY`](#generating-script_secrets_key) below. It decrypts only the global variables explicitly opted into scripts (see [Script jobs](/guides/script-jobs/#opting-a-secret-variable-into-a-scripts-environment)) — never `auth_config`, the SMTP password, or a secret that hasn’t opted in. Each of the three script-worker binaries fails fast at startup if it’s unset. `zilean-mcp` and `zilean-webhook` do the opposite: they refuse to start if it is set. | ### Generating `SCRIPT_SECRETS_KEY` [Section titled “Generating SCRIPT\_SECRETS\_KEY”](#generating-script_secrets_key) The key is fully determined by `SETTINGS_ENCRYPTION_KEY`: HKDF-SHA256 over the decoded master key, with the fixed salt `zilean-script-secrets-v1` and info `zilean-script-secrets`, producing 32 bytes. Derive it with OpenSSL 3, passing in the deploy’s existing `SETTINGS_ENCRYPTION_KEY`: ```sh SETTINGS_ENCRYPTION_KEY='...' openssl kdf -keylen 32 -kdfopt digest:SHA256 \ -kdfopt hexkey:"$(printf %s "$SETTINGS_ENCRYPTION_KEY" | base64 -d | xxd -p -c 64)" \ -kdfopt salt:zilean-script-secrets-v1 -kdfopt info:zilean-script-secrets \ -binary HKDF | base64 ``` Set the output as `SCRIPT_SECRETS_KEY` for the three script workers. Because it’s derived, rotating `SETTINGS_ENCRYPTION_KEY` means regenerating `SCRIPT_SECRETS_KEY` from the new value too. ## Listen addresses [Section titled “Listen addresses”](#listen-addresses) | Variable | Default | Read by | Description | | ---------------- | -------------------------------- | ------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `CORE_ADDR` | `:8091` | core | HTTP listen address for the admin API and `/prometheus/metrics`. | | `UI_ADDR` | `:8090` | ui | HTTP listen address for the admin SPA, River UI and the login routes. | | `MCP_ADDR` | `:8092` | mcp | HTTP listen address for the MCP endpoint at `/mcp`, plus an unauthenticated `/healthz`. | | `MCP_HEALTH_URL` | `http://zilean-mcp:8092/healthz` | core | Optional. Where core probes the MCP service so **Settings → MCP** can warn when it isn’t reachable. Set it only if you renamed the service or changed `MCP_ADDR`. | | `WEBHOOK_ADDR` | `:8093` | webhook | HTTP listen address for the public inbound webhook endpoint at `/api/webhooks/{job_id}/{token}`, plus an unauthenticated `/healthz`. | The four job-executing services (`worker` and the three script workers) run **no HTTP server** and have no listen address. ## Invites (core only) [Section titled “Invites (core only)”](#invites-core-only) | Variable | Default | Description | | ----------------- | ------- | ------------------------------------------------------------------------------------------------------------------ | | `PUBLIC_BASE_URL` | `""` | Externally reachable base URL — scheme + host, **no trailing slash** — used to build absolute invite-accept links. | SMTP itself is not an environment variable — it is configured from **Settings → SMTP** in the admin UI and takes effect immediately, with no restart. See [Configuration](/start/configuration/#email-is-a-settings-page-feature). Caution Set `PUBLIC_BASE_URL` whenever SMTP is configured. Without it, invite links stay relative (`/admin/accept-invite?token=...`) — fine inside the SPA, unusable in an email. ## Optional [Section titled “Optional”](#optional) | Variable | Default | Read by | Description | | ------------------------------ | --------------------------- | ---------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `SEED_JOB` | unset | core | Exactly `"1"` inserts one sample job (`smoke-test-httpbin`, `GET https://httpbin.org/get`, cron `*/5 * * * *`) at startup. **Re-seeds on every restart while set** — unset it once you have real jobs. | | `DEMO_MODE` | unset | core, ui | Exactly `"1"` turns on demo mode. Core then refuses writes to script jobs (`bash`, `javascript`, `typescript`, `groovy`), workflow script blocks, SMTP settings and test email, notification targets, MCP settings, password changes, user management, role management, and sending invites with a 403 (`... is not available in the demo`); seeds sample demo data idempotently at startup; and the admin UI shows a demo banner (the flag reaches the UI through `/api/meta`). Any other value leaves it off. | | `DEMO_DISPLAY_PASSWORD` | unset | core, ui | Display-only text shown with the admin username in the demo banner and on the login page. Never used for authentication, and ignored unless `DEMO_MODE=1`. **Never set it on a real deployment.** | | `DEMO_ECHO_BASE_URL` | `http://127.0.0.1:8099` | core | Base URL of the echo server that the seeded demo jobs call. Only meaningful with `DEMO_MODE=1`. | | `DEFAULT_AUDIT_RETENTION_DAYS` | `180` | core | Seeds `app_settings.audit_retention_days` **once**, at first startup. No effect afterwards — a Super Admin changes it in the admin UI. Unset, invalid, or `< 1` falls back to 180 with a warning; never fatal. | | `TZ` | container system time (UTC) | all services; forwarded into script jobs | The IANA zone that cron expressions are evaluated in, and what the admin UI reports as “Scheduler TZ”. Also one of only two variables forwarded into a script job’s child environment (the other is `PATH`). | | `UI_TRUST_PROXY_HEADERS` | `false` | ui | Optional. When `true`, the in-process per-IP limiters guarding login and invite acceptance key off the leftmost `X-Forwarded-For` entry, falling back to the connection’s remote address (also when the header is absent or its entry is blank or invalid). Enable it **only** when a reverse proxy in front actually sets and sanitizes `X-Forwarded-For` — overwriting rather than appending to the client-supplied value — otherwise a forged header trivially bypasses the limit. Leave it `false` only if every client that reaches zilean-ui connects directly; behind a reverse proxy with it unset, every client keys to the proxy’s address and shares one bucket, so an attacker making about 10 POSTs a minute could lock every admin out of sign-in. IPv4 clients are limited per address and IPv6 clients per `/64` prefix. | | `UI_LOGIN_RATE_PER_MINUTE` | `10` | ui | Optional. The sustained rate, in POSTs per minute, allowed by each in-process limiter guarding login and invite acceptance. Must be a positive integer; any other value makes ui fail at startup with a message naming the variable and the value. Both limiters honor it and keep separate buckets. | | `UI_LOGIN_RATE_BURST` | `5` | ui | Optional. The burst size allowed by each in-process limiter guarding login and invite acceptance. Must be a positive integer; any other value makes ui fail at startup with a message naming the variable and the value. Both limiters honor it and keep separate buckets. | | `WEBHOOK_TRUST_PROXY_HEADERS` | `false` | webhook | Optional. When `true`, the in-process per-IP rate limiter keys off the leftmost `X-Forwarded-For` entry, falling back to the connection’s remote address if the header is absent or its leftmost entry is blank or not a valid IP. IPv4 clients are limited per address and IPv6 clients per `/64` prefix. Enable it **only** when a reverse proxy in front actually sets and sanitizes that header — otherwise a forged header trivially bypasses the limit. Default `false` keys off the real TCP peer address, so a client cannot spoof its way around the limiter. | | `WORKER_MAX_CONCURRENCY` | `20` | worker | Optional. How many HTTP jobs `zilean-worker` runs at once — River’s `MaxWorkers` for the default queue — **per replica**. Unset or empty means 20. Must be an integer from 1 to 200; any other value makes the worker exit at startup with a message naming the variable and the value (it is not clamped). Only the count is configurable: the queues the worker subscribes to are fixed, and the script workers’ concurrency is fixed too. Keep it unquoted in an `env_file`. | ## Script volumes (script workers only) [Section titled “Script volumes (script workers only)”](#script-volumes-script-workers-only) | Variable | Default | Read by | Description | | --------------------- | ------- | -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `SCRIPT_VOLUMES_ROOT` | unset | script workers | Optional. The path, inside the container, of the directory mounted from the host that holds every [script volume](/guides/script-jobs/#volumes). Resolved once at startup. It cannot be `/`. When it is unset, the directory is missing, or it is not writable by UID `65532`, the worker still starts and runs every job that declares no volume; a job that declares one fails without retrying. Changing it or the mount needs a redeploy of the script workers. | ## Bundled Postgres container only [Section titled “Bundled Postgres container only”](#bundled-postgres-container-only) These are read by the `postgres` image, not by any Zilean code. They are unnecessary if you point `DATABASE_URL` at an external database. | Variable | Reference value | | ------------------- | --------------- | | `POSTGRES_USER` | `zilean` | | `POSTGRES_PASSWORD` | `changeme` | | `POSTGRES_DB` | `zilean` | ## Which service needs what [Section titled “Which service needs what”](#which-service-needs-what) * **core only** — `ADMIN_USERNAME`, `ADMIN_PASSWORD_HASH`, `CORE_ADDR`, `SEED_JOB`, `DEMO_MODE`, `DEMO_DISPLAY_PASSWORD`, `DEMO_ECHO_BASE_URL`, `PUBLIC_BASE_URL`, `DEFAULT_AUDIT_RETENTION_DAYS`, `MCP_HEALTH_URL`. * **core and worker** — `SETTINGS_ENCRYPTION_KEY`. Worker needs it to decrypt a job’s `auth_config` secret and any secret global variable before applying them to a real outgoing HTTP call. * **mcp only** — `MCP_ADDR`, alongside `DATABASE_URL` and `TZ`. The MCP service deliberately does **not** receive `SETTINGS_ENCRYPTION_KEY` or `SCRIPT_SECRETS_KEY`, so it cannot read a job’s `auth_config` secret or a secret global variable. It refuses to start, naming the variable, if either key is set in its environment. * **webhook only** — `WEBHOOK_ADDR`, and optionally `WEBHOOK_TRUST_PROXY_HEADERS`. The webhook service deliberately does **not** receive `SETTINGS_ENCRYPTION_KEY` or `SCRIPT_SECRETS_KEY` either, and refuses to start if either is set — the same posture as mcp: it verifies a token by its sha256 digest and reads a job through the registry’s redacted path, so it never holds a job’s credentials, even as ciphertext. * **ui only** — `UI_ADDR`, and optionally `UI_TRUST_PROXY_HEADERS`, `UI_LOGIN_RATE_PER_MINUTE` and `UI_LOGIN_RATE_BURST`. The UI service needs **no** admin credentials; its auth is entirely database-backed. * **worker** — `DATABASE_URL`, `TZ` and `SETTINGS_ENCRYPTION_KEY` (see above), plus the optional `WORKER_MAX_CONCURRENCY`. Those are the entire contents of `.env.worker.example`. * **script workers** — `DATABASE_URL` (functional), `TZ` (log timestamps, plus forwarded to scripts) and `SCRIPT_SECRETS_KEY` (see above), deliberately **without** `SETTINGS_ENCRYPTION_KEY` itself. That is the entire contents of `.env.workers.example`. The least-trusted containers never hold the master key or admin credentials; the derived `SCRIPT_SECRETS_KEY` is scope-limited to the secrets explicitly opted into scripts. The optional `SCRIPT_VOLUMES_ROOT` is set on these three services only. See [Script jobs](/guides/script-jobs/) for the explicit per-variable, per-job opt-in that can place a secret variable’s value in a script’s environment. # Job Fields > Every field on a job, with its type, default, constraints and meaning. A job is one row in the registry. Fields are grouped by what they govern; a job only reads the group matching its `job_type` and `schedule_type`. `PUT` expects the **full** object — there are no partial updates. `GET` the current row, change what you mean to, and send the whole thing back. ## Identity [Section titled “Identity”](#identity) | Field | Type | Default | Notes | | --------------------------- | --------- | --------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `id` | uuid | server-assigned | | | `name` | string | — | Required. | | `group_name` | string | `""` | Optional. Groups can be enabled/disabled together, and a disabled group gates its jobs. A `/` nests groups, up to 16 levels: `Backups/Nightly` appears as `Nightly` inside `Backups`. Limited to 255 bytes and 16 levels deep — the name is normalized first (each `/`-separated segment is trimmed of whitespace and blank segments are dropped), and the limits apply to the normalized name. | | `enabled` | bool | `true` | Checked both by the scheduler and again by the worker at execution time. | | `created_at` / `updated_at` | timestamp | server-managed | | ## Scheduling [Section titled “Scheduling”](#scheduling) | Field | Type | Default | Notes | | -------------------------- | --------- | -------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `schedule_type` | enum | `cron` | `cron`, `rate`, `one_off`, or `webhook`. | | `cron_expression` | string | — | Standard **5-field** cron (no seconds) or a descriptor. For `rate`, must match `@every `. Validated at write time. Ignored for `one_off` and `webhook`. | | `run_at` | timestamp | — | `one_off` only. Must be in the future until fired, then immutable. | | `run_at_fired_at` | timestamp | server-managed | `null` until the one-off run has been inserted. Ignored on write. Cleared if `schedule_type` changes away from `one_off`. | | `one_off_after_run` | enum | `keep` | `keep` disables the row and leaves it; `delete` removes it. Unknown values fall through to `keep`. | | `webhook_token_hint` | string | server-managed | First 8 characters of the job’s webhook token, for identification. Set only when `schedule_type` is `webhook`; ignored on write. | | `webhook_token_rotated_at` | timestamp | server-managed | When the current webhook token was issued. Ignored on write. | A webhook token’s plaintext is shown **exactly once** — in the response to the create or update that set `schedule_type` to `webhook`, or from `POST /jobs/{id}/webhook/rotate`. Only a sha256 digest is stored, and it is never returned by the API, so the token cannot be recovered afterwards; `webhook_token_hint` is what identifies it from then on. See [Scheduling](/concepts/scheduling/). ## Type [Section titled “Type”](#type) | Field | Type | Default | Notes | | ---------- | ---- | ------- | ------------------------------------------------------------------------------------------------------------- | | `job_type` | enum | `http` | `http`, `bash`, `javascript`, `typescript`, `groovy`. **Immutable after creation** — changing it returns 409. | These fields are also the columns of a [CSV import](/guides/csv-import/) file — the same immutability, validation, and script-permission rules apply to an imported row. ## HTTP jobs [Section titled “HTTP jobs”](#http-jobs) Must all be empty on a script job. | Field | Type | Default | Notes | | -------------- | -------- | ------- | -------------------------------------------------------------------------------------------------------------------- | | `http_method` | string | `GET` | | | `url` | string | — | Required for HTTP jobs. Supports `{{variable}}`. | | `query_params` | object | `{}` | Substitution applies to **keys and values**. Overrides same-named params already in the URL. | | `body` | raw JSON | — | Sent verbatim. Substitution applied to the whole serialized body. | | `headers` | object | `{}` | Keys and values substituted. `Content-Type: application/json` is added only if a body exists and you didn’t set one. | | `auth_type` | enum | `none` | `none`, `bearer`, `basic`, `api_key`. | | `auth_config` | object | `{}` | Shape depends on `auth_type`; see below. Every field is substituted. | ### `auth_config` shapes [Section titled “auth\_config shapes”](#auth_config-shapes) | `auth_type` | Shape | Validation | | ----------- | ------------------------------------------ | ------------------------------------------------------------------- | | `bearer` | `{"token": "..."}` | `token` must be non-empty. | | `basic` | `{"username": "...", "password": "..."}` | `username` must be non-empty; empty password allowed. | | `api_key` | `{"api_key": "...", "header_name": "..."}` | `api_key` must be non-empty; `header_name` defaults to `X-API-Key`. | Unknown keys in `auth_config` are ignored. `token`, `password`, and `api_key` are encrypted at rest (see [Environment Variables](/reference/environment-variables/#required)) and are never returned by the admin API. A `GET`/list response’s `auth_config` carries only the non-secret fields (`username`, `header_name`); a `has_auth_secret: true`/`false` field alongside it says whether a secret is currently set. Sending an empty string for the secret field on `PUT` keeps the existing value rather than clearing it — send a non-empty value to change it. This does not apply to `create`, where there is no existing value to keep: an empty secret there just means no secret. ## Script jobs [Section titled “Script jobs”](#script-jobs) Must both be empty on an HTTP job. | Field | Type | Default | Notes | | -------------------- | --------- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `script_body` | string | — | The whole script, written to a temp file at run time. Used verbatim: `{{variable}}` is not substituted; read variables from `GLOBALVAR_*` environment variables instead. | | `script_args` | string\[] | `[]` | Appended after the script path. Each element is passed as written, with no `{{variable}}` substitution. | | `script_secret_vars` | string\[] | `[]` | Names of secret global variables this job is allowed to receive as `GLOBALVAR_`. A named variable is injected only when it also has `allow_in_scripts` set on the global variable itself — see below. | | `capabilities` | object | `{}` | What the script may reach. Shape: `{"volumes": [{"name": "reports", "mode": "rw"}]}`. `mode` is `ro` or `rw` and cannot exceed the volume’s `max_mode`; each volume appears once; every volume must exist under **Settings → Script volumes**. `{}` and `{"volumes": []}` mean none. Unknown keys are rejected, and `egress` is reserved and rejected as not yet supported. Re-checked against the current volume list on every attempt. | Requires the `jobs:script` permission. See [Script jobs](/guides/script-jobs/). `capabilities` must be empty on an HTTP job (`400` otherwise). See [Volumes](/guides/script-jobs/#volumes). Responses include a read-only `capabilities_invalid` boolean. When it is `true`, the stored volume declaration cannot be read and `capabilities` shows `{}`. The job is cancelled each time it would run, rather than running without its volumes. The job list shows a “volumes unreadable” tag, and the edit form warns that saving replaces the declaration. Setting `script_secret_vars` requires **both** `jobs:script` and `globalvars:write` — distinct from turning `allow_in_scripts` on for a global variable itself, which needs only `globalvars:write`. See [Opting a secret variable into a script’s environment](/guides/script-jobs/#opting-a-secret-variable-into-a-scripts-environment). ## Execution [Section titled “Execution”](#execution) | Field | Type | Default | Constraints | | ----------------- | ---- | ------- | ------------------------------------------------------------------------------------------------------------------------ | | `exec_timeout_ms` | int | `60000` | `> 0` and `≤ 1800000` (30 min). Groovy jobs have a **minimum of 10000**. Bounds one attempt. | | `pool_count` | int | `1` | `1 … 20`. Must be `1` for `one_off`. Fans one firing into that many parallel, independently locked/looped/retried calls. | | `overlap_policy` | enum | `skip` | `skip` takes a per-pool-index lock and drops a firing for an index still running; `allow` never locks. | ## Loop (success path) [Section titled “Loop (success path)”](#loop-success-path) | Field | Type | Default | Notes | | --------------------- | ------ | ------------------ | --------------------------------------------------------------------------------------------------------------------------------------------- | | `loop_enabled` | bool | `false` | | | `loop_response_path` | string | `result.remaining` | **gjson** path, evaluated against the response body (HTTP) or **stdout** (script). Continues only on a strictly positive number. | | `loop_delay_ms` | int | `1000` | Flat delay before the next iteration. | | `loop_jitter_ms` | int | `0` | Random 0–N ms added to the delay. **No applied default** — 0 means no jitter. | | `loop_max_iterations` | int | `50` | Bounds **one logical run**; resets on the next firing. On hit: logs an error, releases the lock, and returns cleanly without failing the job. | ## Retry (error path) [Section titled “Retry (error path)”](#retry-error-path) | Field | Type | Default | Notes | | --------------------- | ----- | ------- | ----------------------------------------------------------------------------------------------------------------- | | `retry_enabled` | bool | `true` | When false, a failure is cancelled outright — 1 attempt, no backoff. | | `retry_base_delay_ms` | int | `1000` | Coalesced to 1000 if `≤ 0`. | | `retry_multiplier` | float | `2` | Coalesced to 2 if `≤ 1`. | | `retry_max_delay_ms` | int | `30000` | Ceiling on the computed delay. Coalesced to **300000** if `≤ 0` — note this differs from the create-time default. | | `retry_max_attempts` | int | `5` | Becomes the queue job’s attempt budget **at insert time**; editing it doesn’t affect already-queued jobs. | Backoff is `min(base × multiplier^(attempt−1), max_delay)`. No jitter. ## Notifications [Section titled “Notifications”](#notifications) Configured through the job wizard’s **Notify** step, backed by targets and the Apprise server set up under **Settings → Notifications** — see [Configuration](/start/configuration/#notifications-is-a-settings-page-feature-too). | Field | Type | Default | Notes | | -------------------- | ------ | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- | | `notify_on` | enum | `none` | `none`, `success`, `failure`, or `both`. Fires once a run has *fully* finished (not on every retry/loop iteration). Does not gate invocation alarms. | | `notify_all_targets` | bool | `false` | When `true`, notifies every configured target and `notify_targets` is ignored. | | `notify_targets` | int\[] | `[]` | Target IDs to notify when `notify_all_targets` is `false`. | Targets themselves are configured under **Settings → Notifications** (Apprise and email) or **Settings → Webhooks**. A webhook target carries its own event filter (`success`, `failure`, `retry_exhausted`, `invocation_alarm`) that applies on top of this job-level gate: a target subscribed to `failure` also receives `retry_exhausted`, and leaving the filter empty selects all four. Metrics alarms reuse the same registry but offer only Apprise and email targets. The same target fields govern the job’s **invocation alarms**, which fire when the job’s invocation count reaches a threshold over a rolling window. `notify_all_targets`/`notify_targets` choose their targets, and `notify_on` does **not** gate them — a job with `notify_on: none` still notifies its selected targets when an alarm breaches or recovers. Like success/failure notifications, alarm delivery is subject to a webhook target’s own event filter. See [Invocation alarms](/guides/monitoring/#invocation-alarms). Changing any of these three fields requires the `notify:write` permission — a caller without it can still edit every other field of a job that already has notifications configured; the stored notify config is carried forward unchanged on that edit rather than silently reset. See [Permissions](/reference/permissions/). A custom role can end up with an empty target picker `notify:write` lets a role change a job’s notify config, but the target picker itself is populated from the target list, which requires `notify_settings:read`. A custom role holding `notify:write` without also holding `notify_settings:read` sees an empty picker in the job wizard — it can still choose “All targets,” just not name specific ones. Grant both together for a role that should manage per-job notifications. ## Server-managed [Section titled “Server-managed”](#server-managed) Set by the server and ignored on `POST`/`PUT`: | Field | Notes | | ----------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- | | `running_since_by_pool` | Overlap locks — one entry per pool index, keyed by its string form (`"0"`, `"1"`). A lock older than 1 hour is stolen. | | `invocations_total` | Lifetime attempt count. **One invocation is one attempt** — a retry or loop iteration each count separately, not one cron-triggered run. | | `invocations_completed` | Lifetime successes. | | `invocations_failed` | Lifetime failures. | ## Validation summary [Section titled “Validation summary”](#validation-summary) Rejected at write time: * Invalid cron or rate expression. * `exec_timeout_ms` outside `1 … 1800000`, or under 10000 for Groovy. * `pool_count` outside `1 … 20`, or `> 1` on a one-off. * Unknown `auth_type`, `overlap_policy`, `schedule_type` or `job_type`. * `run_at` in the past on an unfired one-off, or changed after firing. * `group_name` over 255 bytes or 16 levels deep. * HTTP fields set on a script job, or script fields set on an HTTP job. * A `job_type` change (409), or a script job without `jobs:script` (403). # Permissions > All 24 permission keys, what each gates, and which built-in roles carry them. Permission keys are defined in Go, not in a database table — so a newly added permission is automatically available to the Super Admin without a migration. The live list is available at `GET /api/roles/permissions`, which is what the admin UI’s role editor is built from. ## The keys [Section titled “The keys”](#the-keys) | Key | Gates | | ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `jobs:read` | List and get jobs, recent calls, heartbeats, per-job activity, a job’s invocation alarms, the disabled-group list. | | `jobs:write` | Create, update, delete a job; enable/disable (single, all, by group); run-now. **Also the `/riverui/` mount.** | | `jobs:script` | Creating or editing a **script** job — *in addition to* `jobs:write`. Also gates `GET /api/scripts/liveness`. Also required, on top of `workflows:write`, to save a workflow whose graph contains a `script` block. | | `workflows:read` | List and get workflows, their runs, and their run steps. | | `workflows:write` | Create, update, delete, enable/disable, and manually run a workflow. Distinct from `jobs:write` — a workflow is its own schedulable entity, not a `scheduled_jobs` row. | | `globalvars:read` | List global variables. | | `globalvars:write` | Create, update, delete global variables. Also gates a variable’s `allow_in_scripts` flag and, combined with `jobs:script`, a job’s `script_secret_vars` list. | | `metrics:read` | `/api/metrics/live`, `/api/metrics/history`, and the Prometheus `/prometheus/metrics` endpoint. | | `users:read` | List users. | | `users:write` | Create a user, change role or status, admin password reset, delete a user. | | `roles:read` | List roles and list permission keys. | | `roles:write` | Create, update, delete roles. Only permissions the caller holds can be added to a role. | | `invites:create` | Create an invite, list pending invites, resend an invite. | | `audit:read` | Read the audit log and the audit retention setting. | | `audit:write` | Change the audit retention setting. | | `settings:smtp` | Read and change the outbound email (SMTP) settings, and send a test message. | | `settings:river_jobs` | Read and change the River job (`river_job`) retention setting. | | `settings:mcp` | Manage MCP tokens (create, list, revoke) and toggle the MCP endpoint on/off. Gates the token-issuing surface only — what a token can then do is controlled by its own linked role’s permissions, not by this key. | | `scriptvolumes:read` | List script volumes (with how many jobs and workflows use each) and read script worker status. | | `scriptvolumes:write` | Create, update and delete script volumes. Declaring an existing volume on a script job needs only `jobs:script`, not this key. | | `notify:read` | View a job’s per-job notification config (`notify_on`, `notify_all_targets`, `notify_targets`). | | `notify:write` | Change a job’s per-job notification config, and create, update or delete the job’s invocation alarms. Gates only an actual change to those fields — a caller without it can still edit every other field of a job that already has notifications configured. | | `notify_settings:read` | List notification targets, the Apprise server connection settings, and the metrics alarms (Settings → Notifications). Also populates the job wizard’s target picker. | | `notify_settings:write` | Create, update, delete, and test notification targets; change the Apprise server connection and metrics alarms. | Custom roles receive neither `scriptvolumes` key; add them to any custom role that should see or manage **Settings → Script volumes**. ## Built-in roles [Section titled “Built-in roles”](#built-in-roles) A user can be assigned more than one role — the admin UI’s create/invite form is a checkbox multi-select, not a single dropdown — and always holds at least one; the API rejects any create or update that would leave a user with zero roles. When a user holds several roles, their effective permissions are the **union** of what each assigned role grants. Three system roles are seeded and cannot be renamed or deleted. Super Admin’s permissions can’t be touched either — its access is computed from the user’s super-admin flag, not role rows. Admin and Read-Only permissions *can* be edited by a caller holding `roles:write`; the table below shows each one’s seeded default. | | Super Admin | Admin (default) | Read-Only (default) | | ----------------------- | ----------- | --------------- | ------------------- | | `jobs:read` | ✓ | ✓ | ✓ | | `jobs:write` | ✓ | ✓ | | | `jobs:script` | ✓ | | | | `workflows:read` | ✓ | ✓ | ✓ | | `workflows:write` | ✓ | ✓ | | | `globalvars:read` | ✓ | ✓ | ✓ | | `globalvars:write` | ✓ | ✓ | | | `metrics:read` | ✓ | ✓ | ✓ | | `users:read` | ✓ | | ✓ | | `users:write` | ✓ | | | | `roles:read` | ✓ | | ✓ | | `roles:write` | ✓ | | | | `invites:create` | ✓ | | | | `audit:read` | ✓ | ✓ | | | `audit:write` | ✓ | | | | `settings:smtp` | ✓ | ✓ | | | `settings:river_jobs` | ✓ | ✓ | | | `settings:mcp` | ✓ | ✓ | | | `scriptvolumes:read` | ✓ | ✓ | ✓ | | `scriptvolumes:write` | ✓ | ✓ | | | `notify:read` | ✓ | ✓ | | | `notify:write` | ✓ | ✓ | | | `notify_settings:read` | ✓ | ✓ | | | `notify_settings:write` | ✓ | ✓ | | Five permissions are in no built-in role `jobs:script`, `users:write`, `roles:write`, `invites:create` and `audit:write` reach a non-Super-Admin **only** through a custom role. This is intentional — particularly `jobs:script`, which is the ability to execute arbitrary code. `globalvars:write` deserves similar care. On its own it can mark an existing secret variable as eligible for script exposure (`allow_in_scripts`), even though actually delivering that secret to a job still requires `jobs:script` to set the job’s `script_secret_vars` list. It is the variable-side half of a two-gate control, not an inert config permission. ## Enforcement [Section titled “Enforcement”](#enforcement) Checks happen per route: ```plaintext 403 {"error":"you do not have permission to perform this action"} ``` `jobs:script` is the exception. It can’t be a route-level check, because the job type is only known after the request body is decoded, so it’s enforced inside the create and update handlers: ```plaintext 403 {"error":"script jobs require the jobs:script permission"} ``` ## Routes that need no permission [Section titled “Routes that need no permission”](#routes-that-need-no-permission) | Route | Why | | ------------------------------------------------------------- | ----------------------------------------------------------------------------------------------- | | `GET /api/meta` | Server timezone. Needed by the UI shell before any permission check. | | `GET /api/account/me` | The caller’s own identity and permission list. | | `POST /api/account/change-password` | Self-service; must work for a user who is required to change their password. | | `/login`, `/logout`, `/admin/assets/`, `/admin/accept-invite` | Registered outside the auth middleware so login stays reachable. `/logout` accepts `POST` only. | Everything else requires a valid session plus the permission in the table above. ## Super Admin protections [Section titled “Super Admin protections”](#super-admin-protections) These return **409** regardless of the caller’s permissions: * Changing a Super Admin’s role or status. * Deleting a Super Admin. * An admin password reset targeting a Super Admin. * Minting an invite for an already-active user (which closes an account-takeover path available to a caller holding only `invites:create`). The first-boot seeding path sets the super-admin flag once, for the bootstrap account. Beyond that, an account gains it whenever it’s assigned the **Super Admin** role — by accepting an invite into that role, by being created directly into it (`POST /api/users`), or by having its roles changed to include it (`PATCH /api/users/{id}`). All three routes — plus `POST /api/invites` — only let an existing Super Admin assign that role, or assign any role whose permissions exceed the caller’s own, so a plain `invites:create` or `users:write` holder can’t use them to mint a Super Admin or escalate past their own access. `POST /api/mcp-tokens` applies the same role-grant check as those routes — a caller can’t mint a token bound to a role whose permissions exceed their own. It goes one step further than the human-user routes: a token can never be bound to the **Super Admin** role at all, even by an actual Super Admin, since the token’s hidden linked account is created without the super-admin flag that role normally carries — binding it anyway would silently issue a token with zero effective permissions. # Configuration > How Zilean is configured, and which service reads what. Zilean is configured entirely through environment variables. There is no config file. The [environment variable reference](/reference/environment-variables/) has the complete table; this page covers how to think about it. ## A minimum configuration [Section titled “A minimum configuration”](#a-minimum-configuration) .env ```ini DATABASE_URL=postgres://zilean:changeme@zilean-postgres:5432/zilean?sslmode=disable ADMIN_USERNAME=you@example.com ADMIN_PASSWORD_HASH=$2a$10$... SETTINGS_ENCRYPTION_KEY=<32-byte key, base64> TZ=America/Los_Angeles ``` That is enough to boot. Everything else has a working default. ## Configuration is split by trust, not by convenience [Section titled “Configuration is split by trust, not by convenience”](#configuration-is-split-by-trust-not-by-convenience) Services do not share one environment. There are four tiers, deliberately: **`zilean-core` and `zilean-ui`** get the full `.env`. Core is the only service that reads the admin credentials, and it reads them once at startup to seed the Super Admin. **`zilean-worker`** gets a narrowed environment, one variable wider than the three script workers below: .env.worker ```ini DATABASE_URL=postgres://... TZ=America/Los_Angeles SETTINGS_ENCRYPTION_KEY=<32-byte key, base64> # WORKER_MAX_CONCURRENCY=20 ``` It needs `SETTINGS_ENCRYPTION_KEY` because it applies a job’s `auth_config` and any secret global variables to real outgoing HTTP calls, and those secrets are encrypted at rest — decrypting it happens here, not in core. `WORKER_MAX_CONCURRENCY` is optional: it sets how many HTTP jobs the worker runs at once **per replica** (20 when unset or empty, at most 200, and any other value stops the worker at startup naming the variable and value), it applies to `zilean-worker` only, and it must stay unquoted because `env_file` does not strip surrounding quotes. **`zilean-mcp` and `zilean-webhook`** are zero-key services. Each gets a narrowed environment and no encryption key at all: ```yaml # zilean-mcp environment: DATABASE_URL: ${DATABASE_URL} TZ: ${TZ:-UTC} MCP_ADDR: ${MCP_ADDR:-:8092} # zilean-webhook environment: DATABASE_URL: ${DATABASE_URL} TZ: ${TZ:-UTC} WEBHOOK_ADDR: ${WEBHOOK_ADDR:-:8093} WEBHOOK_TRUST_PROXY_HEADERS: ${WEBHOOK_TRUST_PROXY_HEADERS:-false} ``` Neither reads a job’s credentials: MCP goes through the registry’s redacted path, and the webhook endpoint verifies a token by its sha256 digest. Both refuse to start, with a non-zero exit and an error naming the variable, if `SETTINGS_ENCRYPTION_KEY` or `SCRIPT_SECRETS_KEY` is set to a non-empty value in their environment, so a stray `env_file: [.env]` fails loudly instead of quietly widening what a public-facing container holds. Note that the `environment:` blocks above are interpolated, so they depend on where `.env` lives — see the caution below. **The three script workers** — bash, Node/TypeScript, and Groovy — get the narrowest environment of all: .env.workers ```ini DATABASE_URL=postgres://... TZ=America/Los_Angeles ``` `DATABASE_URL` is the only variable any of them actually reads for function; `appboot.Open` is fatal without it. `TZ` only affects their log timestamps — and is forwarded into script jobs. They deliberately do **not** get `SETTINGS_ENCRYPTION_KEY`, even though they already hold `DATABASE_URL` and so can read an `auth_config` secret’s ciphertext directly: every script job has `auth_type: none` by construction, so they never need to decrypt one, and withholding the key is what stops a compromised script from decrypting some *other* job’s secret. The reason for the split: these are the least-trusted containers, the ones that run operator-supplied code, and there is no reason for them to hold admin credentials or a key they have no legitimate use for. Derive the narrowed files from the real one rather than retyping a password: ```bash grep -E '^(DATABASE_URL|TZ|SETTINGS_ENCRYPTION_KEY|WORKER_MAX_CONCURRENCY)=' .env > .env.worker grep -E '^(DATABASE_URL|TZ)=' .env > .env.workers ``` Compose interpolation does not read a service’s `env_file` An `environment: { DATABASE_URL: ${DATABASE_URL} }` block — the form `zilean-mcp` and `zilean-webhook` use above — interpolates `${VAR}` from the `.env` file **beside the compose file**, never a service’s own `env_file`. It therefore only works while `.env` sits next to `docker-compose.yml`; if yours lives elsewhere, `${DATABASE_URL}` becomes an empty string and that service dies at startup. In that case give those two a narrowed `env_file` instead, derived from the real `.env`: ```bash grep -E '^(DATABASE_URL|TZ|MCP_ADDR)=' .env > .env.mcp grep -E '^(DATABASE_URL|TZ|WEBHOOK_ADDR|WEBHOOK_TRUST_PROXY_HEADERS)=' .env > .env.webhook ``` Never point them at the full `.env`: both refuse to start when `SETTINGS_ENCRYPTION_KEY` or `SCRIPT_SECRETS_KEY` is set. The script workers are unaffected either way — they use `env_file: .env.workers`, which is never interpolated. Also note `env_file` does **not** strip surrounding quotes the way a shell does. Keep values unquoted. Script workers may also set `SCRIPT_VOLUMES_ROOT`, an optional path to the directory that holds script volumes. It is not a secret, so set it on the service rather than in `.env.workers`. See [Volumes](/guides/script-jobs/#volumes). ## Timezone [Section titled “Timezone”](#timezone) `TZ` decides the zone cron expressions are evaluated in — there is no per-job timezone, and no `WithLocation` override, so the scheduler runs in the process’s `time.Local`. Because this is easy to get wrong and invisible when it is, the admin API exposes what the server actually resolved: ```bash curl https:///api/meta # {"timezone":"America/Los_Angeles"} ``` The admin UI displays this too. If it says `UTC` and you expected otherwise, `TZ` isn’t reaching the container. ## Email is a Settings page feature [Section titled “Email is a Settings page feature”](#email-is-a-settings-page-feature) SMTP isn’t set through the environment at all — it’s configured from **Settings → SMTP** in the admin UI, and takes effect immediately, with no restart. Fill in the host, port, credentials and from address, then use the page’s test-email button to confirm delivery before relying on it for invites. The test button reuses the saved password only while the host and port are unchanged; if you point it at a different host or port, re-enter the password. Zilean refuses to connect to link-local (`169.254.0.0/16`, `fe80::/10`), unspecified and multicast addresses — never a legitimate mail relay, and where cloud metadata services live. Private and loopback relays (an SMTP server on your LAN, or on the same host) are allowed. If you need stricter isolation, apply egress rules on the host. Leave it unconfigured and the invite flow still works — the admin UI shows the invite link for an admin to copy and send by hand. The SMTP password is encrypted at rest, keyed by the `SETTINGS_ENCRYPTION_KEY` environment variable (required on `zilean-core`, and also on `zilean-worker` — see [Configuration is split by trust](#configuration-is-split-by-trust-not-by-convenience) above — for the same reason a job’s `auth_config` secret is encrypted). If you configure SMTP, also set `PUBLIC_BASE_URL` — without it, invite emails contain a bare relative path instead of a clickable link. ## Notifications is a Settings page feature too [Section titled “Notifications is a Settings page feature too”](#notifications-is-a-settings-page-feature-too) Apprise/email notifications aren’t set through the environment either, and Zilean does not bundle an Apprise server — run one yourself, reachable from wherever `zilean-core` runs, then configure it from **Settings → Notifications**: * **Apprise server connection** — the server’s URL and whether to verify its TLS certificate. When a URL is set it must end in `/notify`; the form and the server both reject anything else (the server answers `400`), whether or not Apprise is enabled. Saving with Apprise enabled and an empty URL is rejected with `server_url is required when Apprise is enabled`; with Apprise off, an empty URL saves fine. The **Test** button probes the URL currently entered in the form, so you can confirm the connection before enabling Apprise or saving. * **Notification targets** — named Apprise/email destinations a job or metrics alarm can notify: * An **Apprise** target holds a [Service URL](https://github.com/caronc/apprise/wiki) (Discord, Slack, Telegram, and everything else Apprise supports) and is delivered through the server above. * An **email** target is just an address, delivered through the SMTP settings described above — no separate email configuration. * **Alarms** — an optional CPU%/memory alert per service (`core`, `ui`, `worker`, `mcp`, `webhook`, and the three script workers), notifying a chosen set of targets when a service crosses the line and again when it recovers. These metrics alarms notify Apprise and email targets only: they are not event-driven, so a webhook target is not offered for one — see [Webhooks is a Settings page feature too](#webhooks-is-a-settings-page-feature-too) below. Per-job [invocation alarms](/guides/monitoring/#invocation-alarms) are configured from the job’s **Activity** modal and can notify webhook targets. A job’s own **Notify** step (in the job wizard) picks which outcomes — success, failure, or both — notify which targets, or all of them. See [Job Fields](/reference/job-fields/#notifications). Standing up an Apprise server is one container: ```bash docker run -d -p 8000:8000 caronc/apprise ``` The image is `caronc/apprise` — `caronc/apprise-api` is not a real image. See the [apprise-api repository](https://github.com/caronc/apprise-api) for the server itself. ## Webhooks is a Settings page feature too [Section titled “Webhooks is a Settings page feature too”](#webhooks-is-a-settings-page-feature-too) Webhook targets have their own **Settings → Webhooks** page, separate from Notifications — they’re delivered independently of Apprise/email and don’t share that page’s Apprise server connection or SMTP dependency. A **webhook** target is a URL Zilean POSTs a JSON payload to when a job reaches one of the states its **Notify on** filter selects. The filter is per-target: **Success**, **Failure**, **Retry exhausted**, and/or **Invocation alarm**. A target subscribed to **Failure** also receives `retry_exhausted` events, matching the job-level `notify_on: failure` gate; select **Retry exhausted** without **Failure** to receive only exhausted retries. A target subscribed to **Invocation alarm** receives the deliveries from that job’s [invocation alarms](/guides/monitoring/#invocation-alarms). Leaving every box unchecked stores all four (the default). A target saved with an explicit filter keeps it, so tick **Invocation alarm** on any target that should receive these deliveries. A webhook delivery is fire-once — a non-2xx response or a transport error is logged and dropped, never retried — and redirects are not followed (a 3xx is a failed delivery). Webhook deliveries are signed when the target has a **signing secret**. The secret is write-only: set it when adding the target, and the API reports only whether one is stored (`has_signing_secret`), never its value. Leaving the field blank while editing keeps the stored secret. ### Receiving a webhook [Section titled “Receiving a webhook”](#receiving-a-webhook) Zilean sends one `POST` per matching terminal state. The JSON body carries only non-secret fields: ```json { "event": "failure", "job_id": "3f2b…", "job_name": "nightly-backup", "group_name": "Ops/Backups", "job_type": "http", "occurred_at": "2026-09-22T20:00:00Z", "snippet": "endpoint returned status 500" } ``` The body above is a terminal-state delivery. An `invocation_alarm` delivery carries the same job-identifying fields plus five flat string keys, present only on that event: | Key | Value | | ---------------------- | -------------------------------------- | | `alarm_state` | `breached` or `ok` (a recovery). | | `alarm_metric` | `total` or `failed`. | | `alarm_count` | The count that triggered the delivery. | | `alarm_threshold` | The alarm’s configured threshold. | | `alarm_window_minutes` | The alarm’s window in minutes. | …alongside these request headers: | Header | Value | | -------------------- | --------------------------------------------------------------------- | | `Content-Type` | `application/json` | | `X-Zilean-Event` | `success`, `failure`, `retry_exhausted`, or `invocation_alarm` | | `X-Zilean-Delivery` | a unique id (UUID) per delivery, for de-duplication | | `X-Zilean-Timestamp` | the signing timestamp, Unix seconds | | `X-Zilean-Signature` | `sha256=`, present only when a signing secret is configured | To verify a delivery, recompute HMAC-SHA256 over the string `.` using the shared signing secret, hex-encode it, and compare it (constant time) against the value after `sha256=` in `X-Zilean-Signature`. The timestamp is part of the signed string, so reject a delivery whose timestamp falls outside your replay tolerance before trusting it. Because a delivery is never retried, treat `X-Zilean-Delivery` as the idempotency key. Target URLs are stored in plaintext A notification target’s URL, and the Apprise server URL, are stored and used as plaintext — the same trust posture as `auth_config` before its encryption. An Apprise Service URL often embeds a webhook token or similar credential (e.g. `slack://TokenA/TokenB/...`), so treat access to the target list the same way you’d treat access to a job’s HTTP auth config. A webhook target’s signing secret is the exception: it is AES-256-GCM-encrypted at rest under `SETTINGS_ENCRYPTION_KEY` (the same key the SMTP password and a job’s `auth_config` secret use), and is never returned by the API. ## Settings that live in the database, not the environment [Section titled “Settings that live in the database, not the environment”](#settings-that-live-in-the-database-not-the-environment) Some configuration is deliberately *not* environment-driven, because it is meant to change without a restart: * **Job definitions** — schedule, retry, loop, pooling, auth. All rows. That’s the whole point; see [Introduction](/start/introduction/). * **Users, roles and permissions** — database-backed, managed in the admin UI. See [Users & permissions](/concepts/users-and-permissions/). * **Audit log retention** — `DEFAULT_AUDIT_RETENTION_DAYS` seeds the `app_settings` row *once* on first startup. After that, changing the environment variable does nothing; a Super Admin changes the window in the admin UI. * **Global variables** — values shared across jobs, stored in `global_variables`. * **Script volumes** — the named directories script jobs may be given, set from Settings → Script volumes and effective on the next run. The host directory they live under and `SCRIPT_VOLUMES_ROOT` are container settings that need a redeploy. See [Volumes](/guides/script-jobs/#volumes). * **SMTP settings** — host, port, credentials and from address, set from Settings → SMTP. See [Email is a Settings page feature](#email-is-a-settings-page-feature) above. * **Apprise/email notification targets, the Apprise server connection, and metrics alarms** — set from Settings → Notifications. See [Notifications is a Settings page feature too](#notifications-is-a-settings-page-feature-too) above. * **Webhook targets** — set from Settings → Webhooks. See [Webhooks is a Settings page feature too](#webhooks-is-a-settings-page-feature-too) above. ## Once you’re running [Section titled “Once you’re running”](#once-youre-running) * [Your first job](/start/your-first-job/) — register a job and watch it execute. * [Deploying](/guides/deploying/) — Compose topology, image targets, resource limits and reverse-proxy routing. # Installation > Get a Zilean instance running — prerequisites, migrations, and first boot. Zilean runs as eight service containers, published as images you pull and run with Docker Compose — no build step required. ## Prerequisites [Section titled “Prerequisites”](#prerequisites) * **Postgres.** The reference deployment uses `postgres:17.10-alpine`. No minimum version is formally declared; the application’s own SQL deliberately avoids `date_bin` so it does not require Postgres 14+. * **Docker and Docker Compose.** ## Images [Section titled “Images”](#images) Every tagged release publishes multi-arch (`linux/amd64` + `linux/arm64`) images to GHCR: | Image | Contains | | --------------------------------------------- | ------------------------------------------------------------------------------------------------------------------ | | `ghcr.io/patrickpingol/zilean-slim` | `zilean-core`, `zilean-ui`, `zilean-worker`, `zilean-mcp`, `zilean-webhook` — `command:` selects which binary runs | | `ghcr.io/patrickpingol/zilean-scripts-bash` | `zilean-script-worker-bash` | | `ghcr.io/patrickpingol/zilean-scripts-node` | `zilean-script-worker-node` | | `ghcr.io/patrickpingol/zilean-scripts-groovy` | `zilean-script-worker-groovy` | Each is tagged `vX.Y.Z`, `vX.Y`, `vX`, and `latest`. Pin to a `vX.Y.Z` tag for anything but throwaway testing — `latest` moves. Deploy only the script runtimes you actually use. A script job whose runtime has no deployed worker queues indefinitely with nothing to pick it up — visible in River UI as stuck in `scheduled`/`available`. ## Two migration systems, both automatic [Section titled “Two migration systems, both automatic”](#two-migration-systems-both-automatic) Zilean’s schema and River’s schema are applied by different tools, but both run automatically from `zilean-core` startup — there is nothing to run by hand. ### 1. River’s tables [Section titled “1. River’s tables”](#1-rivers-tables) River’s own tables (`river_job`, `river_queue`, …) are migrated by `zilean-core` itself, via River’s Go migration API (`rivermigrate`) rather than the `river` CLI. This runs on every startup and is idempotent — an already-current schema is a no-op. ### 2. Zilean’s own schema [Section titled “2. Zilean’s own schema”](#2-zileans-own-schema) Zilean’s tables (`scheduled_jobs`, `worker_metrics`, `job_call_stats`, `global_variables`, users/roles/sessions/invites, `audit_log`, `app_settings`) are embedded in the `zilean-core` image and applied by `golang-migrate` the moment core starts. * Tracked in the `schema_migrations` table. * **`zilean-core` is the only service that applies migrations** (both River’s and Zilean’s own) — one owner, so several containers never race for the advisory lock. * Startup logs say which case you’re in: `migrations already up to date` (with the version) or `migrations applied` (from → to) for Zilean’s schema, and similarly for River’s. * A **dirty** migration state (Zilean’s schema only) aborts startup with `schema is marked dirty at version N; resolve by hand before starting`. Migrations are up-only There are no down migrations. Rollback is not supported by the tooling — going back a version means restoring the database or hand-writing the reverse SQL. ## Generate the admin password hash [Section titled “Generate the admin password hash”](#generate-the-admin-password-hash) The first Super Admin account is seeded from environment variables on first boot. Generate its bcrypt hash with `htpasswd` (from the public `httpd` image, so there’s nothing to install locally): ```bash docker run --rm httpd:2.4-alpine htpasswd -bnBC 10 "" 'the real password' | cut -d: -f2 ``` It prints a bcrypt hash, ready to paste into `.env`: ```plaintext ADMIN_PASSWORD_HASH=$2y$10$... ``` Escape `$` for Docker Compose, or the hash gets silently corrupted Compose’s `.env` loader interpolates `$`-prefixed tokens **within `.env` values themselves** — not just in `docker-compose.yml`. A bcrypt hash contains several literal `$` characters (`$2y$10$...`), and Compose treats each one as the start of a variable reference. Anything after it that looks like a variable name (e.g. the `NrWPv` in `$2y$10$NrWPv...`) gets silently replaced with an empty string if no such variable is set — with no error, no warning, just a truncated hash that will never validate a login. Escape every `$` as `$$` before writing the value to `.env`: ```bash docker run --rm httpd:2.4-alpine htpasswd -bnBC 10 "" 'the real password' \ | cut -d: -f2 | sed 's/\$/$$/g' ``` If you generate `.env` with a script rather than by hand, apply the same escaping to every secret you write, not just this one — anything could coincidentally contain `$`. ## Generate the settings encryption key [Section titled “Generate the settings encryption key”](#generate-the-settings-encryption-key) `zilean-core` and `zilean-worker` both refuse to start without `SETTINGS_ENCRYPTION_KEY` — a 32-byte key, base64-encoded, that encrypts secrets at rest (the SMTP password and a job’s `auth_config` secret). Generate one: ```bash openssl rand -base64 32 ``` ## Set the required configuration [Section titled “Set the required configuration”](#set-the-required-configuration) At minimum, create a `.env` file: ```ini DATABASE_URL=postgres://zilean:changeme@zilean-postgres:5432/zilean?sslmode=disable ADMIN_USERNAME=you@example.com ADMIN_PASSWORD_HASH=$2y$10$... SETTINGS_ENCRYPTION_KEY=... ``` `ADMIN_USERNAME` must be an email address `zilean-core` validates this at startup and **refuses to start** if the value has no `@` in it: ```plaintext ADMIN_USERNAME must be an email address (got "admin") — update it in .env before starting zilean-core ``` Fix it *before* first boot. Once a Super Admin row exists, `HasSuperAdmin` prevents re-seeding, so a wrong value caught afterwards needs a manual `UPDATE users SET email = ...` rather than an `.env` edit and restart. See [Configuration](/start/configuration/) for everything else, and the [environment variable reference](/reference/environment-variables/) for the full table. ## Start the services [Section titled “Start the services”](#start-the-services) Create a `docker-compose.yml` alongside your `.env`: ```yaml services: zilean-postgres: image: postgres:17.10-alpine restart: unless-stopped env_file: [.env] volumes: - ./postgres/data:/var/lib/postgresql/data zilean-core: image: ghcr.io/patrickpingol/zilean-slim:v1.0.0 command: ["/usr/local/bin/zilean-core"] restart: unless-stopped env_file: [.env] depends_on: - zilean-postgres zilean-ui: image: ghcr.io/patrickpingol/zilean-slim:v1.0.0 command: ["/usr/local/bin/zilean-ui"] restart: unless-stopped env_file: [.env] depends_on: - zilean-core zilean-worker: image: ghcr.io/patrickpingol/zilean-slim:v1.0.0 command: ["/usr/local/bin/zilean-worker"] restart: unless-stopped env_file: [.env] depends_on: - zilean-core zilean-mcp: image: ghcr.io/patrickpingol/zilean-slim:v1.0.0 command: ["/usr/local/bin/zilean-mcp"] restart: unless-stopped # Zero-key service: a narrowed environment, not env_file. It never # receives SETTINGS_ENCRYPTION_KEY or SCRIPT_SECRETS_KEY and refuses to # start if either is set. environment: DATABASE_URL: ${DATABASE_URL} TZ: ${TZ:-UTC} MCP_ADDR: ${MCP_ADDR:-:8092} depends_on: - zilean-core zilean-webhook: image: ghcr.io/patrickpingol/zilean-slim:v1.0.0 command: ["/usr/local/bin/zilean-webhook"] restart: unless-stopped # Zero-key service: a narrowed environment, not env_file. It never # receives SETTINGS_ENCRYPTION_KEY or SCRIPT_SECRETS_KEY and refuses to # start if either is set. environment: DATABASE_URL: ${DATABASE_URL} TZ: ${TZ:-UTC} WEBHOOK_ADDR: ${WEBHOOK_ADDR:-:8093} WEBHOOK_TRUST_PROXY_HEADERS: ${WEBHOOK_TRUST_PROXY_HEADERS:-false} depends_on: - zilean-core # Deploy only the script runtimes you actually use — see "Images" above. zilean-script-worker-bash: image: ghcr.io/patrickpingol/zilean-scripts-bash:v1.0.0 restart: unless-stopped env_file: [.env] # Always-on hardening — subtractive only, so it needs no host # prerequisite and applies whether or not the bwrap sandbox is enabled. # The root filesystem is read-only, so the image cannot be modified; # /tmp is the only writable path — a size-capped, non-executable tmpfs # that each run gets its own working directory under, so relative-path # writes work. Every capability is dropped, and no-new-privileges # blocks setuid/file-capability privilege gain on exec. The size is a # suggested cap; see the note below on how it couples to mem_limit. read_only: true tmpfs: - /tmp:size=128m,noexec,nosuid,nodev cap_drop: - ALL # The two entries below are needed only for the bwrap sandbox, and only # once SCRIPT_SANDBOX_ENABLED=1 is also set in .env (see "Recommended: # enable script sandboxing" below). Both are relaxations of the # *container's* own confinement — neither loosens what a sandboxed # script can reach, which is governed by bwrap's bind list and # namespaces. no-new-privileges is not one of those relaxations: it is # part of the always-on hardening above and needs nothing from the host. # seccomp:unconfined - bwrap's --unshare-user needs # unshare(CLONE_NEWUSER), which Docker's # default seccomp profile blocks. One syscall # family. # systempaths:unconfined - bwrap mounts a fresh /proc, which the kernel # refuses while Docker's masked /proc paths are # mounted over it. security_opt: - seccomp:unconfined - systempaths:unconfined - no-new-privileges:true depends_on: - zilean-core zilean-script-worker-node: image: ghcr.io/patrickpingol/zilean-scripts-node:v1.0.0 restart: unless-stopped env_file: [.env] # Always-on hardening — subtractive only, so it needs no host # prerequisite and applies whether or not the bwrap sandbox is enabled. # The root filesystem is read-only, so the image cannot be modified; # /tmp is the only writable path — a size-capped, non-executable tmpfs # that each run gets its own working directory under, so relative-path # writes work. Every capability is dropped, and no-new-privileges # blocks setuid/file-capability privilege gain on exec. The size is a # suggested cap; see the note below on how it couples to mem_limit. read_only: true tmpfs: - /tmp:size=256m,noexec,nosuid,nodev cap_drop: - ALL # The two entries below are needed only for the bwrap sandbox, and only # once SCRIPT_SANDBOX_ENABLED=1 is also set in .env (see "Recommended: # enable script sandboxing" below). Both are relaxations of the # *container's* own confinement — neither loosens what a sandboxed # script can reach, which is governed by bwrap's bind list and # namespaces. no-new-privileges is not one of those relaxations: it is # part of the always-on hardening above and needs nothing from the host. # seccomp:unconfined - bwrap's --unshare-user needs # unshare(CLONE_NEWUSER), which Docker's # default seccomp profile blocks. One syscall # family. # systempaths:unconfined - bwrap mounts a fresh /proc, which the kernel # refuses while Docker's masked /proc paths are # mounted over it. security_opt: - seccomp:unconfined - systempaths:unconfined - no-new-privileges:true depends_on: - zilean-core zilean-script-worker-groovy: image: ghcr.io/patrickpingol/zilean-scripts-groovy:v1.0.0 restart: unless-stopped env_file: [.env] # Always-on hardening — subtractive only, so it needs no host # prerequisite and applies whether or not the bwrap sandbox is enabled. # The root filesystem is read-only, so the image cannot be modified; # /tmp is the only writable path — a size-capped, non-executable tmpfs # that each run gets its own working directory under, so relative-path # writes work. Every capability is dropped, and no-new-privileges # blocks setuid/file-capability privilege gain on exec. The size is a # suggested cap; see the note below on how it couples to mem_limit. read_only: true tmpfs: - /tmp:size=256m,noexec,nosuid,nodev cap_drop: - ALL # The two entries below are needed only for the bwrap sandbox, and only # once SCRIPT_SANDBOX_ENABLED=1 is also set in .env (see "Recommended: # enable script sandboxing" below). Both are relaxations of the # *container's* own confinement — neither loosens what a sandboxed # script can reach, which is governed by bwrap's bind list and # namespaces. no-new-privileges is not one of those relaxations: it is # part of the always-on hardening above and needs nothing from the host. # seccomp:unconfined - bwrap's --unshare-user needs # unshare(CLONE_NEWUSER), which Docker's # default seccomp profile blocks. One syscall # family. # systempaths:unconfined - bwrap mounts a fresh /proc, which the kernel # refuses while Docker's masked /proc paths are # mounted over it. security_opt: - seccomp:unconfined - systempaths:unconfined - no-new-privileges:true depends_on: - zilean-core ``` ### Optional: script volumes [Section titled “Optional: script volumes”](#optional-script-volumes) Script volumes are opt-in. The compose file above does not mount a volumes directory and does not set `SCRIPT_VOLUMES_ROOT`, so no script job can reach persistent storage, and nothing changes unless you add the mount. To let jobs be given named directories that survive container recreation, add a bind mount and the variable to each script worker service (`zilean-script-worker-bash`, `-node` and `-groovy`, whichever you run): ```yaml environment: SCRIPT_VOLUMES_ROOT: /volumes volumes: - ./script-volumes:/volumes ``` Merge these into the `environment` and `volumes` keys the service already has. Create the host directory before the workers first start, owned by the user the script worker containers run as (UID `65532`). Docker creates a missing bind-mount directory owned by `root`, which the worker cannot write to: ```bash mkdir -p ./script-volumes sudo chown -R 65532:65532 ./script-volumes ``` The root directory itself must be writable by UID `65532` and mounted read-write (not `:ro`). The worker probes the root once, at startup. A root that is unset, missing, or unwritable is treated as not mounted: the worker still starts and runs jobs that declare no volume, but reports `volumes_root_mounted` as `false` (logging a warning when the value is set but unusable), and every job that declares a volume fails. After fixing ownership or permissions, restart the worker (`docker compose restart zilean-script-worker-bash`) so it re-probes the root. The mount and `SCRIPT_VOLUMES_ROOT` are container settings, so adding or changing them needs a redeploy of the script workers. Defining volumes and giving them to jobs is covered in [Script jobs](/guides/script-jobs/#volumes). `SCRIPT_VOLUMES_ROOT` cannot be `/`. Use the sandbox with volumes With the `bwrap` sandbox off (the default), every script job on a worker with a mounted root can read and write every volume, and `ro` is advisory. A volume is also persistent and is not mounted `noexec`, so it is the one place a script’s writes survive and where a written file could be executed. Enable the sandbox below when you use volumes. ### Recommended: enable script sandboxing [Section titled “Recommended: enable script sandboxing”](#recommended-enable-script-sandboxing) The Bash, Node, and Groovy workers can each confine every script run with `bwrap` (bubblewrap): filesystem and pid-namespace isolation so a script sees only its own runtime binaries and a writable temp directory scoped to that one run. **This is opt-in and off by default** — set in `.env`: ```bash SCRIPT_SANDBOX_ENABLED=1 ``` on top of the `security_opt` entries in the Compose snippet above (and the host prerequisite in the next section). The hardening in that snippet is always on and needs no host prerequisite — `bwrap` is what adds per-job separation on top of it, not what switches the hardening on. If you skip this, every Bash/JavaScript/TypeScript/Groovy script job still runs — without per-job isolation, with real consequences worth knowing before you decide: * **One script’s temp files are visible to every other concurrently-running script job** on the same worker, not just its own. * **A script can read the whole container filesystem** it runs in — there is nothing stopping it from walking outside its own working directory. Its *writes* are confined to `/tmp` by the always-on hardening, but its reads are not. That hardening limits what a script can do to the container — it cannot modify the image, gain privileges, or persist writes outside `/tmp` (other than to a [script volume](#optional-script-volumes) you have mounted) — but it does not separate one script from another. That separation is what `bwrap` adds. Leaving `SCRIPT_SANDBOX_ENABLED` unset is a deliberate choice to accept that, not a default that assumes it away — enable it unless you have a specific reason not to. ### Check the host’s user-namespace policy first [Section titled “Check the host’s user-namespace policy first”](#check-the-hosts-user-namespace-policy-first) `bwrap` needs an unprivileged user namespace to do its confinement. Some distributions restrict those — Ubuntu 23.10 and newer ship the restriction on by default: ```bash cat /proc/sys/kernel/apparmor_restrict_unprivileged_userns ``` If that prints `1`, the namespace is created with no capabilities in it and `bwrap` cannot confine anything on this host — no `security_opt` setting can compensate. Set it to `0` (`sysctl -w kernel.apparmor_restrict_unprivileged_userns=0`, plus a `/etc/sysctl.d/` entry so it survives reboot), or give those three containers an AppArmor profile granting `userns create`. If the file prints `0` or does not exist, there is nothing to do. A restricted host does not fail your script jobs: with `SCRIPT_SANDBOX_ENABLED=1` set, each worker runs a real bwrap self-test once at startup, and if it fails (this sysctl being the most common reason), it logs a clear warning and falls back to running scripts without per-job confinement (the container hardening still applies) rather than breaking them. Check `docker compose logs zilean-script-worker-bash zilean-script-worker-node zilean-script-worker-groovy` after a deploy for that warning if you expected sandboxing to be active. Bring up **core first** — it owns migrations, and the other services exit non-zero and restart until the schema exists. ```bash docker compose pull docker compose up -d zilean-postgres docker compose up -d zilean-core docker compose logs -f zilean-core # confirm migrations applied docker compose up -d zilean-ui zilean-worker zilean-mcp zilean-webhook \ zilean-script-worker-bash zilean-script-worker-node zilean-script-worker-groovy ``` See [Deploying](/guides/deploying/) for the full Compose topology, resource limits, and reverse-proxy routing. ## Smoke test [Section titled “Smoke test”](#smoke-test) Set `SEED_JOB=1` on core for one boot. It inserts a single sample job — `smoke-test-httpbin`, `GET https://httpbin.org/get`, cron `*/5 * * * *` — which exercises the whole path: scheduler tick → job insert → HTTP call → visible in River UI and the admin UI. ```bash docker compose run --rm -e SEED_JOB=1 zilean-core ``` **Unset it once you’re managing real jobs** — it re-inserts the same row on every restart while set. # Introduction > What Zilean is, who it's for, and what makes it different from cron or a workflow engine. Zilean is a self-hosted scheduled job runner. It keeps the definition of every job — its schedule, its retry backoff, its success-path looping, its parallelism — in a Postgres table, and it applies changes to that table on the job’s very next run. It is a Go application built on the [River](https://riverqueue.com/) job queue, with an embedded admin UI and an embedded copy of River UI behind a single login. ## The core promise [Section titled “The core promise”](#the-core-promise) **Every behavior change is a row write, never a code or deploy change.** Editing a job’s row takes effect on the next scheduled run or the next job attempt. No redeploy. No editing crontab files on a host. No re-uploading a DAG or workflow YAML and waiting for a scheduler to pick it up. That is the one thing Zilean is built to do well, and it is the reason to reach for it over the neighboring tools: | | How a change is made | | ------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | **[cron](https://man7.org/linux/man-pages/man8/cron.8.html)** | Edit a crontab file on the host; needs shell access, and the job definition lives wherever the host is. | | **[Zapier](https://zapier.com/) / [n8n](https://n8n.io/)** | Edit in a hosted UI you don’t run, with your credentials and payloads on someone else’s infrastructure. | | **[Airflow](https://airflow.apache.org/)** | Change Python, re-upload the DAG, wait for the DAG processor to reparse it. | | **[Obsidian Scheduler](https://web.obsidianscheduler.com/)** | A web UI or its API, no redeploy needed — the same category as Zilean here. It’s a self-hosted JVM process, though, with the heavier baseline footprint that implies; see [Deploying to AWS](/guides/deploying-to-aws/) for a sizing comparison. | | **Zilean** | Write a Postgres row (via the admin UI or the JSON API). Effective on the next run. | Zilean is also deliberately lightweight: eight small Go binaries with no JVM or other managed runtime underneath them, sized to run comfortably on a single small instance rather than a cluster. See [Deploying to AWS](/guides/deploying-to-aws/) for concrete sizing. ## Who it’s for [Section titled “Who it’s for”](#who-its-for) A small group of people who trust each other and their network, managing their own scheduled jobs on their own infrastructure. Zilean has real users, roles and permissions — it is not single-user — but its design assumptions are “a handful of trusted operators,” not anonymous public internet users. Concretely, that shapes two things you should know up front: * **Global variables and script bodies are stored as plaintext.** A job’s `auth_config` secret (bearer token, basic auth password, API key) is encrypted at rest, but global variable values and script job bodies are not — anything that can read the database can read those. * **Script jobs have limited sandboxing.** Bash, Node/TypeScript, and Groovy script jobs can all run confined in a `bwrap` (bubblewrap) sandbox with no access to the broader container filesystem, once you opt in. See [Script jobs](/guides/script-jobs/) for the full security posture and deployment checklist. ## What a job is [Section titled “What a job is”](#what-a-job-is) A job is one row in the registry, and it is one of two kinds: * An **HTTP job** calls a URL on a schedule. * A **script job** runs a Bash, JavaScript/TypeScript, or Groovy script on a schedule, each runtime in its own isolated worker. Both kinds share the same scheduling, looping, retry and pooling machinery. See [Job types](/concepts/job-types/). ## Three mechanisms, deliberately kept separate [Section titled “Three mechanisms, deliberately kept separate”](#three-mechanisms-deliberately-kept-separate) Most schedulers give you one general-purpose “retry policy.” Zilean gives you three independent per-job mechanisms, and never merges them into one control: * **Loop** re-calls a job along its **success** path, until a value in the response says to stop. This is how you drain a queue or paginate a backlog in one firing. * **Retry** re-calls a job along its **error** path, with exponential backoff. * **Pooling** fans a single firing out into *N* parallel calls, each looping and retrying independently. A job can loop without retrying, retry without looping, do both, or do either across a pool. See [Loop, retry & pooling](/concepts/loop-retry-and-pooling/). ## Visibility over automation [Section titled “Visibility over automation”](#visibility-over-automation) Zilean surfaces what is happening rather than hiding execution behind an abstraction. Every attempt is recorded, every job has an activity chart and a heartbeat, worker processes report their own CPU and memory, and the embedded River UI exposes the raw queue internals — the actual job rows, states, attempts and errors. ## Next steps [Section titled “Next steps”](#next-steps) * [Installation](/start/installation/) — get a Zilean instance running. * [Configuration](/start/configuration/) — the environment variables it reads. * [Your first job](/start/your-first-job/) — register a job and watch it run. * [Architecture](/concepts/architecture/) — how the services fit together. ## Docs for AI tools [Section titled “Docs for AI tools”](#docs-for-ai-tools) These docs are also published in the [llms.txt](https://llmstxt.org/) format, so you can hand them to an LLM or coding agent: * [`llms.txt`](/llms.txt): an index of the documentation sets. * [`llms-full.txt`](/llms-full.txt): the complete documentation in one file. * [`llms-small.txt`](/llms-small.txt): a trimmed version for smaller context windows. # Try the live demo > Start a private, temporary Zilean instance in your browser, with sample jobs and a sample workflow. You can try Zilean without installing anything at . ## What you get [Section titled “What you get”](#what-you-get) Select **Start demo** on the landing page, complete the verification check, and wait for the instance to start (usually under a minute). You receive: * **A private instance.** It is yours alone; nothing you do affects another visitor. * **Automatic sign-in.** You land in the admin UI already signed in as a Super Admin. The banner at the top of the page shows the login, and if you log out, the login page shows it too, so you can sign back in. * **Sample data.** Six sample HTTP jobs in the groups `Demo/Payments`, `Demo/Marketing` and `Demo/Ops` (cron, rate, looping, failing, one-off and disabled examples), 24 hours of call history so the activity bars are populated, and a manual workflow named `Demo/Health probe`. * **A demo banner** with **Reset demo** (discards your changes and starts a fresh instance) and **End demo** (closes your session and discards its data). Both ask for confirmation. ## Time limits [Section titled “Time limits”](#time-limits) A demo ends after **15 minutes without activity**, or after **1 hour** in total, whichever comes first. When it ends, the instance and everything you created in it are destroyed. Opening the demo address afterwards shows an expired page, and you can start a new demo. ## What is not available [Section titled “What is not available”](#what-is-not-available) The demo never runs operator-supplied code and sends nothing outward, so these actions are refused with a message ending in `is not available in the demo`. Direct requests receive a `403`. A job import (NDJSON) still responds `200`, reports each refused row as a row error with the same message, and imports the other rows. Listing users, roles and invites still works, and the admin UI shows the refused controls disabled with a hint: | Area | Refused | | -------------------- | ------------------------------------------------------------------------------------------------------------------- | | Script jobs | Creating or editing (`403`), or importing (row error), a job of type `bash`, `javascript`, `typescript` or `groovy` | | Workflows | Saving a workflow that contains a script block | | SMTP | Changing SMTP settings; sending a test email | | Notification targets | Creating, changing or deleting targets; testing a target | | MCP | Changing MCP settings (creating and revoking MCP tokens still works) | | Passwords | Changing a password | | Users | Creating, editing, resetting the password of, or deleting a user | | Roles | Creating, editing or deleting a role | | Invites | Creating or resending an invite | The demo instance also has no access to the internet. HTTP jobs that call external addresses fail; the sample jobs call a built-in echo service instead. ## Limits per visitor [Section titled “Limits per visitor”](#limits-per-visitor) * **One demo at a time per network.** The limit applies to your IPv4 address, or to your IPv6 `/64` prefix, so a different browser or tab on the same network does not get a second demo. Ending or resetting your demo frees the slot. * **A global capacity cap.** Only a fixed number of demos run at once across all visitors. * **A start rate limit.** Starts and resets are limited separately: more than three starts, or more than three resets, in a minute from the same network is refused. When a request is refused, the page shows a message; clients that ask for JSON receive an `error` code instead: | Situation | HTTP status | `error` code | Page message | | -------------------------------------------------------------- | -------------- | -------------- | --------------------------------- | | You already have a demo running | `429` | `ip_limit` | You already have a demo running | | All demo slots are in use | `503` | `busy` | Demos are busy, try again shortly | | More than three starts, or more than three resets, in a minute | `429` | `rate_limited` | (JSON response only) | | New demos are paused | `503` | `paused` | Demos are paused | | Verification check failed | `403` | `turnstile` | The landing page is shown again | | Demo could not be prepared | `503` or `502` | `unavailable` | Demo temporarily unavailable | Pausing stops new starts; demos already running are not ended by the Worker. To run Zilean yourself, see [Installation](/start/installation/). # Your First Job > Register a job, watch it run, then make it loop. This walks through creating an HTTP job, confirming it ran, and editing it live — the thing Zilean exists to make cheap. You’ll need a running instance ([Installation](/start/installation/)) and an account with `jobs:write`. ## 1. Log in [Section titled “1. Log in”](#1-log-in) Open your instance root. You’ll be redirected to `/login`. Sign in with the `ADMIN_USERNAME` email and the password you generated the hash from. The API uses the same session cookie, so anything below works from the browser or from `curl` with a session. ## 2. Create a job [Section titled “2. Create a job”](#2-create-a-job) The simplest possible job — call a URL every five minutes: ```bash curl -X POST https:///api/jobs \ -H 'Content-Type: application/json' \ -d '{ "name": "hello-httpbin", "job_type": "http", "schedule_type": "cron", "cron_expression": "*/5 * * * *", "http_method": "GET", "url": "https://httpbin.org/get" }' ``` Everything else takes a default: 60-second timeout, retry enabled with 5 attempts, no looping, no pooling, `skip` overlap policy. In the admin UI, the same thing is **Jobs → New job**, a wizard that steps through job type, basics, request, authorization, loop & retry, and a review screen. ## 3. Confirm the schedule is understood [Section titled “3. Confirm the schedule is understood”](#3-confirm-the-schedule-is-understood) Cron expressions are evaluated in the **server’s** timezone, not yours: ```bash curl https:///api/meta # {"timezone":"America/Los_Angeles"} ``` If that says `UTC` and you expected otherwise, `TZ` isn’t reaching the container — see [Configuration](/start/configuration/). ## 4. Wait for the reconciler — or don’t [Section titled “4. Wait for the reconciler — or don’t”](#4-wait-for-the-reconciler--or-dont) The scheduler picks up new and changed rows every **30 seconds**. You can skip the wait: ```bash curl -X POST https:///api/jobs//run ``` A manual trigger fires immediately, deliberately skipping the deduplication guard that protects scheduled firings. ## 5. See what happened [Section titled “5. See what happened”](#5-see-what-happened) ```bash curl 'https:///api/jobs//calls?limit=5' ``` Each entry carries the queue state, attempt number, timestamps, and a `last_call` object with the method, URL, status code, duration and a snippet of the response. In the UI: the job’s row has a heartbeat bar, and the row menu offers **Calls** (recent attempts) and **Activity** (volume and latency over time). For raw queue internals — states, attempts, errors — use the embedded River UI at `/riverui/`. ## Working with many jobs in the UI [Section titled “Working with many jobs in the UI”](#working-with-many-jobs-in-the-ui) On the Jobs page, groups appear as a tree in a sidebar; select one to filter the table. Tick the checkboxes on several rows to enable, disable or delete them together with the batch actions, and use **Import CSV** to create jobs in bulk ([CSV import](/guides/csv-import/)). First-run and What’s new tours, a Settings overview tour, and a short tour for every Settings section can be replayed any time from **Settings → Tours**, which keeps the What’s new tours for the three most recent releases. Each Settings section also has a small “?” button beside its heading that starts that section’s tour in place. ## 6. Edit it, live [Section titled “6. Edit it, live”](#6-edit-it-live) This is the part worth testing, because it’s the product’s whole premise. Change the URL: ```bash # fetch the current row, change what you want, send the whole thing back curl https:///api/jobs/ > job.json # edit job.json, then: curl -X PUT https:///api/jobs/ \ -H 'Content-Type: application/json' \ --data @job.json ``` `PUT` takes the **full** object — there are no partial updates. The change takes effect on the **next attempt**, with no restart and no reconcile wait. The worker re-reads the job row on every attempt, so only a change to `cron_expression` itself waits for the next 30-second tick. ## 7. Make it loop [Section titled “7. Make it loop”](#7-make-it-loop) Looping is what separates Zilean from a cron entry. Point the job at an endpoint that reports how much work is left, and it will keep calling itself while that number stays positive: ```json { "loop_enabled": true, "loop_response_path": "result.remaining", "loop_delay_ms": 2000, "loop_jitter_ms": 300, "loop_max_iterations": 100 } ``` Your endpoint returns `{"result": {"remaining": 42}}` while there’s work and `{"result": {"remaining": 0}}` when it’s drained. The path is [gjson](https://github.com/tidwall/gjson) syntax, and only a strictly positive **number** continues the loop. Each iteration is its own attempt, so each one counts separately in the invocation counters — and can retry on its own. Caution `loop_max_iterations` hitting its cap logs an error and stops cleanly. There is **no alerting** — the log line is the only signal. Don’t rely on being told. ## 8. Disable it when you’re done [Section titled “8. Disable it when you’re done”](#8-disable-it-when-youre-done) ```bash curl -X POST https:///api/jobs//disable ``` Disabling stops work already queued, not just future firings — the worker re-checks `enabled` at execution time. Or delete it: ```bash curl -X DELETE https:///api/jobs/ ``` ## Where to go next [Section titled “Where to go next”](#where-to-go-next) * [Job fields](/reference/job-fields/) — every field, default and constraint. * [Loop, retry & pooling](/concepts/loop-retry-and-pooling/) — the three mechanisms in depth. * [Script jobs](/guides/script-jobs/) — run Bash, JS/TS or Groovy instead of an HTTP call, and the security caveats that come with it. * [Monitoring jobs](/guides/monitoring/) — heartbeats, metrics and Prometheus.