Script Jobs

A script job runs operator-supplied code as a subprocess on a schedule.

curl -X POST https://<your-host>/api/jobs \
  -H 'Content-Type: application/json' \
  -d '{
    "name": "prune-exports",
    "job_type": "bash",
    "schedule_type": "cron",
    "cron_expression": "0 4 * * *",
    "script_body": "set -euo pipefail\nfind /exports -mtime +7 -delete\necho done",
    "script_args": ["--verbose"],
    "exec_timeout_ms": 120000
  }'

Permissions

Creating or editing a script job requires jobs:script in addition to jobs:write.

This is checked inside the create/update handler rather than at the route, because the job type is only known after the request body is decoded. Without it you get:

403 {"error":"script jobs require the jobs:script permission"}

No built-in role carries jobs:script — not even Admin. It reaches a non-Super-Admin only through a custom role, which is deliberate: the permission is the ability to run arbitrary code on your infrastructure. Grant it as such.

How a script runs

1. script_body and every element of script_args are used exactly as written. {{variable}} placeholders are not substituted in scripts, so editing a global variable can never change the code a script runs. Variables reach a script through its environment instead (see below).
2. The body is written to a temp file (mode 0600, no execute bit) with the runtime’s extension.
3. The interpreter is invoked with the temp file as an argument, followed by script_args.
4. The temp file is removed on every path — success, non-zero exit, or timeout.

A temp file rather than bash -c '<body>' avoids shell-escaping bugs, argument-length limits, and exposing the script body in the process list.

The process runs in its own process group, and a timeout kills the whole group — so a forked child can’t survive to hold the output pipes open. A further 5-second delay guards against a hung wait.

Where a script can write

/tmp is the only path where a script’s writes persist, unless you have mounted a script volume and the job declares it; each run gets its own working directory under it, so relative-path writes work (echo x > out.txt), and writes to the container’s own filesystem fail. The container’s root filesystem is read-only; with bwrap sandboxing enabled, a script’s writes outside its per-run directory land in the sandbox’s own temporary root and are discarded when the run ends, never reaching the container.

That /tmp is a size-capped, noexec tmpfs shared by every job on the worker — see Container confinement settings. Because it is noexec, a file the script writes to scratch cannot be run as a program directly (a volume is not mounted noexec, so a file written there can be): chmod +x helper && ./helper fails with permission denied. Run it through its interpreter instead — bash helper.sh, node helper.js. The interpreters live on the read-only root filesystem, so this does not change how the script itself runs: Bash, Node/TypeScript via tsx, Groovy, stdin payloads and environment variables are unchanged.

Scratch is not private to a job: all jobs on a worker run as the same UID (65532), so concurrent jobs can read each other’s files under /tmp. Per-job separation is what the opt-in bwrap sandbox provides — see Script jobs are not sandboxed.

What the script receives

The child environment is built from scratch — nothing is inherited. A script can rely on:

Variable	Notes
PATH	Copied from the worker process.
ZILEAN_JOB_NAME	The job’s name.
ZILEAN_POOL_IDX	Pool index. Set always, even at pool_count: 1.
ZILEAN_LOOP_ITERATION	Current loop iteration, 0 on the first.
TZ	Only if set on the worker process.
ZILEAN_VOLUME_<NAME>	One per volume the job declares, holding the path where it is mounted. See Volumes.
GLOBALVAR_<NAME>	One per non-secret global variable, plus any secret variable explicitly opted in. A secret is placed in the environment only when the variable is marked “Allow in scripts” (allow_in_scripts) and the job’s allowed-secrets list (script_secret_vars) names it; otherwise it is never injected.

Groovy additionally gets JAVA_HOME and JAVA_OPTS.

GLOBALVAR_ names are upper-cased with every run of non-alphanumeric characters replaced by _. A variable whose name is empty, separators-only, non-ASCII, or starts with a digit is skipped rather than mangled, so it is not exposed to the script. Runner-injected variables cannot be shadowed by a global variable; a collision keeps the injected value and logs a warning.

Opting a secret variable into a script’s environment

Secret global variables stay out of script environments by default. A deliberate, two-sided opt-in can expose specific ones to specific jobs:

1. On the Global Variables settings page, turn on Allow in scripts next to a variable’s secret toggle (allow_in_scripts). This marks the variable as eligible; on its own it does not expose it to any job. The toggle carries a warning that the value becomes visible in the environment and captured output of any script job it is injected into.
2. In the job wizard’s script step, select the variable’s name in the allowed-secrets multi-select (script_secret_vars). This lists which secret variable names this job may receive. The picker is only shown to a caller who holds globalvars:write in addition to jobs:script.

A value is injected as GLOBALVAR_<NAME> only when both sides are set. Setting allow_in_scripts requires globalvars:write; setting a job’s script_secret_vars list requires both jobs:script and globalvars:write. Both changes are recorded in the audit log.

Injected secret values are redacted to REDACTED if they appear in the job’s captured output in the Recent Calls / run-log view. That masking is display-only: the running script itself sees the real value, so this is a controlled exposure, not prevention.

HTTP jobs get the same masking without the opt-in, because their secrets are resolved by Zilean rather than handed to a script: every secret global variable’s value, plus the job’s resolved auth credentials, is redacted from what a call records. See Global variables.

Success, failure and output

• Exit 0 → success.
• Non-zero exit → failure: script exited 2: <stderr, truncated to 2000 chars>.
• Timeout → script timed out: …. Checked before exit status, because the kill surfaces as a signal-terminated exit.

stdout and stderr are each captured up to 64 KiB; beyond that, output is dropped while the full length is still reported. Error snippets kept in metadata are truncated to 2000 characters — four times the HTTP limit, because script failures are stack traces and compiler diagnostics rather than short error bodies.

stdout is also the loop signal. If loop_enabled is set, stdout is evaluated exactly as an HTTP response body would be — see Loop, retry & pooling. That’s why the output memory limit is well above the snippet limit.

Editing a script in the browser

The script field is a code editor rather than a plain text box. It highlights the selected runtime — Bash, JavaScript or TypeScript; Groovy is highlighted with Java rules, so closures, GString interpolation and def highlight approximately. Bracket matching, auto-indent, find/replace and word-based suggestions are available, but there is no language-aware IntelliSense or diagnostics. The editor follows the light/dark theme and needs no internet access.

Key	Effect
Tab	Inserts indentation.
Ctrl+M (Ctrl+Shift+M on macOS)	Toggles whether Tab indents or moves keyboard focus to the next control, so keyboard users can always leave the editor.

Comments and indentation are kept as you write them; mixed line endings are normalised to a single style.

Runtime-specific notes

Runtime	job_type	Notes
Bash	bash	Real bash plus coreutils — not busybox sh — so scripts behave as they do elsewhere.
JavaScript	javascript	Runs under node.
TypeScript	typescript	Runs under tsx; no separate build step.
Groovy	groovy	Minimum exec_timeout_ms of 10 000 ms, enforced at write time, because a fresh JVM costs 1–2 s to start. Grape (@Grab) dependency resolution is disabled at the interpreter.

A job whose runtime has no deployed worker queues indefinitely with nothing to pick it up. Check before creating one:

curl https://<your-host>/api/scripts/liveness

It reports {"live": bool, "last_sample_at": ...} per script type, based on whether that worker has reported a metrics sample in the last two minutes. The new-job wizard surfaces this as a warning.

Volumes

A script can be given persistent storage, and a job declares exactly which storage it may reach. Without a declaration a script sees no volume. Volumes are named directories that live under one host directory you mount into the script workers.

Volumes are opt-in. A default deploy mounts no volumes directory and sets no SCRIPT_VOLUMES_ROOT, and nothing changes until you add both. Until then every script worker reports volumes_root_mounted: false and a job that declares a volume fails.

Set up the host once

1. Create one host directory to hold all volumes, for example /srv/zilean-volumes.
2. Make the root directory itself writable by the user the script worker containers run as — UID 65532, the user all three script worker images run as. Confirm it with docker compose exec zilean-script-worker-bash id -u. Files a script writes are owned by that same ID on the host. Docker creates a missing bind-mount directory owned by root, so create and chown the root before first start.
3. Mount it into each script worker you use and point SCRIPT_VOLUMES_ROOT at the mount:

  zilean-script-worker-bash:
    # ...existing configuration...
    environment:
      SCRIPT_VOLUMES_ROOT: /volumes
    volumes:
      - /srv/zilean-volumes:/volumes

SCRIPT_VOLUMES_ROOT cannot be /. At worker startup the root is probed by creating and removing a temporary directory inside it, so the root itself must be writable: writable subdirectories under a root the worker cannot write to do not help, and a root mounted read-only (:ro) fails the probe.

Repeat for zilean-script-worker-node and zilean-script-worker-groovy if you run them. Mounting the directory and setting SCRIPT_VOLUMES_ROOT are container settings, so they need a redeploy of the script workers. Defining and changing volumes afterwards does not.

A worker whose root is unset, missing, not a directory, /, or not writable by UID 65532 still starts and still runs every job that declares no volume. It treats the root as not mounted: GET /api/script-workers/status reports volumes_root_mounted: false for that queue, and a job that declares a volume fails on that worker (see Failure modes).

The worker also logs a warning naming the path when the value is set but unusable. For an unwritable root the warning reads:

script volumes root is not writable by this process; script volumes will fail closed (likely wrong ownership or permissions, e.g. a Docker-created root-owned directory; UID 65532 needs write access)

The probe runs once, at worker startup, and the heartbeat reports that result for the life of the process — fixing ownership or permissions changes nothing until the worker is restarted. After a fix, restart it (docker compose restart zilean-script-worker-bash) and confirm the status flips to true.

Each volume is one directory directly under the root, named after the volume, created on first use if it does not exist. Keep it a real directory: a volume directory that is a symbolic link, or that resolves outside the root, is rejected.

Define a volume

Open Settings → Script volumes and create a volume. Defining volumes needs scriptvolumes:write; viewing them needs scriptvolumes:read (see Permissions). A volume has:

Field	Notes
name	Lowercase letters, digits and underscores, starting with a letter, at most 63 characters (^[a-z][a-z0-9_]{0,62}$). It cannot be renamed. Create a new volume instead.
max_mode	ro or rw. The most access any job may be granted to this volume.
description	Free text.

Creating a volume is a row write and takes effect on the next run. It does not create the directory. The worker creates it the first time a job mounts it.

Deleting a volume removes only the catalog entry. The files on disk are never deleted. A volume that a job or workflow script block still declares cannot be deleted; the request fails with 409 and lists what references it.

Declare volumes on a job

Pick volumes in the job wizard’s script step, or set capabilities in the API:

{
  "capabilities": {
    "volumes": [
      { "name": "reports", "mode": "rw" }
    ]
  }
}

• mode is required and is ro or rw. A volume whose max_mode is ro cannot be declared rw.
• Each volume may be declared once.
• {} and {"volumes": []} both mean no capabilities.
• Any other top-level key is rejected. egress is reserved and is rejected as not yet supported.
• Only script jobs and workflow script blocks can declare capabilities. An HTTP job, or any other workflow block, with a non-empty declaration is rejected with 400.
• Editing a declaration needs the same jobs:script permission as editing the script. Declaring an existing volume needs no volume permission, so a role holding jobs:script alone can use volumes but cannot create storage.

What the script sees

For each declared volume the script gets the directory mounted at <SCRIPT_VOLUMES_ROOT>/<name>, read-only for ro and writable for rw, and an environment variable with its path:

echo "report" >> "$ZILEAN_VOLUME_REPORTS/today.txt"

The variable is ZILEAN_VOLUME_ followed by the upper-cased volume name. A global variable cannot shadow it. Volumes the job did not declare do not exist inside the sandbox, and the volumes root itself is not mounted.

The declaration is checked again on every attempt against the current catalog, not only when the job is saved. A volume removed from the catalog, or lowered from rw to ro after a job declared rw, fails the attempt rather than being silently downgraded.

Enforcement needs the sandbox

Volume confinement is carried out by bwrap, so it is enforced only on a worker where the sandbox is active (see Installation). When the sandbox is not active, for example SCRIPT_SANDBOX_ENABLED is unset or the bwrap self-test failed:

• The declaration is still validated, so a bad declaration fails the same way on every host.
• The script runs unconfined with the same environment variables. Every script job on a worker with a mounted root can read and write every volume, and ro is advisory. Enable the sandbox when you use volumes.
• Each worker process logs one warning, the first time any job with a declaration runs unenforced. It is not logged again per job.

Settings → Script volumes shows, per script worker queue, whether the sandbox is enforced, whether the volumes root is mounted, and whether the worker has reported recently. Each worker reports every 60 seconds, and a report older than 3 minutes is shown as stale. The job wizard warns when you declare a volume for a queue that is not enforcing.

Volume failure modes

• The volume cannot be used (no usable root on this worker — unset, missing, or not writable by UID 65532; volume removed; mode above the ceiling; or a volume directory that is a symlink): the attempt fails immediately and is not retried, so it does not spend the retry budget. The error names the volume and the reason (volume root is not configured when the root is not mounted), and never contains file contents. Check the queue’s status in Settings → Script volumes (a worker without a usable root shows Root not mounted), or the worker’s log, to tell which.
• Volumes are persistent and executable. A volume is not mounted noexec, so it is the one place a script’s writes survive and where a file a script wrote could be executed. Grant rw sparingly.
• Concurrent writers. A volume is shared, persistent storage. Pool members, loop iterations and different jobs that write the same rw volume at once are your script’s concern. overlap_policy: skip serialises a single job.
• Egress is not supported. A declaration cannot restrict outbound hosts. The Docker network split described below remains the only network boundary.

Script jobs are not sandboxed

Worth stating exactly, because the partial confinement described below is easy to mistake for a full sandbox. What the design does close:

• The application’s own credentials are absent from the child environment — it’s built from scratch, not inherited.
• Execution in the HTTP worker’s container is impossible, because queue subscriptions are disjoint. This is structural, not a bypassable check.
• Services on other Docker bridge networks are unreachable, because Docker doesn’t forward between bridges.
• Writes to the container’s own filesystem are blocked. The worker’s root filesystem is read-only, so a script cannot modify the image, and /tmp is the only path where its writes persist (other than to a volume you mounted and the job declares) — a size-capped, non-executable tmpfs with a fresh working directory per run, so relative-path writes work. A helper written to scratch must be run through its interpreter rather than executed directly. See Where a script can write.

What remains open, and is explicitly accepted:

A script can recover DATABASE_URL, and therefore global variables and script bodies

The script runs as the same UID as the script-runtime worker process, so it can read /proc/<pid>/environ and recover that process’s environment — including DATABASE_URL. From there it can connect to Postgres as the application. Non-secret global variable values and other jobs’ script_body are plaintext and readable this way.

auth_config secrets (a bearer token, basic password, or API key) and secret global variables are not — they’re AES-256-GCM-encrypted at rest, keyed by SETTINGS_ENCRYPTION_KEY. Recovering DATABASE_URL gets the ciphertext, not the master key: the script-runtime workers deliberately never hold SETTINGS_ENCRYPTION_KEY itself, since every script job has auth_type: none by construction and so never needs to decrypt auth_config. This is the actual reason for that omission, not an oversight — see Configuration.

What a script worker does hold is SCRIPT_SECRETS_KEY — a sub-key derived from SETTINGS_ENCRYPTION_KEY via HKDF-SHA256 (see Generating SCRIPT_SECRETS_KEY for the command). It can decrypt only global variables with allow_in_scripts set — never auth_config or the SMTP password. This is what makes the opt-in mechanism above (see Opting a secret variable into a script’s environment) work: both gates must be set by someone holding the right permissions before the real value reaches a script, with only the captured-output view redacted afterward.

The per-job script_secret_vars allowlist is an authorization boundary, not a cryptographic one: SCRIPT_SECRETS_KEY can decrypt every variable with allow_in_scripts set, not just the ones the currently-running job names. A well-behaved worker only ever fetches the values its own job’s allowlist names, but a compromised script-worker process holding this key could, in principle, decrypt any opted-in secret directly — not just the ones belonging to its own job. Every variable that has not opted in remains ciphertext-only to a script worker, undecryptable by either key it might hold.

The curated child environment is blast-radius reduction, not a boundary. What actually contains the remaining exposure (global variables, script bodies) is the container’s own narrowed environment plus Postgres authentication. Closing it properly needs a restricted database role; that doesn’t exist today.

• Every script runtime’s filesystem access can be confined via bwrap (bubblewrap): the script sees only its own runtime binaries, a writable temp directory for that run alone, and any volumes the job declares, and nothing else — but only when you set SCRIPT_SANDBOX_ENABLED=1. This is opt-in and off by default, so an existing deployment never has script sandboxing silently turned on for it by an upgrade. Without it, one script’s temp files are visible to every other concurrently-running script job, and a script can read the whole container filesystem — the always-on hardening restricts writes to /tmp, but it is not a per-job boundary — enabling it is recommended; see Installation for how. The bwrap wrapper also fails open even when enabled: if the binary is missing, or a real self-test of it fails on this host (most commonly a distro that restricts unprivileged user namespaces — see Deploying), the worker logs a warning and scripts run without per-job confinement (the container hardening still applies) rather than failing.
• Outbound internet access is intentional — the application network is deliberately not internal.
• Host-published ports defeat Docker network isolation. A service published on the host’s address is reachable from a script regardless of which Docker network it’s on. The real control is host-level egress filtering, not Docker networking, and it must be in place before you enable script execution.
• Nothing bounds CPU, memory or process count beyond the per-attempt timeout and whatever container limits you set, and disk writes only as far as the size-capped /tmp tmpfs — 128m for Bash, 256m for Node and Groovy, charged to the container’s memory limit (see Container confinement settings). Groovy’s MaxRAMPercentage is a per-JVM heap ceiling read from the cgroup limit, so mem_limit must cover five concurrent JVMs, not one — see Deploying.

What the sandbox does not cover

• There is no syscall or seccomp filtering inside the sandbox. The confinement is filesystem and pid-namespace only: a script cannot see paths outside the bind list or other processes, but it can make any system call the container itself can.

Before enabling script jobs in production

1. Put host-level egress filtering in place, and verify it by IP, not hostname — a hostname probe from an isolated container fails on DNS whether or not isolation exists, so it proves nothing while reading as a pass.
2. Confirm the script workers’ container environments really are narrowed to DATABASE_URL, TZ and SCRIPT_SECRETS_KEY.
3. Confirm mem_limit on the Groovy worker if you use Groovy.
4. Grant jobs:script to a specific role, deliberately — never alongside jobs:write by default.