Monitoring Jobs

Zilean surfaces execution detail rather than hiding it. There are six places to look, each answering a different question.

Question	Where
Has this job been running, and succeeding?	Heartbeat bar on the jobs list
What did the last few attempts actually do?	Calls modal / /api/jobs/{id}/calls
How has this job’s volume and latency trended?	Activity modal / /api/jobs/{id}/activity
Are the worker processes healthy?	Metrics page / /api/metrics/* and /prometheus/metrics
Are all eight services reachable at a glance?	Settings → System Health / /api/system/health
What is the queue actually doing right now?	River UI at /riverui/

What counts as one invocation

One attempt. A retry and a loop iteration each count separately — so a single cron firing can produce many invocations.

Every terminal outcome is recorded, including transport errors, non-2xx responses, body-read failures and configuration errors. Recording writes to three places: queue job metadata, lifetime counters on the job row, and a per-minute bucket table.

A failure to record is logged but never propagated — losing a count must not change a job’s retry or loop decision.

Call history

curl 'https://<your-host>/api/jobs/<id>/calls?limit=20'

Default limit 20, maximum 100. Returns a bare JSON array ([], never null), newest first.

Each entry carries the queue state, attempt number, scheduled/attempted/finalized timestamps, the last error, the pool index, the loop iteration, and a last_call object:

For HTTP jobs	For script jobs
method, url, status_code, response_snippet (≤500 chars)	interpreter, script_args, exit_code, stdout_snippet, stderr_snippet (≤2000 chars each)

Both share job_name, timeout_ms, duration_ms, and error when one occurred.

For HTTP jobs, secret values are redacted from what’s recorded: every secret global variable’s value and the job’s resolved auth credentials are replaced with REDACTED in url, response_snippet, error and last_error. Script jobs do the same for the secret variables their job was allowed to receive. The endpoint and the script itself still see the real values.

exit_code is only present when the process actually ran to completion — which is how you distinguish a genuine exit 0 from “never started.”

Note

This route reads River’s own job table, not the registry, so job_id in a response is the queue job id — not the registry UUID you used in the request path.

Heartbeats and activity

Heartbeats are fetched for every job in one request, because the jobs list polls on a timer and per-job requests would be N requests per tick. A job with no history returns an empty bucket list, which renders as an all-grey “never ran” bar — deliberately distinguishable from a failed fetch.

Per-job activity adds lifetime counters (invocations_total, invocations_completed, invocations_failed) alongside the windowed buckets. Those counters cover the job’s whole life, not the selected window.

Bucketing

Calls accumulate into fixed 1-minute buckets, which keeps row count independent of call volume — roughly 1,440 rows per job per day whether the job runs once an hour or loops constantly.

Reads aggregate those into wider display buckets. Every series response includes the window geometry:

{ "window_start": "...", "bucket_width_seconds": 60, "slot_count": 40 }

The UI needs this because the queries are GROUP BY aggregates — they emit nothing for an empty window — so gaps are filled client-side. It can’t re-derive the boundaries itself: its clock may differ, and buckets are epoch-aligned rather than now-relative.

Param	Default	Cap	Invalid
since	24h	720h (30 days), clamped silently to match retention	400 since must be a positive duration, e.g. 24h
buckets	40	500, clamped silently	400 buckets must be a positive integer

Latency is only populated on the per-job route

duration_ms_avg and duration_ms_max are filled in by the per-job activity query only. The heartbeat and worker-wide series leave both at 0, which is indistinguishable from a genuinely zero-latency bucket.

Call history buckets are retained for 30 days.

Worker metrics

Each service samples itself once a minute — CPU as a percentage of one core, Go heap allocation, bytes obtained from the OS, and goroutine count — and writes a row tagged with its service name.

Service labels: core, ui, worker, mcp, webhook, script-worker-bash, script-worker-node, script-worker-groovy.

curl 'https://<your-host>/api/metrics/live?service=worker'
curl 'https://<your-host>/api/metrics/history?since=24h&buckets=40&service=all'

live returns the latest sample, or JSON null with a 200 if that service has never sampled. service=all returns a map keyed by service instead of one series. An unknown service value is a 400, not a silent fallback.

Retained for 30 days. The metrics page also charts a worker-wide invocation series summed across all jobs.

Memory figures are Go-runtime, not container RSS

mem_sys_bytes is what the Go runtime obtained from the OS. Summing it across services would not give you “Zilean’s memory” — per-service lines are the honest reading. CPU percentages do sum meaningfully, since they’re percent-of-one-core.

Alerting on worker metrics

A per-service CPU%/memory threshold is created under Settings → Notifications with Add alarm: choose the service (core, ui, worker, mcp, webhook, or one of the three script workers), set a CPU% and/or memory (MB) threshold, and pick which targets to notify.

Thresholds accept fractional values such as 0.5 or 0.01. Leaving both empty shows “Set a CPU or memory threshold.”; a value that is not a number, or is negative, shows “CPU threshold must be 0 or greater (for example 0.5).” for the CPU field and “Memory threshold must be 0 or greater (for example 256).” for the memory field.

There is one alarm per service, and each is listed with its current state (OK or Breached). When an enabled alarm’s latest sample is at or above either threshold, Zilean notifies the alarm’s chosen targets once on the way up (state becomes breached) and once on the way back down (state returns to ok) — not on every sample while still breached, so a sustained outage doesn’t spam the same channel every minute.

This reuses the same target registry as a job’s own Notify step, but only its Apprise and email targets: the metrics alarms are not event-driven, so webhook targets are not offered for one. Per-job invocation alarms are state-change driven and do support webhook targets. See Configuration.

Invocation alarms

A per-job alarm that fires when a job’s invocation count over a rolling window reaches a threshold. Where a worker-metrics alarm watches a service’s CPU or memory, an invocation alarm watches how often a job runs.

Open a job’s Activity modal and use its Invocation alarms section to add one. Each row shows the metric, threshold and window, and the current state (OK or Breached). A breached alarm also shows the time it has been breached since. Each row has controls to edit, disable or delete the alarm.

Setting	Meaning
Metric total	Every attempt — each loop iteration and each retry counts separately.
Metric failed	Failed attempts only, counted the same way.
Threshold	Fires when the count reaches at or above this value.
Window	1 … 1440 minutes, counted in 1-minute buckets including the current partial minute.

An alarm is evaluated every 60 seconds. The count crossing the threshold moves the alarm to breached and sends a notification; falling back below it moves the alarm to ok and notifies again. Only those transitions notify — a sustained breach does not re-notify on every evaluation.

A breach notification contains:

• Title: Zilean: <job name> invocation alarm breached
• Body: <job name> had <count> <total|failed> invocations in the last <N> minutes (alarm at >= <threshold>).

A recovery notification puts recovered in the title and keeps the same body shape.

Invocation alarms notify the job’s own targets — the ones picked in the job wizard’s Notify step — whether they are Apprise, email or webhook targets. That target picker is shared with the job’s success/failure notifications, but notify_on does not gate an invocation alarm: a job with notify_on: none still delivers to the targets selected for it. A webhook target receives these deliveries only when its own event filter includes Invocation alarm (see Receiving a webhook). A job with no targets selected and All targets off has nowhere to deliver an alarm, so set at least one target on that step.

Prometheus

/prometheus/metrics on zilean-core exposes four gauges, each labelled by service:

• zilean_process_cpu_percent
• zilean_process_mem_alloc_bytes
• zilean_process_mem_sys_bytes
• zilean_process_goroutines

Core re-exports the other services’ samples on its own endpoint, because the workers run no HTTP server and can’t be scrape targets. Values refresh on a one-minute ticker, so a scrape never issues a database query.

A service with no samples yet is skipped, never published as zero — “not running” must not look like “running at 0% CPU”.

/prometheus/metrics is authenticated

It requires a valid session cookie and metrics:read. An unauthenticated scrape gets a 302 to the login page, not a 401. Plan your scrape configuration accordingly, and keep the endpoint off any internet-facing hostname.

Script worker liveness

curl https://<your-host>/api/scripts/liveness

Reports per script job type whether that runtime’s worker looks alive, based on whether it has reported a metrics sample within the last 2 minutes (two sampling intervals — tolerating one missed tick while still catching a real outage in about a minute).

Gated on jobs:script rather than metrics:read: the people who need this signal are the ones who can create script jobs. The new-job wizard uses it to warn when the runtime that would execute a job isn’t running.

System Health

curl https://<your-host>/api/system/health

Returns one entry per running service — worker, script-worker-bash, script-worker-node, script-worker-groovy, core, ui, mcp, webhook — each with a live boolean and a last_sample_at timestamp (null if that service has never sampled). A service is live if it has reported a metrics sample within the last 2 minutes, the same staleness window script worker liveness uses.

This is a simplified status roll-up across every service Zilean runs, built from the same stored samples the Worker Metrics chart already has — it adds no new sampling of its own. Where it differs from the other two liveness-shaped signals on this page: Worker Metrics charts CPU/memory over time for services you pick one at a time, and Script worker liveness covers only the three script runtimes, gated on jobs:script. System Health is a live/stale/never-seen status for all eight services at once, gated on metrics:read, and is what Settings → System Health in the admin UI renders as a table. A read failure for one service degrades just that entry to “never seen” rather than failing the whole request.

River UI

Mounted at /riverui/, this is the third-party River dashboard showing raw queue internals — job rows, states, attempt counts, errors, scheduled times.

Use it when you need to know what the queue itself is doing: a job stuck in scheduled/available (usually a runtime with no deployed worker), retry timing, or the full error on a discarded job.

River UI can mutate the queue

It offers retry, cancel and delete, which is why it’s gated behind jobs:write rather than session validity alone.

Finished job rows shown here don’t stick around forever: how long completed, cancelled, and discarded rows are kept is an admin-configurable retention window (Settings → River Jobs, settings:river_jobs), swept daily alongside the audit log’s own retention sweep. See River job retention in the API reference.

Known gaps

• Loop-max-iterations has no notification. Reaching loop_max_iterations logs an error and is otherwise silent — no target is notified. Other terminal job states do notify, through the job’s own Notify step (Apprise, email, or webhook targets; see Configuration).
• Failed logins for unknown emails aren’t recorded (deliberately — it would be an enumeration oracle).
• Audit retention changes aren’t audited.