Loop, Retry & Pooling

These are three separate mechanisms with three separate purposes. Zilean deliberately never merges them into one “retry policy.”

	Triggered by	Purpose
Loop	Success	Keep calling while there’s more work — drain a queue, page a backlog.
Retry	Failure	Back off and try again.
Pooling	A firing	Do N of it at once, in parallel.

Loop

When loop_enabled is true, a successful attempt inspects the response and decides whether to run again.

How the decision is made

The response body is evaluated with gjson at the path in loop_response_path (default result.remaining):

• The loop continues only if the path resolves to a strictly positive number.
• The loop stops if the path is absent, or the value is zero or negative.

For a script job, the same evaluation is applied to the script’s stdout, with no script-specific handling.

gjson paths, not JSONPath

loop_response_path uses gjson syntax — dot notation, array indices, wildcards, # queries. result.remaining, data.items.#, queue.0.depth. It is not JSONPath and has no leading $.

Because the check reads gjson’s numeric field, a JSON true or a quoted "5" does not continue the loop — return an actual number.

How the next iteration is scheduled

A continuation is a fresh queue insert, scheduled at:

delay = loop_delay_ms + random(0, loop_jitter_ms)

A flat delay plus jitter — completely independent of the retry backoff settings. Note that loop_jitter_ms has no applied default: leaving it at 0 means no jitter at all.

The pool index is carried forward unchanged, so a pool member’s loop stays addressed to its own index for the life of that run.

Hitting the iteration cap

loop_max_iterations (default 50) bounds one logical run, not the job’s lifetime — the iteration counter resets to 0 on the next scheduled firing.

When the cap is hit:

• An error is logged (loop max iterations hit, stopping) with the job id, name, pool index and cap.
• The overlap lock is released and any one-off completion is applied.
• The queue job is not failed. It returns cleanly, so there is no retry and no discarded job.

There is no alerting on this yet

Hitting the cap produces a log line only. No Slack, no email, no notification. If a looping job silently stops making progress, the log is the only signal.

Retry

When retry_enabled is true, a failed attempt is rescheduled with exponential backoff. Zilean overrides River’s default retry policy entirely.

The backoff formula

delay = min( retry_base_delay_ms × retry_multiplier ^ (attempt − 1),
             retry_max_delay_ms )

attempt is 1-based. There is no jitter on the retry path — unlike loop.

Retry settings are re-read from the registry on every retry decision, so editing them takes effect on the next backoff calculation. Values that are unusable are coalesced: base ≤ 0 → 1000 ms, multiplier ≤ 1 → 2, max delay ≤ 0 → 5 minutes.

Attempt budget

retry_max_attempts (default 5) becomes the queue job’s MaxAttempts:

• retry_enabled: false → 1 attempt. A failure is cancelled outright rather than handed to any retry schedule.
• retry_max_attempts ≤ 0 → 5.

MaxAttempts is fixed at insert time

Editing retry_max_attempts affects jobs inserted after the change. Jobs already queued or mid-retry keep the budget they were created with.

What counts as a failure

For an HTTP job: any status outside 200–299, plus transport and body-read errors. For a script job: a non-zero exit code, or a timeout.

A configuration error — an unparseable URL, an unknown auth type, a missing interpreter, an unsupported job type — is cancelled rather than retried. It will never succeed, so it doesn’t burn the retry budget.

Pooling

pool_count (default 1, max 20) fans one firing out into that many parallel jobs, one per pool index 0 … pool_count − 1.

Each pool member is fully independent: its own queue job, its own overlap lock, its own loop, its own retry budget. One member being blocked does not prevent the others from firing.

How a job knows its index

Job type	Surface
HTTP	The pool_idx query parameter — added only when pool_count > 1, so a non-pooled job’s request is byte-identical to a pre-pooling one.
Script	The ZILEAN_POOL_IDX environment variable — set always, even at pool_count: 1.

The index is also carried through every loop continuation unchanged.

Partial firings are normal

If some indices are already running and others aren’t, the firing proceeds for the ones that are free and logs which were skipped. It only reports “run in progress” (HTTP 409 on a manual trigger) when every index was already locked.

One-off jobs must have pool_count: 1.

Overlap policy

overlap_policy decides what happens when a job is still running as its next firing comes due.

Value	Behavior
skip (default)	A lock is taken per pool index before inserting. If that index is still running, that index’s firing is dropped. The lock is held for the entire run — every loop iteration and every retry — and released only when that member has fully finished.
allow	Fire regardless. No lock is ever taken.

Under skip, the lock is a per-index entry in running_since_by_pool, acquired with a single guarded statement so two concurrent firings cannot both decide the job is free.

Locks are released on every terminal path, including retry exhaustion — which is handled by a dedicated error handler, because when River exhausts attempts it discards the job without calling the worker again. Without that handler, the lock would sit until the one-hour stale timeout.

Why uniqueness isn’t enough

The insert-time uniqueness guard stops one firing being duplicated across two close reconcile ticks. It cannot stop a firing that overlaps a previous run’s loop, because later loop iterations are separate queue rows with a different iteration number. That’s what skip is for.

How they compose

• Loop and retry are orthogonal axes. Each loop iteration is its own queue job with its own attempt budget, so a failing iteration retries within that iteration. The loop path never goes through the retry scheduler.
• A live loop holds its overlap lock. The lock is not released between iterations, and its timestamp is refreshed on every attempt — success or failure — so a genuinely progressing multi-hour loop never trips the stale-lock backstop.
• Pooling multiplies both. Each of the pool_count members loops and retries on its own, and under skip holds its own independent lock.
• Every attempt counts separately. A loop iteration and a retry attempt are each their own invocation, so the count grows with the work a job does rather than with the number of firings. An invocation alarm reads that count the same way, whether it watches every attempt or failures only.