> ## Documentation Index > Fetch the complete documentation index at: https://docs.reducto.ai/llms.txt > Use this file to discover all available pages before exploring further. # Operations > Runtime configuration for on-premise Reducto worker health and restart behavior Runtime knobs operators use to keep on-prem Reducto worker pods healthy: liveness behavior, restart policy, and watchdog tuning. Related references: * [Observability → Pod stack trace dumps](/onprem/observability#pod-stack-trace-dumps-sigusr2) for live thread-stack diagnostics via `SIGUSR2`. * [Database configuration](/onprem/database_configuration) for DB pool sizing and timeouts. * [LLM options → Azure Vision](/onprem/llm_options#azure-vision-ocr) for OCR provider timeouts, retries, and failover. ## v1.12.6 to v1.13 streaq migration v1.13 requires a healthy Redis-backed streaq deployment. Upgrade to the stable v1.12.6 release (not a prerelease) first and keep the legacy PostgreSQL queue workers running while streaq is validated and traffic drains. Streaq is opt-in on v1.12.6 patches: a routine install or upgrade that never sets `WORKER_PROVIDER: STREAQ_LOCAL` is not affected by anything below and will not have its release blocked. The checks described here only activate once you configure streaq, and they stay mandatory from that point on — including for any later v1.12.6+ patch upgrade, not just the v1.13 cutover. Configure v1.12.6 with a small, fixed streaq pool: ```yaml theme={null} env: WORKER_PROVIDER: STREAQ_LOCAL streaqWorkers: io: enabled: true workerName: io replicaCount: 1 kedaScaler: false cpu: enabled: true workerName: cpu useFullImage: true workerCount: 1 replicaCount: 1 kedaScaler: false ``` Also either set `redis.enabled: true` or supply `REDIS_URL` through `env`, `secretEnv`, or `envFrom`. The v1.12.6 Helm post-upgrade check sends a real `ping` task through both the `io` and `cpu` queues. The release fails if Redis is unreachable, either worker is missing, or the values above are incomplete. Before the v1.13 upgrade, run several representative `/parse` requests with the request-scoped streaq canary enabled: ```json theme={null} { "input": "https://example.com/document.pdf", "settings": { "alpha": { "use_streaq_parse": true, "compute_target_override": "STREAQ_LOCAL" } } } ``` Perform the v1.13 cutover as separate Helm applies: Use `--timeout 10m` for each Helm upgrade in this migration. The preflight Job has a nine-minute deadline, including pod scheduling and up to six minutes for worker health checks, so Helm's five-minute default timeout is insufficient. On v1.12.6, the health check is a post-upgrade hook that only runs once streaq is configured: a failure marks the Helm release failed after its resources are applied. Correct the reported topology or Redis issue and rerun the upgrade, or roll back. The v1.13 check runs pre-upgrade, and runs unconditionally, regardless of whether streaq was ever configured, blocking resource changes when its prerequisites fail. 1. Upgrade to v1.13 with `worker.enabled: true`. The pre-upgrade check requires an installed release marker from v1.12.6 or a later v1.12 patch and repeats the live Redis worker round trips before changing resources. 2. Scale the streaq pools for production load, then set both the trainable and non-trainable `*_STREAQ_*_ROLLOUT_RATIO` values to `1.0` for parse, extract, split, edit, and pipeline. Keep the legacy worker enabled while existing PostgreSQL-queue work drains. At full parse cutover, asynchronous requests with `use_reducto_lite` or `reducto_lite.enabled` are rejected because those modes require the legacy worker; the synchronous variants remain available because they run inline or route their base arm through streaq. 3. Poll `GET /drain_status` until `drained` is `true`. Then set `worker.enabled: false`, `worker.priorityCount: 0`, and `gpuWorker.enabled: false`. The Helm pre-upgrade check blocks this final apply unless the currently running HTTP deployment reports every effective rollout ratio as `1.0` and the live drain check passes. Ratios supplied via `env`, `secretEnv`, or `envFrom` are all read from that running process, so deploy the ratio change before the separate worker-disable apply. The drain check deliberately has no age-based exception for PostgreSQL queue rows. A 12-hour batch-queue SLA is not a row TTL, and the legacy queue has no default global queue timeout, so an old row can still represent valid work. The v1.12.6 bridge keeps the stale-job recovery CronJob and legacy worker alive until the queue is empty; the final apply removes both. If drain remains blocked, inspect the reported queue and active-job counts instead of deleting or ignoring rows based only on `created_at`. This drain check is not limited to the v1.13 cutover: once streaq is configured, setting `worker.enabled: false` on any v1.12.6+ patch is treated the same way — the apply is blocked unless every rollout ratio is `1.0` and `GET /drain_status` reports drained. There is no earlier, unguarded way to disable the legacy worker after streaq is armed. Re-enable `worker.enabled` and restore the rollout ratios to roll back while the v1.13 chart still contains the legacy worker templates. ## Worker liveness probe The DB-queue worker pods (`reducto-worker`, `reducto-priority-worker`, `reducto-gpu-worker`) ship with a Kubernetes liveness probe that restarts a pod **only** when it is stuck mid-processing, not when it is idle waiting for work. ### How it works 1. An asyncio `WorkerWatchdog` task runs alongside the worker's job loops. 2. Every 5 seconds it writes a heartbeat file `/tmp/worker-state` containing ` `. Idle workers emit `-1` for the age. 3. The kubelet runs `bin/worker-liveness.sh` as an `exec` probe. The script fails (exit non-zero) and triggers a restart when **either**: * the heartbeat file itself is older than `WORKER_LIVENESS_WATCHDOG_STALE_SEC` (event loop is wedged so the watchdog can't tick); **or** * the oldest in-flight task age exceeds `WORKER_LIVENESS_STUCK_THRESHOLD_SEC` (a real job has hung beyond the threshold). 4. Idle workers always pass the probe. The watchdog ticks even with no work, and the age sentinel `-1` is always treated as healthy. A file-based heartbeat is used rather than an in-process HTTP `/health` endpoint because an HTTP server can keep returning `200` while the asyncio event loop is blocked on a syscall. The watchdog has to be alive to refresh the file, so the probe directly tests the thing we care about. ### Helm configuration Configure the probe via `worker.livenessProbe.*` in your Helm values: ```yaml theme={null} worker: livenessProbe: enabled: true # set to false to disable the probe entirely stuckTaskThresholdSec: 1800 # restart if any in-flight task runs longer than this watchdogStaleSec: 60 # restart if the watchdog heartbeat hasn't ticked in this long periodSeconds: 30 # how often kubelet runs the probe timeoutSeconds: 5 # exec probe timeout initialDelaySeconds: 120 # grace period after pod start before probing begins failureThreshold: 2 # consecutive probe failures before pod restart ``` The same Helm partial applies the probe to all three worker deployments, so a single block configures `reducto-worker`, `reducto-priority-worker`, and `reducto-gpu-worker` together. ### Defaults | Knob | Default | When to change | | ----------------------- | --------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `stuckTaskThresholdSec` | `1800` (30 min) | Raise if your workload includes legitimately long single-task work (large multi-thousand-page documents, long extraction prompts). Lower if you'd rather fail fast and rely on client retries. | | `watchdogStaleSec` | `60` | Rarely needs tuning. Lower bounds how quickly an event-loop wedge is caught; should stay several times larger than the 5s tick interval to avoid false positives. | | `periodSeconds` | `30` | Lower for faster detection at the cost of more probe overhead. | | `initialDelaySeconds` | `120` | Raise if your pods take longer to come up (large image pulls, slow init containers). | | `failureThreshold` | `2` | Raise to make restart decisions more conservative. | ### Environment variables The Helm chart pipes the values above into env vars that the Python watchdog and shell probe both read, so the two stay in sync. You normally configure these via Helm, but you can override directly when running outside the chart: | Variable | Read by | Default | Purpose | | ------------------------------------- | ------------------------ | ------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `WORKER_STUCK_TASK_THRESHOLD_SEC` | Python watchdog | `1800` | Drives the per-task age comparison **and** the `logfire.warn` emitted when oldest age exceeds 80% of threshold (early breadcrumb for SRE before kubelet restarts the pod). | | `WORKER_LIVENESS_STUCK_THRESHOLD_SEC` | `bin/worker-liveness.sh` | `1800` | Per-task age limit used by the exec probe itself. Helm sets this from `worker.livenessProbe.stuckTaskThresholdSec`. | | `WORKER_LIVENESS_WATCHDOG_STALE_SEC` | `bin/worker-liveness.sh` | `60` | Maximum heartbeat-file age before the probe fails. Helm sets this from `worker.livenessProbe.watchdogStaleSec`. | | `WORKER_WATCHDOG_STATE_PATH` | Both | `/tmp/worker-state` | Heartbeat file location. Almost never needs to change. | ### Disabling the probe If you're operating in a constrained environment that can't run exec probes, or you'd rather rely on external orchestration, set: ```yaml theme={null} worker: livenessProbe: enabled: false ``` The Python watchdog still runs and emits `logfire.warn` when tasks exceed 80% of the threshold. The probe just doesn't trigger restarts. ### Verifying it's installed After deploy, confirm the probe is wired up: ```bash theme={null} kubectl describe pod -n reducto -l app=reducto-worker | grep -A 4 Liveness # Liveness: exec [bin/worker-liveness.sh] delay=120s timeout=5s period=30s ... ``` And confirm the watchdog file is being refreshed: ```bash theme={null} kubectl exec -n reducto -- cat /tmp/worker-state # 1778631873 -1 <- idle (age sentinel -1 is healthy) # 1778631878 42 <- busy with a 42-second-old task ```