Skip to content

Troubleshooting

What to do when the stack or a filter is not behaving. Each entry says what you see, why it happens and what to do. Sift’s general rule helps when reading them: when something fails, Sift sends more events to the agent, not fewer. A failure costs agent runs; it does not silently hide events.

Start with three commands:

docker compose ps
curl -fsS http://localhost:8080/api/readyz
make logs SERVICE=sift-api

Every warning code and filter health code is listed, with its meaning, in States, badges and health codes.

The stack does not start

What you see	Why	What to do
`docker compose up` stops with set SIFT_SECRET_KEY in .env	There is no `.env`, or it has no secret key.	Run `make init`. If you are restoring an existing deployment, put its original key back instead: a new key cannot read the stored secrets.
Every backend container exits with Refusing to start in production with credentials anyone can read…	`SIFT_ENVIRONMENT=production` and a published or weak credential. The message names every such setting.	Fix each setting it names. See the production checklist.
Every backend container exits naming another setting	A value Sift refuses at start: a short metrics token, an inverted savings band, a malformed console URL, a webhook range outside private networks, the decision model on without its key.	Correct the setting; see Configure Sift.
`sift-api` exits with `model_weights_digest_mismatch`	The weights in the `model-weights` volume are not the pinned ones. Sift stops rather than score with a different model, because that would invalidate every trained model and approved threshold.	Re-stage the weights (Install without internet access). Do not edit the pin to match.
The workers and `sift-beat` stay in Created; `sift-api` is still starting	On a first start `sift-api` downloads the weights (about 134 MB) before it serves, and the others wait for it to be healthy. Its health check allows 15 minutes.	Wait, and watch its log. On a slow or blocked link, stage the weights yourself and set `SIFT_MODEL_FETCH_ON_START=false`.
`sift-web` restarts under the TLS overlay	`tls.crt` or `tls.key` is missing from `SIFT_TLS_DIR`.	Put both files in place (Serve over HTTPS).
`/api/readyz` answers `degraded` while `/api/healthz` is fine	A dependency is down. The API stays up on purpose.	Read the `checks` block: it names `postgres`, `redis` or `object_storage`, with a detail.

Signing in

What you see	Why	What to do
Sign-in succeeds, then every page sends you back to sign in	Outside development mode the session cookie is `Secure`, and the browser does not send it over plain HTTP.	Serve the console over HTTPS (Serve over HTTPS).
A banner that cannot be dismissed: Default administrator password still in use	An active account still has the shipped password `admin`.	Select Change password on the banner. The banner clears when no active account still has it.
Sign-in answers Too many failed sign-in attempts. Try again shortly.	Ten failed sign-ins for one username, or thirty from one address, within 15 minutes. There is no permanent lockout.	Wait for the window to pass. Behind a load balancer, check `SIFT_TRUSTED_PROXY_HOPS`: set too low, every user shares one address’s budget.

Warnings in the console

What you see	Why	What to do
No Kindo connection, or the connection not verified	Sift cannot reach your tenant, or has not confirmed it can.	An admin opens the gear icon (Administration) › Kindo connection. See The Kindo connection.
`escalations_undelivered` (critical) or `escalations_failing`	Escalations are not reaching the agent: Kindo unreachable, the key rejected, or the agent refusing runs. With `escalations_undelivered`, the named filters have stopped filtering.	Check the Kindo connection, then review Filtered events for the affected window. Once delivery works, escalations from the last 24 hours on running filters are sent again within about twenty minutes.
`polls_overdue` (critical)	Running filters have not been polled when due. `sift-beat` starts every poll, and nothing else reports it stopping.	Check that `sift-beat` is running, as one replica, and that `sift-worker-io` is healthy.
`scoring_backlog` (critical)	Events read from a source were never scored. The sweep that catches them is not running, or cannot score them.	Check `sift-beat` and `sift-worker-io`. If only some filters are named, search the worker log for `redrive_rescore_failed`.
`deliveries_overdue` (critical)	Shape delivery’s groups are not being released on time.	Check `sift-beat` and `sift-worker-io`.
`worker_pool_silent` or `broker_unreachable` (critical)	A worker pool, or Redis itself, is not answering. No webhook ever sends these.	Check the containers named, including whether one was killed for memory. See Monitor Sift.
`webhook_failing`	A receiver is refusing or unreachable, or its secrets cannot be decrypted.	The gear icon (Administration) › Notifications shows what each receiver answered.
`agent_runs_backlogged`	Agent runs waited more than 5 minutes on the deployment’s Kindo rate limit in the last 15 minutes. Every filter, and Sift’s own Kindo calls, share that limit.	Set an agent budget on the busiest filters (see Put Sift in front of a big stream). Raise `SIFT_KINDO_MAX_REQUESTS_PER_SECOND` only if your Kindo tenant’s quota allows.

A filter is not doing what you expect

What you see	Why	What to do
No events scored, and no errors	Only filters in shadow, proposed or active are polled. The filter may be in another state, held, not yet due, or its source has nothing new.	Open the filter. Its Actions tab says what it is waiting on.
Check the source (`polling.failing`)	The source has failed to poll three times in a row.	The filter’s source settings and its integration in Kindo.
Check delivery (`delivery.failing`)	Escalations are not reaching the agent.	As for `escalations_undelivered` above.
Raise the budget or the queue (`queue.losing`)	The filter’s agent budget held more than its queue could keep: escalations aged out or were pushed out, and the agent never saw them.	Settings › Delivery › Agent load: raise the budget, the queue’s depth or its longest wait, or accept the loss. See Watch a running filter.
Configure its agent (`agent.unconfigured`)	The paired agent answers that it has no instructions, so its runs teach Sift nothing.	Import or fix the agent in Kindo. See Connect a filter to its agent.
`agent.no_verdict`: the agent’s newest runs returned no verdict Sift could read	A step was added after Sift’s verdict step, the agent’s prompt was edited, or an agent Sift did not write is not asked for a verdict.	The card’s button says which. See Connect a filter to its agent and the agent contract.
The filter was filtering and is back in shadow, Resume or retrain (`fell_back.drift` or `fell_back.ceiling`)	Sift stopped it: its traffic drifted from what was approved, or it reached its trigger ceiling. The approval was withdrawn.	Read the alarm, then resume with a reason or retrain. See Hold, stop, resume or retire a filter.
`fell_back.unapproved`	The running version has no approved operating point.	Approve one, or retrain, from the card.
The filter stays in shadow on Collect N labels	It needs enough labels before it trains: 400 by default, or the recipe’s own minimum if higher.	Wait for verdicts. If the count stands still for seven days the step turns into a problem; check the source and the agent. See Verdicts, labels and training.
The agent acted on an old event before the filter was in shadow, or on the first poll’s backlog	A warm start, and a filter’s first poll, send past events. An agent generated at contract 5 or later is told they are history and takes no action on them; an older one, or one Sift did not write, is not told.	Regenerate the agent and pair the new one (a newer agent shape is available on Settings › Delivery), or choose None for the warm start where that matters.
The agent judged one event twice: in a warm start and again in shadow	The event changed in the source after the warm start read it, so shadow sent the new version. An unchanged one is recorded Judged in a warm start and not sent.	Expected.
The agent ran twice for one event	A send whose answer was lost is sent again. Kindo does not de-duplicate runs, so Sift cannot tell whether the first send started one. The second send tells the agent it may have seen the event before.	Expected after network trouble or a restore. An agent can check for an earlier action on the same event.
Training fails at once	The model weights are not staged, or there are too few labels.	`make check-models`, then the `sift-worker-ml` log.

Recipes and the assistant

What you see	Why	What to do
Text analysis (small sample) is shown as unavailable	The default backend image cannot train this recipe; it needs the larger `setfit` variant.	Set `SIFT_EXTRAS=setfit` in `.env`, then `make build` (or load the variant from the release bundle) and `make up`. A model of this recipe already trained still scores on the default image.
The `setfit` variant runs but refuses to train Text analysis (small sample); `fine_tune_weights_unavailable` in the `sift-api` log	The extra weights this recipe trains from are not staged.	`make models`, or stage them with `--fine-tune` on an isolated host.
Sift’s assistant is slow, or a turn fails after minutes	Each model call may take up to 180 seconds (`SIFT_KINDO_INFERENCE_TIMEOUT_S`) and is retried. Sift’s own nginx waits up to 900 seconds for assistant requests.	If a proxy of your own sits in front, raise its read timeout and turn off buffering for `text/event-stream`.

After a restore or a key change

What you see	Why	What to do
The Kindo credential looks normal but every call fails; webhook deliveries are abandoned	`SIFT_SECRET_KEY` is not the key the secrets were stored under. Verify on the connection says the credential could not be decrypted.	Put the original key back if you have it. If not, an admin replaces the Kindo key and re-enters each webhook endpoint’s secrets.
A filter does not retrain after a restore	A training job that was running at backup time came back as running.	Sift gives up on it four hours after it started; then retraining resumes.