Monitor Sift

Sift reports on itself in four ways: two health endpoints for orchestration and uptime checks, a list of deployment warnings in the console, Prometheus metrics, and structured logs. This page covers each, and the two failures you must watch from outside Sift because Sift cannot report them itself.

Health endpoints

Both are served through the web port and need no sign-in.

curl -fsS http://localhost:8080/api/healthz
curl -fsS http://localhost:8080/api/readyz

Endpoint	Answers	Use it for
/api/healthz	{"status": "ok", "service": "kindo-sift", "version": "…"} from the API process alone. It touches no dependency.	Is the API process up.
/api/readyz	One check each for postgres, redis and object_storage. All ok gives 200 and "status": "ok"; any failure gives 503 and "status": "degraded", with the failing check’s detail.	Is the deployment able to work. Gate on this one.

The object_storage check asks for the model bucket with Sift’s own key, so a wrong key or a missing bucket fails it as well as a stopped store; its detail is the storage error code, such as 403 or NoSuchBucket. Each check has three seconds.

Every service in the stack also has a Docker health check, so docker compose ps shows healthy or unhealthy per container.

Deployment warnings in the console

Sift turns conditions an operator should act on into warnings: the default admin password still in use, no Kindo connection or an unverified one, escalations not reaching agents, events left unscored, polls overdue, a webhook receiver failing, and others. They appear in the console’s alerts (the bell and the Alerts page), and the dashboard’s system line names which of Kindo, Workers, Ingest and Delivery is not healthy. Each warning clears on its own once the condition is fixed, and none can be dismissed.

The same list is at GET /api/v1/system/status for any signed-in user. Every warning code, its severity and what to do is in States, badges and health codes. For what each filter’s own health means, see Health, Actions and alerts.

Warnings also reach webhook receivers an admin configures, raised once a condition has held for two minutes and cleared ten minutes after it goes. See Notifications and webhooks.

Watch these from outside Sift

Two warnings, broker_unreachable (Redis is not answering) and worker_pool_silent (a worker pool is not answering), are never sent by webhook: Sift sends alerts through Redis and its workers, so when those fail nothing can send them. The console still shows them, but only to someone looking. worker_pool_silent appears up to 15 seconds after a pool stops. SIFT_WORKER_HEARTBEAT=false makes the status ask the workers directly instead, which adds about a second to every status call.

Cover them with checks that do not depend on Sift’s workers:

• An uptime check on /api/readyz. It fails when Redis does.
• docker compose ps, or your container platform’s health status, for each worker.
• With metrics on, sift_celery_queue_length rising on one queue for 10 minutes: a queue nothing consumes only grows. Sift queues alert work on notify several times a minute, so a dead sift-worker-notify shows within minutes.

To prove a pool takes work, an operator or admin can send it a test job: POST /api/v1/system/ping?queue=io (or ml, notify). It answers within ten seconds, or fails with 503 naming the queue.

Prometheus metrics

Metrics are off until you set a token. Without one, /api/metrics answers 404; with one, a request without it gets 401.

1. Generate a token and add it to .env:

   echo "SIFT_METRICS_TOKEN=$(openssl rand -hex 32)" >> .env

   Sift refuses to start with a token shorter than 32 characters.

2. Apply it:

   make up

3. Point Prometheus at /api/metrics on the web port, sending the token as a bearer token:

   scrape_configs:
     - job_name: sift
       scheme: https
       metrics_path: /api/metrics
       authorization:
         credentials_file: /etc/prometheus/sift-metrics-token
       static_configs:
         - targets: ["sift.example.internal:443"]

Every series is read from Postgres and Redis when the scrape arrives, and every one is a gauge but sift_page_cache_hits_total, a counter. Filters are labelled by their public id (flt_…), never by name.

Group	Series
Filters	sift_filters{state}
Scoring	sift_scored_events_last_hour{filter,decision}: filter is what was filtered; escalate and leak_through reached the agent.
Delivery to agents	sift_escalations_undelivered{filter}: escalations from the last day, older than 15 minutes, with no agent run.
Sources	sift_poll_consecutive_failures{filter}, sift_poll_last_success_timestamp_seconds{filter}, sift_poll_failures_last_hour{filter}
Verdicts	sift_agent_runs_outstanding, sift_harvest_oldest_outstanding_age_seconds
Training	sift_training_jobs{status}
Shape delivery	sift_deliver_groups_overdue, sift_deliver_groups_held, sift_deliver_events_held, sift_deliver_group_runs, sift_deliver_runs_saved, sift_deliver_char_cap_releases, each by filter
Always escalate	sift_guarantee_rescues{filter}, sift_guarantee_not_asked{filter}
Agent budget	sift_agent_budget_runs_per_hour, sift_agent_runs_last_hour, sift_held_queue_events, sift_held_queue_oldest_seconds (absent with nothing waiting), sift_held_queue_lost_24h{outcome} (expired, displaced, cleared, recorded), each by filter, for filters with an agent budget
Kindo rate limit	sift_kindo_bucket_wait_seconds_max: the longest an agent run waited on the deployment’s Kindo rate limit in the last 15 minutes
Decision model	sift_decisions_calls{feature,provider}, sift_decisions_failures{feature,provider} (today, UTC)
Console overview cache	sift_page_cache_age_seconds{read}: seconds since the newest cached copy of each overview read was computed (absent until one is); sift_page_cache_hits_total{read,outcome}, a counter of reads served as hit, stale or miss; sift_page_cache_refresh_seconds{read}: how long the last computation that filled each took. read is dashboard, fleet, activity, flow, asks, alerts or filters.
Queues	sift_celery_queue_length{queue}, sift_celery_unacked_messages
The scrape itself	sift_metrics_source_up{source}, sift_metrics_collect_seconds

When sift_metrics_source_up is 0 for a store, that store’s series are missing, not zero.

Alerts worth starting with

Alert on	Means
sift_escalations_undelivered > 0 for 15 minutes	Events are decided and not sent. When this persists, Sift stops the affected filters from filtering; find out why first.
time() - sift_poll_last_success_timestamp_seconds > 3600, or sift_poll_consecutive_failures >= 5	A source has stopped being read.
sift_harvest_oldest_outstanding_age_seconds > 7200	Verdicts are arriving late, so labels and training are too.
sift_deliver_groups_overdue > 0 for 5 minutes	Grouped escalations are not being released.
sift_celery_unacked_messages > 0 and sum(sift_celery_queue_length) == 0 for 30 minutes	A worker died holding work.
A sharp rise in sift_scored_events_last_hour{decision="filter"} for one filter	The filter is filtering more than it did. The direction worth paging on.
sift_guarantee_not_asked rising for 15 minutes	The decision model is not answering a filter’s always-escalate rules.
sift_held_queue_lost_24h{outcome=~"expired|displaced"} rising	A held queue is losing escalations the agent never sees (queue.losing).
sift_kindo_bucket_wait_seconds_max > 300	Agent runs are waiting on the deployment’s Kindo rate limit (agent_runs_backlogged).
sift_metrics_source_up == 0, or the scrape failing	You are blind.

Logs

Every service logs to standard output, and the backend logs one JSON object per line (SIFT_LOG_FORMAT=console for readable lines instead). Follow one service:

make logs SERVICE=sift-worker-io

Each API request gets a request id, returned in the X-Request-ID response header and in the body of every error (or taken from the request’s own header, if it sends one of up to 64 characters). Every log line the API writes for that request carries it, so you can find a failed request’s lines from the id a person reports.

A backstop removes values whose key looks like a secret (password, token, API key, cookie and similar) from every log line.

Lines worth searching for:

Event	Where	Means
"logger":"entrypoint"	sift-api	The start-up steps: migrations_done, buckets_done, seed_admin_done, model_weights_ready, or what failed.
model_weights_digest_mismatch	sift-api and the io and ml workers	The staged weights are not the pinned ones; the container stopped.
task_failed_permanently	workers	A background job gave up after its retries. The line names the job and what it was working on.
readiness_degraded	sift-api	/readyz failed, with the failing checks.
metrics_refused	sift-api	A metrics request without a valid token.
startup	sift-api	The API started, with its environment and version.

Related

• Troubleshooting
• States, badges and health codes
• Notifications and webhooks