How Sift works

Sift runs one loop for every filter: read events, decide which reach the agent, read the agent’s verdicts back, and learn from them. This page follows that loop, then the path one filter takes from the day you create it to the day it starts filtering. Each step links to the page that covers it in depth.

The loop

  Kindo integration (your source)
        │  polled on the filter's schedule
        ▼
  ┌──────────────────── pipeline ─────────────────────┐
  │ Always escalate → Collapse repeats → Decide →      │
  │                                   Shape delivery   │
  └───────────────────────────────────────────────────┘
        │                         │
   escalate                    filter ──► Filtered events
        │                         │
        │      leak-through ◄─────┘  (a small random share is sent anyway)
        ▼
  Kindo agent run ──► verdict: actionable | benign | insufficient data
        │
        ▼
  Sift learns ──► a person approves a threshold ──► Decide uses it

1. Read the source. Each filter reads one tool of one Kindo integration, on its own schedule (every 60 seconds by default). If a read fails, Sift tries again from where it left off, so no event is skipped.

2. Run the pipeline. Every event passes through up to four slots, in order. A filter uses the slots it needs. See The pipeline.

Slot	What it does	Can it filter?
Always escalate	Rules that send an event whatever the later slots decide.	No, it only adds escalations.
Collapse repeats	Folds an event into a near-identical earlier one, so the agent runs once for the group.	Yes
Decide	Scores the event and filters it if the score is below the threshold.	Yes
Shape delivery	Groups escalations into fewer agent runs.	No

3. Escalate or filter. An escalated event starts an agent run. A filtered event is listed under Filtered events, without its content. A small random share of the events a filter would filter, 3% by default, is sent anyway as leak-through. The agent cannot tell those apart from any other event, which is how Sift measures what filtering misses.

4. Read the verdict back. Sift reads each finished run and takes the agent’s verdict: actionable, benign or insufficient data. A run that failed or gave no verdict is never counted as benign. See Verdicts, labels and training.

5. Learn. Each verdict becomes a label. Once a filter has enough labels, Sift trains a model on them and measures it on recent labels it did not learn from. A new model always starts in shadow: it is measured, and decides nothing, until a person approves its threshold.

One filter, start to finish

A filter moves through these states. The console shows each as a badge and lists the next step on the filter’s Actions tab. See Filters and their lifecycle.

 authoring ──(gold set approved, then an operator starts shadow)──► shadow
                                                                      │
                                                               (model trained)
                                                                      ▼
 active ◄───────────────────(threshold approved)───────────────── proposed
   │
   └──(drift, trigger ceiling, or stopped by hand)──► shadow

1. Create. You describe what the filter should catch. Sift’s assistant reads samples of the source and proposes a filter; you review it and press Create filter. The filter is in authoring and filters nothing. See Create a filter.
2. Connect the agent. Sift generates the paired agent for the filter. You create it in Kindo from the console, or import it by hand, and Sift pairs the filter to it. A filter that feeds an agent you already have skips this. See The paired agent.
3. Approve the gold set. Sift picks about fifty events from the live source and proposes for each whether it matches the filter’s intent. A person judges every one, and an operator or admin approves the set. This is the first of two approvals. It allows the filter into shadow; it does not move it there. See The gold set.
4. Shadow. An operator or admin presses Start in shadow on the filter’s Actions tab, once the gold set is approved and, for a filter with a paired agent, the agent is connected. The filter now reads and scores every event and records what it would filter, but sends everything to the agent. On a stream too big for the agent, an agent budget or sampled shadow can send less, if you set one; see Put Sift in front of a big stream. See Shadow and filtering.
5. Collect labels. Verdicts accumulate. Sift trains once the filter reaches its label floor, 400 labels by default, and once the newest labels, which Sift sets aside to measure the model on, hold at least 20 actionable events. How long that takes depends on the source’s volume.
6. Proposed. Sift trains a model on its own, measures it, and proposes an operating point: a threshold, with the share of events it would filter and the actionable events it would have missed. It moves the filter to proposed, which still sends everything.
7. Approve the threshold. An operator or admin reads the evidence and approves the threshold with a written reason. This is the second approval, and the only way a filter starts filtering. See Approve a threshold.
8. Active. The filter filters events below the threshold. Sift keeps measuring it through leak-through and watches for drift. If the traffic changes from what was approved, or the filter starts more agent runs in an hour than its trigger ceiling allows, Sift moves it back to shadow on its own, and it sends everything again until a person resumes it or approves again. (An operator can set a filter to keep filtering through drift and the ceiling, and raise an alert instead; see Shadow and filtering.) See Watch a running filter.

Retraining follows the same rule: a new model enters shadow, and replaces the running one only through another threshold approval.

Related pages

• What Sift guarantees: the safety properties this loop is built around.
• Getting started: who does each step on a new install.
• Health, Actions and alerts: how Sift tells you a filter needs something.