How Sift measures a filter

Sift reports two kinds of figure about a filter: what it saved, and what that saving cost in missed events. The dashboard and every filter’s Overview show both together, and never the first without the second. This page defines each figure in the console’s words, says where it comes from, and explains why some familiar figures are missing.

What a filter did

Each figure covers the window you pick: 7, 30 or 90 days on the dashboard (30 by default, headed Last 30 days), and 1, 7, 30 or 90 days on a filter’s Overview (7 by default).

Figure	What it counts
Events	Events read from the source in the window. The dashboard shows it under runs avoided, as of N events.
Runs avoided (dashboard: Agent runs avoided)	Events filtered: below the operating point, or folded into an earlier near-duplicate by Collapse repeats. Leak-through events reach the agent, so they count as escalated, never as avoided.
Share filtered	Runs avoided as a share of events.
Would filter (dashboard: Would have filtered)	What a filter in shadow or proposed would have filtered. On an active filter, what it would have filtered but let through, for example while a delivery fault or an agent budget held it back. A forecast, not a saving: it is always shown apart from what was filtered, never added to it. On the dashboard, once anything filters, it appears as Waiting on approval.
Agent runs	Runs that exist in Kindo, not decisions to send. With Shape delivery, one run can carry many events.
Not seen by the agent (24 h)	Escalations the agent never received because sampled shadow did not draw them or an agent budget had no room. Shown on a filter’s Is it safe beside filtered, on a filter with a budget or a sample. Never counted as filtered, as runs avoided, or as a miss in recall.
Escalations undelivered	Decisions to escalate that, 15 minutes later, still have no run. Shown only when it is not zero, beside how many were decided. Any undelivered escalation is serious: it is an event nobody looked at.
Runs saved by grouping	For each group run Shape delivery sent, its events less one: a run carrying ten events saved nine. Absent for a filter that has never grouped.
Agent runs saved	Runs avoided plus runs saved by grouping, shown only with its two parts beneath it.

Runs saved by grouping are never added into runs avoided. A filtered event can be a miss; a grouped one reached the agent and cannot.

Live recall and the miss rate

The case against a filter is the actionable events it filtered. Sift measures this continuously from two samples of the agent’s verdicts:

the verdicts on the events the filter escalated, which show how many of those were actionable;
the verdicts on the leak-through sample, a random share of the filtered events that was sent anyway, which show how many of the filtered events were actionable.

From those, over the window:

Live recall is the estimated share of the window’s actionable events the filter sent. Higher is safer.
The miss rate is the estimated share it filtered: one minus live recall. Lower is safer.

Both are estimates, so the console shows each with a 95% range as well as a single figure. The fewer verdicts behind them, the wider the range.

Live recall reads only what the filter filtered. An escalation an agent budget or sampled shadow kept from the agent is never counted as filtered or as a miss: the filter decided to send it. Sift counts it apart, as not seen by the agent.

Recall is not yet measurable until there are at least 20 leak-through verdicts and 20 verdicts on escalations (or a verdict on every escalation, if there were fewer). The console then shows not yet measurable with the sample size and the reason, never zero and never 100%.

Recall counts only events scored since the approval the filter filters on now, or since its agent was paired, when either is inside the window. A model approved after a fall-back is judged on its own misses, not its predecessor’s. A filter with no standing approval has no recall to read.

If a person corrects an agent verdict in the verdict check, the figures use the corrected answer.

The omission rate

The omission rate is the leak-through sample’s own actionable share: of what was filtered, how much mattered. It is shown beside the miss rate but not judged. Where few events matter, it stays small however much the filter misses. On a stream where 1 event in 100 is actionable it cannot pass about 1% even if the filter filters every one of them.

What the miss rate is held to

Each filter that is filtering has its own limit, set by its approval:

with a posture, the recall the approved point measured on the holdout (the console marks it approved);
without one, the spec’s recall floor, acceptance.min_value (marked target).

The miss-rate limit is one minus that figure. A filter is over its limit when its live recall is clearly below what it was approved on, allowing for how uncertain both figures are. The drift alarm uses the same rule, so the badge and the alarm never disagree. See Shadow and filtering.

Where you see them

On a filter’s Overview, the Is it safe panel shows:

a verdict badge: a raised condition in its own words, Drift breached, Above limit, Within limits, or Not yet measurable;
Miss rate on a meter against its limit, with the omission rate and the sample size beneath;
Recall estimate with its 95% range, on a scale from 85% to 100%, against the approved or target figure;
the four drift readings, model age against the retrain schedule, and any circuit-breaker trips.

On the dashboard, The risk taken for it shows the fleet’s miss rate: each measured filter’s estimated misses over its catches and misses, summed. Under it, the filters that are filtering are counted against their own limits, for example 3 within their limits · 1 over · 1 not yet measurable. There is no fleet-wide target, because each filter’s limit is its own.

The flow diagram

The top of each filter’s Overview, and the dashboard’s window panel, draw the window’s events as a flow from left to right:

Events in (or Holdout, when the figures come from the holdout).
A column for each stage that filtered anything: Filtered there, Leaked through, or on to the next stage.
Escalated, or Would send on a filter that is not filtering.
The agent’s verdict: Actionable, Benign, Insufficient data, or a reason there was none.

An escalation that did not reach the agent is drawn as its own node, with no verdict: Waiting in the queue while an agent budget holds it, Over budget when the budget let it go unsent, and Not sent when sampled shadow did not draw it.

On a filter that is not filtering, the labels say would: Would filter events were still sent, so they go on to a verdict. Any actionable verdict on an event that was filtered, would have been, or leaked through is a miss, drawn in the danger colour. The value and the risk taken for it are one picture.

Under the flow, a filter that is filtering shows Runs avoided and Share filtered; one that is not filtering shows Actionable it would filter, for example 3 of 140.

Spend

The console shows no money figures. Sift cannot know what a run costs you, and an invented price shown as a saving is worse than none. If an operator sets an assumed cost per agent run, the API reports an estimated spend saved (runs avoided times that price). It leaves that figure out while the agent the filter replaced may still be running, because those events may still have been handled by it. See the configuration reference.

Why Sift never reports accuracy

Accuracy is never computed or shown. When one event in a thousand matters, a filter that filters everything is 99.9% accurate and useless. Recall and the miss rate ask the question that matters: of the events that needed the agent, how many reached it.

Recall is measured against the agent

Every recall figure, on the holdout and live, is measured against the agent’s verdicts, not against the truth. Agents make mistakes. An agent that calls some benign events actionable makes recall read lower than it is, never higher: its false alarms look like misses.

Two things keep this honest. The approved figure and the live one come from the same agent, so the comparison between them holds even where the absolute number is low. And a verdict a person corrects in the verdict check replaces the agent’s answer. See Verdicts, labels and training.