Shadow and filtering

A filter filters events only while it is active, and it reaches active only through a person’s threshold approval. Every other state sends the agent every event. This page explains how that switch works, what keeps measuring a filter once it filters, and how Sift takes a filter out of service when the measurements go wrong.

The state is the switch

Sift asks one question before it filters anything: is this filter active, and can its operating point filter something? If not, every event goes to the agent. The console asks the same question, so the badge and the behaviour cannot disagree.

In shadow and proposed, a filter does everything an active one does except filter. It scores every event, and records what it would have done. That record is what the threshold screen and the filter’s Overview show as would filter, and it is how you see what a filter will do before you let it. Every new filter and every retrained model runs in shadow first.

A filter’s operating point has its own alarm settings, and each stage has a mode, but none of them overrides the state: a filter that is not active filters nothing.

The first poll reads history

A filter’s first poll has no place in the stream to start from, so it reads the backlog the source already holds: up to 20 pages and 1,000 events. Those events are history, not the live stream, and every one goes to the agent in shadow. An agent Sift generated at contract 5 or later is told so (see History): it judges them as if they were live and takes no action outside the run. Any other agent is sent them as live events. Every poll after the first reads only what is new.

After a warm start, the first poll starts where the warm start’s read ended, so its backlog is small or empty. An event shadow reads that the agent already judged, unchanged, in the warm start is not sent again, and the warm start’s verdict is its label. It is not filtered and not not seen by the agent, since the agent saw it; the flow draws it as Judged in a warm start, on to that verdict. An active filter decides as usual.

Sampled shadow

On a stream too big for the agent to take whole, a filter in shadow or proposed can send the agent a sample instead of every event. You set it in Settings › Delivery › In shadow, under In shadow, send the agent: Send everything (the default), A tenth, One in twenty, One in fifty, or Custom, which opens the four settings below. The new-filter flow suggests it for a big stream (see Create a filter). In the spec it is operating_point.shadow_sample (see Filter settings). Off means every event reaches the agent in shadow. Where the stream’s size and its share of positives are known, each preset says about how many positives a day it would bring labelled.

share is the expected share of events sent, from 0.5% to 50%.
guided (on by default) sends the events the filter’s newest model scores higher more often. floor (0.3 by default) keeps every event’s chance at least floor × share. With no model yet, or with guided off, every event has the same chance.
min_positives (50 by default, at least 20): until the sample holds this many events the agent called actionable, training proposes no threshold.

An event that is not drawn is never counted as filtered: the filter decided nothing about it. Recall, the flow and the cost figures show it as not seen by the agent.

Sift allows for the sampling when it trains and measures, so the recall and the share filtered describe the whole stream, not the sample. The threshold screen shows the recall with its 90% interval (“97.6%, 90% interval 93–100%”) and what it was measured on (“Measured on 2,483 judged of 105,958 events”). When the interval’s lower end is below the filter’s recall bar, approving asks you to tick that you accept the interval, and the tick is recorded with the approval. A sample that caught every positive it judged still has an interval: with 34 positives judged, all caught, it reads about 92–100%, so a 99% bar asks for the tick.

Approval ends sampling: an active filter filters by its threshold, with leak-through, whatever shadow_sample says. The setting stays on the version, so a fall-back to shadow samples again at the same share. Changing it never needs an approval.

Not seen by the agent

An event the filter decided to send, and the agent never saw, is not seen by the agent. It is never counted as filtered, because the filter filtered nothing: sampled shadow did not draw it, or an agent budget had no room for it. The console keeps it apart everywhere:

the Overview’s flow draws it as its own path, by what kept it back: Not sent (the sample did not draw it), Over budget (the budget let it go unsent), or Waiting in the queue while a budget holds it;
Is it safe shows Not seen by the agent (24 h) beside Filtered (24 h) on a filter with a budget or a sample;
Filtered events lists only what the filter filtered.

The agent budget

An agent budget is the most agent runs a filter starts in a rolling hour, in every state: shadow, proposed and active, re-sends included. A run of one event and a grouped run each count as one. It is off by default, and off is exactly the filter without one. You set it in Settings › Delivery › Agent load; the new-filter flow suggests one when the stream is bigger than the agent can take.

What the budget has no room for is never filtered. It goes by the budget’s overflow:

Preset	What happens to the overflow
Hold in a queue (default)	It waits, and is sent when there is room, the likeliest incidents first: rescues, then the measuring sample, then by score. It leaves unsent only if it waits past the longest wait (a day by default) or a full queue pushes it out.
Group the overflow	It goes in one grouped run, as Shape delivery sends. Needs whole events to an agent Sift generated at contract 4 or later.
Record only	It is recorded and never sent. Use it when a late answer is useless.

Custom opens every setting: the overflow, the queue’s depth and longest wait, the pace and the warning share (see Filter settings). A held escalation neither trips the delivery check nor is sent twice. The Overview’s Agent load panel shows the budget and its queue filling, and Sift alerts when it binds, fills or loses events.

Under a budget, a fall-back to shadow sends at most the budget, and the rest waits. That is what makes Stop filtering safe on a big stream.

What keeps measuring an active filter

The leak-through sample

An active filter sends a small random share of the events it would filter to the agent anyway: 3% by default, set between 1% and 5% under Settings › Delivery › Leak-through (%). It cannot be set to zero.

These events are the only unbiased view of what filtering misses. The agent cannot tell them from any other escalation, so its verdict on them does not depend on the filter’s decision. Every actionable verdict among them is a measured miss, and live recall is computed from them.

Which events leak is random to anyone outside your deployment, and recorded, so an audit can see which events leaked. Each filter draws independently.

Leak-through events count as escalated, never as runs avoided. Raising the rate needs no approval; lowering it filters more, so it needs a new threshold approval.

The trigger ceiling

The ceiling is the most escalations an active filter may send in one clock hour: 200 by default, set between 1 and 100,000 under Settings › Delivery › Max escalations per hour. A group run from Shape delivery counts one escalation per event it carries. The count runs only while the filter filters, and starts again each time the filter enters active.

A filter that fires far more than usual is not working hard; its model or threshold no longer describes the traffic. Reaching the ceiling counts as an alarm.

On a big stream, falling back defeats the ceiling: shadow sends every event. Where On alarm is Stop filtering, no budget is set and the stream is at least four times the ceiling, the On alarm setting and the new-filter flow say so in one line: “If the ceiling trips, this filter falls back to sending about N an hour, N× what tripped it.” It decides nothing; an agent budget or the safer point is the answer.

Drift signals

Once an hour Sift checks every filter in shadow, proposed or active against four drift signals. Each looks at how the deciding model scores the traffic, not at what the filter did, so a move to shadow or a stalled delivery does not look like drift.

Signal	Console field	Default alarm	What it compares
Firing rate	Escalation change (%)	50% relative change	The share of events the operating point escalates over the last 7 days against the 7 before. Needs 50 events in each.
Missed actionable events	(set by the approval)	Below the approval’s line	Live recall over the last 14 days against what the approval was held to. Needs 20 leak-through verdicts.
Score shift	Score-shift alarm	0.15	How far the deciding model’s average score moved week on week, relative to how spread out the earlier week’s scores were.
Feature PSI	Feature drift alarm	0.25	How much the values of the fields the recipe reads changed week on week (0.25 is a large change). Recipes that read structured fields only; needs 500 events.

You set the levels under Settings › General › Monitoring. A signal without enough data reads as not measured, never as fine.

On alarm

Each filter says what it does when a drift signal breaches or it reaches its ceiling, under Settings › General › On alarm:

Option	Setting	What happens
Stop filtering (default)	`fall_back`	The filter moves to shadow and its approval is withdrawn. Everything is escalated until a person resumes it or approves a retrain.
Keep filtering	`keep_filtering`	The filter keeps filtering at its approved point and raises an alert, a fault on its Actions tab, until the alarm clears.
Filter at a safer point	`loosen`	On a drift alarm, the filter moves to a second, looser point approved with its threshold, and keeps its approval. A second drift alarm there moves it to shadow. On the ceiling it keeps filtering and raises an alert, as Keep filtering does.

Switching to Keep filtering keeps the filter filtering through an alarm, so it needs a threshold approval, or a written reason that re-affirms the standing one. So does switching from Stop filtering to Filter at a safer point. A switch toward Stop filtering needs nothing.

The safer point

Filter at a safer point is for a stream the agent cannot take whole, where moving to shadow would send it everything. The safer point is chosen when a threshold is approved: a point on the same trade-off curve with a lower threshold and at least the same recall, such as the one catching 99.5% of what matters. A threshold that already catches everything it was measured on can still have one: a lower threshold sends more, so it misses no more where the measurement was optimistic. One approval covers both points.

On a drift alarm Sift writes a new version itself, at the safer point, and the filter stays active. The breach that moved it does not move it again. A breach of another signal, or of the same one after it has read normal, moves it to shadow: nothing safer is left. To go back, use Return to the approved point on the filter’s Actions tab, with a reason. It acknowledges the signals breached then, as a resume does.

Falling back

A fall-back is Sift moving an active filter to shadow on its own. There are three causes:

Cause	When	Respects On alarm
Drift	The hourly check finds a signal past its alarm level.	Yes. Under Filter at a safer point, only at the safer point.
Trigger ceiling	The hour’s escalations reach the ceiling.	Yes. Filter at a safer point keeps filtering.
Missing approval	The version being scored has no standing threshold approval.	No: always falls back

A fall-back withdraws the approval the filter was running on. Sift records the withdrawal with its cause, the model and threshold that were running, and the readings that tripped it, in the audit log as the system’s act. The approval stays withdrawn so that nothing, not even a pause and resume, can put the filter back into service on evidence a measurement has contradicted. The filter’s Actions tab says why it fell back and what you can do.

When you move a filter to shadow yourself, nothing is withdrawn.

A stalled delivery is not a fall-back

If escalations keep failing to reach the agent, Sift stops filtering and escalates everything, without changing the filter’s state. Filtering is safe only while the events the filter sends arrive. As soon as an escalation lands again, filtering resumes at the approved point. The audit log records both edges (filter.delivery_stalled, filter.delivery_restored).

Stopping a filter yourself

The filter header’s Stop or hold menu has two moves. Neither needs an approval or a reason, because nothing should stand between you and turning filtering off.

Move	State	What the agent gets	What carries on
Stop filtering	active to shadow	Every event, unfiltered	Scoring and labels. The approval stands, so Resume puts the filter back on the same model.
Hold events	to paused	Nothing from Sift	Nothing: Sift stops reading the source. On resuming it reads from where it stopped, up to 1,000 events a poll, if the source can be asked for events since a point.
Retire (Settings › Retire)	to retired	Nothing, for good	Nothing. History, approvals and the filtered events are kept.

Choose Stop filtering when a filter is filtering what it should not. Choose Hold events when the agent should get nothing at all for a while.

Resuming

Resume puts a filter in shadow, proposed or paused back into active on the model that was running, at the operating point already approved. It asks for a written reason (Why (required)) and records it as a re-affirmation of the threshold approval. It approves nothing new.

After a fall-back, the Actions tab offers Resume on the current model and lists each reading you are overriding against its alarm level. The drift signals breached at that moment are acknowledged so the next hourly check does not take the filter straight out again. An acknowledgement ends at the first of:

7 days;
the signal reading within bounds;
a new model;
a change that needs a new approval;
another fall-back;
a later resume.

A breach of any other signal still falls back. The trigger ceiling is never acknowledged.

Resume is refused, with the reason, when there is no approval to re-affirm: after a fall-back for a missing approval, after a save that needed a new approval, after a person rejected the threshold, or when the model that was approved is no longer the one running. In those cases the threshold screen offers Approve the running model, a fresh approval of the model already serving, with no retrain. See Operating point and approval.