Put Sift in front of a big stream

A new filter starts in shadow, where it sends the agent every event until you approve a threshold. On a stream bigger than the agent can take, that is where the cost is. This guide walks the Agent load section of Review and create: what its figures mean, and when to tick each protection it suggests.

Sift suggests; you decide. Every suggestion starts unticked, marked Recommended. Nothing is switched on unless you tick it, and a filter with none of these settings behaves exactly as one created without the section.

When you’d do this

The source is busy: thousands of events an hour or more.
Each agent run costs enough that a day of shadow on the whole stream matters.
The agent, or the deployment’s Kindo rate limit, cannot keep up with every event.

Before you start

Build the filter as in Create a filter, up to Review and create. You need the operator or admin role. Agent load sits between What it will do and Posture. It reads the draft as you edit it, so the figures follow your changes.

Read the figures

Figure	What it says
Events an hour	The stream’s size, as a range. Under it, what it was read from: measured (at least 24 hours of polls in the last 7 days), as you said, from a read of the source, from the agent’s past runs (when you started from an agent), or not known with the reason.
Agent runs an hour in shadow	What shadow would start, as drafted. Under it, what bounds it: every event, until a threshold is approved, the sample’s share, or held to the budget.
Cost a day	Those runs priced. Shown only when the deployment sets a cost per agent run.
The meter	Shadow’s runs against what the agent can take, with the Kindo limit marked. It reads over capacity when the low end is over it.
The agent can take	The agent’s capacity, in runs an hour. Empty uses the deployment’s default.

The capacity is what Sift sizes its suggestions from. It is the number you type here, or else the deployment’s default: one the operator sets, or half of what the deployment’s Kindo rate limit allows in an hour (9,000 runs at the default limit), because Sift’s own Kindo calls share that limit. The capacity limits nothing by itself: only an agent budget does. Operators set the defaults; see the Configuration reference.

When nothing has read the stream, the section asks About how many events an hour?: Not sure (the default), Under 1,000, 1,000–10,000, 10,000–100,000, Over 100,000, or Custom with a number. Not sure suggests nothing, and says so.

What Sift suggests, and when

The stream	Sift suggests
Its low end is within what the agent can take	Nothing. A line says the agent can take it whole in shadow.
Its low end is over the agent’s capacity	An Agent budget at the capacity, holding the rest in a queue; and On alarm, a safer point.
Its low end is over four times the capacity	Both of the above, and Sampled shadow at the preset nearest capacity ÷ volume.
The above, with a decision model that answers or a built gold set	Warm start as well.
Its high end is over the Kindo limit, or has no high end	A line, not a suggestion: the deployment’s Kindo rate limit, not the agent, is the bound. See The Kindo limit.

Ticking a row writes its preset into the draft and opens its settings under it. Unticking puts the draft back as it was. Whatever you tick is saved with the filter when you select Create filter, and can be changed later in Settings › Delivery and General.

The agent budget

Tick it when the agent cannot take the whole stream, in shadow or after an alarm. A budget is the most agent runs the filter starts in a rolling hour, in every state, re-sends included. A grouped run counts as one.

Tick Agent budget. It starts at the agent’s capacity with Hold in a queue.

Pick a preset under Agent budget:

Preset	What happens to what the budget has no room for
Hold in a queue (default)	It waits in a queue and is sent when there is room, the most important first. Nothing is lost unless it waits a day, or a full queue pushes it out.
Group the overflow	It goes as one grouped run. Shown disabled, with why, unless the filter sends whole events to an agent Sift generated at contract 4 or later.
Record only	It is recorded and never sent. Use it when a late answer is useless.
Custom	Opens Overflow, Queue depth (24 hours of the budget by default, at most 1,000,000), Longest wait (a day by default, 15 minutes to 7 days), Pace and Warn at.

Set Runs an hour. Its track marks the projected shadow rate, the capacity and the Kindo limit (and, on a filter that has run, its busiest hour).

What the budget keeps back is never counted as filtered: it is not seen by the agent. Once the filter runs, the Overview’s Agent load panel shows the runs this hour against the budget, the queue’s depth and oldest wait against their limits, and what was sent late, aged out, pushed out or recorded.

Under a budget, Start in shadow says “At most N agent runs an hour; the rest wait in a queue”. Without one, on a stream over the capacity, it suggests a budget, links to set one, and its button reads Start sending everything. It never blocks.

On an active filter, setting a budget, lowering it, shortening its queue or moving its overflow away from Hold in a queue is gated; see Change a filter’s settings.

On alarm, a safer point

Tick it when a drift alarm on an active filter should not send the agent the whole stream. By default an alarm moves the filter to shadow, which on a big stream is the most expensive place it can go.

Ticking it sets On alarm to Filter at a safer point. It does not pick the point: there is nothing to choose from until a model is trained. You choose it when you approve a threshold, on the threshold screen, under On an alarm: 99.5% recall (the default), 99% recall, or Custom. Each shows what it filters and its runs an hour. One approval covers both points. See Approve a threshold. It needs a Decide recipe operated by a score threshold.

Once it is set:

A drift alarm moves the filter to the safer point. It stays active, with its approval, and filters less. A second drift alarm there moves it to shadow.
The trigger ceiling does not loosen it: on the ceiling the filter keeps filtering and raises an alert. A budget, if set, is what bounds it.
A missing approval, a gated save and a stalled delivery still stop it filtering, as they always do. A budget bounds what that sends.
To filter at the approved point again, use Return to the approved point on Actions, with a reason. See After Sift falls a filter back.

Sampled shadow

Tick it when even a budget would leave most of the stream waiting: the stream is over four times what the agent can take. In shadow and proposed, the filter sends the agent a sample instead of every event, and trains and measures on it.

Preset	Share sent
Send everything (default)	Every event. Fastest to approve.
A tenth	10%. About a tenth of the runs; approval in about twice the time.
One in twenty	5%.
One in fifty	2%. For very big streams; expect a wide recall interval at first.
Custom	0.5% to 50%. Opens Share sent, Draw (Guided or Uniform), Floor and Positives before a proposal.

Where the stream’s size and a gold set’s share of positives are known, each preset says about how many positives a day it would bring labelled.

What a smaller sample costs is certainty, and a slower approval:

Training declines to propose a threshold until the sample holds Positives before a proposal (50 by default).
The threshold screen shows recall with its 90% interval, such as “97.6%, 90% interval 93–100%”. When the interval’s low end is below the recall the filter must meet, approving asks you to accept the interval, and that is recorded with the approval.
Approval ends sampling: an active filter filters by its threshold, with leak-through. A fall-back to shadow samples again at the same share.

An event the sample did not draw is never counted as filtered. Changing the sample never needs an approval. See Sampled shadow.

Warm start

Warm start has no box. It is an action that spends agent runs, and a filter being created has no approved gold set or paired agent yet. Its row says where it is offered: on the filter’s Actions, beside Start in shadow, once the gold set is approved.

There you choose None (the default), 200 events, 500 events, or Custom (50 to 5,000). The agent judges that many past events, and once there are enough labels Sift trains a first model, which is in use from the first shadow poll. Before you start one:

It runs once for a gold set. A new approved gold set allows another.
The agent is told these are history if Sift generated it at contract 5 or later, and takes no action on them outside the run. Any other agent acts on them as on live events: one that opens a ticket for what it judges actionable opens one for these too, and the confirmation asks you to accept that.
Shadow does not send them again. An event the agent already judged, unchanged, is not sent a second time.
Its runs count against the budget, and wait for room rather than queue.

See Warm start.

The Kindo limit

Every filter on the deployment, and Sift’s own Kindo calls, share one Kindo rate limit, set by the operator. When a stream could pass it, the section says that limit, not the agent, is the bound. A filter’s budget cannot see it: runs over it are not lost, they wait their turn, for every filter at once.

When agent runs wait more than 5 minutes on that limit, the deployment raises the warning Agent runs backlogged. Set budgets on the busiest filters, so their overflow waits in their own queue, most important first. Ask the operator to raise the limit only if your Kindo tenant’s quota allows.

When a protection binds

Sift raises a condition on the filter’s Actions, and an alert, when a protection works hard or falls short. The load conditions are also the badge of the Overview’s Agent load panel. Each one below is an alert of its own, even when another condition leads the filter’s health:

Condition	What happened
At budget, Queue filling, Recording overflow	The budget is spent and escalations wait, the queue is near its limits, or overflow was recorded and not sent.
Losing events (a fault)	Escalations aged out or were pushed out of a full queue, unseen by the agent.
Safer point (a fault; At its safer point under Is it safe)	Sift moved the filter to its safer point on a drift alarm.
Over capacity	A filter in shadow or proposed with no budget started more runs than its agent can take for two hours. It needs none of these settings to be raised.

Near budget also shows, but is alerted only when it leads the filter’s health. Agent runs backlogged is a deployment warning, alerted like the others. States, badges and health codes lists each with its code. To send these to a webhook, see Notifications and webhooks.