Use Collapse, Always escalate and Shape delivery

Besides Decide, a filter’s pipeline has three optional stages. Collapse repeats folds near-identical events into one agent run. Always escalate sends on events that match your rules, whatever Decide would do. Shape delivery groups escalations into fewer agent runs. This guide shows when to use each, how to set it up in Settings › Pipeline, and what approval each change needs.

Before you start

• You need the operator or admin role to change a stage, and to approve one.
• Each stage is a block in Settings › Pipeline, in the order an event meets them. Every block has a Stage setting, a Review change button that shows a badge (Free, Needs approval, Waits for approval, Cannot change or No change), a required Why, and a save. See Change a filter’s settings for how saving works.
• You can also set these stages up when you create a filter, on the review step, where their evidence comes from the sample Sift’s assistant read (below).

Stage	Settings	Filters?
Collapse repeats	Off, Measuring, Filtering	Yes: the repeats it folds.
Always escalate	Off, Watching, Escalating	Never. It only adds escalations.
Shape delivery	Off, Measuring, Grouping	Never. Every escalation still reaches the agent.

A measuring or watching stage records what it would do and changes nothing the agent receives. The last stage that can filter cannot be turned off.

Collapse repeats

Use it when a source sends bursts of near-identical events (the same alert from many hosts, a ticket re-filed) and the agent needs to see one of them. Collapse folds an event into a near-identical earlier one; the one that escalates tells the agent how many events it stands for. The repeats are filtered.

Set it up

1. In the Collapse repeats block, set Stage to Measuring.
2. Pick Fields compared: the event fields whose text decides whether two events repeat.
3. Set Similarity (0.5 to 1, default 0.92). Higher folds only near-identical events.
4. Set Window (1 minute to 1 day, default 60 minutes): how far back an event looks for one it repeats. The track marks the filter’s poll interval; a window much longer than it rarely changes anything.
5. Set Max cluster size (2 to 10,000, default 500): the most events one may stand for, a cap on the damage a false merge can do.
6. Review the change and save. Adding Collapse in measuring is free.

Read its evidence

Under the similarity slider, and in the stage’s section on Overview, the compression curve shows how much each similarity setting would fold of the filter’s recent events, at the window set. Settings that would make a runaway cluster are hatched runaway. Until the stage has been measured, select Measure now (or wait for the next training run).

On the review step of a new filter, the curve is the sample sweep: the same measure over the sample Sift’s assistant read, labelled Sample folded. When it reads up to, the sample carried too few times for an exact figure, and the true fold is at most that. If it does not finish, select Measure again.

The section on Overview shows Folds away, Largest cluster, Purity and Repeats agreeing. Purity is the share of clusters whose repeats the agent judges the same way as the one that escalated, allowing for the agent’s own inconsistency. It needs at least ten clusters with a verdict on both. Show examples lists the largest collapses.

Switch it on

• Beside a Decide stage: set Stage to Filtering and save. This is gated. On an active filter it waits as a version for approval while the filter keeps filtering; on any other filter it filters once a threshold is approved.
• On its own (a filter with no Decide stage, or Decide still learning): Collapse is approved on its own evidence. The setup steps are Measure collapsing, Collect 30 cluster verdicts and Approve collapsing repeats. Once 30 clusters (by default; the operator can change it) have a verdict on the first event and on a repeat, the threshold screen offers it; the approval confirms with Approve — begin filtering repeats.

Loosening or tightening Collapse while it filters is gated.

Warnings

Condition	What it means	What to do
Collapse runaway	One cluster holds more than half of the last day’s events (General › Runaway share, default 50%): almost certainly distinct events merged.	Retune or switch it to shadow: raise Similarity, or set the stage to Measuring.
Compression shift	Collapse folded a share of the last day’s events that differs from the week before by 15 points or more (General › Compression shift).	Check whether the source changed, then retune.

Always escalate

Use it for events that must reach the agent whatever their score: a field value that always matters, or a question a person could answer yes or no. It never filters.

Field rules

Field rules need no decision model.

1. In the Always escalate block, set Stage to Watching.
2. Under Field rules, select Add a field rule and name it.
3. Set Match to All conditions or Any condition, and add conditions with Add a condition.
4. Review the change and save.

A field rule escalates every event it matches, whatever the other stages decide.

Plain-language rules

Plain-language rules need the decision model, and may escalate only where it reads the filter at least partly. Otherwise Escalating is unavailable, with the reason.

1. Under Plain-language rules, select Add a plain-language rule.
2. Give it a Name and a Question: one sentence a person could answer yes or no about an event.
3. Pick its Reach: Narrow (0.1%), Normal (0.5%) or Wide (2%) of filtered events. Reach is a ceiling, not a target: a rule with little to find sends little.
4. Select Try it. It asks the rule of about 200 recent filtered events and shows the extra escalations it would add, with examples. It works once the filter runs.
5. Set Max rescues per hour (1 to 200, default 20). Past it, events the rules fire on are recorded and not sent.
6. Review the change and save.

A plain-language rule is asked of events Decide filtered, and sends one on (a rescue) when the model answers yes. On Overview, the stage’s section shows Would rescue while watching, or Rescued, and each rule’s verdict, such as Earning its keep or A looser threshold would do better. Show examples opens Activity narrowed to that rule’s rescues.

Switch it on and change it

Set Stage to Escalating and save. Anything that can only escalate more is free: adding the stage or a rule, switching to Escalating, widening a reach, raising the hourly cap. Anything that can escalate less is gated: removing the stage or a rule, switching back to Watching, narrowing a reach, editing a question or a field rule, lowering the cap.

If the decision model stops answering, plain-language rules pause. After 15 minutes, on an active filter, Actions raises the fault Always-escalate rules are paused: … with See the paused rules. Events they would have rescued stay filtered; field rules keep running.

Shape delivery

Use it when the agent can judge several events in one run and you want fewer runs. Its recipe is Group escalations. It never filters: every escalation still reaches the agent, each with its own verdict, and some wait up to the preset’s stated delay.

Before it can group

Grouping needs a paired agent that takes groups: an agent Sift generated at contract 4 or later. Until then Grouping shows Unavailable: the paired agent takes single events., with Regenerate the agent. Regenerating creates a new agent in Kindo; steps you added to the old one in Kindo are not carried over. See Connect a filter to its agent.

The setup steps are Pair an agent that takes groups, Measure what grouping saves and Switch grouping on.

Set it up

1. In the Shape delivery block, set Stage to Measuring and save. This is free. Every escalation is still sent singly, and Sift measures on the last 7 days’ escalations what each preset would save. It needs 200 escalations in that window.

2. Pick a preset under Grouping:

   Preset	What it does
   No added delay	Groups what one poll brings; at most 25 events a run.
   Storms only	Sends singly until 30 runs an hour, then waits up to 5 minutes.
   Short wait	Waits up to 5 minutes; at most 25 events a run.
   Not urgent	Waits up to 15 minutes; at most 10 events a run.
   Custom	Your own Wait, Runs per hour before grouping, Budget counts and Max events per run.

   Once measured, each preset’s note shows its saving, such as 58 in 100 runs saved. Under the presets, Runs saved, Added delay, p95 and Largest group show what the chosen setting would do.

3. Under Advanced settings, set Group by (event fields whose values decide which events share a run; empty puts all escalations in one stream), Max characters per run (default 32,000) and Split rate (default 2%).

On the review step of a new filter, the figures are measured on the sample Sift’s assistant read, as if every event in it escalated (Sampled events).

Switch grouping on

1. Set Stage to Grouping, review and save. This is gated: grouping changes what the agent sees and can change its verdicts.
2. The change waits as a version for approval, in any state, while the running version goes on as it is. Actions shows Review grouping.
3. On the threshold screen, now titled Grouping approval, review what is approved: runs saved, added delay and the effect on verdicts. Grouping is approved on what it would save, never on a threshold. See Approve a threshold.

Grouping less later is free; grouping more is gated.

Effect on verdicts

The Split rate sends a small random share of events that would be grouped singly instead, so Sift can compare the two. Effect on verdicts shows how often the agent finds grouped events actionable against single ones, over 14 days. If grouped events are clearly found actionable less often (by more than 3 points), Sift raises the fault Grouping hides actionable events, with Group less, or switch it back to measuring. The comparison is not measurable until the single events have 20 actionable verdicts.

Two other conditions: Held groups, when a group is past its due time and nothing has released it (a fault; See held groups on Actions), and a warning when more than a quarter of group runs are sent at the character cap.

With On alarm set to Stop filtering, either fault makes Sift switch the stage back to measuring itself, which sends every escalation singly at once. With Keep filtering, the fault stays open until it clears.

Related

• The pipeline: slots and stages
• Recipes
• Filter settings reference