Verdicts, labels and training
A filter learns what to filter from the agent’s own verdicts. Every run the agent finishes ends with a verdict; Sift reads it, turns it into a training label, and once there are enough labels it trains a model. This page explains where labels come from, which runs are left out, and when Sift trains and retrains.
From a run to a verdict
Section titled “From a run to a verdict”Sift collects finished runs every 60 seconds. For each run it looks for the agent’s verdict in three places, strongest first, and records which one answered:
| Route (console word) | Where Sift found the verdict |
|---|---|
| Written | In the conversation metadata, where a paired agent records it on purpose. |
| Stated | In the agent’s final message, as JSON or a disposition: line. |
| Guessed | Read from the final message’s prose. Only for agents Sift did not write, and the route most often wrong. |
The Overview’s What became of what was sent panel shows the share of verdicts that were stated rather than guessed. When most are guesses, the console warns that a better recipe will not help: tell the agent how to report its verdict instead, with Briefing for the agent.
From a verdict to a label
Section titled “From a verdict to a label”A label is one event with a yes or no attached: worth the agent’s run, or not.
| Verdict | Label | Why |
|---|---|---|
| Actionable | Positive | The event needed doing something about. |
| Benign | Negative | The event needed nothing. Only benign verdicts teach Sift what can be filtered. |
| Insufficient data | Positive | An event the agent could not clear is not one to filter. Counting it as benign would teach the model that unanswerable events are safe to filter. |
A run with no verdict is never a negative label. A run that failed, was cancelled, returned nothing, could not be parsed, gave no disposition, or is still running yields no label at all; Sift counts it by reason instead. Reading a broken run as benign would teach the model that whatever broke the agent is safe to filter, which is the worst direction to be wrong in. The same goes for a generated agent whose instructions were never filled in: it answers insufficient data to everything, and Sift recognises that and excludes those runs rather than learning from them. Five runs in a row with no readable verdict raise a condition on the filter’s Actions tab.
Where labels come from
Section titled “Where labels come from”| Source | API value | What it is |
|---|---|---|
| Escalation | escalation | The agent’s verdict on an event the filter sent it. |
| Leak-through | leak_through | The agent’s verdict on an event the filter would have filtered but sent as part of the random sample. |
| Analyst | analyst | A person’s judgement: the reviewed gold set, once it is approved, and answers in the verdict check. |
| Sample | sample | The agent’s verdict on an event sampled shadow drew, with the chance it had of being drawn. |
| Backtest | backtest | The agent’s verdict on an event of history a warm start chose before shadow, with the chance it had of being chosen. |
| Historical | historical | The past verdicts of the agent the filter replaces, imported on request. |
Leak-through labels are the only unbiased negatives. Every escalation label describes an event the filter already chose to send. Leak-through events are drawn at random from what it would filter, so their verdicts are the only honest measure of what filtering misses. Sift keeps them apart from escalation labels for that reason. See How Sift measures a filter.
Historical labels are a bridge. They are another agent’s judgement of whatever it was sent, with no filter in front of it: the weakest evidence in the set. Sift imports them only when you ask (Settings › Labels, Add its past verdicts), only from runs whose input carries a field the filter reads, and counts them only while the filter’s own labels are below the floor. You can withdraw them.
Sample labels stand for more than themselves. Each event sampled shadow drew had a known chance of being drawn, so its verdict does not depend on the filter’s choice. When Sift measures a model on them, it counts each one for the events it stands for, so recall describes the stream, not the sample. Backtest labels count the same way: each verdict stands for the stretch of history it was chosen from.
When one event has labels from several sources, Sift keeps one, in this order: analyst, leak-through, sample, escalation, backtest, historical. An event a warm start judged and the poll later delivered is one event, labelled by its later verdict. A person’s label, from the gold set or the verdict check, always outranks the agent’s verdict on the same event.
Settings › Labels counts the labels by where they came from: Reviewed in the gold set, From its own agent, Judged in a warm start (once there are any), and The agent’s past verdicts.
When Sift trains
Section titled “When Sift trains”A filter is trained once two things hold:
- It has reached its label floor. The floor is the higher of the filter’s Min labels to train (400 by default) and the recipe’s own minimum. See Recipes.
- The holdout, the newest 20% of labels (or 30%), holds at least 20 actionable events, and the earlier labels hold some too. A model measured on fewer positives has no meaningful recall behind it, and one trained on none has nothing to learn from. Until then, the Setup step reads “Collect 20 actionable events”.
Sift starts the first training as soon as both hold. A warm start is the one way to train before shadow: when its labels reach the floor, Sift starts a training at once, while the filter is still being authored. You can also start one yourself from the Actions tab: Train a model on the Setup step, or Train now where Sift is still waiting to train. Recipes that learn nothing from labels (Collapse repeats, Fixed rule, Anomaly detection) still wait for the floor, but not for positives. Training needs an approved gold set for any recipe that learns from labels.
The holdout. Sift keeps the newest 20% of labels out of training and measures the model on them; Holdout in Settings › General can make it the newest 30%. It is always the newest labels, so the figures describe how the model does on traffic it has not seen.
What training produces. Every new model starts in shadow, where it decides nothing. If it can propose an operating point, a filter in shadow moves to proposed on its own and waits for a person’s threshold approval. If the holdout held too few actionable events to measure recall, the model is marked underpowered and proposes nothing; Sift trains again within a day once enough are in.
Retraining
Section titled “Retraining”Each filter has a Retrain schedule (Settings › General › Monitoring): Daily, Weekly (the default), Monthly or Never. Sift checks once a day which filters are due. With Never, nothing trains automatically, the first training included; start each training from the Actions tab.
A retrained model enters shadow beside the running one and never replaces it on its own. On a filter that is filtering, it takes over only when a person approves its operating point, and the approval screen says whether it was shown to beat the model it would replace. A running model that goes twice its schedule without a retrain (two weeks under Never) is flagged on the Actions tab, with Retrain it.
Candidates and the bake-off
Section titled “Candidates and the bake-off”Decide can carry candidates: other recipes trained on the same labels and scored on the same events as the deciding one. A candidate never filters anything; it is only measured. You manage the slate in Settings › Pipeline, in the Decide block.
The bake-off compares them by measured recall, each at its own proposed operating point. When a candidate beats the deciding recipe, the Actions tab shows a recommendation under Sift recommends, marked Advisory. Sift never switches recipe on its own. You can Promote… it, which, on a filter that is proposed or filtering, returns the filter to shadow until the new recipe’s operating point is approved; or Decline… it with a reason. A declined recommendation returns only when a different recipe is recommended, or the holdout it was measured on grows by a tenth. See Compare recipes in a bake-off.
How good the labels are
Section titled “How good the labels are”Every measured figure counts the agent’s verdicts as the truth, except where a person corrected one in the verdict check. An agent’s mistakes do not cancel out. Where it over-calls (says actionable when the event was not), the recall Sift reports reads lower, and the precision higher, than the model really achieves. Mistakes the other way barely move either figure. The threshold approval screen says so under Measured against the agent’s verdicts. Once a filter is filtering, the leak-through sample gives the steadier reading of what it misses.
If a decision model is configured, the verdict check puts the verdicts it most strongly disputes in front of a person, and the answers become analyst labels.