Compare recipes in a bake-off

A filter’s Decide stage can carry candidates: other recipes trained on the same labels and scored on the same events as the one deciding. Comparing them is the bake-off. Candidates never filter anything; they only record what they would have done. Sift may recommend switching to one, and never switches on its own. This guide shows how to read the comparison, choose the candidates, and promote one to decide.

When you’d do this

Actions shows a Sift recommends card naming a recipe to promote.
You want to know whether another recipe would serve this stream better.
You want to add, turn off or retrain candidates.

Before you start

Anyone can read the comparison. Changing the slate, training it and promoting a recipe need the operator or admin role.
The filter needs a Decide stage. A filter that only collapses repeats has no contest.
A recipe that learns from labels trains only after the gold set is approved, and each needs its own number of labels before it can train. See Verdicts, labels and training and Recipes.

Read the comparison

On a filter in shadow, Overview’s Labels toward the floor panel shows each recipe’s progress: its labels against its floor while it is short, then its recall against the spec’s target, marked ready or below target.

The full comparison is under Overview’s How it decides, in the Decide stage’s section (In full):

Column	What it means
Recipe	The recipe deciding now first, then each candidate, ranked by recall.
Recall	The share of the holdout’s actionable events the recipe would still escalate at the threshold it proposed for itself.
Would filter	The share of live events it would have filtered (Filters for the deciding recipe while the filter is filtering). Shown as evidence, never a ranking: the recipe that filters the most is one that filters everything.
Badge	Deciding, Recommended, Below target, or Not ready.

There is no accuracy column. Events that matter are rare, so a recipe that filtered every event would score near-perfect accuracy.

A candidate is not ready until it is turned on, has its labels, has a trained model, and has a proposed operating point. Too few actionable events to measure it on is the usual reason a trained candidate has no proposal; it is flagged underpowered.

How Sift recommends

Sift recommends the candidate with the highest measured recall at its own proposed threshold, and only if that beats the recall of the recipe deciding now. Recall is the comparison because it is the figure an approver is asked to accept. If no candidate beats the incumbent, there is no recommendation, and leaving the recipe in place is the result.

The recommendation appears on Actions under Sift recommends, titled Promote {recipe}, with the two recipes’ Recall and Filters bars and the margin in points. It is advisory: a recipe decides only because an operator says so, and filters only after a threshold approval.

Promote a recipe

On the recommendation card, select Promote…. On a filter that is filtering, the button reads Promote and stop filtering….

To promote a candidate Sift does not recommend, open Settings › Pipeline, and in the Decide block open Promote a candidate.
In Recipe to make the decider, keep the recommended recipe or pick another. Only candidates that can be promoted now are listed.
Write a Rationale. It is required and recorded in the audit log.
Select Make this the deciding recipe, read the consequence, and confirm with Promote to deciding recipe, or Promote and stop filtering on a filter that is filtering.

The new recipe decides from the next spec version, and the filter moves to shadow. A filter that was filtering stops filtering at once: the approved threshold was measured on the old recipe’s model and says nothing about the new one. It filters again only after a new threshold is approved; see Approve a threshold. The recipe it replaced stays on the slate as a turned-off candidate. The gold set is untouched.

Decline a recommendation

On the card, select Decline….
Write your reason in Why not, on this evidence (required).
Select Decline the recommendation.

It comes back if a different recipe is recommended, or once the holdout has grown by a tenth.

Change the slate

Open Settings › Pipeline and find the Decide block. The slate is under Compare against.
Turn a candidate on or off with its checkbox. Each shows measured or off and who added it. Turning one off keeps it on the slate as a record that it was considered.
To add one, choose a recipe under Add a candidate and select Add in shadow. Candidates you added can be removed with Remove.
Write a Rationale and select Save the slate, then confirm.

Saving writes a new spec version. It is a free change: candidates never filter, so what the filter filters does not change and its approval carries over. Nothing is trained until you train the slate.

Every candidate scores every event the filter reads and is trained at every retrain, so each one adds cost. Sift’s assistant proposes at most three; past that the editor marks the extra as your headroom. A recipe this install cannot train cannot be turned on.

Train the slate

In the Decide block, select Train the slate.
Confirm with Queue training.

Sift trains the deciding recipe and every enabled candidate on the same labels and measures them on the same events, so they are compared on equal terms. The more candidates, the longer training takes. Every model enters shadow; nothing becomes active, and what the filter filters today does not change. Each model joins the comparison as it finishes.

What can go wrong

Train the slate is disabled with Waits for the gold set to be approved. Approve the gold set first.
The deciding recipe cannot be trained on this install. The block says so in place of the button; a recipe can need something this deployment lacks, such as a decision model. See Filter recipes.
No recommendation appears. No candidate is eligible, none has a measured recall yet, or none beats the recipe deciding now.