Answer the verdict check

The verdict check puts in front of a person the agent’s verdicts that the decision model strongly disagrees with. You say whether the agent was right, and your answer becomes that event’s training label. It catches verdicts that would otherwise teach the filter the wrong thing. This guide covers switching it on and answering the queue.

When you’d do this

A filter shows a Verdict check tab, or Actions shows Check the verdicts (raised once 10 or more items are waiting).
You want to switch the check on for a filter the decision model reads well.

Before you start

The verdict check needs the decision model, which an admin turns on; it is off by default. See Decision model settings.
It is offered only on a filter the decision model reads it well, and is off for each filter until someone switches it on.
Any role, a reviewer included, can answer items. Switching the check on or off, and resuming it, need the operator or admin role.

Switch it on

Open the filter’s Settings › Labels and find the Verdict check panel.
Check the Decision model: line. It shows how well the model reads this filter: reads it well, reads it partly or does not read it, or not enough verdicts yet. Select Measure now to read it again.

If the check is not offered, the panel says Not offered: and why.
Select Switch it on, read the consequence, and confirm with Switch on.

From then on, Sift asks the decision model about each new verdict the agent returns on an escalation or a leak-through event, up to 200 a day per filter by default (the operator can change it; verdicts past the cap that day are not checked). It puts in the queue those the model disputes most strongly. An insufficient data verdict is not checked. The filter gains a Verdict check tab, beside Gold set, with a count of open items.

The panel shows To check, Overturned (the share of answered items where the agent was wrong) and Asked today against the daily cap.

Answer the queue

Open the filter’s Verdict check tab. The filter’s intent is pinned at the top: Each verdict is checked against this policy.
Read the card, Verdict n of N: the event, what the agent said (Needs a person or Can be closed) with its reasoning, and what the decision model reads it as.

Choose your answer:

Answer	Key	What it writes
The agent was right	R	A training label with the agent’s verdict.
The agent was wrong	W	A training label with the opposite verdict.
Can’t tell	C	No label (and takes back one an earlier answer wrote).

The page moves to the next unchecked item.

Move between items with ← Previous (K), Next → (J) and Next unchecked (U), or the strip of cells. Press ? for every key.

Your answer is final as given: there is no approval step, unlike the gold set. Your label is a person’s label, so it takes precedence over the agent’s verdict on that event when the filter trains. Each card records who answered and when.

Item states read Not yet checked, Agent right, Agent wrong, Can’t tell or Expired. An item expires when the event’s payload has been purged, so there is nothing left to judge.

When it pauses itself

The check pauses itself once 30 items have been answered since it was switched on or last resumed, with fewer than 10% overturned: at that rate it is costing people time without changing many labels. The panel and tab show Paused with the reason.

To resume it, an operator or admin selects Resume in Settings › Labels and confirms with Resume the check. The stop rule counts answers from then on.

If the decision model stops answering, the check also reads Paused. Nothing new is asked, but open items stay answerable, because answering asks no model. Verdicts that arrive meanwhile are checked when it answers again. It reads Waiting while it waits for a fresh reading of the filter.

Switch it off

Select Switch it off in Settings › Labels and confirm with Switch off. No new verdicts are checked and the tab goes. Answers already given stay as training labels.