Review the gold set

The gold set is about fifty events from the filter’s live source, each with a proposed label. You judge every one against the filter’s intent, then an operator or admin approves the set. Approval is the first of the two approvals a filter needs, and it allows the filter to start in shadow.

When you do this

When you create a filter, the console starts building its gold set straight away; a build usually takes a few minutes. Sift does not build one on its own otherwise: if the build was not started, or failed, an operator builds it from the filter’s Gold set tab. The Actions tab shows Review the gold set as a step, and on the dashboard, Needs attention lists waiting gold sets under For reviewers.

Before you start

Task	Who
Build or rebuild the set	Operator or admin, while the filter is in draft or authoring
Judge examples and write notes	Anyone signed in, reviewers included
Approve the set	Operator or admin. The form is shown to reviewers, but the server refuses their approval.

Judge the examples

1. Open the filter and select the Gold set tab.

   If there is no set yet, This filter needs a gold set offers operators and admins Build the gold set; a reviewer sees “An operator or admin can build it.” While it builds, the page says it is reading events and proposing a label for each, and updates on its own.

2. Read the sentence under You are labelling against this sentence.

   This is the intent the set was built for. Every judgement is about whether an event matches it, not whether the event looks important in general.

3. Read the example: the event, the Proposed label (Escalate, Filter, or No proposal when none could be made), and the reasoning behind the proposal.

4. Under Your judgement, select Escalate (key E) if the agent should see this event, or Filter (key W) if it should not.

   You can instead press A to agree with the proposal or D to disagree; these do nothing on an example with no proposal. Each of the four records your judgement and moves to the next example you have not judged. The card is then badged You agreed or You disagreed.

5. Optionally, write a Note (key N) on why an example was not obvious.

   The note is saved with the judgement. To change the note on an example you already judged, edit it and select Save note. Shortcuts do nothing while a text field has focus; press Esc to leave it.

6. Repeat until every example is judged.

   Move with ← Previous (K), Next → (J), Next unreviewed (U) or the numbered strip of examples. All keys (?) lists every shortcut. The line under the progress bar says how many are reviewed and how many are left.

The disagreement rate

Disagreement rate is how often your judgements went against the proposal. A high rate means the intent and the assistant’s reading of it have diverged, and training on the set would encode that. At 20% or more the page says “The proposed reading has drifted from the intent. Re-wording the intent and building another set usually beats pushing through.” The rate is kept with the approval.

When a decision model ranked the pool

If a decision model is on, the build reads a wider stretch of the source and has the model pick out the events most likely to match the intent. Half of the set comes from those, and half from across the stretch. The page then shows Ranked half and Spread half, with how many likely matches each found. More in the ranked half means the model reads this filter well.

Approve the set

Until every example is judged, the side panel says “Review all gold set examples.”

1. Optionally, write a Note (optional) about the set.

2. Select Approve the gold set.

   The confirmation says what you are signing off: all examples as the training target, unlocking shadow, and that filtering still needs a separate threshold approval.

3. Select Approve all {N} examples.

   The approval is recorded with your name, your note, and the disagreement rate at that moment, and shows on the filter’s Activity tab. The audit log records the decision and your note.

What you should see

The panel becomes Approved, with Signed off by, When, Over, Disagreement and your note. The intent heading reads This set was labelled against this sentence. Labels are now closed: re-labelling would change what was signed off. Your judgements become training labels for the filter.

Approval does not start the filter. It lets an operator start it in shadow:

1. Open the Actions tab.

   Start watching in shadow offers Start in shadow, once the paired agent, if the filter needs one, is connected (see Connect a filter to its agent).

2. Optionally, write a Note (optional), then select Start in shadow and confirm.

   The filter starts polling its source and scoring every event. It sends every event to the agent and filters nothing, so agent runs are billed at the unfiltered rate. Each verdict becomes a label. See Shadow and filtering.

Start over

If the set is not what the intent means, an operator or admin can discard it and build another, while the filter is in draft or authoring, the set is not approved, and at least one example has been judged:

1. Select Start over with a new set.
2. Select Discard {n} and rebuild.

The judgements already made are discarded and not recovered, and Sift reads the source again for a fresh set. The intent and source are unchanged, so if one of those is what is wrong, fix it first in Settings.

What can go wrong

You see	What it means	What to do
The gold set could not be built (badge Build failed)	Reading the source or proposing labels failed. The reason is shown.	Fix the source if it was the problem, then select Try again.
This set no longer describes the filter	The filter’s intent or source changed after the set was built. Labels are closed and it cannot be approved.	While the filter is in draft or authoring, select Build a new gold set and review that one.
An approval is refused	You are a reviewer, or an example is still unjudged.	Ask an operator or admin to approve, or finish the remaining examples.
“None of the … labelled looked like a match”	Every example is proposed as one to filter. Approving says the stream is quiet for this intent.	Check that is what you expect, and that the intent and source are right, before approving.

Related

• The gold set
• Verdicts, labels and training
• Approve a threshold, the second approval