The gold set

The gold set is about fifty events from a filter’s own source, each with a proposed reading of whether it matches the filter’s intent, for a person to check one at a time. Approving it is the first of the two approvals a filter needs, and what allows it to be started in shadow. This page explains how the set is built, what you check, and what the approval covers.

Why it exists

A filter is built from one or two sentences, its intent. Everything after that depends on Sift reading those sentences the way you meant them. The gold set is a short check, about fifteen minutes, that catches a misread intent before any model is trained on it and before any event is scored.

What you are really checking is not fifty separate labels. It is whether the intent at the top of the screen and Sift’s reading of it are the same thing. That is why the disagreement rate is the headline figure.

How the set is built

The console starts the build when it creates a filter. An operator or admin can also start it from the filter’s Gold set tab with Build the gold set. A build takes minutes:

Read a pool. Sift reads up to 500 recent events from the filter’s source. This is a look, not a poll: nothing is stored as a filter event and the source’s read position does not move.
Propose a reading for each. The model chosen on the Kindo connection reads the events and proposes for each whether it matches the intent, with a one-sentence reason. Event content is treated as data, never as instructions.
Choose about fifty. Up to half from the proposed matches and the rest from the proposed non-matches, spread across the pool. If one side runs short, the other fills in.

With the decision model turned on, Sift reads a wider pool and uses the model to find the events most likely to match, so more of them reach the set. That helps on streams where matches are rare.

The review screen shows how many events were read and how many were proposed as matches, because fifty events chosen for balance say nothing about how rare a match is.

A set can be built only while the filter is in draft or authoring. Once the filter is scoring, a new set would be evidence about nothing.

If a build fails, the tab says The gold set could not be built, with the reason, and offers Try again. A build that does not finish within an hour is marked abandoned.

Reviewing

Each example shows the event as it came from the source, the proposal with its reason, and two buttons for your judgement:

Escalate (key E): the agent should see this event.
Filter (key W): the event may be filtered.

You can also answer against the proposal: A agrees with it, D disagrees. Either advances to the next unreviewed example. You can add a note to an example.

Anyone signed in can label examples, including a reviewer. Building and approving a set need an operator or admin.

The set’s status reads Not started, then In review once the first example is judged, then Approved.

The disagreement rate

The disagreement rate is the share of your judgements that went against the proposal. A high rate means the intent and Sift’s reading of it have drifted apart, and training on the set would build that gap into the model. From 20% the screen says so and suggests rewording the intent and building another set, which usually works better than pushing through.

The rate is recorded with the approval. A set signed off at 40% disagreement is a different decision from one signed off at 2%.

Approving

Approval is refused while any example is unreviewed: the rate means something only over the whole set, and a gate that can be skipped is not a gate.

Once every example is judged, the form Approve the gold set appears with an optional Note. If you leave it blank, the record says Approved without comment. The confirmation, Approve all 50 examples (or however many there are), records your name, the time and the disagreement rate with the approval; the audit log records the decision and your note.

Approving does not start shadow by itself. It opens the way: the filter’s Actions tab then offers Start watching in shadow, and an operator or admin presses Start in shadow (for a filter with a paired agent, once that agent is connected). In shadow the filter scores every event and still sends every one. Approving the gold set never lets a filter start filtering; that needs the separate threshold approval. See Operating point and approval.

After approval, labels are closed: changing one would change what was signed off.

What the approval covers

An approval covers the set and what it was built from: the intent its proposals answered and the source its events came from. If you change either while the filter is in authoring, the set no longer describes the filter. The tab says This set no longer describes the filter, the approval stops counting, and you build and review a new one. Changes to anything else, such as the recipe, the target or the operating point, leave it standing.

Once a filter is scoring, its gold set is the record of what was reviewed before scoring began. Sift does not check a scoring filter against it again.

Starting over

While the filter is in draft or authoring, an operator or admin can discard a set with Start over with a new set. The confirmation, Discard N and rebuild, says how many judgements are lost. The intent and source do not change, so if one of them is what is wrong, fix it first.

What happens to the judgements

Approving turns each reviewed example into a training label carrying your judgement, not the proposal’s, from a person rather than the agent. These labels count while their set is the filter’s latest, approved, and still describes the filter. See Verdicts, labels and training.