Back up and restore

Two scripts back Sift up and put it back: scripts/backup.sh copies the database and the trained model files into one directory, and scripts/restore.sh loads such a directory into a stopped stack and then checks that the result works. Run both from the directory you deploy from. Sift does not schedule backups; you do.

What a backup holds

	In the backup	Notes
Postgres	Yes, all of it	One consistent `pg_dump` in custom format.
Object store	Yes, every object in every bucket	One file per object.
`SIFT_SECRET_KEY` and `.env`	No	Back them up separately. See below.
Redis	No	Queues and caches; nothing in it needs to survive.
`model-weights` volume	No	Reproducible from its pinned digest.
Container images	No	Keep the release’s image bundle.

The backup itself is sensitive: it holds the encrypted credentials, password hashes, the audit log and up to 30 days of event content. Store it as you would the database.

Take a backup

./scripts/backup.sh

The script needs postgres and seaweedfs running, and the backend image on the host. By default it stops sift-worker-io, sift-worker-ml, sift-worker-notify and sift-beat for the length of the backup and starts them again afterwards, so nothing writes or deletes between the database dump and the object copy. It restarts only the services it stopped. The API and console stay up throughout. Polling pauses and resumes from where it left off, so events arrive late, not lost.

Option	Default	Meaning
`-p`, `--project`	`$COMPOSE_PROJECT_NAME`, else `kindo-sift`	The Compose project to back up.
`-o`, `--output`	`$SIFT_BACKUP_DIR`, else `./backups`	Where to write the backup directory.
`--env-file`	none	An env file passed to `docker compose`.
`--no-quiesce`	off	Leave the workers and scheduler running.

The script reads COMPOSE_PROJECT_NAME from your shell, not from .env. If the directory you deploy from is not named kindo-sift, pass -p with your project’s name (docker compose ls lists it).

Each run writes one directory, sift-<project>-<UTC time>, readable only by you:

File	Contents
`postgres.dump`	The database.
`objects/`	Every object, as `<bucket>/<key>`.
`manifest.json`	When and from where; the Postgres version and schema revision; the bucket list, object count and size; a row count for every table; whether the backup was quiesced; and a one-way fingerprint of `SIFT_SECRET_KEY` (not the key).
`SHA256SUMS`	A checksum of every file above.

The script prints PASS or FAIL for each step and exits non-zero if any step failed.

When `--no-quiesce` is safe

Every night at 04:20 UTC Sift deletes the stored files of models retired more than 30 days earlier (SIFT_ARTIFACT_RETENTION_DAYS). It is the only thing that deletes from the object store. An unquiesced backup taken across that moment can name model files it does not contain. Only long-retired models are affected, and the restore’s checks name them, but the backup you would actually restore from should be a quiesced one. Take --no-quiesce copies clear of 04:20 UTC.

Schedule backups

Sift has no scheduler for this. A cron example, from the deployment directory, for a host running on UTC:

# Hourly copies with nothing stopped, skipping the 04:00 hour.
15 0-3,5-23 * * *  cd /opt/kindo-sift && ./scripts/backup.sh --no-quiesce -o /srv/sift-backups >> /var/log/sift-backup.log 2>&1
# The nightly backup to restore from: quiesced.
30 2 * * *         cd /opt/kindo-sift && ./scripts/backup.sh -o /srv/sift-backups >> /var/log/sift-backup.log 2>&1

Cron uses the host’s local time and Sift’s nightly jobs use UTC, so shift the skipped hour if the host is not on UTC.

Nothing deletes old backups. Prune them yourself, for example with find /srv/sift-backups -maxdepth 1 -type d -mtime +N -exec rm -rf {} +. Choose N knowing that filtered-event records older than SIFT_FILTERED_EVENTS_RETENTION_DAYS (400 days) are deleted from the database, so an old backup may be the only copy of them.

Restore

A restore drops the target database and makes the object store match the backup exactly. It refuses to start, changing nothing, if:

any file fails its checksum;
the SIFT_SECRET_KEY the target runs with does not match the fingerprint in the manifest (unless you pass --allow-key-mismatch);
any container of the target project is running;
you did not pass --confirm.

Make sure .env holds the SIFT_SECRET_KEY that was in force when the backup was taken.
Stop the stack:
Terminal window
```
make down
```

Restore:

./scripts/restore.sh -i backups/sift-kindo-sift-<time> --confirm

The script starts postgres and seaweedfs, recreates the database empty, loads the dump, gathers the database’s statistics, mirrors every bucket back, starts the whole stack, waits for sift-api to be healthy, and then verifies. Running it twice lands the same state.

Option	Meaning
`-i`, `--input`	The backup directory. Required.
`-p`, `--project`	The target Compose project. Same default as the backup.
`--env-file`	An env file for the target. The restored containers read it too.
`--confirm`	Required for a restore.
`--allow-key-mismatch`	Restore even though `SIFT_SECRET_KEY` differs. Every stored credential then needs re-entering.
`--verify-only`	Run only the checks below, against a running stack. Changes nothing.

What the restore checks

Check	What it shows
`sift-api` healthy, `/healthz` and `/readyz` answer `ok`	The stack came up and reaches Postgres, Redis and the object store.
The schema revision matches the manifest, or was migrated forward from it	The database is the one the dump was taken from.
No table has fewer rows than the manifest records	Nothing is missing. Growth is reported, not failed: a stack in use writes audit rows.
Every model’s file reads back byte for byte	The database and the object store agree.
The stored Kindo key and each webhook endpoint’s secrets decrypt	`SIFT_SECRET_KEY` is the right key. No call goes to Kindo.

Run --verify-only at any time to check a live stack the same way. After a restore, also select Verify on the gear icon (Administration) › Kindo connection, and Send a test to each webhook endpoint under Notifications.

What a restore rewinds

Everything since the backup is gone, and some of it has consequences:

Sources are read again from where the backup left off. An event already in the backup is not scored twice, but an event whose agent run happened after the backup is sent to the agent again, so expect duplicate runs for that window.
The Filtered events log loses that window. Record the gap if you need it for review.
Approvals and state changes made after the backup are undone. Each filter comes back with the model, threshold and state it had at the backup. A filter that was filtering then and has since been stopped, held or fallen back is filtering again, so check the Filters page after a restore.
A training job that was running when the backup was taken comes back as running, and that filter does not retrain until Sift gives up on the job after four hours.

There is no point-in-time recovery between backups: your recovery points are exactly the backups you took.

Rehearse it

Restore into a separate project on the same host, with its own env file:

./scripts/restore.sh -i backups/<backup directory> -p sift-dr --env-file .env.dr --confirm

.env.dr needs a different SIFT_WEB_PORT and the same SIFT_SECRET_KEY as the source. A backup that has never been restored is not yet a backup procedure.

Limits

The backup is not encrypted beyond its directory’s permissions. Encrypt it at rest and in transit yourself if it leaves the host.
There is no point-in-time recovery.
The object store’s bucket settings are not backed up, only the objects. Sift sets none.
The weights are not in the backup. On a connected host sift-api downloads them again on first start; on an isolated host, stage them as in Install without internet access.