- Python 94.1%
- Shell 5.9%
| Filename | Latest commit message | Latest commit date |
|---|---|---|
Snapshots on the same disk don't survive that disk. Each [[zfs.replicate]] entry in the host config (from, to, exclude, schedule, before) gets a kivotos-replicate-<name> service and timer from `kivotos units`, which also removes units for entries that are gone. `kivotos replicate [NAME]` runs the `before` commands as root, then syncoid --recursive --no-sync-snap --create-bookmark, so only sanoid's snapshots are sent. `before` is refused unless the config is owned and writable only by root. `kivotos plan` now shows each replication with the sanoid retention of both sides and of every source's dataset, read from sanoid.conf (templates and recursive parents resolved). It warns when nothing prunes the target, when sanoid would take its own snapshots there, and when the timer hasn't been written yet. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
| .gitignore | ||
| config.toml | ||
| install.sh | ||
| kivotos | ||
| LICENSE | ||
| README.md | ||
kivotos
kivotos (Κιβωτός) - the ark keeping things safe through the flood
Warning
This was made with Claude Opus 5.5 mostly for personal use.
Backups of service directories. Each service's backup rules live next to it in a kivotos.toml,
so a service directory carries its own backup rules wherever you move it.
It has two backends, picked with backend in /etc/kivotos/config.toml:
restic(the default): kivotos copies every service into a restic repo on its own timer. Works on any filesystem and to any restic target (local, sftp, S3, ...), so it's the one for offsite copies.zfs: the sources are on ZFS, and sanoid snapshots the whole dataset. kivotos runs each service's dumps and hooks right before and after every snapshot. See ZFS backend.
One Python file, standard library only. It needs restic ≥ 0.17 and Python ≥ 3.11.
Install / update
sudo sh ~/Packages/kivotos/install.sh
The first run installs /usr/local/bin/kivotos and writes /etc/kivotos/config.toml, with user set to whoever ran sudo. Then it stops.
Check the repository and the sources (sudoedit /etc/kivotos/config.toml), then run it again.
The second run creates the password and the repo, and writes the systemd service and timer from the config.
You can run it again at any time, and you should after changing the program in this folder. It never overwrites the config, the password or a repo it can open.
| What | Where | Owner |
|---|---|---|
| Program | /usr/local/bin/kivotos |
root |
| Host settings (repo, schedule, retention, sources) | /etc/kivotos/config.toml |
root |
| Repo password | /etc/kivotos/password (keep a copy in your password manager) |
root, 600 |
| Repo | repository in the config, e.g. /tank/backups/kivotos (a ZFS dataset there can also be snapshotted by sanoid) |
root |
| Per-service rules | ~/Containers/<svc>/kivotos.toml |
you |
Usage
kivotos plan # what a run would do, per service (no sudo needed)
sudo systemctl start kivotos # back up now, the same as the timer (logs: journalctl -u kivotos)
sudo kivotos run NetBird Postgres # back up only these services (no retention or check)
sudo kivotos run --dry-run # what restic would back up; no hooks, dumps or stopping
systemctl list-timers kivotos # next and last run
sudo kivotos restic <anything> # any restic command against the repo, for example:
sudo kivotos restic snapshots --tag NetBird
sudo kivotos restic ls latest --path ~/Containers/NetBird
sudo kivotos restic stats --mode raw-data
Change the schedule in /etc/kivotos/config.toml (sudoedit it), then run sudo kivotos units to rewrite the units.
What a run does
Every directory directly inside a source (~/Containers/*) is a service, and each one gets its own snapshot.
Snapshots are tagged with the source and the service name, for example containers,NetBird.
Loose files at the top (AGENTS.md, TODO.md, ...) go in one extra snapshot tagged _files.
Symlinked directories are ignored.
For each service, in this order:
- Dumps run while everything is still up. Each one is streamed straight into the repo, with no temp files, as
<svc>/<file>, taggeddump. beforehook, then stop the listed containers.- Back up the directory, minus its excludes.
- Start the stopped containers again, then the
afterhook. These always run, even if the backup failed.
Then comes retention (per backed-up path: 7 daily, 4 weekly, 6 monthly by default), then restic check.
kivotos doesn't stop at the first problem: it carries on and backs up everything else.
Afterwards it lists every problem and exits with 1, so kivotos.service fails.
Examples of problems: a failed hook or dump, a broken kivotos.toml, or an unreadable file.
To be alerted, hang an OnFailure= on the service with sudo systemctl edit kivotos.service.
That drop-in survives kivotos units, which only rewrites the unit files themselves.
sema does this for every system service and sends the alert to Matrix, so with its onfailure module on, there's nothing to set up here.
A container that isn't running is not an error. Its dumps and if_running hooks are just skipped.
ZFS backend
With backend = "zfs", kivotos has no timer, repo or retention of its own. sanoid snapshots the
dataset your sources live on (/etc/sanoid/sanoid.conf), and calls kivotos around each snapshot:
[zroot/home] # the dataset ~/Containers is on (`kivotos plan` tells you)
pre_snapshot_script = /usr/local/bin/kivotos hook
post_snapshot_script = /usr/local/bin/kivotos hook
force_post_snapshot_script = yes # run `after` hooks even if a `before` one failed
script_timeout = 900
sudo kivotos units removes the restic timer and tells you if that section is missing. kivotos plan does too.
A ZFS snapshot is the whole dataset at one instant, like pulling the power cord, in zero time. That changes what the rules mean:
| Rule | On zfs |
|---|---|
[hooks] |
Run as usual: before right before each snapshot, after right after it. Every snapshot (hourly), so keep them quick. |
[[dump]] |
Written to <svc>/.kivotos/<file> right before the snapshot, so the snapshot holds it. Only on the snapshot types in [zfs] dump_on (default daily). A failed dump keeps the last good one. |
exclude |
Does nothing. ZFS can't leave out part of a dataset. Mount caches outside the snapshotted dataset instead (/var/cache/containers/<svc>/... on the SSD, /tank/cache/<svc>/... on the HDD, neither snapshotted). kivotos plan warns about leftover excludes. |
stop |
Ignored. Nothing needs stopping for an instant snapshot. |
skip |
Does nothing, for the same reason as exclude. |
Databases are fine without dumps: Postgres, SQLite and RocksDB all recover from a power-cut copy, which is what a snapshot is. A dump is still handy to restore into a newer Postgres version, or one database alone.
Replication
Snapshots on the same disk don't survive that disk. [[zfs.replicate]] in /etc/kivotos/config.toml
sends a dataset tree's sanoid snapshots to another pool with syncoid (incremental: only changed blocks):
[[zfs.replicate]]
from = "zroot" # sent recursively
to = "tank/backups/ssd/zroot" # created on the first run
exclude = ["zroot/var/cache"] # datasets (and their children) left out
schedule = "*-*-* *:20" # systemd OnCalendar, default "hourly"
before = ["mountpoint -q /boot && rsync -a --delete /boot/ /tank/backups/boot/"] # optional, as root
# name = "zroot" # unit name, default `from` with / -> -
sudo kivotos units writes kivotos-replicate-<name>.service and .timer for each entry (and removes ones
you deleted). sudo kivotos replicate [NAME] runs them now. Only sanoid's own snapshots are sent, so the copy
needs its own sanoid section with autosnap = no to prune it, with longer retention than the source, e.g.:
[tank/backups/ssd]
use_template = replica # hourly = 48, daily = 30, weekly = 8, monthly = 12, autosnap = no
recursive = yes
before commands run as root, so kivotos refuses to run them unless the config is owned and writable only
by root. Guard anything destructive: mountpoint -q /boot && stops rsync --delete from emptying the copy
when /boot isn't mounted. kivotos plan shows each side's sanoid retention and warns when nothing prunes to.
kivotos plan # dataset per source, what runs, warnings
sudo kivotos snapshot pre-upgrade # hooks + a snapshot of every source's dataset now, named
# <dataset>@kivotos_pre-upgrade_<date>. sanoid never prunes these.
journalctl -u sanoid # what the hooks did (they log there)
Restoring: every snapshot is browsable read-only at ~/Containers/../.zfs/snapshot/<name>/ (the dataset's
root, e.g. /home/.zfs/snapshot/). Copy files back with sudo cp -a to keep their subuid owners,
or roll the whole dataset back with sudo zfs rollback.
Per-service rules: kivotos.toml
No file means the whole directory is backed up as is. So a new service needs nothing, as long as its data is in bind mounts under its directory.
Add a kivotos.toml only for special cases. Every key is optional:
# "./name" or "a/b" is relative to this dir, a bare "name" matches at any depth.
# No '!' negation, no '..' and no leading '/'. Patterns can't reach outside the service dir.
exclude = ["./cache", "data/dind", "node_modules"]
skip = false # true: don't back up this service at all
stop = ["foo", "foo-worker"] # stopped for this service's backup (only if running), started again after
[hooks] # shell commands, run as the configured `user` in this service's dir
if_running = "foo" # only run the hooks while this container is up
before = "podman exec foo some-flush-command"
after = "podman exec foo some-resume-command"
[[dump]] # as many as you like
name = "foo-db" # letters, digits, . _ -
container = "foo-db" # run with `podman exec` in this container; leave it out to run on the host as `user`, in this dir
command = "pg_dumpall -U postgres" # stdout is what gets stored
file = "foo.sql" # stored as <svc>/foo.sql (default: <name>.dump)
Security
kivotos runs as root, for two reasons. It has to read files owned by container subuids, and the repo must be out of reach of your user, so ransomware running as your user can't touch it.
kivotos.toml files belong to your user, so root treats them strictly as data:
- Hooks, dump commands and every
podmancall run as the source'suser, throughrunuserwith a clean environment. Nothing from akivotos.tomlis ever executed as root. Root only stores the output. - A
kivotos.tomlis only honoured if it's owned by thatuseror by root. Anything else (another user's file, a file a container wrote with its subuid) is ignored and reported, and the service is backed up with no rules. So nobody can get commands run as someone else. - Excludes are anchored inside the service directory.
..and!are rejected. - Symlinked service dirs and a symlinked
kivotos.tomlare refused. Root never follows them out of the tree. - The worst a tampered
kivotos.tomlcan do is skip or exclude its own service's data. A*exclude is reported as a problem.kivotos planshows everything that's set.
Keep hook scripts next to compose.yaml, never inside a directory that's mounted into a container.
Otherwise a compromised container could rewrite what runs as your user.
Databases in services
Copying a database's files while it's writing to them can give a backup that won't open:
| Kind | Examples here | What to do |
|---|---|---|
| Postgres / MySQL / MariaDB | postgres-db |
Add a [[dump]], and exclude the raw data dir |
| SQLite | Grafana, NetBird, Mumble | A [[dump]] with the host's sqlite3 .dump (recipe below): a consistent snapshot while the app keeps running, whatever is in the image |
| RocksDB / LevelDB / embedded KV | tuwunel (on /tank) |
stop the container, or keep the data on /tank: ZFS snapshots are atomic, like a power cut, which databases are built to survive |
| Redis / Valkey | searxng-valkey (cache) |
Usually not worth it. Otherwise use a before hook with redis-cli SAVE |
Recipes:
[[dump]] # MariaDB / MySQL
name = "mariadb"
container = "foo-db"
command = 'mariadb-dump --all-databases -uroot -p"$MARIADB_ROOT_PASSWORD"'
file = "mariadb.sql"
[[dump]] # SQLite, using the host's sqlite3. podman unshare lets it read files owned by container subuids.
name = "app-db"
command = "podman unshare sqlite3 data/app.db .dump" # relative to this service's dir
file = "app.sql" # restore: sqlite3 app.db < app.sql
Rule of thumb: a busy, important database goes on /tank/services/<svc>. A small app database can stay in ~/Containers. A database server gets a [[dump]].
Restoring
Stop the service first: cd ~/Containers/<svc> && podman compose down.
# One service, in place. Files come back with their original subuid owners.
sudo kivotos restic restore latest --path ~/Containers/NetBird --target /
# Somewhere else first, to look around
sudo kivotos restic restore latest --path ~/Containers/NetBird --target /tmp/restore
# A single file or folder from an older snapshot
sudo kivotos restic snapshots --tag NetBird
sudo kivotos restic restore <id> --target /tmp/restore --include ~/Containers/NetBird/config.yaml
# Browse everything as a filesystem (Ctrl-C to unmount)
sudo mkdir -p /mnt/kivotos && sudo kivotos restic mount /mnt/kivotos
Postgres from the dump (<db-user> is your POSTGRES_USER). Use it on an empty or fresh cluster, because roles and databases that already exist will error:
sudo kivotos restic dump latest Postgres/postgres-all.sql --path /Postgres/postgres-all.sql > ~/postgres-all.sql
podman exec -i postgres-db psql -U <db-user> -d postgres < ~/postgres-all.sql
Adding a source
To back up another tree of services, add a [[source]] block to /etc/kivotos/config.toml, then run sudo kivotos units:
[[source]]
name = "projects"
path = "~/Projects" # ~ is the home of this source's user
exclude = ["node_modules", "target", ".venv"] # applied inside every service of this source
Several users
Each source can have its own user, which defaults to the top-level one.
Rootless podman is per user, so a source's hooks and dumps have to run as the user who owns its containers:
[[source]]
name = "trowe"
path = "~/Containers" # ~ is this source's user, so /home/trowe/Containers. "~nn/x" names a user explicitly.
user = "trowe"
All sources share one repo. Only root can read it, so users can't see each other's backups, and deduplication works across everyone.
The flip side is that only root can restore. Snapshots carry the source name as a tag (--tag trowe).
☦
Ὤ
Ὁ Ν
Ι̅Ϲ̅ │ Χ̅Ϲ̅
───┼───
ΝΙ │ ΚΑ
☦
Εἰς δόξαν τοῦ Θεοῦ
To the glory of God
Τῇ Ὑπεραγίᾳ Θεοτόκῳ δόξα
Glory to the Most Holy Theotokos
Δόξα τῷ Θεῷ πάντων ἕνεκεν
Glory to God for all things
ΑΜΗΝ
☦