rcmd/keeporg
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Keep server data in an encrypted database, and harden every part

Browse Source 
Storage. The server keeps its data in SQLite (a file in the data
directory, the default) or PostgreSQL (database, KEEPORG_DATABASE_URL or
-database). Every row, the KDBX versions included, is sealed with
AES-256-GCM under a key derived from the master key and named by an HMAC.
The master key is required to start and a different one is refused. A data
directory from 0.10 or earlier is imported on first start. Several servers
may share one PostgreSQL database. Go goes to 1.26 for the pure-Go
drivers (modernc.org/sqlite, jackc/pgx).

Hardening, after an audit of the server, both clients and the deployment:
- Single sign-on approves a device only after an explicit Approve in the
  browser that started it, bound by a cookie and PKCE; identities are never
  linked by username, by verified email only when allowed, never to admins.
- Sign-in attempts are limited per address and per account, passwords are
  checked outside the store lock with uniform timing, one-time codes work
  once, and starting two-factor needs the password.
- Merge vaults bound KDF parameters, payload size and XML shape, check
  pushes before storing them, and run at most two merges at once. A version
  is written in the same transaction as its metadata.
- Tokens in URLs only on the event stream, which ends on revocation; CSP,
  frame-ancestors, no-store and HSTS; sign-in CSRF; 404 for databases
  without access; new trusted_proxies.
- Clients open only the server's approval page, keep case-variant and
  retired file names apart, never overwrite a save made during a pull,
  require https except locally, send the stream token in a header, bound
  their reads and escape notifications and terminal output.
- The .deb creates the master key, systemd is sandboxed further, nginx sets
  modern TLS and HSTS, and CI deploys from main only.

Server 0.12.0, desktop 0.6.0, Android 0.4.0 (versionCode 4).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Reactorcoremeltdown
2026-11-02 15:38:32 +01:00
co-authored by Claude Opus 5.5
parent f8f634fbf7
commit 4ffc434b1b
77 changed files with 4192 additions and 776 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/HANDOVER.md
+280 -84
View File
@@ -28,8 +28,11 @@ cmd/keeporgd/ server entry point: flags, config file, bootstrap
cmd/keeporg/ desktop client: the tray application (agent.go is its
logic, tray.go its menu), login, run, sync, status,
administration, start-at-login per platform
internal/store/ domain model, JSON metadata store, ACL resolution
internal/blob/ versioned KDBX payload storage on disk
internal/store/ domain model, the in-memory state and its sealed rows,
ACL resolution
internal/blob/ versioned KDBX payloads, sealed, in the database
internal/sqldb/ SQLite or PostgreSQL: connection, dialect, schema,
generation lock, key check; sqldbtest for tests
internal/kdbx/ KDBX 4 codec and the merge engine, with its own crypto
internal/server/ HTTP surface: auth, OIDC, sync, admin, SSE hub, web,
web administration, page templates
@@ -52,21 +55,33 @@ Makefile the only entry point for building and publishing
### Server (`keeporgd`)
One process, one data directory, no database engine:
One process, one database, one master key:
- `state.json` holds users, groups, memberships, databases, grants, invites,
sessions and pending device logins. For a merge vault it also holds that
vault's password, encrypted under the server's master key.
- `blobs/<database-id>/<version>.kdbx` holds the payloads, keeping the last
20 versions of each database.
- The master key itself lives **outside** the data directory, so a copy of
the data directory alone yields ciphertext.
- **The database** is SQLite (`sqlite:<path>`, by default
`<data_dir>/keeporg.db`) or PostgreSQL (`postgres://…`), chosen by the
`database` setting, `KEEPORG_DATABASE_URL` or `-database`. It holds
everything: users, groups, memberships, databases, grants, invites,
sessions, pending device logins, and the last 20 versions of every KDBX
payload.
- **Every row is sealed** with AES-256-GCM under a key derived from the
master key, and named by an HMAC rather than by anything readable. The
database, its backups and its administrators see three tables of opaque
rows; see [Encryption at rest](#encryption-at-rest).
- **The master key** lives outside the database and outside the data
directory, so neither a database dump nor a copy of the data directory
yields anything readable. The server does not start without it, and does
not start with a different one.
`internal/store.Store` is the only writer of `state.json`. Every mutation
runs through `Update(func(*State) error)`, which applies the change to a
private copy, writes it with `write` + `fsync` + `rename`, and only then
publishes it. A failed mutation changes nothing, and a crash leaves either
the old document or the new one.
`internal/store.Store` keeps the metadata in memory as one `State` and is
the only writer of it. Every mutation runs through `Update(func(*State)
error)`, which applies the change to a private copy, writes the rows that
changed in one transaction, and only then publishes the copy. A failed
mutation changes nothing, and a crash leaves the last committed state.
`internal/sqldb` is the database underneath: connection, dialect, schema.
`internal/blob` keeps the payloads in the same database. Both engines run
through `database/sql` with pure-Go drivers (`modernc.org/sqlite`,
`jackc/pgx`), so the server still builds with `CGO_ENABLED=0`.
Reads go through `Read(func(*State))` under a read lock. Access decisions are
pure functions over `*State` in `internal/store/acl.go`, which is why they
@@ -318,6 +333,7 @@ Three different things are encrypted here, and they are worth keeping apart:
fails to open instead of unlocking the wrong database. It is never
returned by the API, never logged, and never leaves the process.
3. **Account passwords.** Not stored at all; only PBKDF2-SHA256 verifiers.
4. **Everything else the server keeps**, in the database: see below.
The master key is looked for in three places, in order:
@@ -330,28 +346,56 @@ The master key is looked for in three places, in order:
Only the configured path is ever created, which is what makes a plain
package install work with no setup. The two environment variables name a key
that something else owns, so a missing one is reported rather than replaced:
minting a key in its place would open none of the existing vaults. Without
any of the three the server still runs and still serves every key-less
strategy; only vault creation and merging are refused, with a line in the
log saying so.
minting a key in its place would open nothing that is already stored. The
server refuses to start without a key, and refuses one that is not the key
the database was written with (`sqldb.CheckKey` opens a sealed marker stored
at the first start). Carrying on would write rows nobody could read beside
rows this key cannot.
### Encryption at rest
The database holds three tables, and none of them says anything readable:
| Table | Row | Name of the row | Content |
| --- | --- | --- | --- |
| `keeporg_records` | one user, group, database record, grant, membership, invite, session or device login | `HMAC(records key, kind ‖ natural key)` | the object as JSON, sealed |
| `keeporg_blobs` | one stored version of one KDBX file | `HMAC(blobs key, database id)` and the version number | the KDBX bytes, sealed |
| `keeporg_meta` | schema version, write generation, key check | plain names | numbers, and the sealed key-check marker |
The records key and the blobs key are derived from the master key with
HMAC-SHA256 under distinct names (`secret.Keyring.Derive`), so a ciphertext
made for one purpose does not open under another. Each row is sealed with
AES-256-GCM, its own name as the authenticated data: a row copied over
another, or a version copied over another version, fails to open instead of
being served. A row's name is stable for the same object, which is what
lets a write touch only the rows that changed, and says nothing to anyone
without the key — session and invite tokens, which are bearer credentials,
appear only inside sealed rows.
What the database reveals is the number of rows, their sizes, and how many
versions each payload has. Nothing is queryable by SQL: the server reads
every record at start and answers from memory, which is also why access
decisions stay pure functions over `*State`.
KDBX is already encrypted by KeePass; sealing it again means the database
holds no readable KeePass headers either (KDF parameters, cipher choice),
and one backup of the database is the whole server.
What this buys and what it does not:
- A stolen copy of `/var/lib/keeporg` is useless on its own. The vaults are
ciphertext and the wrapped keys do not open without the master key.
- A stolen copy of the data directory **and** the master key opens every
merge vault. Keep them on different media, or feed the key in through
`KEEPORG_MASTER_KEY` from a secret manager or a systemd credential so it
never touches the disk at all.
- Backups must cover both, separately. A backup of the data directory with no
master key cannot be used to merge after a restore; the vaults still open
for their users, and the keys can be stored again with
`keeporg db -id <id> -store-password`.
- A database dump, a backup, a replica or a curious database administrator
gets ciphertext. So does a stolen copy of the data directory.
- The database **and** the master key together open everything, including
every merge vault. Keep them on different media, or feed the key in
through `KEEPORG_MASTER_KEY_FILE` from a secret manager, a Kubernetes
secret or a systemd credential so it never sits beside the data.
- Backups must cover both, separately, and a database backup without its
key restores nothing. Losing the key loses the server.
## Authentication
Three ways in, one kind of credential out: an opaque bearer token stored in
`state.json` with an expiry (90 days by default).
Three ways in, one kind of credential out: an opaque bearer token, stored
sealed in the database with an expiry (90 days by default).
- **Password** (`POST /api/v1/auth/login`), with the built-in TOTP second
factor when the account has enrolled one. The server answers
@@ -369,6 +413,36 @@ only. The server is a confidential client: it exchanges the authorization
code over TLS with the token endpoint and reads the identity from the
userinfo endpoint, so there is no local JWT verification to get wrong.
### Proving who someone is
Every password and one-time code the server checks — API login, the sign-in
form, the device page, a password change, a second-factor change, a vault
deletion — goes through `verify` in `internal/server/authn.go`:
- **The hash is checked outside the store's lock.** It is deliberately
slow; under the write lock it stalled every other request, so a few
guesses a second were a denial of service. `commit` re-checks inside the
write that the account and its hash did not change in between.
- **Failures are limited** per client address (30 per 15 minutes) and per
account (10), and a client or account over budget gets `429` with
`Retry-After` before any hash is computed. Unauthenticated device and
single sign-on starts are limited per address too. The address is the
peer's, or, for a peer in `trusted_proxies` (`KEEPORG_TRUSTED_PROXIES`),
the right-most untrusted address in `X-Forwarded-For`; nobody else gets to
name their own address. Behind nginx, set it, or every client shares the
proxy's budget.
- **Unknown, disabled and password-less accounts take as long to refuse as
a wrong password** (a dummy hash is verified), so timing does not list the
usernames.
- **A one-time code is good once.** The step of the last accepted code is
stored on the account (`TOTPLastStep`) and anything at or before it is
refused, including the code that confirmed enrolment.
- **Re-authentication does not say which factor was wrong.** For a change
made from a live session (password, second factor, deletion), a wrong
password and a wrong code give the same answer: a stolen session must not
become a password oracle. Starting a second factor needs the password too,
since on a stolen session it would lock the owner out.
### Setting up OIDC
At the identity provider, register a confidential client with one redirect
@@ -388,7 +462,8 @@ and restart:
"client_secret": "...",
"scopes": "openid profile email",
"auto_create": false,
"default_group_id": ""
"default_group_id": "",
"link_verified_email": false
}
```
@@ -404,20 +479,33 @@ Two entry points use it, and both end at the same callback:
starts a sign-in that ends in a session cookie.
- **A client**, from the device flow. `keeporg login -browser=sso` prints a
short code; the device page at `/device?code=…` offers the same link, this
time as `/oidc/login?code=<user code>`, and approving it releases the
token the waiting client is polling for.
time as `/oidc/login?code=<user code>`. After the provider, the person is
shown which device is asking, as whom, and its code, and must press
**Approve**: only then is the token released to the waiting client.
Arriving at the callback proves that a browser with a provider session was
sent there, not that its owner meant to let a device in, and before 0.12 a
single link click was enough to hand an attacker's device a session.
Every round trip is bound to the browser that started it (a `keeporg_oidc`
cookie holding the state, checked at the callback; the approval's nonce in
the same cookie, checked at `/oidc/approve`) and uses PKCE, so a forwarded
callback URL or a code intercepted on the way back is useless.
How an identity becomes an account:
1. If some account is already linked to this issuer and `sub`, that one.
2. Otherwise an unlinked local account whose username equals
`preferred_username`, or whose email matches `email`, is linked to it.
This is how someone invited by link later moves to single sign-on.
2. Otherwise, only when `link_verified_email` is set and the provider says
`email_verified`, an unlinked local account with the same email — never a
server administrator's, never a disabled one. **Never by username**:
whoever could choose `preferred_username=admin` at the provider would
otherwise have become the server administrator.
3. Otherwise, with `auto_create` set, a new account is provisioned and put
in `default_group_id` when one is given. Without it, sign-in is refused
with "no account is linked to this identity; redeem an invite link
first", which is the setting to keep if membership should stay by
invitation.
in `default_group_id` when one is given. Without it, sign-in is refused.
An existing account moves to single sign-on from its own **Account** page:
**Link single sign-on** runs the round trip with that session's CSRF token
and links whatever identity comes back to it. That replaces the automatic
username matching that used to do this.
An account created this way has no password, so it cannot sign in with one
and cannot set one through the password form.
@@ -489,7 +577,7 @@ the `/api/v1/auth/sessions` and `/api/v1/auth/totp` endpoints:
out, or all but the current one. Pages and responses refer to a session by
`sessionID`, a digest of its token, so the list never carries a token.
`LastSeenAt` is refreshed by `markSeen` at most every five minutes, because
each refresh rewrites the state document.
each refresh is a database write.
- *Two-factor authentication* turns on with nothing more than the session,
since it only adds a check. Replacing or removing an active factor asks
for the password and a current code: before this, `totp/enroll` reset the
@@ -527,16 +615,63 @@ database, encrypts it with the password from the form, stores it as version
state update fails, the blob is removed again, so a half-created vault cannot
be left behind.
## Security hardening (0.12)
An audit of every part — server authentication, the data plane and its
cryptography, both clients, and the deployment — found the problems below,
and all of the critical, high and medium ones were fixed with a regression
test each. The authentication changes are described under
[Proving who someone is](#proving-who-someone-is) and
[Setting up OIDC](#setting-up-oidc).
| Area | Was | Now |
| --- | --- | --- |
| Single sign-on | one click on a link approved an attacker's device | an explicit Approve in the browser that started it; state cookie; PKCE |
| Single sign-on | an identity named `admin` claimed the admin account | no linking by username; verified email only when allowed; never an administrator |
| Passwords | no attempt limit; each guess held the global write lock | per-address and per-account limits; hashing outside the lock |
| TOTP | a code could be replayed for its 90-second window | each code is accepted once |
| Merge vaults | a writer could push a header asking for 2⁶² AES rounds or 2 GiB of Argon2 memory and stop the server, or a gzip bomb of nested XML that overflowed the stack | KDF, payload and XML bounds in `internal/kdbx/limits.go`; merge-vault pushes are checked (`kdbx.Inspect`) before they are stored; at most two merges at a time |
| Several nodes | a stale node's push could overwrite a version another node had committed | the version is written in the same transaction as its metadata, after the generation lock |
| Tokens | `?access_token=` worked on every endpoint, and the clients put it in URLs that reached logs | only on the event stream, for old clients; current clients send the header |
| Event streams | a revoked device kept receiving events | the session is re-checked at every keepalive; disabled users get no events |
| Web pages | no CSP, framing allowed, cacheable, login CSRF | CSP, `frame-ancestors 'none'`, `no-store`, HSTS behind https; the sign-in form needs a pre-session token |
| Enumeration | admin endpoints said 403 for someone else's database, 400 for none | 404 for both; group administrators no longer see other people's email, sign-in methods or 2FA |
| Desktop client | the server chose what the desktop opened at sign-in (any URL, `file:`, a network share) | only the server's own `/device` page |
| Desktop client | a database whose name differed in case, or matched a file left by a lost database, adopted and uploaded another vault | names compare case-insensitively; files of lost databases stay retired; renames never replace a file; reserved suffixes and Windows device names are avoided |
| Clients | a save made while a pull ran could be overwritten without a trace; Android could read half a save | the file is re-hashed before it is replaced; Android leaves files open for writing alone |
| Clients | `http://` servers accepted silently; unbounded reads from the server; markup and options in notifications; control characters printed to the terminal | `https` required except for loopback (`allow_insecure_http`); bounded reads; `--` and escaping; control characters stripped |
| Deployment | the `.deb` service could not create its master key under `ProtectSystem=strict`, so a fresh install did not start | the package creates the key; more systemd sandboxing |
| Deployment | nginx: host TLS defaults, no HSTS, 8 GiB uploads; CI ran as root on any branch | TLS 1.2+ and modern ciphers, HSTS, 64 MiB; CI on `main` only |
Left for later, deliberately:
- **Rollback of database rows.** Someone with write access to the database
itself can replay an older sealed row (a revoked session, a user before a
demotion). A sealed digest over every row, checked at load, would catch
it; until then, write access to the database is as sensitive as the
master key.
- **The master key also keys the subkey derivation and the vault-key
wrapping directly.** Not exploitable, but cleaner as separate derived keys;
changing it means re-wrapping every stored vault key.
- **A cap on event streams per user**, and pinning the container's base
image by digest.
- **The Kubernetes NodePort** (32970) reaches the server without nginx,
Vouch or TLS. Make sure only nginx can reach it.
- **KDBX magic is not required of pushes to key-less databases.** The server
treats their bytes as opaque by design.
## Decisions worth knowing
**No third-party Go dependencies in the server, no cgo outside the macOS
tray.** The server builds with `CGO_ENABLED=0` from the standard library
alone: SSE instead of a WebSocket library, `crypto/pbkdf2` (new in Go 1.24)
**No cgo, and as few dependencies as the job allows.** The server builds
with `CGO_ENABLED=0`: SSE instead of a WebSocket library, `crypto/pbkdf2`
for account passwords, hand-written ChaCha20, BLAKE2b and Argon2 for the
KDBX format, 60 lines of HMAC for TOTP, `html/template` for the pages. There
is no dependency tree to audit in the program that holds the vault keys;
`go list -deps ./cmd/keeporgd` shows only the standard library and this
module, and the container builds without fetching anything.
KDBX format, 60 lines of HMAC for TOTP, `html/template` for the pages. Its
only third-party code is the two database drivers, both pure Go:
`modernc.org/sqlite`, SQLite translated to Go, and `jackc/pgx` for
PostgreSQL. The standard library has no SQL driver, and writing either
would be a project of its own. Keeping them pure Go keeps static binaries,
cross-compilation and the `scratch` container; a cgo driver such as
SQLCipher was rejected for exactly that.
The desktop client takes exactly one, `fyne.io/systray` (with
`godbus/dbus` and `golang.org/x/sys` under it), because a tray icon cannot
@@ -550,13 +685,34 @@ darwin client with cgo when it is built on a Mac; built elsewhere it keeps
every command but `tray` (`tray_nocgo.go`). The client only moves
ciphertext, so the trade is in the program that never sees a key.
**A JSON document instead of SQLite.** The metadata is hundreds of records,
never thousands: users, groups, grants. One writer with atomic rename gives
durability and consistency without cgo or a migration story, and an operator
can read and back up the whole server state with `cat`. The cost is a single
node and a full rewrite per mutation. If this ever needs several servers or
tens of thousands of users, replace `internal/store` — the rest of the code
only touches it through `Read`, `Update` and the ACL functions.
**A database, held in memory, sealed row by row.** Until 0.11 the server
kept one JSON document and a directory of KDBX files. Enterprise use needs a
real database — managed backups, replication, a PostgreSQL the operations
team already runs — and data that is encrypted at rest under a key the
database never sees. Three choices follow from that:
- *The state stays in memory.* The metadata is hundreds of records, never
thousands, and every request makes access decisions over it. Loading it
once and answering from memory kept `Read`, `Update` and every ACL function
unchanged, so the rest of the server did not move. The database is
durable storage, not a query engine.
- *Encryption is in the application, not the engine.* Sealing each row with
the master key works the same on SQLite and PostgreSQL and protects dumps
and backups, which disk encryption and PostgreSQL's own options do not.
The price is that nothing is queryable in SQL — which, for a server whose
rows are bearer tokens and password verifiers, is the point.
- *Writes are diffs.* `Update` compares each row's plaintext digest with the
last one stored and writes only what changed, so a session touch is one
row, not the whole state.
Several processes may share one PostgreSQL database. A write locks the
`generation` row (`SELECT … FOR UPDATE`), and if another process has
committed since this one loaded, the state is reloaded and the mutation
applied again on top. A watcher reloads within two seconds of another
process's write, so a session revoked through one process is refused by all
of them. What is not shared yet is the event stream: a client connected to
one process hears about pushes made through that process only, and picks up
the rest at its next periodic rescan.
**A mirror directory instead of FUSE.** The requirement is that a KeePass
application sees a regular `.kdbx` file. A real file in a real directory is
@@ -598,7 +754,7 @@ keeporgd: created the first administrator: username "admin" password "13e4…"
```
Read it with `journalctl -u keeporgd`, or `podman logs <container>`. It is
never written anywhere else: the state file holds only a PBKDF2 verifier.
never written anywhere else: the database holds only a PBKDF2 verifier.
Set `KEEPORG_ADMIN_PASSWORD` before the first start to choose it instead, in
which case nothing is printed. The variable is read only when the server has
no accounts at all, so leaving it set afterwards does nothing.
@@ -615,10 +771,12 @@ sudo -u keeporg keeporgd -data /var/lib/keeporg -reset-admin
```
It prints a new one, clears any second factor on that account, and revokes
its sessions. The server must be stopped: a running one holds the state in
memory and would write over the change. Nothing else is touched, and no
vault is affected — account passwords and vault passwords are different
things.
its sessions. It needs the same `database` and master key as the server
(pass `-config /etc/keeporg/server.json`, or the same environment), and the
server may keep running: it writes to the database like any server process,
and the running one picks the change up within seconds. Nothing else is
touched, and no vault is affected — account passwords and vault passwords
are different things.
`/etc/keeporg/server.json`:
@@ -626,6 +784,7 @@ things.
{
"addr": ":8765",
"data_dir": "/var/lib/keeporg",
"database": "sqlite:/var/lib/keeporg/keeporg.db",
"external_url": "https://keeporg.example.com",
"session_ttl_hours": 2160,
"max_db_size_mb": 32,
@@ -649,17 +808,19 @@ the server, or set `tls_cert` and `tls_key`.
### Environment variables
The server reads four variables of its own: the public URL, and secrets or
first-run material. Everything else — address, data directory, OIDC,
The server reads six variables of its own: where its database and its
public address are, and secrets or first-run material. Everything else — address, data directory, OIDC,
limits — is set only in `server.json` or by flag, with flags winning; there
is no environment equivalent for those. An empty or whitespace-only value
counts as unset.
| Variable | Read by | Meaning |
| --- | --- | --- |
| `KEEPORG_MASTER_KEY` | every start | The master key itself, base64 of 32 bytes. Takes precedence over everything below. Never written anywhere. A value that does not decode has the same effect as a missing key file. |
| `KEEPORG_MASTER_KEY_FILE` | every start, when `KEEPORG_MASTER_KEY` is unset | Path to a file holding the key, as base64 text or exactly 32 raw bytes. The file must exist and is never created, since a new key would open none of the existing vaults. If it is missing or malformed the server logs `no master key, so this server cannot create or merge vaults` and runs without one: key-less strategies keep working, vault creation and merging are refused. Takes precedence over `master_key_file` / `-master-key-file`. |
| `KEEPORG_MASTER_KEY` | every start | The master key itself, base64 of 32 bytes. Takes precedence over everything below. Never written anywhere. A value that does not decode stops the server from starting. |
| `KEEPORG_MASTER_KEY_FILE` | every start, when `KEEPORG_MASTER_KEY` is unset | Path to a file holding the key, as base64 text or exactly 32 raw bytes. The file must exist and is never created, since a new key would open nothing already stored. If it is missing or malformed the server refuses to start. Takes precedence over `master_key_file` / `-master-key-file`. |
| `KEEPORG_DATABASE_URL` | every start, and `-reset-admin` | Where the data lives, as `database`: `postgres://user:password@host/db?sslmode=…` or `sqlite:<path>`. Overrides the file; `-database` overrides it. A PostgreSQL URL carries a password, so this belongs in a secret. The password is masked in the log. |
| `KEEPORG_EXTERNAL_URL` | every start | The base URL a browser reaches the server at, as `external_url`. Overrides the file; `-url` overrides it. Invite links, the device-approval page every client sign-in uses, and the OIDC redirect URI are built from it, and without any of the three it is `http://localhost<addr>`, which is wrong behind any proxy. |
| `KEEPORG_TRUSTED_PROXIES` | every start | Comma-separated addresses or CIDR ranges of reverse proxies whose `X-Forwarded-For` is believed, as `trusted_proxies`. Sign-in rate limits count per client address; behind a proxy without this, every client shares the proxy's budget. |
| `KEEPORG_ADMIN_PASSWORD` | first start on an empty state, and `-reset-admin` | The password for the built-in `admin` account instead of a generated one, which is then not logged. Ignored once any account exists, so leaving it set is harmless but pointless; unset it after the first start. |
With neither master-key variable set, the key comes from `master_key_file`
@@ -688,11 +849,27 @@ nginx `server_name`; the image's command passes neither `-config` nor
`KEEPORG_ADMIN_PASSWORD` is not set there, so the first start logs a
generated password once.
Back up `/var/lib/keeporg` **and** `/etc/keeporg/master.key`, separately.
The data directory is the whole server: `state.json` plus the blobs. The
master key is what turns the wrapped vault keys in it back into vault
passwords; without it a restored server can still serve, but can no longer
merge.
Back up the database **and** `/etc/keeporg/master.key`, separately. The
database is the whole server: with SQLite that is `keeporg.db` (copy it with
`sqlite3 keeporg.db ".backup …"` while the server runs, or stop it and copy
the file with its `-wal`); with PostgreSQL, whatever backs that database up
already. Every row in it is sealed under the master key, so a backup without
the key restores nothing.
### Moving from 0.10 or earlier
A server up to 0.10 kept `state.json` and `blobs/` in the data directory.
On the first start of 0.11 with an empty database, it imports both — every
account, session, grant and stored version — so clients and their tokens
carry on without signing in again. The old files are then renamed to
`state.json.imported-<time>` and `blobs.imported-<time>` and the log says
so. **Delete them once the server works**: `state.json` holds session
tokens in the clear. A database that already holds data is never imported
into again; the log warns if the old files are still lying around.
The import runs against whichever database is configured, so moving
straight to PostgreSQL is one start with `KEEPORG_DATABASE_URL` set and the
old data directory mounted.
### Setting up a team
@@ -724,7 +901,7 @@ keeporg db -id <db-id> -forget-password # the server can no longer read it
### Running it in a container
```sh
make docker # keeporgd:<version>, about 9 MB
make docker # keeporgd:<version>, about 25 MB
make docker-save # dist/keeporgd-<version>-oci.tar
```
@@ -735,10 +912,11 @@ uid 65532. There is no shell in it, so `HEALTHCHECK` runs the binary with
needs `--format docker` for the image to keep its healthcheck, which the
Makefile passes.
Two directories matter, and the whole point is that they are separate:
Two things matter, and the whole point is that they are separate:
- `/var/lib/keeporg` — state and blobs. Mount a volume.
- `/etc/keeporg` — `server.json` and, by default, the master key.
- the database — `/var/lib/keeporg/keeporg.db` on a volume, or PostgreSQL
through `KEEPORG_DATABASE_URL`;
- the master key — `/etc/keeporg` by default, or a provisioned secret.
**Give the master key somewhere durable.** The best option is a provisioned
secret plus `KEEPORG_MASTER_KEY_FILE` naming where it landed, which is the
@@ -747,17 +925,14 @@ convention compose, swarm and Kubernetes all follow:
The key is then neither in the process environment nor on a disk the
container can write, and the file may hold base64 or 32 raw bytes.
`KEEPORG_MASTER_KEY` and a volume on `/etc/keeporg` also work. A container
started with none of those mints a key
into its own layer and loses it on the next `run`. That is survivable but
annoying: the vaults still open for their users, and only merging stops. The
server says so at startup —
started with none of those mints a key into its own layer and loses it on
the next `run`, and the next start then refuses to run:
```
WARNING: 1 of 1 stored vault keys do not open with this master key.
start: database …: this master key does not open this database; start with the key it was created with
```
— and a merging push then answers `409 merge_failed` rather than corrupting
anything. Restore the original key and it resumes.
Nothing is overwritten. Restore the original key and it starts.
`packaging/docker/compose.yaml` is a worked example with `read_only`,
`cap_drop: ALL` and `no-new-privileges`; the server writes one directory and
@@ -807,13 +982,29 @@ Then sign and index on the repository host (`fdroid publish && fdroid update`).
```sh
make check # gofmt, go vet, go test ./...
make test-race # the same suite under the race detector
# the same suite on PostgreSQL: each test gets a fresh database there
KEEPORG_TEST_POSTGRES='postgres://postgres:secret@127.0.0.1:5432/postgres?sslmode=disable' go test ./...
```
Every test that touches storage goes through `sqldbtest.URL`, which hands
out a temporary SQLite file, or a new PostgreSQL database dropped afterwards
when `KEEPORG_TEST_POSTGRES` is set. The suite has passed on both.
What the suite actually covers:
- `internal/totp` against the RFC 6238 vectors.
- `internal/store` for grant inheritance, sibling isolation, admin
inheritance, cycle safety, update rollback and expiry.
inheritance, cycle safety, update rollback and expiry; that no row id or
body carries a username, email or token in the clear; that a change
rewrites only its own row; that two stores on one database see each
other's writes and re-apply a stale one on top; and that another master
key is refused.
- `internal/blob` for versioning, pruning, sealing, a version's row copied
over another failing to open, and importing the old on-disk layout.
- `internal/server` for importing a data directory from 0.10 (accounts,
sessions, versions; the old files set aside; no second import), and for
refusing to start without the master key or with another one.
- `internal/server` for every push verdict as a table, and an end-to-end HTTP
pass over login, stale push, forced push, idempotent push, read-only
grants, invite redemption and reuse, TOTP enrolment, device login and the
@@ -866,8 +1057,13 @@ what an arm64 Linux build host needs.
- **The DMG is per architecture.** A universal `.app` needs `lipo` over both
cgo builds, which `midi-webhook`'s Makefile shows how to do.
- **One server node.** The JSON store is single-writer by design. Horizontal
scale means replacing `internal/store`.
- **Several server nodes share a database, but not their event streams.**
Writes are serialized and every node sees every write within two seconds,
but a client hears another node's pushes only at its next rescan. A
PostgreSQL `LISTEN`/`NOTIFY` relay into `Hub` would close that.
- **The metadata lives in memory.** Fine for thousands of accounts; far
beyond that, the per-write clone and the full load at start would want
the store to become per-row.
- **Merging requires the server to hold the vault key.** That is inherent, not
an implementation gap: the alternative is a client-side merge, which would
mean the pushing client fetching the server copy, merging locally and