rcmd/keeporg
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Merge concurrent saves on the server, with vaults created from the web

Browse Source 
Why: the point of a sync server is that two people can work at once. Picking
a winner was all this one could do while it could not read a vault.

What, end to end:
- A vault is created at /web or with `keeporg vault -create`: the server
  builds an empty KDBX 4 file, encrypts it with the password from the form,
  and stores it as version 1. An existing database can adopt a stored key
  with `keeporg db -id <id> -store-password`, which is verified against the
  stored file before it is accepted.
- A stale push to a `merge` vault is no longer refused. The server opens the
  stored version and the pushed one in memory, folds them together, and
  stores the re-encrypted result. Clients push as usual and pull what comes
  back. Any failure answers 409 merge_failed and the client keeps its copy
  aside, so a failed merge still loses nothing.
- At rest: vault contents stay KDBX-encrypted as always. A merge vault's
  password is kept AES-256-GCM-wrapped under a master key that lives outside
  the data directory, bound to the vault id, never returned by the API. A
  stolen data directory is ciphertext; a stolen data directory plus the
  master key is not. The server runs without a master key and simply refuses
  to create or merge vaults.

This moves the server inside the trust boundary for vaults it can merge,
which is inherent to merging and is stated plainly in the README, the
handover notes and the web form itself. Key-less strategies are unchanged and
remain the default for a database created without a password.

Proved: two clients editing the same version are merged rather than refused,
the newer edit of one entry wins, a merge failure leaves the stored vault
untouched, `merge` is refused without a stored key, the wrapped key never
appears in state.json or the API, and the web interface rejects a form
without its CSRF token, withholds its cookie over plain http, and only offers
groups the user administers. Checked by hand against the real thing: a vault
created in a browser, edited by keepassxc-cli on two sides from the same
version, merged by the server, and both entries read back out with
keepassxc-cli. make check, make test-race, android lint and both .deb builds
are clean.
This commit is contained in:
root
2026-10-29 17:34:47 +01:00
parent 4d2f0b047b
commit a9308a3e89
33 changed files with 2168 additions and 88 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/HANDOVER.md
+205 -22
View File
@@ -5,9 +5,21 @@ server holds many KDBX files and arbitrates who wrote last; clients present
each assigned database as an ordinary `.kdbx` file that any KeePass
application can open, and synchronize it on every save.
The server never opens a database. It moves encrypted bytes and metadata
about them. Master passwords and key files stay on the clients, so a
compromised server leaks the ciphertext and the group structure, nothing else.
There are two kinds of database here, and the difference is the whole
security model:
- **A key-less database** is one the server only ever moves. It cannot be
opened here, so a compromised server leaks ciphertext and the group
structure and nothing else. Conflicts are settled by picking a winner.
- **A merge vault** is one the server can open, because it keeps that
vault's password. That is the price of folding two people's edits
together: merging means decrypting. Whoever controls the server and its
master key can read every merge vault on it.
Both are first-class. A deployment that wants no server-side key uses
`server-wins`, `client-wins` or `manual` and loses nothing but the merge. A
deployment that wants merging accepts that the server is now part of its
trust boundary, and should treat the host accordingly.
## Repository map
@@ -16,7 +28,8 @@ cmd/keeporgd/ server entry point: flags, config file, bootstrap
cmd/keeporg/ desktop client: login, run, sync, status, administration
internal/store/ domain model, JSON metadata store, ACL resolution
internal/blob/ versioned KDBX payload storage on disk
internal/server/ HTTP surface: auth, OIDC, sync, admin, SSE hub, pages
internal/kdbx/ KDBX 4 codec and the merge engine, with its own crypto
internal/server/ HTTP surface: auth, OIDC, sync, admin, SSE hub, web, pages
internal/client/ shared client engine: API wrapper, merge policy, mirror
internal/secret/ password verifiers, tokens, identifiers
internal/totp/ RFC 6238 second factor
@@ -34,9 +47,12 @@ Makefile the only entry point for building and publishing
One process, one data directory, no database engine:
- `state.json` holds users, groups, memberships, databases, grants, invites,
sessions and pending device logins.
sessions and pending device logins. For a merge vault it also holds that
vault's password, encrypted under the server's master key.
- `blobs/<database-id>/<version>.kdbx` holds the payloads, keeping the last
20 versions of each database.
- The master key itself lives **outside** the data directory, so a copy of
the data directory alone yields ciphertext.
`internal/store.Store` is the only writer of `state.json`. Every mutation
runs through `Update(func(*State) error)`, which applies the change to a
@@ -74,6 +90,33 @@ event stream open so colleagues' changes arrive without opening the app.
`Mirror.decide` is a transcription of the Go decision table. **Keep the two in
step**: a change to one belongs in the other in the same commit.
### The KDBX codec (`internal/kdbx`)
A real KeePass 2 implementation, because merging cannot be done on
guesswork. It reads and writes KDBX 4.0 and 4.1 with AES-256-CBC or ChaCha20
payloads, AES-KDF or Argon2d/Argon2id derivation, and the ChaCha20 inner
stream that protects individual values. ChaCha20, BLAKE2b and Argon2 are
implemented in the package: the standard library has none of them, and this
repository carries no third-party dependencies. Each is checked against its
RFC's test vectors.
Two decisions in there are worth keeping:
**The XML payload is a generic node tree, not Go structs.** A merge server
that dropped elements it did not model would silently destroy data written by
a newer KeePass. Everything unrecognised round-trips untouched, and the merge
logic reaches only into the paths it understands: UUID, Times, String,
History, Binary, DeletedObjects, Meta.
**Anything unsupported is reported, never guessed.** A KDBX 3 file, a Twofish
payload or a Salsa20 inner stream comes back as `*kdbx.ErrUnsupported`, and
the server then falls back to a non-merging strategy rather than mangling the
file.
The format is verified against the real thing: the test suite has
keepassxc-cli read, decrypt and re-save vaults this code writes, and has this
code read what KeePassXC saved.
## Access control
Groups form one tree. Two rules decide everything:
@@ -122,19 +165,91 @@ Set per database, changed by an administrator of the owning group
| Strategy | Server behaviour on a stale push | Client behaviour |
| --- | --- | --- |
| `server-wins` (default) | rejects with `409 conflict` | keeps the local save as `<name>.conflict-<timestamp>.kdbx`, then writes the server copy |
| `merge` | opens both copies in memory, folds them together, stores the result as a new version | pushes as usual, then pulls the merged result |
| `server-wins` | rejects with `409 conflict` | keeps the local save as `<name>.conflict-<timestamp>.kdbx`, then writes the server copy |
| `client-wins` | accepts a retry carrying `If-Match: *` | retries with force; the server copy is overwritten and kept in history |
| `manual` | rejects, and refuses force | writes the server copy as `<name>.server-v<n>.kdbx` beside the file, touches nothing, reports a conflict once |
`merge` is the default for a vault created through the web interface;
`server-wins` is the default for a database created without a password,
because the server has no key for it.
A losing save is never discarded: it is always written somewhere the user can
find it. A read-only user who edits their copy gets the same treatment, with
the server copy restored.
**There is no entry-level merge.** The server cannot perform one, because it
cannot decrypt a database. A true three-way merge belongs in a client that
already holds the master password; the strategy field is the hook for it, and
`manual` is the honest behaviour until then. KeePassXC's own "Merge from
database" on the two files is the current answer.
### What merging actually does
`kdbx.Merge` follows KeePass's own synchronization rules. Objects are matched
by UUID across the two copies:
- An object only one side has is added, unless the other side deleted it
more recently than it was last modified.
- An object both sides changed is resolved by `Times/LastModificationTime`.
The newer version wins and **the loser is filed in the entry's history**,
so a merge never destroys an edit, it demotes one.
- A deletion applies only when it is newer than the other side's edit.
Otherwise the edit resurrects the object and the stale deletion record is
dropped.
- Groups keep their nesting; an object moves when the other side moved it
more recently, by `Times/LocationChanged`.
- Database settings follow `Meta/SettingsChanged`; custom data and icons are
unioned by key.
- Attachments are remapped from the incoming pool into the target's. An entry
that kept its old index would silently gain the wrong file.
The ordering of objects inside a group is not merged, and neither does
KeePass reconcile it.
Where it runs: `internal/server/merge.go` decrypts the stored version and the
pushed one, merges, re-encrypts, and stores the result as a new version. The
plaintext exists only as two `kdbx.Database` values in memory for the
duration of one request; nothing is ever written to disk in the clear. If any
step fails — most often because a client changed the vault's own password, so
the stored key no longer opens it — the push is answered with
`409 merge_failed` and the client falls back to keeping its copy aside. A
failed merge never loses an edit.
**The cost.** A merge runs the vault's key derivation three times: the stored
version, the pushed version, and the re-encryption, each with a different
salt. At the default 32 MiB / 2 iterations that is roughly 150 ms and 100 MB
of transient memory per merging push. `vault_kdf` in the configuration is
the dial; a transformed-key cache would remove most of it and has not been
written.
## Key custody and encryption at rest
Three different things are encrypted here, and they are worth keeping apart:
1. **The vault contents.** Always encrypted, by KDBX itself: AES-256 under a
key derived with Argon2d from the vault password. This is true on the
server's disk, on every client's disk and on the wire. The server stores
exactly the file a KeePass application would.
2. **The vault password**, for merge vaults only. Encrypted with AES-256-GCM
under the server's master key, with the vault's own id as additional
authenticated data, so a wrapped secret moved to another vault's record
fails to open instead of unlocking the wrong database. It is never
returned by the API, never logged, and never leaves the process.
3. **Account passwords.** Not stored at all; only PBKDF2-SHA256 verifiers.
The master key comes from `KEEPORG_MASTER_KEY` (base64) or from
`master_key_file`, which defaults to `/etc/keeporg/master.key` and is created
with mode 0600 on first start. Without it the server still runs and still
serves every key-less strategy; only vault creation and merging are refused,
with a line in the log saying so.
What this buys and what it does not:
- A stolen copy of `/var/lib/keeporg` is useless on its own. The vaults are
ciphertext and the wrapped keys do not open without the master key.
- A stolen copy of the data directory **and** the master key opens every
merge vault. Keep them on different media, or feed the key in through
`KEEPORG_MASTER_KEY` from a secret manager or a systemd credential so it
never touches the disk at all.
- Backups must cover both, separately. A backup of the data directory with no
master key cannot be used to merge after a restore; the vaults still open
for their users, and the keys can be stored again with
`keeporg db -id <id> -store-password`.
## Authentication
@@ -159,12 +274,34 @@ userinfo endpoint, so there is no local JWT verification to get wrong. An
identity links to an existing account by `preferred_username` or `email`, or
provisions one when `oidc.auto_create` is set.
## The web interface
The server has one browsable page set, at `/web`, and it exists for the one
job a command line does badly: creating a vault and setting its password.
Sign in, see which vaults you can reach, create one. It never shows a vault's
contents — the server opens a vault only to merge it — and there is no
entry browsing, by design.
Browser sessions are the same session tokens the API uses, carried in a
cookie that is `HttpOnly`, `SameSite=Strict`, `Path=/`, and `Secure` whenever
`external_url` is https. Every form carries a per-session CSRF token that is
checked on each POST. The group selector only offers groups the signed-in
user administers, and a user who administers none is told so instead of being
shown a form that would fail.
Creating a vault is one transaction: the server builds an empty KDBX 4
database, encrypts it with the password from the form, stores it as version
1, and — for a merge vault — wraps the password under the master key. If the
state update fails, the blob is removed again, so a half-created vault cannot
be left behind.
## Decisions worth knowing
**No third-party Go dependencies, no cgo.** The whole server and desktop
client build with `CGO_ENABLED=0` from the standard library: SSE instead of a
WebSocket library, `crypto/pbkdf2` (new in Go 1.24) instead of an Argon2
package, 60 lines of HMAC for TOTP, `html/template` for the three pages.
WebSocket library, `crypto/pbkdf2` (new in Go 1.24) for account passwords,
hand-written ChaCha20, BLAKE2b and Argon2 for the KDBX format, 60 lines of
HMAC for TOTP, `html/template` for the pages.
Cross-compiling to five platforms is then one `go build` each, and there is
no dependency tree to audit for a program that guards passwords.
@@ -218,6 +355,8 @@ an `everyone` group, and logs the generated password once. Set
"external_url": "https://keeporg.example.com",
"session_ttl_hours": 2160,
"max_db_size_mb": 32,
"master_key_file": "/etc/keeporg/master.key",
"vault_kdf": { "memory_mb": 32, "iterations": 2, "lanes": 2 },
"oidc": {
"issuer": "https://id.example.com/realms/main",
"client_id": "keeporg",
@@ -233,23 +372,37 @@ an `everyone` group, and logs the generated password once. Set
pages and the OIDC redirect URI are built from it. Terminate TLS in front of
the server, or set `tls_cert` and `tls_key`.
Back up `/var/lib/keeporg`. It is the whole server: `state.json` plus the
blobs. There is nothing else to dump.
Back up `/var/lib/keeporg` **and** `/etc/keeporg/master.key`, separately.
The data directory is the whole server: `state.json` plus the blobs. The
master key is what turns the wrapped vault keys in it back into vault
passwords; without it a restored server can still serve, but can no longer
merge.
### Setting up a team
```sh
keeporg login -server https://keeporg.example.com -user admin
keeporg group -create engineering -parent <everyone-id>
keeporg db -create "Engineering" -group <engineering-id>
keeporg vault -create "Engineering" -group <engineering-id> # prompts for the vault password
keeporg invite -group <engineering-id> -role member -uses 5 -hours 72
keeporg grant -group <readers-id> -db <db-id> -perm read
keeporg db -id <db-id> -strategy manual
```
A new database has no bytes until someone saves one. Create the KDBX in your
KeePass application under the exact name the server shows, inside the mirror
directory; the client adopts it and pushes it as version 1.
Or create the vault at `https://keeporg.example.com/web`, which is the same
operation with a form around it.
`keeporg vault` creates the KDBX file itself, so the vault has content from
the start: it appears in every member's mirror directory and opens with the
password that was set. `keeporg db -create` is the other path — it registers
a name with no bytes, and the first client to save a KDBX of that name in its
mirror directory adopts it as version 1. A database created that way has no
key on the server, so it cannot merge until one is stored:
```sh
keeporg db -id <db-id> -store-password # checked against the stored file
keeporg db -id <db-id> -strategy merge
keeporg db -id <db-id> -forget-password # the server can no longer read it
```
## Versioning and publishing
@@ -294,6 +447,21 @@ What the suite actually covers:
pass over login, stale push, forced push, idempotent push, read-only
grants, invite redemption and reuse, TOTP enrolment, device login and the
event stream.
- `internal/kdbx` for ChaCha20 (RFC 8439), BLAKE2b (RFC 7693) and all three
Argon2 variants (RFC 9106) against their published vectors; round trips
that preserve unknown elements; rejection of a wrong password, a flipped
payload byte and a flipped header byte; and twelve merge cases covering
both conflict directions with history, deletion, resurrection, nesting,
relocation, attachment remapping, idempotence and order independence.
Three tests drive keepassxc-cli: it reads what this code writes, this code
reads what it writes back, and it decrypts an entry that reached a vault
only through a merge.
- `internal/server` additionally for vault creation, two clients editing the
same version being merged rather than refused, the newer edit of one entry
winning, a merge failure falling back to a conflict with the stored vault
untouched, the refusal to switch to `merge` without a stored key, and the
web interface: cookie hardening, CSRF rejection, group scoping and that a
vault created from the form opens with its password.
- `internal/client` for the decision table, and for the mirror driven against
a real in-process server: adopting a local file, following a remote push,
server-wins, client-wins, manual, read-only restore, server-side rename,
@@ -308,8 +476,23 @@ what an arm64 Linux build host needs.
- **One server node.** The JSON store is single-writer by design. Horizontal
scale means replacing `internal/store`.
- **No entry-level merge.** See the strategies section. The pluggable place
is the client, which holds the key.
- **Merging requires the server to hold the vault key.** That is inherent, not
an implementation gap: the alternative is a client-side merge, which would
mean the pushing client fetching the server copy, merging locally and
pushing the result. That is a worthwhile future option and would let a
deployment keep the key off the server entirely.
- **KDBX 3 files cannot be merged.** They are reported as unsupported, not
mangled; re-save such a file as KDBX 4 in KeePassXC, or keep it on a
key-less strategy. Reading KDBX 3 is a contained addition: a different
payload layout and the Salsa20 inner stream.
- **A merge vault must keep one password.** If a client changes the vault's
own password, the server's stored key stops working and pushes come back as
`merge_failed` until an administrator stores the new one. The server does
not detect the change in advance.
- **Key files and hardware keys are not supported for merge vaults**, because
the server would have to hold those too. They work fine on any key-less
strategy, where the server never opens the file.
- **No transformed-key cache.** Each merge pays the Argon2 cost three times.
- **The Android app has never run on a device.** It compiles clean, lint
reports no issues and the DEX carries every class the manifest names, but
no emulator or phone was available here. The first real run is where to
docs/PROTOCOL.md
+40 -5
View File
@@ -5,8 +5,10 @@ Every `/api/` endpoint takes and returns JSON and needs
endpoints. Errors carry `{"code": "...", "error": "..."}`; `code` is stable
and meant to be matched on, `error` is for humans.
Three endpoints serve HTML, because a browser is the right place for them:
`/device`, `/invite/{token}` and `/oidc/callback`.
HTML is served at `/web` (sign in, list vaults, create one) and at `/device`,
`/invite/{token}` and `/oidc/callback`. The web pages use the same session
tokens in a cookie that is `HttpOnly`, `SameSite=Strict` and `Secure` behind
https, and every form carries a per-session CSRF token.
## Authentication
@@ -40,7 +42,9 @@ A device code is single use: the first successful poll consumes it.
| `If-Match` missing | `428 precondition_required` | – |
| version matches, bytes differ | `200` | `{version, sha256, unchanged: false}` |
| version matches, bytes identical | `200` | `{version, sha256, unchanged: true}` |
| version stale | `409 conflict` | `{current_version, merge_strategy}`, plus `ETag` and `X-KeepOrg-Strategy` |
| version stale, strategy is `merge` | `200` | `{version, sha256, merged: true, merge_summary, based_on}` — the two copies were folded together and the result stored as a new version |
| version stale, strategy is `merge`, but merging failed | `409 merge_failed` | `{error, current_version, merge_strategy}`; the stored vault is untouched |
| version stale, any other strategy | `409 conflict` | `{current_version, merge_strategy}`, plus `ETag` and `X-KeepOrg-Strategy` |
| `If-Match: *`, strategy is `client-wins` | `200` | the server copy is replaced |
| `If-Match: *`, any other strategy | `409 force_not_allowed` | – |
| read-only access | `403 forbidden` | – |
@@ -50,6 +54,10 @@ A device code is single use: the first successful poll consumes it.
`X-KeepOrg-SHA256` on a `PUT` is optional; when present it must match the
body or the request is rejected with `400 checksum_mismatch`.
After any accepted push, compare the `sha256` in the answer with what you
sent. If they differ the server stored something else — a merge — and the
local copy must be replaced by a fresh `GET`.
### Events
```
@@ -65,6 +73,31 @@ rename, strategy change or deletion. A comment line (`: keepalive`) arrives
every 25 seconds. Reconnect on any error and reconcile fully; events are
notifications, never state.
## Vaults
A vault is a database the server created and can open. Creating one is the
only way the server comes to hold a KDBX key, and the only strategy that
needs one is `merge`.
| Method | Path | Body | Needs |
| --- | --- | --- | --- |
| POST | `/api/v1/vaults` | `{name, owner_group_id, merge_strategy, password}` | group admin of the owner |
| POST | `/api/v1/databases/{id}/key` | `{password}` or `{forget: true}` | database admin |
`POST /api/v1/vaults` builds an empty KDBX 4 database, encrypts it with
`password`, stores it as version 1 and answers with the database view. The
password must be at least 12 characters. For `merge_strategy: "merge"`
(the default here) the password is kept, encrypted under the server's master
key; for every other strategy it is used once and forgotten. A server with no
master key refuses with `400`.
`POST /api/v1/databases/{id}/key` stores the password of a database that
already has content, after checking that it opens the stored version, which
is how an uploaded database becomes mergeable. `{"forget": true}` removes it;
that is refused while the strategy is still `merge`.
The wrapped key is never returned by any endpoint.
## Administration
Permission needed is in the last column. "Group admin" means administering
@@ -87,5 +120,7 @@ the group named or any group above it.
| GET | `/api/v1/users` | – | yourself, plus members of groups you administer |
| GET | `/healthz` | – | unauthenticated liveness |
`merge_strategy` is one of `server-wins`, `client-wins`, `manual`. `perm` is
one of `read`, `write`, `admin`. `role` is `member` or `admin`.
`merge_strategy` is one of `merge`, `server-wins`, `client-wins`, `manual`.
Switching to `merge` requires a stored key and is otherwise refused with
`400`. `perm` is one of `read`, `write`, `admin`. `role` is `member` or
`admin`.