EverestLinux/libglacier-ng
2
0 Fork 0
Code Issues 1 Pull Requests Actions Packages Projects 1 Releases Wiki Activity

10/14

Browse Source
This commit is contained in:
lw-everestlinux committed 2026-10-14 00:50:56 -04:00
1 parent ee4bcc3b77
commit ae54f74070
53 files changed
+8916 -1424
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

REPO_DESIGN.txt
+446
View File
@@ -0,0 +1,446 @@
+--------------------------------------------+
| Remote repositories and signing |
+--------------------------------------------+
STATUS: DESIGN, 2026-10-06. ROADMAP 3.1. Steps 1 (Ed25519, signify
formats, gsign), 2 (keyring, the local base rule, gbuild -k,
bootstrap carrying .sig, Everest's key) and 3 (grepo, the index,
refresh with -y) of section 9 DONE 2026-10-06, step 4 (-f, -u,
dependency closure, the cache) DONE 2026-10-07; sections 7, 5 and 6
say how they came out. Step 5 next. See CLAUDE.md.
Sections 1 and 10 are decided; the only open item is 10a's TLS library.
Adds AUTHENTICITY to glacier: SECURITY_DESIGN.txt section 1 deferred it
to "a future phase, tied to remote-repo support". This is that phase.
It also unblocks gpkg/syspkg -f (ROADMAP 2.2), -u (2.1), automatic
dependency closure (2.3) and glacier-bootstrap -r (GLACIER_BOOTSTRAP
section 5), and closes Finding 1 of SECURITY_REVIEW_2026-07-16.md (a
manifest that says PKG_REPO = "base" gets linked into /usr).
Companion documents: SECURITY_DESIGN.txt (integrity, refusal matrix),
PACKAGE_SCOPES.txt (repo routing), GLACIER_BOOTSTRAP.txt (pins).
+--------------------------------------------+
| 1. Decisions (the user, 2026-10-06) |
+--------------------------------------------+
a. Signatures: Ed25519, vendored (Monocypher's Ed25519 variant, which
brings its own SHA-512), compiled into the private
libglacier_verify.so next to sha256. Key and signature files are
signify-compatible, so OpenBSD's signify can make and check them
too. No new base dependency.
b. What is signed: one INDEX per repository (repo.idx + repo.idx.sig),
listing every package it serves with its sha256. Packages are
verified through the hash in the signed index, i.e. through the
existing sha256 path. The index has a serial; a client refuses an
index older than the last one it accepted (rollback).
c. Local base packages: once signing exists, `syspkg -l FILE` of a
package whose manifest says base requires a signature by a key
trusted for base. extra/community local installs stay
sidecar-only (integrity). Manual installation keeps working: a
builder makes a key and root adds it to the keyring.
d. Users may add their own repositories and keys, for their own scope
only (gpkg). Never read by syspkg.
+--------------------------------------------+
| 2. Keys and signatures (proposal) |
+--------------------------------------------+
signify's formats, byte for byte (two lines: "untrusted comment: ..."
then base64):
public key "Ed" | keynum[8] | pubkey[32]
signature "Ed" | keynum[8] | sig[64]
secret key "Ed" | "BK" | rounds[4] | salt[16] | checksum[8] |
keynum[8] | seckey[64]
- keynum is a random 8-byte key id; a signature names the key that
made it, so verification picks the key by keynum and never tries
every key in the keyring.
- Signing is pure Ed25519 over the WHOLE message, as signify does. So
we never sign a .gpkg directly (it would have to be read into memory
whole). What gets signed is small:
- a repository's repo.idx;
- a local package's SIDECAR (NAME-VER.gpkg.sha256, the one line
"sha256:<hex>"), as NAME-VER.gpkg.sha256.sig. Checking the
sidecar against the file is the existing path; the signature
makes the sidecar authentic. `signify -V -p KEY -m X.sha256`
checks the same file.
- Secret keys: glacier's tools read and write only UNENCRYPTED secret
keys (rounds = 0, what `signify -G -n` makes): passphrase support
needs bcrypt_pbkdf, which is more vendored crypto for a build-host
convenience. A passphrase-protected key works by signing with
signify itself (10b).
New tool `gsign` (C, built with gsha256sum; a build/maintainer tool,
later in "glacier-tools", never in base):
gsign -G -p KEY.pub -s KEY.sec [-c COMMENT] make a keypair
gsign -S -s KEY.sec -m FILE [-x SIGFILE] sign (default FILE.sig)
gsign -V -p KEY.pub -m FILE [-x SIGFILE] verify
gbuild gains `-k KEY.sec`: sign the sidecar it writes.
+--------------------------------------------+
| 3. Keyring and repository configuration |
+--------------------------------------------+
System (root-owned; syspkg applies its trust checks to these files and
every ancestor, like its own lib/):
/etc/glacier/keys/NAME.pub
/etc/glacier/repos.conf
repos.conf (libconfig, like glacier.conf):
repos = (
{ name = "base"; url = "https://pkg.example.org/base";
keys = [ "everest-2026" ]; },
{ name = "extra"; url = "https://pkg.example.org/extra";
keys = [ "everest-2026" ]; }
);
- A key is trusted ONLY for the repos that list it. That binding is
what makes "base" mean something: the base index must be signed by a
key listed for base, and a local base package's sidecar likewise.
Several keys per repo allow rotation (old and new side by side).
- Order = priority, when one name is served by several repos.
- name follows the gl_valid_pkg_ident rules. url is https:// or
file:// (local mirrors, tests); http:// is refused.
- url is optional: an entry without one only says which keys are
trusted for that repo, for a system installed and kept up by hand
from signed local files (section 7).
- Missing file = no remote repositories; -f/-u then refuse in one
line. No defaults, as with KEEP_OLD.
User (gpkg only):
~/.config/glacier/repos.conf ($XDG_CONFIG_HOME)
~/.config/glacier/keys/NAME.pub
- Merged AFTER the system list (lower priority).
- A user repo named "base", or named like a system repo, is refused.
- System keys are trusted for system repos only; a user's keys for the
user's repos only.
+--------------------------------------------+
| 4. The index |
+--------------------------------------------+
URL/repo.idx and URL/repo.idx.sig. libconfig, parsed only AFTER its
signature checks:
REPO = "base";
SERIAL = 2026100601L; # strictly increases with each publish
VALID_UNTIL = 1761955200L; # unix time; required (10c)
PKGS = (
{ NAME = "musl"; VER = "1.2.5"; FILE = "musl-1.2.5.gpkg";
SHA256 = "sha256:3a5f..."; SIZE = 812345L;
DEPS = (); CONFS = (); },
...
);
- REPO must equal the configured name the index was fetched for.
- Every version a repo serves is listed (so a pinned or older version
can still be fetched by -f NAME=VER later).
- DEPS/CONFS repeat the manifest's, so dependency closure can be
planned before downloading anything. After download, the manifest
must match the entry (NAME, VER, PKG_REPO = REPO, DEPS, CONFS):
a mismatch refuses the package. That is the Finding 1 rule for
remote packages: a package is "base" because the base index lists
it AND it says so, never because it says so.
- FILE is a bare file name (gl_valid_pkg_ident rules plus ".gpkg"):
no paths, no URLs.
New tool `grepo` (maintainer side, glacier-tools): `grepo -s KEY.sec
-r REPO DIR` reads every NAME-VER.gpkg in DIR (manifests, hashes),
writes DIR/repo.idx with SERIAL = the previous one + 1 (or the date
form), and signs it. The server is then any static file host.
+--------------------------------------------+
| 5. Client state and fetching |
+--------------------------------------------+
Inside each scope's base directory (one more entry beside index, store,
...; same ownership and modes as the rest):
<base>/repos/REPO.idx, REPO.idx.sig the last ACCEPTED index
<base>/cache/REPO/FILE downloaded packages
Refresh (every -f / -u; indexes are small; 10d):
1. curl --proto =https --proto-redir =https --fail (or a file:// copy)
URL/repo.idx and .sig into a temporary file in <base>/repos/.
2. Verify the signature with a key listed for that repo (by keynum).
3. Parse. REPO must match. SERIAL must be > the stored one, or equal
with identical bytes. VALID_UNTIL must be in the future.
4. rename() over the stored copy (never written in place: bug 25).
Any failure keeps the old index and refuses the operation in one line:
[x] Index for extra is older than the one already accepted.
Fetch: download FILE into the cache under a temporary name, hash it
while downloading, rename into place only if the hash matches the
index, then write the sidecar FROM THE INDEX and hand the cached file
to the existing install path (which re-hashes it, as today). The cache
is a download optimization, not a trusted store: a cache hit is hashed
again.
As implemented (2026-10-06, step 3; the user chose a refresh flag,
the date-form SERIAL and grepo in C):
- grepo is C (src/grepo.c, libglacier-ng; built from source like
gsign, installed to bin; not in base): `grepo -s KEY.sec -r REPO
[-n SERIAL] [-e DAYS] DIR`. SERIAL = max(previous + 1, today (UTC)
* 100 + 1), e.g. 2026100601; -n must still be above the previous
one. VALID_UNTIL = now + DAYS (default 30, 1..3650). Packages are
sorted by file name; each must be a regular file, its manifest must
say REPO and its name must be NAME-VER.gpkg (VER as the store names
versions, "%d.%d.%d"). An existing DIR/repo.idx must be a
well-formed index of REPO (its SERIAL is the previous one). It is
written to a mkstemp file, signed, and both renamed into place, the
.sig first.
- The index format is section 4's, with DEPS and CONFS as lists of
{ NAME = "x"; VER = "1.0.0"; } (the manifest's PKG_DEPS shape).
gl_index_parse (pkg.c) takes the BYTES the signature was checked
over, never a path, and is strict: exactly REPO, SERIAL > 0,
VALID_UNTIL > 0, PKGS; each package exactly NAME, VER, FILE, SHA256,
SIZE >= 0, DEPS, CONFS; FILE must be exactly "NAME-VER.gpkg"; SHA256
"sha256:" + 64 lowercase hex; no NAME + VER twice. A NUL or an '@'
anywhere is refused before libconfig sees it ('@include').
GL_INDEX_ERR_REPO when REPO is not the repo it was fetched for.
- `syspkg -y` / `gpkg -y` (--refresh) refresh every repos.conf entry
with a url and stop (decision 10d still holds for -f/-u). gpkg
reads the SYSTEM repos.conf and keyring until step 6 (user repos),
and stores into the user's base. No repos.conf, or no entry with a
url: `[x] No remote repositories are configured.` (exit 1).
- <base>/repos is made by gl_scope_state_dir (transaction.c): the
base must be the scope's; repos is created with the scope's mode
(0755 root / 0700 uid) and refused unless it is a real directory
with exactly that owner and mode. Files are 0644 / 0600. -y holds
the scope's lock (gl_begin_transaction on a live context); under
-R the indexes go under the install root, checked with the install
root's OWN keyring (DIR/etc/glacier; changed 2026-10-07, see
"Keyrings under -R" below).
- file:// is read directly (absolute paths only); https:// goes
through `curl -sf --proto =https --proto-redir =https`. 64 MiB at
most (GL_SIGN_MAX_MSG). Every repo is tried; exit 1 if any failed.
- One line per repo:
[✓] Index for base: serial 2026100601, 11 packages.
[i] Index for base is up to date: serial 2026100601.
[x] Cannot fetch the index for base. (a missing .sig too)
[x] Index for base is not signed by a key trusted for base.
[x] Malformed index for base.
[x] Index fetched for base is the index of another repository.
[x] Index for base is older than the one already accepted.
[x] Index for base has the serial of the one already accepted, but other contents.
[x] Index for base has expired.
[x] Malformed /glacier/sys/repos/base.idx. (left for the admin)
- The accepted copy is trusted by where it lives (root's or the
user's private repos dir): it is parsed for its SERIAL, not
re-verified.
- Test: glacier-ng/tests/repo_alpine.sh (103 checks; 18 mutants, 12 C
and 6 Lua, all caught).
+--------------------------------------------+
| 6. -f, -u and dependency closure |
+--------------------------------------------+
-f NAME... (ROADMAP 2.2): like -l, with the file found by name in the
accepted indexes (highest version, first repo by priority). Missing
dependencies are added automatically (ROADMAP 2.3's closure): the
highest version satisfying the minimum, recorded `auto`. The
summary lists them, marked as dependencies. Conflicts and every
2.3 check run as today, before the prompt.
-u (ROADMAP 2.1): every installed package whose repo is configured and
whose index has a higher version is replaced, as -lO does (backups,
KEEP_OLD). Packages from repos not configured are left alone.
`-u NAME...` limits it to those names.
syspkg -f/-u fetch only from system repos; gpkg -f/-u from system and
the user's repos. Base is still refused in user scope (library rule).
As implemented (2026-10-07, step 4; M.fetch / M.upgrade in
glacier-ng/lib/gpkg_common.lua). The user decided three details:
- Which repo: the FIRST repo by repos.conf order that serves NAME, at
its highest version. A lower-priority repo never wins by publishing
a bigger version. -u stays within the installed package's repo.
- A dependency installed at too old a version is UPGRADED in the same
transaction from its own repo (backed up per KEEP_OLD, like -lO),
keeping its auto/explicit mark; the summary shows
"old -> new (dependency)".
- Cache: downloads stay in <base>/cache/REPO while a run is declined
or refused; the files a commit installed are deleted after it.
The rest:
- Every -f/-u runs -y's refresh first; if any repo fails, the
operation is refused (decision 10d). The accepted indexes are then
read from <base>/repos (gl_index_parse again; trusted by location).
- Closure: for each planned package, each DEPS entry of its INDEX
entry is met by a planned package, an installed one at >= VER
(this scope; for a user, the system's index too), else upgraded (an
installed one too old) or added (pick(): first repo serving it,
recorded auto). What can't be met is left to gl_check_plan, which
prints the 2.3 refusal. The summary marks additions
"(dependency)".
- Download: gl_scope_cache_dir (transaction.c) makes and checks
<base>/cache and <base>/cache/REPO as gl_scope_state_dir checks
repos (owner, exact mode, not a symlink). A cached file whose hash
matches the index is reused; otherwise file:// is read, https://
goes through `curl -sf --proto =https --proto-redir =https` on a
pipe; at most SIZE + 1 bytes are ever read, hashed on the way, into
a dot-named temporary, renamed into place only if size and hash
match. 0644 / 0600.
- No sidecar is written: the index's sha256 is handed to
verify_and_copy directly (its `vouched` argument), which re-hashes
while copying into the private dir, as for -l. A base package needs
no .sig of its own: the base index vouches for it.
- The copy's manifest (library-read) must say exactly what the index
entry says: NAME, VER, PKG_REPO = the repo, DEPS and CONFS (same
order, "NAME MAJ.MIN.PAT"). Else, and for a bad hash or size:
[x] extra::evil 1.0.0 does not match its index.
and the cached file is deleted. Nothing to download:
[x] Cannot fetch extra::foo 1.0.0.
No repo serves a name:
[x] foo is not in any repository.
- -f of a name installed at another version needs -O (as -l);
dependency upgrades and -u don't (meta.may_replace). -O still also
means displacing foreign files.
- -u [NAME...]: a named package not installed, or whose repo has no
url, is refused (`[x] foo is not installed.` / `[x] community::foo
is not in any configured repository.`); plain -u skips the latter.
Nothing newer: `[✓] Nothing to do.`, exit 0.
- gpkg: user scope never fetches from base (base indexes are not even
considered, so `[x] core is not in any repository.`); the system's
repos.conf until step 6; indexes and cache in the user's base.
- syspkg -R DIR -f/-u: indexes and cache under DIR, DIR's keyring.
Keyrings under -R (decided 2026-10-07, for glacier-bootstrap
configuring the system's repositories): -y, -f and -u read
DIR/etc/glacier/repos.conf and DIR/etc/glacier/keys, with the same
trust checks, plus: each file's real path must lie inside DIR
(`[x] Refusing P: it resolves outside DIR.`), so a symlink in DIR
can't make the host's file stand in for DIR's. The indexes are DIR's
state and DIR checks them with its own keys from then on, and a host
with no glacier configuration can still bootstrap. A local `-R -l`
base package's .sig is still checked with the HOST's keyring (section
7). Tests: repo_alpine.sh / fetch_alpine.sh give the image its own
keyring; bootstrap_alpine.sh sections 14-16.
- Test: glacier-ng/tests/fetch_alpine.sh (84 checks; 16 Lua mutants
and 1 C mutant, all caught except the download-level hash check,
which is EQUIVALENT: the private copy's re-hash refuses the same
package and the mismatch path deletes the cached file. Kept as
defence in depth: a bad download never sits in the cache under its
final name).
- Known gap: a run killed mid-download leaves its dot-named temporary
in the cache (never read; the next fetch writes its own).
+--------------------------------------------+
| 7. Local installs (decision 1c) |
+--------------------------------------------+
syspkg -l FILE, manifest says base:
FILE.sha256 must match (as today) AND FILE.sha256.sig must verify
with a key listed for base in repos.conf. Otherwise:
[x] base::musl is not signed by a key trusted for base.
syspkg -l, other repos; gpkg -l: unchanged (sidecar only).
glacier-bootstrap -d: carries each package's .sig into its private
copies, so syspkg -R checks it. The keyring checked is the HOST's
/etc/glacier (the build host's trust, as KEEP_OLD is today).
Consequence for ~/Projects/everest: it needs a key; container.sh signs
every base package (gbuild -k); the build host's /etc/glacier/keys and
repos.conf list it. The image ships the PUBLIC key and a repos.conf so
the installed system trusts its own base (since 2026-10-07 in the
everest-repos package, split out of everest-bootscripts).
As implemented (2026-10-06, step 2):
- The repo is the one the LIBRARY reads from the verified copy
(gl_read_pkg_manifest, as gl_install_pkg), never gpkg_common's
pattern-matching read_manifest_fields: a manifest comment
`# PKG_REPO = "extra"` would make that one say extra.
- The sidecar is read ONCE; those bytes are both what the copy's hash
is checked against and what the signature is checked over. So
glacier-bootstrap copies the sidecar's bytes as read (it used to
write a new one from the hash) next to the .sig.
- The key is the one, among those listed for base, whose keynum the
signature names. Any other problem with the signature (missing,
malformed, other bytes, other key) is the one-line refusal above,
before the prompt; one refused package refuses the batch.
- repos.conf is read by gl_conf_repo_keys (pkg.c), strictly: `repos`
is the only setting; each entry has name, keys and optionally url,
nothing else; names and key names follow gl_valid_pkg_ident; a
duplicate name, or a url other than https://X or file://X, is
refused. Any of that: `[x] Malformed /etc/glacier/repos.conf.`
- repos.conf and each listed key, every ancestor of them and of their
real paths, must be root-owned and not group/world-writable:
`[x] Refusing F: P is not root-owned, or is group/world-writable.`
A listed key that is missing (`Cannot read F.`) or not a public key
(`Malformed F.`) stops the run: the admin's configuration is at
fault, not the package. Read only if some package is base.
- gbuild -k KEY.sec signs the sidecar (gsign from $GSIGN_BIN,
build/bin, or PATH), trying the key before building; a build
without -k removes a stale .sig. tests/mkpkg -k likewise.
- Test images (Dockerfile.*) carry a TEST key, glacier-test, listed for
base, its secret at /root/glacier-test.sec. tests/keyring_alpine.sh
covers this section (17 mutants, all caught).
- KEEP_OLD backups are re-packed from the store, so a base backup has
no signature: `syspkg -lO` refuses it until the restore interface
exists (decision 10f).
+--------------------------------------------+
| 8. Where the checks live |
+--------------------------------------------+
Signature verification is C (libglacier_verify.so). The POLICY (which
key for which repo, the base rule, rollback state) lives in the CLIs'
shared module, like hash verification today, so a third-party caller of
gl_install_pkg still isn't checked. ROADMAP 3.2 (move verification into
the library) now covers signatures too.
+--------------------------------------------+
| 9. Order of work |
+--------------------------------------------+
1. Vendored Ed25519 + signify formats in libglacier_verify.so; gsign;
tests including interop with signify (Alpine's `signify` package)
and hostile inputs (bad base64, wrong keynum, truncated files).
2. Keyring + repos.conf + the local base rule (section 7); gbuild -k;
glacier-bootstrap carries .sig; Everest gets a key.
3. Index: grepo; refresh, verify, rollback state. DONE 2026-10-06
(section 5, "As implemented"; syspkg/gpkg -y).
4. -f with closure; then -u. DONE 2026-10-07 (section 6, "As
implemented").
5. glacier-bootstrap -r against an index (pins stay optional extra
assurance).
6. User repos.
Each step is tested in Docker (file:// repos need no network) and
mutation-checked, as usual.
+--------------------------------------------+
| 10. Decisions on the details (2026-10-06) |
+--------------------------------------------+
a. Fetching on Everest: curl plus one TLS library in base. Signatures
make packages authentic regardless; TLS also hides WHAT is fetched
and stops a network attacker serving a stale (still valid) index.
STILL OPEN: which TLS library (BearSSL or LibreSSL); decide when
packaging it. Until then, everything is built and tested with
file:// repos and Alpine's curl.
b. No encrypted secret keys in gsign (no bcrypt_pbkdf). A
passphrase-protected key signs with signify itself.
c. VALID_UNTIL is REQUIRED; an expired index is refused. The
maintainer sets the lifetime; grepo defaults to 30 days.
d. Indexes are refreshed on every -f / -u. Offline, they refuse.
e. Everest's SECRET key lives outside the project tree:
~/.config/everest/keys/everest-2026.sec (EVEREST_KEYS), made by
`everest.sh keygen`, mounted read-only into the build container.
The public key ships in everest-bootscripts (/etc/glacier/keys).
f. Base backups (KEEP_OLD) are refused by `syspkg -lO` for want of a
signature; accepted for now. The restore interface (KEEP_OLD's
"later") can trust backups glacier made in its root-owned backup
dir.