10/4
This commit is contained in:
lw-everestlinux committed
1 parent
c432274936
commit
ee4bcc3b77
27 files changed
+2719
-589
No files matched your search
@@ -1,261 +1,448 @@
|
||||
# CLAUDE.md
|
||||
|
||||
Context for Claude Code sessions on this repo (libglacier-ng, the package
|
||||
manager for the Everest/Glacier Linux distribution). This file exists
|
||||
because there's no built-in way to hand off a claude.ai planning
|
||||
conversation into Claude Code — this is that handoff, written down.
|
||||
Context for Claude Code sessions on libglacier-ng: the package manager
|
||||
for Everest Linux. Kept as a handoff, so a fresh session starts with full
|
||||
continuity instead of re-deriving everything. Last full rewrite:
|
||||
2026-10-04, at the end of a long session. **Update it as you go.**
|
||||
|
||||
For the symlink-routing / package-scope design specifically, see
|
||||
`PACKAGE_SCOPES.txt` at the repo root — that doc is the source of truth
|
||||
for that topic and isn't duplicated here.
|
||||
## Start here
|
||||
|
||||
1. `ROADMAP.txt`: what's left, in priority order, plus the decisions that
|
||||
are still open (section 6). Pick work from it.
|
||||
2. This file: how things work, what the user wants, how to test.
|
||||
3. The design docs, which are authoritative for their topics and not
|
||||
repeated here:
|
||||
- `SECURITY_DESIGN.txt`: integrity verification, scope separation,
|
||||
refusal matrix, syspkg's privilege model.
|
||||
- `PACKAGE_SCOPES.txt`: layout, link routing, `/etc` config handling.
|
||||
- `IMAGE_DESIGN.txt`: PROPOSAL for system images (bootstrap OUTPUT,
|
||||
the sysroot library change). Open decisions are in section 9.
|
||||
- `GLACIER_BOOTSTRAP.txt`: PLAN for `glacier-bootstrap`. Prerequisites
|
||||
are in section 7, decisions in section 9 (9c and 9d are decided).
|
||||
- `SECURITY_REVIEW_2026-07-16.md`: older review findings.
|
||||
|
||||
Nothing from the 2026-10-04 session is committed. libglacier-ng is on
|
||||
branch `dev` with a large uncommitted diff (`src/`, the docs). Commit only
|
||||
when the user asks.
|
||||
|
||||
## The three projects
|
||||
|
||||
| Path | What | Git? |
|
||||
|---|---|---|
|
||||
| `~/Projects/libglacier-ng` | C library, design docs, this file | yes (`dev`) |
|
||||
| `~/Projects/glacier-ng` | Frontend: `gpkg`, `gstore`, `syspkg`, `gbuild`, `gworld`, `lib/*.lua`, `tests/`, Dockerfiles, `pkgs/` | **no**: back files up before editing (`file.old`; if that exists, `.old2`, `.old3`, ...) |
|
||||
| `~/Projects/everest` | The distro build: `everest.sh build\|initramfs\|boot`, packages, bootable QEMU image | no |
|
||||
|
||||
`glacier-ng/lib/glacier/` holds `.so`s built on the host (glibc). They
|
||||
won't load on musl. Every Alpine image and test builds libglacier-ng from
|
||||
source instead.
|
||||
|
||||
## What this project is
|
||||
|
||||
A from-scratch Linux distribution (Everest, built on the Glacier
|
||||
package-management model) with its own package manager
|
||||
(`libglacier-ng` + `gpkg`/`gstore` CLI tools) and its own build
|
||||
tooling (`gbuild`/`gworld`) for bootstrapping a set of packages up to
|
||||
a minimal bootable system.
|
||||
A from-scratch, security-focused Linux distribution (Everest) with its own
|
||||
package manager (libglacier-ng + `gpkg`/`gstore`/`syspkg`) and build tooling
|
||||
(`gbuild`/`gworld`). It's a C library, LuaJIT+FFI CLI tools and bash build
|
||||
scripts. The user writes it; Claude implements and designs alongside them.
|
||||
|
||||
## How the user wants things done
|
||||
|
||||
These are preferences stated in sessions. Follow them.
|
||||
|
||||
- **Output format:** every message uses glacier's `lg_printf` levels:
|
||||
`[x]` is error (2), `[!]` warning (1), `[i]` info (0), `[✓]` success (3).
|
||||
Messages are capitalized. Exact strings the user chose include
|
||||
`Cannot open system index: Permission denied` (no period) and
|
||||
`Nothing to do.` (success level).
|
||||
- The pre-trust-check `die()` in syspkg/gstore reproduces the `[x]`
|
||||
format byte for byte, because it can't load `libglacier_log.so`
|
||||
before that file has been verified.
|
||||
- **No hint messages.** Don't print lines that point users to other flags
|
||||
or commands (for example "use -lO to ..."). The user called these
|
||||
clutter. A refusal states the fact, in one line. The user also removed a
|
||||
`Run doas gstore -n ...` follow-up hint from gstore themselves.
|
||||
- **Ask before settling a design choice that is the user's to make**, and
|
||||
give a recommendation. They have answered every such question promptly.
|
||||
Don't silently pick a layout or semantics.
|
||||
- **Verify by running, in Docker or QEMU, never by inspection alone.** For
|
||||
any security fix, also prove the test catches the bug: remove the
|
||||
defence in a throwaway copy and confirm the test fails ("mutation
|
||||
check"). Several defences here are verified that way.
|
||||
- Use doas, not sudo. syspkg is never setuid.
|
||||
|
||||
## Decisions the user has made (don't relitigate)
|
||||
|
||||
- **Per-uid user layout** (`/glacier/usr/<uid>/...`, root-owned parent).
|
||||
Both root and the user can create it: root runs `gstore -n UID`; a user
|
||||
runs `doas gstore -n` through a doas rule pinned to `args -n`
|
||||
(option "B").
|
||||
- **`syspkg -n` creates the top-level tree itself** (GLACIER_BOOTSTRAP 9d).
|
||||
- **`/etc` config files: option 1.** Copy on install, `.new` on conflict
|
||||
(GLACIER_BOOTSTRAP 9c).
|
||||
- **`-O` with `-l` = replace.** `-O` displaces non-glacier files, and
|
||||
`-lO` also replaces an installed *different version* (upgrade or
|
||||
downgrade from a local file). Plain `-l` with another version installed
|
||||
refuses in one line.
|
||||
- **`KEEP_OLD`** in `/etc/glacier/glacier.conf`: the number of backups of
|
||||
replaced versions to keep per package (0 = none).
|
||||
- It has **no default.** It ships as `KEEP_OLD = 1`.
|
||||
- If it's missing or invalid, an operation that would replace a version
|
||||
errors. Only then: other operations don't need it.
|
||||
- "Backed up" means the old package's files are compressed into a
|
||||
`.gpkg`. A restore interface comes later.
|
||||
- **`gpkg -u` / `syspkg -u` upgrade from a REMOTE repository**, and
|
||||
`gpkg -f` is `-l` but fetched by name from a remote repository. Both wait
|
||||
for remote repos + signing (ROADMAP 3.1). Don't implement them against
|
||||
local files.
|
||||
- `everest-bootscripts` is the name of the base `/etc` package. Its source
|
||||
lives in `~/Projects/everest/pkgsrc/`.
|
||||
|
||||
## Core architecture
|
||||
|
||||
**`gl_context_t`** (transaction.h) is the central abstraction: `uid`,
|
||||
`root_path`, `index_path`, `store_path`, `links_path`, `mode`
|
||||
(`GL_CTX_LIVE` / `GL_CTX_STAGE`), `scope` (`GL_SCOPE_USR` /
|
||||
`GL_SCOPE_SYS`), `lock_fd`.
|
||||
**`gl_context_t`** (transaction.h) is the central abstraction. Fields:
|
||||
`uid`, `root_path`, `index_path`, `store_path`, `links_path`, `mode`
|
||||
(`GL_CTX_LIVE`/`GL_CTX_STAGE`), `scope` (`GL_SCOPE_USR`/`GL_SCOPE_SYS`),
|
||||
`lock_fd`, and `overwrite` (last field, zeroed by the init functions).
|
||||
|
||||
**Transaction lifecycle**: `gl_init_live_context` / `gl_init_stage_context`
|
||||
→ `gl_commit_transaction` or `gl_abort_transaction`. Staged installs
|
||||
hardlink-seed `index`/`store` from live at stage-init time (copy-on-write,
|
||||
so removing a live package inside a transaction doesn't require special
|
||||
casing). The `links` tree is deliberately **not** seeded — see below.
|
||||
**Layout.** Each scope owns ONE base directory holding
|
||||
`{index,store,links,stage,old,lock,backup}`. Every create, rename and lock
|
||||
a transaction does stays inside it.
|
||||
|
||||
**Commit** does atomic rename-based promotion (stage → live, with the
|
||||
previous live moved to `old/` as a rollback point), fsyncs the parent
|
||||
dirs, then calls `gl_relink_store` to regenerate symlinks and
|
||||
`gl_rebuild_index` to regenerate the index — both from a fresh
|
||||
`GL_CTX_LIVE` context, not the just-committed stage context.
|
||||
| Scope | Base dir | Ownership |
|
||||
|---|---|---|
|
||||
| `GL_SCOPE_USR` | `/glacier/usr/<uid>/` | uid, mode 0700, inside a root-owned 0755 `/glacier/usr` |
|
||||
| `GL_SCOPE_SYS` | `/glacier/sys/` | root |
|
||||
|
||||
**Scopes**:
|
||||
- `GL_SCOPE_USR`: per-uid, `/glacier/usr/{index,store,links}/<uid>`.
|
||||
- `GL_SCOPE_SYS`: system-wide, `/glacier/sys/{index,store}`. Symlink
|
||||
destination depends on the package's **repo** — see `PACKAGE_SCOPES.txt`.
|
||||
Short version: `repo == "base"` → `/usr`; anything else →
|
||||
`/glacier/sys/links`. This is decided per-package (in `gl_link_pkg` /
|
||||
`gl_relink_store`), not baked into the context.
|
||||
- The helpers `gl_usr_base`, `gl_usr_base_check` and the layout comment
|
||||
are in `common.h`.
|
||||
- `gl_init_stage_context` refuses a user base that isn't a real directory
|
||||
(lstat) owned by its uid with no group/other bits, or whose
|
||||
`/glacier/usr` is loosened. `gl_usr_base_status` exports that check, so
|
||||
gpkg can say why before any prompt.
|
||||
|
||||
**Repos**: `base` (minimal working system, boot-critical by definition —
|
||||
this is a repo-level classification, not a per-package flag), `extra`
|
||||
(important but not required), `community` (everything else). `extra`
|
||||
and `community` currently behave identically — see open items.
|
||||
**Transaction lifecycle.** `gl_init_live_context` / `gl_init_stage_context`
|
||||
→ `gl_commit_transaction` or `gl_abort_transaction`.
|
||||
- Stage init takes the lock FIRST, then clears any leftover stage.
|
||||
- Stage contexts hardlink-seed `index`/`store` from live, so removing a
|
||||
live package inside a transaction needs no special casing. Seeding
|
||||
copies dotfiles: only `.` and `..` are skipped.
|
||||
- A stage on another filesystem fails at seeding with `GL_TXN_ERR_XDEV`.
|
||||
- **Commit:** rename-based promotion (stage → live, previous live → `old/`),
|
||||
then fsync the base. Then `gl_relink_store` + `gl_rebuild_index`
|
||||
regenerate links and index from a *fresh live context* and rescan the
|
||||
WHOLE store (every commit relinks every package, so link-time logic must
|
||||
be idempotent). `old/` is then deleted.
|
||||
|
||||
## Safety mechanisms (read before touching links/symlink code)
|
||||
**Link routing** (per package, in `gl_link_pkg`):
|
||||
|
||||
`ctx->links_path` is **not always glacier-exclusive territory** —
|
||||
for system-scope `base` packages it's literally `/usr`, which holds
|
||||
plenty of content glacier has no business touching. Everything that
|
||||
reads or writes a links destination has to treat it as "possibly
|
||||
shared":
|
||||
| Scope | Repo | Destination |
|
||||
|---|---|---|
|
||||
| USR | any | `<base>/links` |
|
||||
| SYS | `base` | `/usr` |
|
||||
| SYS | anything else | `/glacier/sys/links` |
|
||||
|
||||
- `is_glacier_symlink(path, store_prefix)` (istoreutils.c) is the one
|
||||
source of truth for "did glacier create this." True only if `path`
|
||||
is a symlink whose target lives under `store_prefix`. Never touch a
|
||||
path this returns false for.
|
||||
- `gl_link_pkg` will not overwrite an existing path unless
|
||||
`is_glacier_symlink` says it's safe to replace.
|
||||
- `gl_relink_store` prunes stale symlinks (`prune_stale_links`) rather
|
||||
than wiping and rebuilding the whole links directory — a full
|
||||
`rm -rf` of `ctx->links_path` would be catastrophic once that path
|
||||
can be `/usr`. It prunes **both** possible system-scope destinations
|
||||
(`/glacier/sys/links` and `/usr`) since a single pass can't know in
|
||||
advance whether any base-repo packages are involved.
|
||||
- The stage links tree is never hardlink-seeded — it's genuinely
|
||||
unused (`gl_link_pkg` only ever runs against `GL_CTX_LIVE`;
|
||||
`gl_relink_store` always builds its own fresh live context). Seeding
|
||||
it would mean hardlinking all of `/usr` on every staged system
|
||||
transaction, and risking `EXDEV` if `/usr` and the stage area are on
|
||||
different filesystems.
|
||||
- A package's leading `usr/` is stripped.
|
||||
- In system scope, a package's top-level `etc/` is never linked: it's
|
||||
COPIED into `/etc` (see `/etc` config handling below).
|
||||
|
||||
## Bug history (so it doesn't get relitigated or reintroduced)
|
||||
**Repos.** `base` (the minimal system; boot-critical by definition, a
|
||||
repo-level rule), `extra`, `community`. `extra` and `community` behave
|
||||
identically.
|
||||
|
||||
Roughly chronological, all already fixed in the current tree:
|
||||
**Hidden files in the store** (dot-prefixed, so linking, index rebuild and
|
||||
the store scanners skip them, while seeding carries them through
|
||||
transactions):
|
||||
|
||||
1. **`gl_abort_transaction` never actually released the lock file** — a
|
||||
copy-paste bug wrote into the wrong variable. Fixed by moving off
|
||||
`O_EXCL`-based locking entirely: `gl_begin_transaction` now just
|
||||
`open(O_CREAT)` + `flock()`, and lock files are never `unlink()`'d
|
||||
(the kernel releases `flock()` automatically on process death, even
|
||||
`SIGKILL`; explicit unlinking has its own TOCTOU race with a second
|
||||
process creating a new inode at the same path).
|
||||
2. **`EXDEV` wasn't checked before renaming** — `gl_commit_transaction`
|
||||
now `stat()`s both sides and fails cleanly with `GL_TXN_ERR_XDEV`
|
||||
before attempting any renames if stage and live aren't on the same
|
||||
filesystem.
|
||||
3. **Segfault on install**, root cause: the Makefile's `transaction`
|
||||
target didn't link `libglacier_transaction.so` against
|
||||
`libglacier_istoreutils.so`/`libglacier_log.so`, even though
|
||||
`transaction.c` calls `gl_relink_store()` and `lg_printf()`. Fixed
|
||||
by adding those `-l` flags and an explicit `transaction: log
|
||||
istoreutils` prerequisite (build order matters now).
|
||||
4. **Symlinks going stale after commit** — `gl_link_pkg` used to run
|
||||
during staged installs too, embedding the *stage* path as the
|
||||
symlink target. Fixed: `gl_link_pkg` only runs for `GL_CTX_LIVE`;
|
||||
links are fully regenerated post-commit by `gl_relink_store`.
|
||||
5. **Dangling symlinks never cleaned up on removal** — see "Safety
|
||||
mechanisms" above; this is what `prune_stale_links` fixes.
|
||||
6. **`GL_TXN_ERR_RENAME` on every commit after the first** — leftover
|
||||
`old/` directory from a previous commit made the next commit's
|
||||
`rename()` fail with `ENOTEMPTY`. Fixed: `gl_commit_transaction`
|
||||
wipes `old_base` if present before using it.
|
||||
7. **`-Wstringop-truncation` errors** (real bug, not noise) — four
|
||||
`strncpy(dst, ctx->foo_path, PATH_MAX - 1)` calls that could leave
|
||||
`dst` unterminated. Fixed by switching to `snprintf(dst,
|
||||
sizeof(dst), "%s", ...)`, which always terminates.
|
||||
(`-Wformat-truncation`, separately, is suppressed via
|
||||
`-Wno-error=format-truncation` in `config.mk` — those warnings are
|
||||
about genuinely-safe `snprintf` truncation, not a real bug.)
|
||||
8. **`gbuild` build-system autodetection** checked for a bare
|
||||
`Makefile` before `configure`/`configure.ac`. Broke musl (and any
|
||||
project with a hand-rolled, non-autotools build system that ships
|
||||
both) since its `Makefile` is non-functional without `configure`
|
||||
generating `config.mak` first. Fixed by reordering the checks.
|
||||
9. **Index only stored `repo::pkgname`, no version.** Now
|
||||
`repo::pkgname::version` (`gl_rebuild_index` looks up the
|
||||
`<pkgname>-<version>` store subdirectory). `struct gpkg_entry` in
|
||||
`istoreutils.h` needs a `char *ver;` field — this was a manual
|
||||
header edit, double check it's actually present.
|
||||
10. **`gworld` recipe parser silently dropped `SYS_PROFILE`** — the
|
||||
key-matching regex only matched uppercase letters, not
|
||||
underscores. Fixed to `^([%u_][%u%d_]*)=(.*)$`. Caught by an
|
||||
actual precedence test, not code review — worth remembering that
|
||||
class of bug (a field silently never parsing, falling back to a
|
||||
default that happens to look right) is easy to miss by inspection
|
||||
alone.
|
||||
| File | Purpose |
|
||||
|---|---|
|
||||
| `<ver>/.glacier-manifest` | the package's manifest, saved at install, used by `gl_backup_pkg` |
|
||||
| `<ver>/.glacier-fresh` | one-shot "installed in this transaction"; link conflicts are only reported when present |
|
||||
| `<ver>/.glacier-overwrite` | one-shot `-O` permission for this package's next link pass |
|
||||
| `<store>/.displaced/<abs path>` | originals displaced by `-O`, restored when the link that replaced them is pruned |
|
||||
| `<store>/.etc-state/<path>` | the last version of each `/etc` file a package offered |
|
||||
|
||||
## CLI additions
|
||||
**Configuration.** `/etc/glacier/glacier.conf` (libconfig syntax).
|
||||
- It's read with `gl_conf_get_int(path, key, &out)` in `pkg.c`. Missing
|
||||
file, missing key and wrong type are distinct statuses, and there are no
|
||||
defaults.
|
||||
- The shipped copy is `glacier-ng/etc/glacier/glacier.conf`. Every
|
||||
Dockerfile installs it.
|
||||
|
||||
- `gpkg -s` / `--system`: operate against `GL_SCOPE_SYS` instead of a
|
||||
uid's tree. Requires root (checked via `geteuid()`). Must appear
|
||||
*before* `-l`/`-x` in a grouped flag string (e.g. `-sl`, not `-ls`) —
|
||||
flags are handled in parse order, same constraint `-V`/`-S` already
|
||||
had.
|
||||
- `gstore -N` / `--new-system`: calls `gl_init_sys()`. Requires root.
|
||||
- `gbuild -P` / `--profile PROFILE`: system profile, `ARCH-LIBC[-FEATURE...]`
|
||||
(e.g. `x86_64-musl`, `x86_64-glibc-multilib`). Derives a cross-compile
|
||||
prefix (translating `glibc`→`gnu` per real GNU triplet convention),
|
||||
detects native-vs-cross against the build host, and only pre-fills
|
||||
`-t`/`-x` defaults for whichever the user didn't already set
|
||||
explicitly. Features export as `GL_FEATURE_<NAME>=1` env vars for
|
||||
custom `-b`/`-i` commands to branch on — gbuild itself doesn't
|
||||
hardcode per-feature behavior, that's intentional.
|
||||
## Safety invariants: read before touching links/symlink code
|
||||
|
||||
## Build system
|
||||
`/usr` is NOT glacier-exclusive territory. Treat every link destination as
|
||||
possibly shared:
|
||||
|
||||
`Makefile` builds separate `.so`s per module
|
||||
(`libglacier_log.so`, `libglacier_pkg.so`, `libglacier_istoreutils.so`,
|
||||
`libglacier_transaction.so`, `libglacier_dag.so`) — **this split is
|
||||
deliberate Unix-philosophy design**, not an accident: a third-party
|
||||
glacier-compatible program should be able to link only
|
||||
`libglacier_log.so` without pulling in everything else. `gpkg`/`gstore`
|
||||
load the specific per-module `.so`s they need directly, matching this.
|
||||
- **Ownership test.** `is_glacier_symlink(path, store_prefix)` is the only
|
||||
test for "did glacier create this". Never touch anything it returns
|
||||
false for, except through `-O`'s displace/restore.
|
||||
- **Overwriting.** `gl_link_pkg` won't overwrite a non-glacier path without
|
||||
the `-O` marker. It never descends through *another package's* directory
|
||||
symlink into that package's store directory.
|
||||
- **Pruning.** `gl_relink_store` PRUNES stale links; it never wipes a links
|
||||
destination. A link is stale only if its own store target is gone
|
||||
(checked with `lstat`, not by following it): a package symlink may
|
||||
legitimately dangle.
|
||||
- **Stage links.** The stage links tree is never seeded.
|
||||
- **Extraction never writes through a symlink.**
|
||||
- Symlink entries are created only after all regular content.
|
||||
- Any entry whose parent is a symlink is refused (`crosses_symlink`,
|
||||
plus `O_NOFOLLOW`).
|
||||
- Any entry with `..` or an absolute path refuses the whole package.
|
||||
- A failed extraction removes its half-made version directory.
|
||||
- **Recursive deletes.** `gl_rmdir_recursive` skips only `.`/`..`. Every
|
||||
caller passes a glacier-only path; keep it that way. `gl_delete_user`
|
||||
lstat-checks its target first, because a symlinked base would otherwise
|
||||
have its *target* emptied.
|
||||
- **Wide wrappers.** Don't add convenience wrappers that silently touch a
|
||||
broader filesystem scope than asked. This codebase has been bitten twice
|
||||
by exactly that.
|
||||
|
||||
There's also a `libglacier-ng` Makefile target producing a **combined**
|
||||
`.so` — this exists *solely* for the Lua FFI test harness's
|
||||
convenience (one thing to `ffi.load()` instead of juggling rpath
|
||||
chains in a throwaway test process). It is **not** in `all` and **not**
|
||||
installed. Don't reach for it outside `tests/` — that's a sign
|
||||
something should be split-loaded instead.
|
||||
## Feature reference (current behaviour)
|
||||
|
||||
`config.mk` holds `CFLAGS`, including `-Wno-error=format-truncation`
|
||||
(see bug #7 above for why that specific warning is suppressed and
|
||||
others aren't).
|
||||
**Integrity verification** (`gpkg_common.lua`: `verify_and_copy`,
|
||||
`verify_all`):
|
||||
- Each package is copied into a private `mktemp -d` directory while being
|
||||
hashed with `libglacier_verify.so`. The result is checked against
|
||||
`FILE.sha256` (`sha256:<hex>`).
|
||||
- Everything after that (summary, manifest read, `gl_install_pkg`) reads
|
||||
ONLY the copy.
|
||||
- A missing or malformed sidecar counts as a mismatch and is refused before
|
||||
the prompt; one bad package refuses the whole batch. No bypass exists.
|
||||
- This lives in the CLIs' shared module, not the library (as
|
||||
SECURITY_DESIGN section 4 specifies), so a third-party caller of
|
||||
`gl_install_pkg` is not verified (ROADMAP 3.2).
|
||||
- The base→USR refusal (`GL_INSTALL_ERR_SCOPE = 7`), by contrast, *is* in
|
||||
the library.
|
||||
|
||||
## Lua tooling
|
||||
**`-O` / displace:**
|
||||
- A displaced foreign file or symlink moves to `<store>/.displaced/`:
|
||||
renamed, or on `EXDEV` copied (keeping mode, owner and mtime) and then
|
||||
unlinked.
|
||||
- Directories, special files and paths that already have a backup are
|
||||
still refused.
|
||||
- In user scope `-O` is effectively a no-op, because the links tree is
|
||||
rebuilt from scratch on every commit.
|
||||
|
||||
- `gpkg`, `gstore`: LuaJIT + FFI, calling directly into the `.so`s.
|
||||
- `lib/glacier_cdef.lua`: single shared source of truth for every
|
||||
`ffi.cdef()` declaration, used by `gpkg`, `gstore`, and the test
|
||||
suite. This used to be duplicated three ways, which is exactly what
|
||||
let a stale signature drift silently into `gpkg` at one point — if
|
||||
you're editing a C function signature, this file needs the matching
|
||||
update, and it's the *only* place that needs it now.
|
||||
- `tests/*.lua`: LuaJIT FFI test suite exercising real filesystem/process
|
||||
behavior (`fork`+`SIGKILL` for lock crash-recovery, real `mount()`
|
||||
for the `EXDEV` check, etc.) — not mocked. Run via:
|
||||
```sh
|
||||
cd tests
|
||||
make -C .. libglacier-ng # builds the test-only combined .so
|
||||
GLACIER_TEST_CONFIRM=yes LIBGLACIER_SO=$(pwd)/../build/lib/shared/libglacier-ng.so luajit run_all.lua
|
||||
```
|
||||
`test_exdev.lua` needs `--cap-add=SYS_ADMIN` in Docker to actually
|
||||
exercise the mount path; it skips cleanly (not a failure) without it.
|
||||
**Version replacement** (`-lO`, `M.install` in `gpkg_common.lua`):
|
||||
1. Installed entries are read from the live index's `current` file
|
||||
(`repo::name::version` lines).
|
||||
2. A same name+repo with a different version is a replacement. Without
|
||||
`-O` it's refused; with `-O`, `keep_old()` reads `KEEP_OLD`, and the
|
||||
summary shows `old -> new`.
|
||||
3. In the transaction: `gl_backup_pkg(live, repo, name, X.gpkg.part)`, then
|
||||
`gl_remove_pkg` from the stage, then install.
|
||||
4. After the commit: rename `.part` → `<scope>/backup/<repo>/<name>/<name>-<ver>.gpkg`,
|
||||
write its sidecar, and run `gl_prune_backups(dir, KEEP_OLD)` (newest by
|
||||
nanosecond mtime).
|
||||
- On any abort, `finish()` deletes the `.part`.
|
||||
- A backup is a complete installable `.gpkg`, so `-lO backup.gpkg`
|
||||
restores it.
|
||||
|
||||
## Build automation (bootstrap toolchain)
|
||||
**`/etc` config handling** (`install_config` / `prune_config` in
|
||||
istoreutils.c), system scope only:
|
||||
|
||||
- `gbuild` (bash): single-package builder. Autodetects build system
|
||||
(cmake/meson/cargo/go/autotools/make), supports cross-compilation
|
||||
(`-t`/`-x`, now also `-P` for profiles).
|
||||
- `gworld` (Lua): batch orchestrator. Reads `recipes/*.recipe`,
|
||||
topologically sorts by `DEPS`, invokes `gbuild` once per package in
|
||||
order. `-s SYSROOT` gives every package in the batch a shared
|
||||
staging dir — relies on `gbuild -p` never wiping its target, so
|
||||
headers/libs accumulate across the batch for free.
|
||||
- Recipe format: `NAME VER REPO URL/REF-or-SRC DEPS BUILD INSTALL
|
||||
FLAGS SYS_PROFILE` (plain `key=value`, see `recipes/musl.recipe` for
|
||||
a real example).
|
||||
- Real multi-pass toolchain bootstraps (LFS-style — gcc built twice,
|
||||
etc.) should use separate recipes per pass (`gcc-pass1`,
|
||||
`gcc-pass2`), publishing only the final pass under the real
|
||||
name/repo. The DAG models build steps, not abstract packages.
|
||||
- **`grootstrap` needs a rebuild, not yet done.** The old design (raw
|
||||
`tar` extraction into a target dir + chroot + register everything
|
||||
under an arbitrary bootstrap uid) predates the `GL_SCOPE_SYS` +
|
||||
base-repo design and is now more complicated than necessary. Once
|
||||
`gpkg`/`gstore` themselves exist on the target (still an unsolved
|
||||
bootstrapping-the-bootstrapper problem — how do they get there
|
||||
before a working package manager exists to install them with),
|
||||
every package including `base` ones can go through the same `gpkg
|
||||
-s -l` path; the repo alone routes symlinks correctly. No more
|
||||
special-casing base packages during bootstrap.
|
||||
| Situation | Result |
|
||||
|---|---|
|
||||
| `/etc/F` missing, no record | install |
|
||||
| missing, record exists | the admin deleted it: stays deleted |
|
||||
| unchanged since glacier wrote it | updated in place |
|
||||
| admin-modified, or never glacier's | never touched; a package change is offered ONCE as `F.new` |
|
||||
| no package ships it any more | removed if unchanged, kept otherwise |
|
||||
|
||||
## Known open items / stubs
|
||||
User-scope `etc/` is still linked (ROADMAP 2.9).
|
||||
|
||||
- `gpkg -f` / `-u` (merge/update) are unimplemented stubs. `-f`
|
||||
references an undefined `uid` variable (should be `uidn`) — will
|
||||
nil-concatenation-error the moment it's actually implemented.
|
||||
- `gpkg -x`'s confirmation summary shows placeholder `unknown::pkg
|
||||
0.0.0 (unknown)` since it has no local manifest to read version/repo
|
||||
from for a bare package name. Would need `resolve_repo` called
|
||||
before building the summary, not just before the removal itself.
|
||||
- `gpkg -x` doesn't print a completion message on success (`-l` does:
|
||||
"Completed with no errors."). Cosmetic inconsistency.
|
||||
- Reinstalling an up-to-date package double-logs ("Installed X" +
|
||||
"staged from X" for the same event).
|
||||
- `extra` vs `community` repos don't currently diverge in behavior —
|
||||
both just mean "not base" for symlink-routing purposes.
|
||||
- `gl_relink_store`/`gl_rebuild_index` are full rescans of the store on
|
||||
every commit — fine at current scale, worth revisiting if large
|
||||
transactions become common (a dirty-set of touched packages instead
|
||||
of full rescans).
|
||||
- Package manifests don't record which `SYS_PROFILE` they were built
|
||||
under — deliberately not added yet (would need a `pkg.c` /
|
||||
`gl_gpm2gpkg` change), flagged as a future option, not a gap that
|
||||
needs fixing now.
|
||||
- A stray nested `lib/glacier/lib/glacier/` directory was spotted once
|
||||
in a `tree` listing — looked like a leftover `make install`
|
||||
artifact, not in any load path, harmless but worth a `rm -rf`
|
||||
eventually.
|
||||
**Top-level tree.** `gl_create_fs_tree(root, verbose)` (exported) creates
|
||||
the merged-/usr tree: `/bin` and `/sbin` → `usr/bin`, `/lib` → `usr/lib`,
|
||||
`/usr/sbin` → `bin`, all RELATIVE links. It also creates `/usr/{bin,lib}`,
|
||||
`/etc`, `/dev`, `/proc`, `/sys`, `/run`, `/tmp` and `/var/tmp` (1777),
|
||||
`/var`, `/root` (0700) and `/home`.
|
||||
- It creates only what's missing, of any type.
|
||||
- `gl_init_sys` runs it before writing the index. The index marks `-n` as
|
||||
complete, so a second `syspkg -n` is refused: `Cannot create
|
||||
/glacier/sys: already exists`.
|
||||
- It takes a root prefix, ready for `syspkg --root`.
|
||||
|
||||
## Conventions worth knowing before editing
|
||||
**Package symlinks** are extracted verbatim and linked like files.
|
||||
Hardlink, device and fifo entries are skipped. `add_to_archive` (used by
|
||||
backups and `gstore -b`) archives symlinks.
|
||||
|
||||
- C source generally uses tabs for indentation, though the codebase is
|
||||
inconsistent about it in places (some blocks use spaces). ALWAYS USE TABS WHEN POSSIBLE.
|
||||
- `PATH_MAX`-sized stack buffers + `snprintf` is the standard pattern
|
||||
for path construction throughout; prefer it over `strncpy` for
|
||||
anything copying into a fixed buffer (see bug #7).
|
||||
- Don't add convenience wrappers that silently touch a broader
|
||||
filesystem scope than what's asked — this codebase has already been
|
||||
bitten twice by exactly that class of mistake (`gl_relink_store`'s
|
||||
wipe, `gl_link_pkg`'s blind `unlink`), both only becoming dangerous
|
||||
once a path assumption (`links_path` is glacier-exclusive) quietly
|
||||
stopped holding.
|
||||
## CLIs (glacier-ng)
|
||||
|
||||
| Tool | Flags | Runs as |
|
||||
|---|---|---|
|
||||
| `gpkg` | `-l FILE...` `-x NAME...` `-O` `-S` `-V` `-h` `-v`; `-f`/`-u` are stubs | the user; per-user scope only, NEVER constructs `GL_SCOPE_SYS` |
|
||||
| `syspkg` | `-n` `-l FILE...` `-x NAME...` `-O` `-S` `-V` `-h` `-v` | root via doas; `-h`/`-v` without root |
|
||||
| `gstore` | `-n [UID]` `-b UID OUTFILE` `-x UID` `-s` (stub) `-V` `-h` `-v` | `-n` needs root (direct, or `doas gstore -n` for yourself) |
|
||||
|
||||
- **gpkg:** modifiers `-O`/`-S`/`-V` are applied in a pre-pass, so their
|
||||
position doesn't matter (`-lO` == `-Ol`). It finds its user via
|
||||
`/proc/self/loginuid`, falling back to `getuid()`.
|
||||
- **syspkg, `-n`:** creates the tree plus `/glacier/sys`.
|
||||
- **syspkg as non-root:** `-n`/`-l`/`-x` are refused after argument
|
||||
parsing with `Cannot open system index: Permission denied`.
|
||||
- **gstore, `-n` without a UID, under doas:** acts for the invoking user
|
||||
(loginuid, then `DOAS_USER`, then getuid). The doas rule
|
||||
`permit nopass :wheel as root cmd /path/gstore args -n` is the
|
||||
boundary that stops a user naming another UID.
|
||||
- **Shared code:** `lib/gpkg_common.lua` (`M.install`, `M.remove`,
|
||||
verification, replacement, backups).
|
||||
- **The `-x` summary** comes from the live index; a name that isn't
|
||||
installed fails before the prompt.
|
||||
- **Hardened loading** (syspkg and gstore): `#!/usr/bin/luajit -E`,
|
||||
`package.path` and every `ffi.load` pinned to their own `realpath`, and
|
||||
when euid is 0, every ancestor directory and every loaded `.lua`/`.so`
|
||||
must be root-owned and not group/world-writable. The preamble is
|
||||
DUPLICATED in both files on purpose: change one, change both.
|
||||
- **gpkg** uses the older CWD-relative convention (`require("lib/...")`,
|
||||
`ffi.load("./lib/glacier/...")`): run it from its own directory.
|
||||
- **Declarations:** `lib/glacier_cdef.lua` is the single source of truth
|
||||
for every `ffi.cdef`. `tests/glacier_ffi.lua` is only a wrapper around
|
||||
it. Declaration ORDER matters: a declaration using `gl_context_t` must
|
||||
come after its typedef.
|
||||
|
||||
## Build tooling
|
||||
|
||||
- **gbuild** (bash, the only copy is `glacier-ng/gbuild`):
|
||||
- `-n -V -g -c -d -m -D -C -b -i -o -p -j -F -t -x -u -r -s`;
|
||||
`-v`/`--version` prints the version, `--verbose` turns on verbose
|
||||
output.
|
||||
- Autodetects the build system, with autotools checked BEFORE a bare
|
||||
Makefile.
|
||||
- Writes `NAME-VER.gpkg` plus a `.sha256` sidecar, using `gsha256sum`
|
||||
from `$GSHA256SUM_BIN`, else `<script dir>/build/bin`, else PATH.
|
||||
- **There is NO `-P` system-profile support, and gworld has no
|
||||
`SYS_PROFILE` field** (ROADMAP 4.1). The target libc is decided by
|
||||
where gbuild runs: Alpine for musl.
|
||||
- **gworld** (Lua): batch orchestrator, reads `recipes/*.recipe`, sorts
|
||||
topologically by `DEPS`, runs gbuild per package.
|
||||
- **`~/Projects/everest`:** the gbuild command for each package lives in
|
||||
`scripts/container.sh`; `everest.sh` runs it inside an Alpine image
|
||||
(`Dockerfile.build`).
|
||||
- **Kernel version:** linux `6.12.112`; plain `v6.12` doesn't build with
|
||||
GCC 15.
|
||||
|
||||
## libglacier-ng build
|
||||
|
||||
- **Separate `.so` per module** (`libglacier_log`, `_pkg`, `_istoreutils`,
|
||||
`_transaction`, `_dag`, `_verify`). This split is deliberate: a
|
||||
third-party program links only what it needs.
|
||||
- **The combined `libglacier-ng.so`** (`make libglacier-ng`) exists only
|
||||
for the FFI tests.
|
||||
- **`libglacier_verify.so`** is private: no public header.
|
||||
- **Warnings:** `config.mk` builds with `-Werror`. `-Wno-format-truncation`
|
||||
is set because that warning is noise; `-Wstringop-truncation` stays
|
||||
fatal.
|
||||
- **Makefile quirk:** `make log` builds only `log.o`; the shared object
|
||||
comes from `make log_test` (or plain `make`).
|
||||
|
||||
## Testing
|
||||
|
||||
Run everything in throwaway containers. Several suites write to `/usr`,
|
||||
`/etc` and `/glacier`.
|
||||
|
||||
**FFI suite** (`glacier-ng/tests/*.lua`; 8 files, all passing):
|
||||
```sh
|
||||
docker run --rm --cap-add SYS_ADMIN -v ~/Projects/glacier-ng:/src:ro \
|
||||
-v ~/Projects/libglacier-ng:/libsrc:ro alpine:latest sh -c '
|
||||
apk add --no-cache build-base luajit luajit-dev libarchive-dev libconfig-dev linux-headers tar >/dev/null
|
||||
cp -a /libsrc /build && cd /build && make clean && make && make libglacier-ng
|
||||
cp -a /src /g && cd /g/tests && GLACIER_TEST_CONFIRM=yes \
|
||||
LIBGLACIER_SO=/build/build/lib/shared/libglacier-ng.so luajit run_all.lua'
|
||||
```
|
||||
GNU `tar` is required: the hostile-archive fixtures use `tar --append`.
|
||||
|
||||
**Shell suites** (`glacier-ng/tests/*.sh`):
|
||||
|
||||
| Suite | Image |
|
||||
|---|---|
|
||||
| `syspkg_alpine.sh`, `overwrite_alpine.sh` | `alpine:latest` with `/src`, `/libsrc` and `/t.sh` mounted (see each header); `overwrite_alpine.sh` runs both as-is and with `--tmpfs /glacier:exec` |
|
||||
| `verify_alpine.sh`, `user_layout_alpine.sh`, `upgrade_alpine.sh` | `glacier-alpine_root` |
|
||||
|
||||
Build the `glacier-alpine_root` image with:
|
||||
```sh
|
||||
cd ~/Projects/glacier-ng && docker build -f Dockerfile.alpine_root \
|
||||
--build-context libglacier=../libglacier-ng -t glacier-alpine_root .
|
||||
docker run --rm -v "$PWD/tests/NAME.sh:/t.sh:ro" glacier-alpine_root sh /t.sh
|
||||
```
|
||||
|
||||
**Interactive images:** `./run_docker.sh Dockerfile.{alpine,alpine_root,arch,debian}`.
|
||||
`run_docker.sh` passes the `libglacier` build context. `alpine`, `arch`
|
||||
and `debian` default to user `glacier` (uid 1000, in wheel); `alpine_root`
|
||||
defaults to root. All of them have:
|
||||
- root-owned `/app`;
|
||||
- no `/glacier` until `gstore -n` / `syspkg -n`;
|
||||
- the doas rule plus `permit nopass glacier`;
|
||||
- `glacier.conf` installed.
|
||||
|
||||
**Boot test:** `cd ~/Projects/everest && ./everest.sh boot` boots QEMU
|
||||
(KVM). Press Enter for a root shell; `poweroff -f` to leave.
|
||||
|
||||
## Bug history: so these don't get reintroduced
|
||||
|
||||
All fixed in the current tree.
|
||||
|
||||
1. `gl_abort_transaction` never released the lock (wrong variable). Now
|
||||
`open(O_CREAT)` + `flock()`, and lock files are never unlinked.
|
||||
2. No `EXDEV` check before renaming. Commit `stat()`s both sides and
|
||||
returns `GL_TXN_ERR_XDEV`.
|
||||
3. Segfault on install: the `transaction` target didn't link
|
||||
`istoreutils`/`log`.
|
||||
4. Stale links post-commit: `gl_link_pkg` ran during staged installs,
|
||||
embedding stage paths. It now runs only for LIVE contexts, and links
|
||||
regenerate post-commit.
|
||||
5. Dangling links were never cleaned on removal → `prune_stale_links`.
|
||||
6. `GL_TXN_ERR_RENAME` after the first commit: a leftover `old/`. It's now
|
||||
wiped first.
|
||||
7. `-Wstringop-truncation`: `strncpy(..., PATH_MAX - 1)` could leave
|
||||
strings unterminated. Use `snprintf(dst, sizeof dst, "%s", ...)`.
|
||||
8. gbuild detected a bare `Makefile` before `configure`. The fix had been
|
||||
lost from the tree and was re-applied 2026-10-04.
|
||||
9. The index had no version field; it's now `repo::pkgname::version`.
|
||||
10. [The code this was about is NOT in the tree: ROADMAP 4.1.] A recipe
|
||||
field silently never parsed and fell back to a plausible default.
|
||||
Remember the class.
|
||||
11. `gsha256sum.c` included `sha256.c` instead of `.h`.
|
||||
12. gbuild told users to run `gpkg -f` (a stub) instead of `gpkg -l`.
|
||||
13. `gpkg -f`'s stub referenced an undefined `uid`.
|
||||
14. Path traversal from manifests/archives: `PKG_NAME`/`PKG_REPO` must
|
||||
pass `gl_valid_pkg_ident`, and `..`/absolute entries refuse the
|
||||
package. Finding 1 (self-declared `base`) stays open until signing.
|
||||
15. Stage seeding and `gl_rmdir_recursive` skipped ALL dot-entries:
|
||||
dotfiles vanished, and `old/` couldn't be removed (bug 6 again).
|
||||
16. Per-user transactions could never work for a real user (shared
|
||||
writable parents needed). Fixed by the per-uid layout. Found alongside:
|
||||
- `gstore -n` never created parent directories;
|
||||
- the stage was wiped before the lock was taken;
|
||||
- cross-filesystem seeding was reported as `MKDIR`;
|
||||
- `gl_restore_istore` doubled the uid and accepted `..` entries;
|
||||
- gstore loaded the combined `.so` via `LIBGLACIER_SO` from the
|
||||
environment.
|
||||
17. Package symlinks were silently dropped at extraction, and only regular
|
||||
files were linked, so musl's loader was never installed. Alpine-based
|
||||
tests passed only because Alpine provides its own loader.
|
||||
18. `add_to_archive` (`gstore -b`, backups) silently dropped symlinks.
|
||||
19. gpkg acted on `-l` as soon as it parsed it, so `-lO` ignored its `O`.
|
||||
20. `tests/glacier_ffi.lua` duplicated the cdefs and had drifted (it was
|
||||
missing `GL_INSTALL_ERR_SCOPE`). It's now a wrapper.
|
||||
21. Link conflicts were re-reported on every system commit, for every
|
||||
package (it now uses the `.glacier-fresh` marker).
|
||||
22. A failed extraction left a half-made version directory, so in a live
|
||||
context every retry reported `ALREADY_INSTALLED`.
|
||||
23. Backups written in the same second were pruned in the wrong order
|
||||
(whole-second mtimes).
|
||||
|
||||
## Conventions
|
||||
|
||||
- **C style:** tabs, but the codebase is inconsistent; match the
|
||||
surrounding function.
|
||||
- **Paths:** `PATH_MAX` stack buffers plus `snprintf` for path
|
||||
construction.
|
||||
- **Lua requires:** gpkg uses bare CWD-relative requires.
|
||||
syspkg and gstore pin their paths (see CLIs).
|
||||
- **Lua numbers in C varargs:** a Lua number reaches C varargs as a
|
||||
double, so never pass one to `lg_printf("%d")`. Use `%s` with
|
||||
`tostring()`, or `ffi.new("int", n)`.
|
||||
- **Shell commands from Lua:** always `shell_quote` interpolated paths
|
||||
(syspkg runs as root).
|
||||
- **Verify by running.** Several real bugs here were found only by
|
||||
executing something.
|
||||
Reference in new issue
Block a user