7/19
This commit is contained in:
261
CLAUDE.md
Normal file
261
CLAUDE.md
Normal file
@@ -0,0 +1,261 @@
|
||||
# CLAUDE.md
|
||||
|
||||
Context for Claude Code sessions on this repo (libglacier-ng, the package
|
||||
manager for the Everest/Glacier Linux distribution). This file exists
|
||||
because there's no built-in way to hand off a claude.ai planning
|
||||
conversation into Claude Code — this is that handoff, written down.
|
||||
|
||||
For the symlink-routing / package-scope design specifically, see
|
||||
`PACKAGE_SCOPES.txt` at the repo root — that doc is the source of truth
|
||||
for that topic and isn't duplicated here.
|
||||
|
||||
## What this project is
|
||||
|
||||
A from-scratch Linux distribution (Everest, built on the Glacier
|
||||
package-management model) with its own package manager
|
||||
(`libglacier-ng` + `gpkg`/`gstore` CLI tools) and its own build
|
||||
tooling (`gbuild`/`gworld`) for bootstrapping a set of packages up to
|
||||
a minimal bootable system.
|
||||
|
||||
## Core architecture
|
||||
|
||||
**`gl_context_t`** (transaction.h) is the central abstraction: `uid`,
|
||||
`root_path`, `index_path`, `store_path`, `links_path`, `mode`
|
||||
(`GL_CTX_LIVE` / `GL_CTX_STAGE`), `scope` (`GL_SCOPE_USR` /
|
||||
`GL_SCOPE_SYS`), `lock_fd`.
|
||||
|
||||
**Transaction lifecycle**: `gl_init_live_context` / `gl_init_stage_context`
|
||||
→ `gl_commit_transaction` or `gl_abort_transaction`. Staged installs
|
||||
hardlink-seed `index`/`store` from live at stage-init time (copy-on-write,
|
||||
so removing a live package inside a transaction doesn't require special
|
||||
casing). The `links` tree is deliberately **not** seeded — see below.
|
||||
|
||||
**Commit** does atomic rename-based promotion (stage → live, with the
|
||||
previous live moved to `old/` as a rollback point), fsyncs the parent
|
||||
dirs, then calls `gl_relink_store` to regenerate symlinks and
|
||||
`gl_rebuild_index` to regenerate the index — both from a fresh
|
||||
`GL_CTX_LIVE` context, not the just-committed stage context.
|
||||
|
||||
**Scopes**:
|
||||
- `GL_SCOPE_USR`: per-uid, `/glacier/usr/{index,store,links}/<uid>`.
|
||||
- `GL_SCOPE_SYS`: system-wide, `/glacier/sys/{index,store}`. Symlink
|
||||
destination depends on the package's **repo** — see `PACKAGE_SCOPES.txt`.
|
||||
Short version: `repo == "base"` → `/usr`; anything else →
|
||||
`/glacier/sys/links`. This is decided per-package (in `gl_link_pkg` /
|
||||
`gl_relink_store`), not baked into the context.
|
||||
|
||||
**Repos**: `base` (minimal working system, boot-critical by definition —
|
||||
this is a repo-level classification, not a per-package flag), `extra`
|
||||
(important but not required), `community` (everything else). `extra`
|
||||
and `community` currently behave identically — see open items.
|
||||
|
||||
## Safety mechanisms (read before touching links/symlink code)
|
||||
|
||||
`ctx->links_path` is **not always glacier-exclusive territory** —
|
||||
for system-scope `base` packages it's literally `/usr`, which holds
|
||||
plenty of content glacier has no business touching. Everything that
|
||||
reads or writes a links destination has to treat it as "possibly
|
||||
shared":
|
||||
|
||||
- `is_glacier_symlink(path, store_prefix)` (istoreutils.c) is the one
|
||||
source of truth for "did glacier create this." True only if `path`
|
||||
is a symlink whose target lives under `store_prefix`. Never touch a
|
||||
path this returns false for.
|
||||
- `gl_link_pkg` will not overwrite an existing path unless
|
||||
`is_glacier_symlink` says it's safe to replace.
|
||||
- `gl_relink_store` prunes stale symlinks (`prune_stale_links`) rather
|
||||
than wiping and rebuilding the whole links directory — a full
|
||||
`rm -rf` of `ctx->links_path` would be catastrophic once that path
|
||||
can be `/usr`. It prunes **both** possible system-scope destinations
|
||||
(`/glacier/sys/links` and `/usr`) since a single pass can't know in
|
||||
advance whether any base-repo packages are involved.
|
||||
- The stage links tree is never hardlink-seeded — it's genuinely
|
||||
unused (`gl_link_pkg` only ever runs against `GL_CTX_LIVE`;
|
||||
`gl_relink_store` always builds its own fresh live context). Seeding
|
||||
it would mean hardlinking all of `/usr` on every staged system
|
||||
transaction, and risking `EXDEV` if `/usr` and the stage area are on
|
||||
different filesystems.
|
||||
|
||||
## Bug history (so it doesn't get relitigated or reintroduced)
|
||||
|
||||
Roughly chronological, all already fixed in the current tree:
|
||||
|
||||
1. **`gl_abort_transaction` never actually released the lock file** — a
|
||||
copy-paste bug wrote into the wrong variable. Fixed by moving off
|
||||
`O_EXCL`-based locking entirely: `gl_begin_transaction` now just
|
||||
`open(O_CREAT)` + `flock()`, and lock files are never `unlink()`'d
|
||||
(the kernel releases `flock()` automatically on process death, even
|
||||
`SIGKILL`; explicit unlinking has its own TOCTOU race with a second
|
||||
process creating a new inode at the same path).
|
||||
2. **`EXDEV` wasn't checked before renaming** — `gl_commit_transaction`
|
||||
now `stat()`s both sides and fails cleanly with `GL_TXN_ERR_XDEV`
|
||||
before attempting any renames if stage and live aren't on the same
|
||||
filesystem.
|
||||
3. **Segfault on install**, root cause: the Makefile's `transaction`
|
||||
target didn't link `libglacier_transaction.so` against
|
||||
`libglacier_istoreutils.so`/`libglacier_log.so`, even though
|
||||
`transaction.c` calls `gl_relink_store()` and `lg_printf()`. Fixed
|
||||
by adding those `-l` flags and an explicit `transaction: log
|
||||
istoreutils` prerequisite (build order matters now).
|
||||
4. **Symlinks going stale after commit** — `gl_link_pkg` used to run
|
||||
during staged installs too, embedding the *stage* path as the
|
||||
symlink target. Fixed: `gl_link_pkg` only runs for `GL_CTX_LIVE`;
|
||||
links are fully regenerated post-commit by `gl_relink_store`.
|
||||
5. **Dangling symlinks never cleaned up on removal** — see "Safety
|
||||
mechanisms" above; this is what `prune_stale_links` fixes.
|
||||
6. **`GL_TXN_ERR_RENAME` on every commit after the first** — leftover
|
||||
`old/` directory from a previous commit made the next commit's
|
||||
`rename()` fail with `ENOTEMPTY`. Fixed: `gl_commit_transaction`
|
||||
wipes `old_base` if present before using it.
|
||||
7. **`-Wstringop-truncation` errors** (real bug, not noise) — four
|
||||
`strncpy(dst, ctx->foo_path, PATH_MAX - 1)` calls that could leave
|
||||
`dst` unterminated. Fixed by switching to `snprintf(dst,
|
||||
sizeof(dst), "%s", ...)`, which always terminates.
|
||||
(`-Wformat-truncation`, separately, is suppressed via
|
||||
`-Wno-error=format-truncation` in `config.mk` — those warnings are
|
||||
about genuinely-safe `snprintf` truncation, not a real bug.)
|
||||
8. **`gbuild` build-system autodetection** checked for a bare
|
||||
`Makefile` before `configure`/`configure.ac`. Broke musl (and any
|
||||
project with a hand-rolled, non-autotools build system that ships
|
||||
both) since its `Makefile` is non-functional without `configure`
|
||||
generating `config.mak` first. Fixed by reordering the checks.
|
||||
9. **Index only stored `repo::pkgname`, no version.** Now
|
||||
`repo::pkgname::version` (`gl_rebuild_index` looks up the
|
||||
`<pkgname>-<version>` store subdirectory). `struct gpkg_entry` in
|
||||
`istoreutils.h` needs a `char *ver;` field — this was a manual
|
||||
header edit, double check it's actually present.
|
||||
10. **`gworld` recipe parser silently dropped `SYS_PROFILE`** — the
|
||||
key-matching regex only matched uppercase letters, not
|
||||
underscores. Fixed to `^([%u_][%u%d_]*)=(.*)$`. Caught by an
|
||||
actual precedence test, not code review — worth remembering that
|
||||
class of bug (a field silently never parsing, falling back to a
|
||||
default that happens to look right) is easy to miss by inspection
|
||||
alone.
|
||||
|
||||
## CLI additions
|
||||
|
||||
- `gpkg -s` / `--system`: operate against `GL_SCOPE_SYS` instead of a
|
||||
uid's tree. Requires root (checked via `geteuid()`). Must appear
|
||||
*before* `-l`/`-x` in a grouped flag string (e.g. `-sl`, not `-ls`) —
|
||||
flags are handled in parse order, same constraint `-V`/`-S` already
|
||||
had.
|
||||
- `gstore -N` / `--new-system`: calls `gl_init_sys()`. Requires root.
|
||||
- `gbuild -P` / `--profile PROFILE`: system profile, `ARCH-LIBC[-FEATURE...]`
|
||||
(e.g. `x86_64-musl`, `x86_64-glibc-multilib`). Derives a cross-compile
|
||||
prefix (translating `glibc`→`gnu` per real GNU triplet convention),
|
||||
detects native-vs-cross against the build host, and only pre-fills
|
||||
`-t`/`-x` defaults for whichever the user didn't already set
|
||||
explicitly. Features export as `GL_FEATURE_<NAME>=1` env vars for
|
||||
custom `-b`/`-i` commands to branch on — gbuild itself doesn't
|
||||
hardcode per-feature behavior, that's intentional.
|
||||
|
||||
## Build system
|
||||
|
||||
`Makefile` builds separate `.so`s per module
|
||||
(`libglacier_log.so`, `libglacier_pkg.so`, `libglacier_istoreutils.so`,
|
||||
`libglacier_transaction.so`, `libglacier_dag.so`) — **this split is
|
||||
deliberate Unix-philosophy design**, not an accident: a third-party
|
||||
glacier-compatible program should be able to link only
|
||||
`libglacier_log.so` without pulling in everything else. `gpkg`/`gstore`
|
||||
load the specific per-module `.so`s they need directly, matching this.
|
||||
|
||||
There's also a `libglacier-ng` Makefile target producing a **combined**
|
||||
`.so` — this exists *solely* for the Lua FFI test harness's
|
||||
convenience (one thing to `ffi.load()` instead of juggling rpath
|
||||
chains in a throwaway test process). It is **not** in `all` and **not**
|
||||
installed. Don't reach for it outside `tests/` — that's a sign
|
||||
something should be split-loaded instead.
|
||||
|
||||
`config.mk` holds `CFLAGS`, including `-Wno-error=format-truncation`
|
||||
(see bug #7 above for why that specific warning is suppressed and
|
||||
others aren't).
|
||||
|
||||
## Lua tooling
|
||||
|
||||
- `gpkg`, `gstore`: LuaJIT + FFI, calling directly into the `.so`s.
|
||||
- `lib/glacier_cdef.lua`: single shared source of truth for every
|
||||
`ffi.cdef()` declaration, used by `gpkg`, `gstore`, and the test
|
||||
suite. This used to be duplicated three ways, which is exactly what
|
||||
let a stale signature drift silently into `gpkg` at one point — if
|
||||
you're editing a C function signature, this file needs the matching
|
||||
update, and it's the *only* place that needs it now.
|
||||
- `tests/*.lua`: LuaJIT FFI test suite exercising real filesystem/process
|
||||
behavior (`fork`+`SIGKILL` for lock crash-recovery, real `mount()`
|
||||
for the `EXDEV` check, etc.) — not mocked. Run via:
|
||||
```sh
|
||||
cd tests
|
||||
make -C .. libglacier-ng # builds the test-only combined .so
|
||||
GLACIER_TEST_CONFIRM=yes LIBGLACIER_SO=$(pwd)/../build/lib/shared/libglacier-ng.so luajit run_all.lua
|
||||
```
|
||||
`test_exdev.lua` needs `--cap-add=SYS_ADMIN` in Docker to actually
|
||||
exercise the mount path; it skips cleanly (not a failure) without it.
|
||||
|
||||
## Build automation (bootstrap toolchain)
|
||||
|
||||
- `gbuild` (bash): single-package builder. Autodetects build system
|
||||
(cmake/meson/cargo/go/autotools/make), supports cross-compilation
|
||||
(`-t`/`-x`, now also `-P` for profiles).
|
||||
- `gworld` (Lua): batch orchestrator. Reads `recipes/*.recipe`,
|
||||
topologically sorts by `DEPS`, invokes `gbuild` once per package in
|
||||
order. `-s SYSROOT` gives every package in the batch a shared
|
||||
staging dir — relies on `gbuild -p` never wiping its target, so
|
||||
headers/libs accumulate across the batch for free.
|
||||
- Recipe format: `NAME VER REPO URL/REF-or-SRC DEPS BUILD INSTALL
|
||||
FLAGS SYS_PROFILE` (plain `key=value`, see `recipes/musl.recipe` for
|
||||
a real example).
|
||||
- Real multi-pass toolchain bootstraps (LFS-style — gcc built twice,
|
||||
etc.) should use separate recipes per pass (`gcc-pass1`,
|
||||
`gcc-pass2`), publishing only the final pass under the real
|
||||
name/repo. The DAG models build steps, not abstract packages.
|
||||
- **`grootstrap` needs a rebuild, not yet done.** The old design (raw
|
||||
`tar` extraction into a target dir + chroot + register everything
|
||||
under an arbitrary bootstrap uid) predates the `GL_SCOPE_SYS` +
|
||||
base-repo design and is now more complicated than necessary. Once
|
||||
`gpkg`/`gstore` themselves exist on the target (still an unsolved
|
||||
bootstrapping-the-bootstrapper problem — how do they get there
|
||||
before a working package manager exists to install them with),
|
||||
every package including `base` ones can go through the same `gpkg
|
||||
-s -l` path; the repo alone routes symlinks correctly. No more
|
||||
special-casing base packages during bootstrap.
|
||||
|
||||
## Known open items / stubs
|
||||
|
||||
- `gpkg -f` / `-u` (merge/update) are unimplemented stubs. `-f`
|
||||
references an undefined `uid` variable (should be `uidn`) — will
|
||||
nil-concatenation-error the moment it's actually implemented.
|
||||
- `gpkg -x`'s confirmation summary shows placeholder `unknown::pkg
|
||||
0.0.0 (unknown)` since it has no local manifest to read version/repo
|
||||
from for a bare package name. Would need `resolve_repo` called
|
||||
before building the summary, not just before the removal itself.
|
||||
- `gpkg -x` doesn't print a completion message on success (`-l` does:
|
||||
"Completed with no errors."). Cosmetic inconsistency.
|
||||
- Reinstalling an up-to-date package double-logs ("Installed X" +
|
||||
"staged from X" for the same event).
|
||||
- `extra` vs `community` repos don't currently diverge in behavior —
|
||||
both just mean "not base" for symlink-routing purposes.
|
||||
- `gl_relink_store`/`gl_rebuild_index` are full rescans of the store on
|
||||
every commit — fine at current scale, worth revisiting if large
|
||||
transactions become common (a dirty-set of touched packages instead
|
||||
of full rescans).
|
||||
- Package manifests don't record which `SYS_PROFILE` they were built
|
||||
under — deliberately not added yet (would need a `pkg.c` /
|
||||
`gl_gpm2gpkg` change), flagged as a future option, not a gap that
|
||||
needs fixing now.
|
||||
- A stray nested `lib/glacier/lib/glacier/` directory was spotted once
|
||||
in a `tree` listing — looked like a leftover `make install`
|
||||
artifact, not in any load path, harmless but worth a `rm -rf`
|
||||
eventually.
|
||||
|
||||
## Conventions worth knowing before editing
|
||||
|
||||
- C source generally uses tabs for indentation, though the codebase is
|
||||
inconsistent about it in places (some blocks use spaces). ALWAYS USE TABS WHEN POSSIBLE.
|
||||
- `PATH_MAX`-sized stack buffers + `snprintf` is the standard pattern
|
||||
for path construction throughout; prefer it over `strncpy` for
|
||||
anything copying into a fixed buffer (see bug #7).
|
||||
- Don't add convenience wrappers that silently touch a broader
|
||||
filesystem scope than what's asked — this codebase has already been
|
||||
bitten twice by exactly that class of mistake (`gl_relink_store`'s
|
||||
wipe, `gl_link_pkg`'s blind `unlink`), both only becoming dangerous
|
||||
once a path assumption (`links_path` is glacier-exclusive) quietly
|
||||
stopped holding.
|
||||
Reference in New Issue
Block a user