Chapter 14: Virtual Filesystem Layer

VFS architecture, dentry cache, mount tree, path resolution, overlayfs, mount namespace operations

---

The Virtual Filesystem Layer provides a unified interface over all filesystem implementations. Dentry caching, inode management, mount tree operations, and path resolution are kernel-internal — filesystems plug in via well-defined traits. FUSE, overlayfs, configfs, and autofs are first-class citizens, not afterthoughts.

14.1 Virtual Filesystem Layer

The VFS (umka-vfs) provides a unified interface over all filesystem types. Like all UmkaOS modules, VFS is tier-agnostic: the same binary runs at any tier. The VFS manifest declares preferred_tier, minimum_tier, maximum_tier, and fallback_bias; the loader computes the effective tier from the manifest plus signing certificate ceiling, license ceiling, hardware capability, and operator override (see Section 11.3 for the selection algorithm and Section 12.6 for the KabiDriverManifest structure). Transport selection happens at bind time — kabi_call! resolves to a direct vtable call when VFS is co-located with the caller and to a ring dispatch when they're in different domains (see Section 12.8). When VFS is cross-domain from the core kernel, the shared VFS+FS domain provides crash containment but does not provide mutual isolation between VFS and the filesystem drivers sharing its domain.

Rationale for placing VFS in a separate isolation domain when the architecture supports it:

The VFS handles complex, security-sensitive operations: path resolution (symlink loops, mount point crossing), permission checks, and filesystem driver coordination. Isolating VFS from Core provides:

1. Attack surface reduction: Path resolution bugs (symlink attacks, directory traversal) are confined to the VFS domain and cannot corrupt Core memory.

2. Domain boundary: When VFS is cross-domain from the core kernel, the caller crosses one domain boundary to reach the VFS+FS domain. A compromised VFS+FS domain cannot corrupt Core memory. VFS and filesystem drivers share a domain by default, so a filesystem driver bug can corrupt VFS metadata within the shared domain. On platforms with sufficient isolation domains (e.g., x86-64 with PKU and few active isolated drivers), VFS and filesystem drivers may be placed in separate domains, providing inter-driver hardware isolation. Rust memory safety — not hardware isolation — is the primary defense against filesystem driver bugs within the shared domain. The hard isolation boundary is between Core and the VFS+FS domain, not between VFS and individual filesystem drivers.

3. Crash containment: A VFS panic (e.g., corrupted dentry cache) is recoverable without rebooting the entire kernel. The recovery protocol:

a. Detection: Core detects VFS domain death (MPK exception, panic handler, or watchdog timeout on the VFS heartbeat ring). b. Block at the boundary: New VFS operations stop at the Core domain boundary. select_ring() rejects producers with a transport-level ENXIO while ring_set.state == VFSRS_RECOVERING (set at Step U3 of the unified recovery sequence, Section 14.3). The Core VFS dispatch wrapper converts this boundary ENXIO into an interruptible sleep on sb.recovery_wait (see the SuperBlock struct) and retries the dispatch after wake — callers BLOCK for the recovery duration; they do not observe an error. A signal arriving during the sleep interrupts it with -ERESTARTSYS, handled by the standard syscall-restart machinery (transparent restart under SA_RESTART, -EINTR to userspace otherwise) — -ERESTARTSYS is never generated without a pending signal. Requests already in flight inside the crashed domain are NOT retried by the kernel: their execution state died with the domain, and they are drained with -EIO (Steps U6-U8 and U11 of the unified sequence). c. Dirty page cache flush: Dirty pages in Core's page cache are flushed to their backing block devices. The page cache is in Core memory (not VFS memory), so it survives the VFS crash. Flush uses the block layer ring directly. d. Dentry/inode cache rebuild: The new VFS instance starts with an empty dentry cache. Dentries are lazily re-populated on the next path lookup (cache miss triggers disk read). Inode cache is similarly rebuilt on demand. The crashed instance's dentry storage slots are returned to the Nucleus tracked allocator before the driver reload (Section 14.1) — without that reclaim, each crash would permanently consume dcache instance budget: a 50-year-uptime resource leak. e. Mount tree reconstruction: Core maintains the Shadow Mount Registry — a Core-domain structure holding, for every namespace-attached mount, both the OWNING Arc<Mount> reference and a Core-resident re-creation record (superblock identity, mountpoint, flags, propagation, owning namespace). The registry is updated ONLY by authority-confirmed commit reports from the mount-tree engine — never by pre-dispatch syscall-layer interception — through the stage/commit/abort bracket defined in Section 14.6. After a crash of the domain hosting the umka-vfs module, the recovery worker and the reloaded VFS instance rebuild every mount namespace's tree from the registry. Full structures, reporting protocol, reconstruction sequence, and trust analysis: Section 14.1. A provider-only crash (a filesystem driver in a different domain than umka-vfs) does not use the registry: the mount tree is structurally intact and only the affected superblock runs the unified per-superblock recovery sequence.

  Pseudo-filesystems (/proc, /sys, /dev/devtmpfs) are re-attached from
  kernel state — they have no on-disk backing. Overlayfs is re-constituted
  from the recorded mount data (lower/upper/work paths). FUSE mounts that
  require a userspace daemon connection receive `-ENOTCONN` until the
  daemon reconnects.

f. Open file descriptor recovery — lazy revalidation: Core's FdTable (in Tier 0 Core, per-task via Task.files: ArcSwap<FdTable>, Section 8.1) and every OpenFile it references are Core memory and survive the VFS crash. No eager re-open pass runs during recovery: each surviving fd is revalidated on its FIRST post-recovery operation by the generation-refresh protocol (Section 14.1), which detects the open_generation mismatch and transparently re-runs FileOps::open() against the reloaded driver instance to regenerate the driver's per-open state (private_data). All Core-resident fd state (f_pos, f_flags, f_mode, f_cred, f_wb_err, ra_state, file lock records) carries over unchanged. File descriptors whose backing inode no longer exists after journal replay (e.g., unlinked-but-open files whose keep-alive was driver-domain state) fail revalidation and receive -EIO on that and every subsequent access. g. Resume: Step U17 stores VFSRS_ACTIVE (Release) and the recovery worker's resume_fn wakes sb.recovery_wait. Every dispatcher blocked in step b retries select_ring() and proceeds — passing through fd revalidation (step f) where its open_generation is stale. No syscall that blocked at the boundary is failed by the recovery itself.

Recovery time: ~100-500ms, dominated by the committed-extent bypass flush (Step U10b) and journal replay (Step U14(e)). Independent of the number of open file descriptors — fd revalidation is lazy, paid per fd on its first post-recovery use (one FileOps::open() round-trip, microseconds). Limitation: In-flight writes that had not yet reached the page cache are lost (the application receives -EIO and must retry).

Domain grouping limitation: The crash recovery protocol above is most effective when the crash originates from VFS logic itself (e.g., a bug in path resolution or dentry management). Because VFS and filesystem drivers share an isolation domain by default, a filesystem driver bug can corrupt VFS metadata (dentry cache, mount tree, inode state) before detection. In this case, the corrupted VFS state may have already produced incorrect I/O (wrong block mappings, stale metadata replies) before the domain crash is detected by Core. Recovery restores VFS to a clean state, but data written to disk under corrupted VFS guidance may be silently wrong. This is a known limitation of domain grouping — the hardware fault boundary catches the crash, but cannot retroactively undo I/O performed with corrupted in-domain state. Rust memory safety mitigates this risk by preventing most classes of memory corruption bugs, but unsafe code within the shared domain remains a vector.

In-flight write definition: In-flight writes are writes that have entered the VFS write path (passed the syscall boundary) but whose data has not yet been inserted into the page cache. This includes: (1) writes buffered in the VFS ring command queue awaiting processing, (2) writes being copied from user buffer to a page that has not yet been marked dirty. Writes that have reached the page cache (page marked PageFlags::DIRTY with a committed dirty extent) are NOT in-flight — they survive VFS crash via the dirty extent protocol (Section 14.4). Error reporting for lost in-flight writes: the write() syscall returns -EIO if the VFS crashes during the write. If write() had already returned success, the data is in the page cache and is safe. Applications should retry write() calls that returned -EIO after VFS recovery completes (the ring is reopened in step g of the recovery protocol).

14.1.1.1.1 Shadow Mount Registry and Mount Tree Reconstruction

Pseudocode convention: Code in this section uses Rust syntax and follows Rust ownership, borrowing, and type rules. &self methods use interior mutability for mutation. Atomic fields use .store()/.load(). See CLAUDE.md §Spec Pseudocode Quality Gates.

Authority analysis — why this design. Three candidate authorities exist for rebuilding the mount tree after a crash of the domain hosting umka-vfs, and two of them are unsound:

1. The surviving Mount/MountNamespace instances themselves. Both types live in Nucleus tracked storage (Section 14.6), so their slots, side-header refcounts, and live-registry membership survive the crash and are trustworthy (Nucleus-owned metadata the driver domain cannot write). Their PAYLOAD contents, however, were writable by the crashed domain — the same trust split the dentry reclamation pass documents (Section 14.1, "Trust note"). Tree linkage, flags, propagation state, and device names read from post-crash payloads may be corrupted, and the mountpoint/root dentry references point into a dcache that recovery discards wholesale. Payloads cannot be the reconstruction authority.
2. A syscall-layer shadow updated before dispatching mount()/umount(). Structurally wrong on two axes. First, failure drift: an update-before- dispatch shadow records a mount that VFS then fails (phantom entry) and deletes a mount whose umount then returns EBUSY (silently dropped live mount). Second — and unfixable by reordering — most mount-tree transitions never pass through a mount(2)/umount(2) interception point at all: shared-subtree propagation creates mounts in OTHER namespaces as a side effect (Section 14.6), clone(CLONE_NEWNS)/unshare clone entire trees via copy_tree, fsmount()/move_mount()/open_tree() create and attach mounts through the new mount API, autofs expiry (MNT_SHRINKABLE) unmounts with no syscall in flight, and namespace teardown destroys whole trees when the last task exits. A syscall-layer shadow would have to re-implement the entire propagation engine to stay accurate.
3. A Core-side registry updated by the mount-tree engine itself, at transaction commit. The engine is the one component that knows the exact delta of every transition — including propagation-created clones — at the moment it commits. This is the design used here.

The rule this section instantiates (SHADOW-STATE-DIVERGENCE class remedy): a Core-side shadow of driver-domain state is updated ONLY on authority-confirmed transitions reported by the subsystem that commits them — never by interception ahead of dispatch, and never by scraping domain-writable memory. Where the shadow exists to survive the authority's death, it must also hold the OWNING reference to the shadowed object, so that (a) a transition that was never reported is a liveness bug caught immediately in normal operation (the object has no owner and is freed while attached), not a silent recovery divergence discovered after a crash, and (b) post-crash reclamation needs no walk over untrusted payloads — the shadow drops the references it itself holds.

Structures. All shadow structures are Core-internal: they never cross a KABI, wire, or userspace boundary, so they are ordinary Rust structs — NOT #[repr(C)], no fixed-size byte arrays, no truncation. (An earlier revision used a #[repr(C)] entry with fstype: [u8; 32] and a truncating fs_data: [u8; 256]; both are gone. fstype[32] silently truncated FUSE types like fuse.<subtype> — recovery would then re-mount an unknown or WRONG filesystem type. Exact heap copies eliminate the entire truncation class. The copies are bounded by the same limits the syscall entry already enforces on the source strings: Linux fs/namespace.c copy_mount_string() in Linux is strndup_user(data, PATH_MAX) and copy_mount_options() copies exactly one page — verified against torvalds/linux master.)

/// Core-domain authoritative shadow of every namespace-attached mount.
/// Survives any driver-domain crash; contents are written only by
/// `shadow_txn_commit()` (below). Warm path only (mount-tree transactions
/// and crash recovery) — never touched by per-syscall hot paths.
pub struct ShadowMountRegistry {
    /// Per-namespace shadow trees, keyed by `MountNamespace.ns_id`.
    namespaces: XArray<Box<ShadowNamespace>>,
    /// Superblock records, keyed by shadow-assigned `sb_key`. Registry-level
    /// (not per-namespace): bind mounts and propagation clones in different
    /// namespaces share one superblock.
    superblocks: XArray<Box<ShadowSuperblockRecord>>,
    /// Identity index over `superblocks`: `DevId` (unique per LIVE
    /// superblock — a real device number or an anonymous one, exactly the
    /// property Linux `sget` dedup relies on) → `sb_key`. Staging resolves
    /// engine-supplied `Arc<SuperBlock>` identities through this index —
    /// the engine never sees or handles sb_keys — double-checking
    /// `Arc::ptr_eq` against the found record's pinning `sb`, so a
    /// wrong-`s_dev` report from a corrupted engine aliases nothing and
    /// fails staging with EINVAL. An index entry lives exactly as long as
    /// its record. Guarded by `registry_lock`.
    sb_index: XArray<u64>,
    /// In-flight transactions (staged, not yet committed), keyed by token.
    staged: XArray<Box<ShadowTxn>>,
    /// Serializes delta application and recovery. Sleeping Mutex: all
    /// callers are process-context warm paths.
    registry_lock: Mutex<()>,
    /// Token allocator for `shadow_txn_begin()`. Monotonic u64, never reused.
    txn_id_allocator: AtomicU64,
    /// `sb_key` allocator. Monotonic u64, never reused.
    sb_key_allocator: AtomicU64,
    /// Incremented once per completed mount-tree RECONSTRUCTION (crash
    /// shape 2 only; provider-only recoveries never touch it). Lets fd
    /// rebind distinguish "the tree was rebuilt and this mount is gone"
    /// (latch EIO) from "the tree is intact and this fd legitimately
    /// references a lazy-unmounted/detached mount outside the shadow"
    /// (keep the existing reference). u64: one increment per VFS-module
    /// crash — no wrap within the operational lifetime.
    crash_epoch: AtomicU64,
}

/// Shadow of one mount namespace's tree.
pub struct ShadowNamespace {
    /// Namespace identity (`MountNamespace.ns_id`).
    pub ns_id: u64,
    /// The surviving namespace instance. Held so that reconstruction can
    /// reinitialize the SAME instance in place — task-side
    /// `NamespaceSet.mount_ns` Arcs and `/proc/PID/ns/mnt` fds keep
    /// pointing at it, so namespace identity survives recovery.
    pub ns: Arc<MountNamespace>,
    /// Shadow entries, keyed by `mount_id` (per-namespace unique).
    pub mounts: XArray<Box<ShadowMountEntry>>,
    /// Post-recovery flag: `mnt_count` pins on reconstructed mounts have
    /// not yet been re-established from surviving `OpenFile`s. Checked by
    /// every `mnt_count == 0` teardown gate (umount EBUSY test, autofs
    /// expiry) — see "Open-file pins after reconstruction" below.
    /// 0 = reconciled (normal), 1 = sweep required. `AtomicU8`, not `bool`,
    /// per the repr(C)-adjacent flag convention.
    pub pins_unreconciled: AtomicU8,
}

/// One namespace-attached mount. Holds THE owning `Arc<Mount>` — the
/// tree-side strong reference that keeps an attached mount alive (see
/// [Section 14.6](#mount-tree-data-structures-and-operations--mount-node), Lifetime).
/// Detached mounts (`MNT_DETACHED`: fsmount(2) pre-attach, OPEN_TREE_CLONE)
/// and lazy-unmount remnants are NOT in the shadow — their owner is the
/// referencing file descriptor's `Arc`, and they intentionally do NOT
/// survive a VFS-module crash (see the EIO rule below).
pub struct ShadowMountEntry {
    /// `Mount.mount_id` — preserved across reconstruction, so
    /// `STATX_MNT_ID` values remain stable over a VFS crash.
    pub mount_id: u64,
    /// Parent mount within the same namespace. 0 = namespace root.
    pub parent_mount_id: u64,
    /// The owning reference. Before a crash: the pre-crash instance.
    /// After reconstruction: the fresh instance. Dropped (via `rcu_call`,
    /// after the engine has unhashed the mount) when a removal commits.
    pub mount: Arc<Mount>,
    /// Mountpoint path RELATIVE to the parent mount's root, exact copy
    /// (≤ PATH_MAX). Relative-to-parent — not absolute — so that stacked
    /// mounts on one dentry and mounts shadowed by later mounts remain
    /// representable; reconstruction attaches parents first, then resolves
    /// this path WITHIN the parent. Empty for the namespace root.
    pub mountpoint: Box<[u8]>,
    /// Which superblock this mount is a view of.
    pub sb_key: u64,
    /// This mount's root, relative to the superblock root. Empty for a
    /// full-filesystem mount; non-empty for bind mounts of subdirectories.
    pub root_path: Box<[u8]>,
    /// `MountFlags` bits at last commit (remount/mount_setattr update this).
    pub flags: u64,
    /// Propagation type at last commit.
    pub propagation: PropagationType,
    /// Peer group identity: `(allocating ns_id, group_id)`. Group IDs are
    /// allocated per-namespace, but peer groups SPAN namespaces (copy_tree
    /// preserves peer membership), so the allocating namespace disambiguates.
    /// `(0, 0)` = no peer group (Private/Unbindable).
    pub peer_group: (u64, u64),
    /// Master peer group for slaves, same encoding. `(0, 0)` = not a slave.
    pub master_group: (u64, u64),
}

/// One live superblock. Pins the SuperBlock instance so reconstruction can
/// re-attach mounts to the SAME superblock object — the fd revalidation
/// protocol depends on inode identity (`OpenFile.inode.i_sb`) surviving
/// recovery, so superblocks are REUSED, not re-created (asymmetric with
/// Mount instances, which are re-created because nothing load-bearing
/// hangs off their identity once fds are rebound).
pub struct ShadowSuperblockRecord {
    pub sb_key: u64,
    /// Pinning reference to the surviving SuperBlock.
    pub sb: Arc<SuperBlock>,
    /// Filesystem type string, exact copy (e.g. "ext4", "fuse.sshfs").
    pub fstype: Box<[u8]>,
    /// Backing device (`SuperBlock.s_dev`; anonymous dev for diskless).
    pub device: DevId,
    /// Device name string as passed to mount ("/dev/sda1", "tmpfs", ...).
    pub device_name: Box<[u8]>,
    /// Filesystem-specific mount data, exact copy (≤ one page — the
    /// `copy_mount_options()` bound), as of the last successful
    /// mount/remount. For overlayfs this carries lower/upper/work paths.
    pub fs_data: Box<[u8]>,
    /// Number of shadow entries (across all namespaces) referencing this
    /// record. Guarded by `registry_lock`; the record is removed when it
    /// reaches zero. Bounded by total mounts, not monotonic — u32 refcount
    /// exemption applies.
    pub refs: u32,
}

/// One in-flight staged transaction (the values of
/// `ShadowMountRegistry.staged`). Created by `shadow_txn_begin()`;
/// consumed whole by `shadow_txn_commit()` / `shadow_txn_abort()`
/// ([Section 14.6](#mount-tree-data-structures-and-operations--shadow-registry-reporting)).
/// ALL allocation and validation happen when an op is staged — commit only
/// links the pre-built records, which is what makes it infallible.
/// Dropping a `ShadowTxn` (abort, or reconstruction step 1) drops every
/// staged record with its `Arc` clones.
pub struct ShadowTxn {
    /// Token value (the key in `staged`); `ShadowTxnToken` payload.
    pub txn_id: u64,
    /// Staged operations in staging order. The engine stages parents
    /// before children and superblocks before the mounts that reference
    /// them, so commit applies `ops` front-to-back with no sorting.
    /// `Vec` bound: the transaction's touched-mount count (worst case the
    /// propagation fan-out of one operation) — warm path, allocation
    /// charged at staging time.
    pub ops: Vec<ShadowOp>,
}

/// One staged shadow operation. Add payloads are already in their
/// committed form: staging converted the engine's `ShadowMountRecord` /
/// `ShadowSbRecord`
/// ([Section 14.6](#mount-tree-data-structures-and-operations--shadow-registry-reporting))
/// into the boxed registry records, resolving the superblock identity to
/// `sb_key` through `sb_index`.
pub enum ShadowOp {
    /// Link a new entry into `ShadowNamespace.mounts`.
    MountAdd { ns_id: u64, entry: Box<ShadowMountEntry> },
    /// Unlink an entry; its owning `Arc<Mount>` is dropped via `rcu_call`
    /// (after the engine unhashed the mount — Teardown Ordering).
    MountRemove { ns_id: u64, mount_id: u64 },
    /// Overwrite exactly the `Some` fields of an existing entry.
    MountModify { ns_id: u64, mount_id: u64, delta: ShadowMountDelta },
    /// Link a new superblock record (and its `sb_index` entry).
    SbAdd { sb_key: u64, record: Box<ShadowSuperblockRecord> },
    /// Create the shadow tree for a new namespace (`copy_tree`).
    NsAdd { ns_id: u64, ns: Arc<MountNamespace> },
    /// Drop a namespace's shadow tree and every entry in it.
    NsRemove { ns_id: u64 },
}

/// The kernel-wide registry instance. Core (Tier 0) static, initialized
/// during VFS subsystem bring-up before the initial rootfs mount, so the
/// boot-time mounts are registered like any others.
static SHADOW_MOUNT_REGISTRY: ShadowMountRegistry = ShadowMountRegistry::new();

Memory cost: one entry per attached mount, dominated by the two path copies — typically well under 1 KiB per mount, ~30-100 mounts per container namespace. The owning Arc<Mount> adds no memory (the instance exists anyway); the Arc<SuperBlock> pins objects that live for the mount's lifetime regardless.

Reporting protocol. The mount-tree engine brackets every mount-tree transaction with the stage/commit/abort interface defined in Section 14.6. The contract, from the registry's side:

• Staged records are created (and their allocations charged) BEFORE the corresponding tree mutation commits; shadow_txn_commit() itself is infallible — it only links already-allocated records into the registry and drops removed entries' owning Arcs via rcu_call. An allocation failure therefore surfaces as ENOMEM from the staging call, while the engine can still roll the transaction back — never after the point of no return.
• The engine issues shadow_txn_commit() AFTER its tree mutation has committed, while still holding the lock(s) that serialized the transaction, and BEFORE returning success to the caller. Per-namespace ordering therefore follows mount_lock ordering; transactions in unrelated namespaces commute.
• The registry is maintained at EVERY tier, unconditionally. Tier assignment is runtime-changeable (Section 11.3); a mount created while VFS shares the Core domain must be recoverable after VFS is later demoted to an isolated domain and crashes. When VFS is co-located, kabi_call! resolves the reporting calls to direct calls — warm-path cost only.
• Crash-window semantics: if the domain dies after the tree mutation but before shadow_txn_commit(), recovery discards the staged transaction and restores the pre-transaction tree; if it dies after commit but before the syscall returns, recovery includes the transition. In both windows the in-flight caller receives -EIO (Step U6-U8 drain), and -EIO from a mount-class syscall is documented as indeterminate — the caller must re-query (statmount(2), /proc/self/mountinfo) before retrying. There is no window in which the caller has observed SUCCESS and recovery then loses the transition.
• Validation: the registry structurally validates every staged record under registry_lock — parent_mount_id must exist in the namespace (or be 0 for a root), an add record's superblock identity must resolve through sb_index to a live record or one staged earlier in the same transaction (with Arc::ptr_eq confirming the identity), path lengths must be within the syscall bounds, mount_id must be fresh for additions and present for removals/modifications, and the reported Arc<Mount> must point into live Nucleus tracked storage for the Mount type (a trusted metadata check). A validation failure rejects the STAGING call (the engine sees EINVAL and rolls back). This bounds — it cannot eliminate — the damage a corrupted-but-not-yet-crashed engine can do to the shadow; a corrupted engine that commits wrong-but-well-formed reports is the same accepted exposure as the rest of the "Domain grouping limitation" above.

Ownership. The shadow entry's Arc<Mount> is the ONE tree-side owning reference of an attached mount. The mount hash table, Mount.children, MountNamespace.mount_list, and the peer/slave rings are non-owning intrusive links (Section 14.6); RCU readers of those structures are protected because a removal commit drops the owning Arc from an rcu_call callback that Core queues only AFTER the engine has unhashed the mount (Section 14.6). Because attachment-without-report leaves a mount with no owner (it is freed at the next grace period and the bug surfaces as an immediate use-after-free in testing, caught by the tracked allocator's live registry), the reporting discipline is self-enforcing rather than best-effort bookkeeping.

Reconstruction sequence (crash shape 2 of Section 14.1 — the crashed domain hosted the umka-vfs module; shape 1 skips all of this). Runs in the recovery worker's process context after Step U13 (old domain memory freed, RCU callback drain complete) and interleaved with Step U14 as described below; all VFS dispatch is blocked at the boundary throughout, so no reader observes intermediate states:

1. Discard staged transactions: every ShadowTxn in staged is dropped — its records were never authority-confirmed. Staged Arc<Mount> clones drop with them (freeing never-attached instances).
2. Drop pre-crash owning references: for every ShadowMountEntry, move the owning Arc<Mount> out and drop it. The unpinned bulk of the old tree reaches strong count zero and frees through normal payload Drop — releasing each old mount's Arc<SuperBlock> contribution and its pinned mountpoint/root dentries (which the subsequent dentry reclamation pass then frees as unpinned). Instances still referenced by surviving OpenFile.mount Arcs become bounded orphans, freed at fd rebind or close — mirroring the dentry orphan rule. Payload Drop reads domain-writable fields; a corrupted pointer faults in the recovery worker and aborts recovery per Reload Failure Handling — the identical trust posture as the dentry pass.
3. Reinitialize each surviving MountNamespace in place (per ShadowNamespace.ns): overwrite the payload field-by-field WITHOUT running Drop on the old field values except where noted —
4. ns_id: restored from the shadow (the payload copy is untrusted).
5. root: the old RcuCell value's strong reference was already accounted by step 2's per-entry drop for the root entry (the root entry's recorded owning Arc and the cell's reference are BOTH dropped there — the shadow records roots with parent_mount_id == 0 and drops two references for them, one for the cell). Overwrite the cell without Drop.
6. hash_table: the bucket array pointer references crashed-domain heap that Step U13 already freed wholesale — overwrite WITHOUT Drop (freeing it would double-free). A fresh minimum-size table is installed; reconstruction re-inserts entries as it attaches.
7. mount_list, lock state, mount_count: reinitialized empty/unlocked/0.
8. id_allocator: restored to max(mount_id over shadow entries) + 1; group_id_allocator analogously from recorded peer-group IDs.
9. event_seq: restored to a value strictly greater than any pre-crash value Core observed (Core tracks the high-water mark at commit reports), then incremented once — pollers of /proc/PID/mountinfo see one coalesced change event for the crash.
10. user_ns: re-resolved from Core-side namespace management (Section 17.1) by ns_id — not read from the untrusted payload.
11. Re-attach superblocks: for each ShadowSuperblockRecord, the surviving SuperBlock is handed to the reloaded driver instance through the unified sequence's per-superblock Step U14 (ring-set inheritance, vfs_init(), mount-RO + fsck_fast() + journal replay + remount-RW). The shadow's fstype/device/fs_data record is the trusted parameter source for that re-attach; the driver never re-derives them from post-crash payloads.
12. Rebuild each namespace tree: iterate ShadowNamespace.mounts in (parent-before-child, then ascending mount_id) order — parents first so mountpoint paths resolve; ascending mount_id within one mountpoint restores stacking order of stacked mounts. For each entry the engine allocates a fresh Mount via mount_alloc() with the RECORDED mount_id, resolves mountpoint within the (already reconstructed) parent and root_path within the superblock — populating the fresh dcache on demand — attaches it (hash insert, children/mount_list links, DCACHE_MOUNTED), and re-establishes propagation state from propagation/peer_group/master_group (peer rings are rebuilt by linking entries sharing a peer_group; slaves link to the ring of their master_group). The new instance's Arc replaces the (moved-out) owning slot in the shadow entry. A mountpoint path that no longer resolves (its directory vanished in journal replay) fails that entry and its descendants: the entries are removed from the shadow, an FMA event records the lost subtree, and fds into it hit the EIO rule below — recovery itself continues (one lost subtree must not take down the machine's remaining mounts).
13. Set pins_unreconciled = 1 on every reconstructed namespace, increment ShadowMountRegistry.crash_epoch (Release — published before U17 unblocks dispatch, so every subsequent revalidation observes the new epoch), and emit the FMA reconstruction accounting event (entries rebuilt, entries failed, orphan count from step 2).

Open-file pins after reconstruction. OpenFile.mount still references the pre-crash orphan instance, and the fresh instance's mnt_count starts at 0 — two consequences, both handled:

• Lazy rebind: OpenFile carries Core-written mount_id/mount_ns_id copies (set once at open, trusted — never read back from Mount payloads) plus a mount_epoch snapshot of ShadowMountRegistry.crash_epoch. vfs_revalidate_open_file() calls vfs_rebind_mount() (below) on first post-recovery use: if the epoch is unchanged (shape-1 recovery — tree never rebuilt), the existing reference is kept, including for fds on lazy-unmounted or detached mounts that are legitimately outside the shadow. If the epoch advanced, shadow lookup (mount_ns_id, mount_id) → found: mnt_count.fetch_add(1, Relaxed) on the reconstructed instance, swap OpenFile.mount (an RcuCell — see the struct), drop the orphan Arc; not found: latch reopen_errno dead — lazy-unmounted and detached mounts do not survive a VFS-module crash, and neither do mounts whose subtree failed reconstruction; fds into them return -EIO until closed.
• Teardown-gate reconciliation: until every surviving fd has lazily rebound, mnt_count on reconstructed mounts undercounts open files, and a umount(2) would wrongly succeed instead of returning EBUSY (Linux semantics: unmounting a filesystem with open files fails without MNT_DETACH). Every teardown gate that tests mnt_count == 0 (do_umount step 3a, autofs expiry of MNT_SHRINKABLE mounts) first checks the namespace's pins_unreconciled flag; if set, it runs the reconciliation sweep — walk all FdTables (Core memory, trusted; cold path, executed at most once per namespace per crash, and only if a teardown races the lazy rebind window), rebinding every fd whose mount_ns_id matches per the rule above — clears the flag, and re-reads mnt_count. Hot paths never see the flag; the cost lands entirely on the rare umount-soon-after-crash interleaving.

Trust summary. Decision data for reclamation and rebuild — entry enumeration, owning references, re-creation parameters, fd rebind keys — comes exclusively from Core-owned memory (the shadow, OpenFile fields, Nucleus slot metadata). Post-crash driver-writable payloads are read only inside Drop cascades of instances the shadow itself owns, where a corrupted pointer surfaces as a recovery-worker fault, never as silent Core corruption. Wrong-but-well-formed commit reports from a corrupted still-running engine remain the documented domain-grouping exposure.

14.1.1.1.2 Dirty Page Handling on VFS Crash

Dirty Page Handling on VFS Crash:

When a Tier 1 VFS driver crashes, UmkaOS Core cannot safely flush dirty pages using the crashed driver's block mapping (the file-offset → block-address translation lives in the now-destroyed VFS domain).

UmkaOS's design: two-phase dirty extent protocol.

The dirty extent protocol must accommodate two fundamentally different filesystem write models:

• In-place / pre-allocated filesystems (ext4, XFS non-reflink, FAT): The block address is known at page-dirty time (the extent tree maps file offsets to fixed block addresses). Both the logical reservation and physical commit can happen in one step.

• Copy-on-Write filesystems (Btrfs, XFS reflink, bcachefs): The physical block address is not known at page-dirty time. CoW filesystems allocate new blocks at writeback time, not when the page is first dirtied. A protocol that requires block_addr at dirty time is incompatible with CoW.

To support both models, UmkaOS uses a two-phase dirty extent protocol: Phase 1 (reserve) records the logical intent at dirty time; Phase 2 (commit) binds the physical block address after writeback allocation.

/// Phase 1: Reserve a dirty extent at page-dirty time.
///
/// Called by VFS drivers when a page is first marked dirty (from the
/// `AddressSpaceOps::dirty_extent()` callback). Records a **logical**
/// writeback intent — the file offset and length that will need to be
/// written back. No physical block address is required at this point.
///
/// The intent is stored in a per-inode **writeback intent list** maintained
/// by UmkaOS Core, anchored in `DIRTY_INTENT_INDEX` (below) under the key
/// `(sb_dev, inode_id)`. `InodeId` alone is NOT a valid key: it is
/// filesystem-private (the same u64 in two `SuperBlock` instances names two
/// different inodes), so every function in this API takes `sb_dev` — keying
/// on the inode number alone would collide across mounts and route
/// crash-flush I/O to the wrong device. The list is protected by its own
/// embedded spinlock (`intent_lock`, level `DIRTY_INTENT_LOCK` — see
/// `DirtyIntentList`), NOT by `i_rwsem`: callers typically hold the page
/// lock (`dirty_extent()` is called from `set_page_dirty()`), and the
/// write path holds `i_rwsem` exclusive — neither is required nor inspected
/// by this API.
///
/// # Parameters
/// - `sb_dev`: Device identity of the owning superblock (`SuperBlock.s_dev`;
///   an anonymous device number for diskless filesystems — see the
///   `SuperBlock.s_dev` field doc). Together with `inode_id` this forms the
///   globally unique key that survives VFS crash.
/// - `inode_id`: Inode number within the filesystem identified by `sb_dev`.
/// - `file_offset`: Byte offset of the dirty range start.
/// - `len`: Length of the dirty range in bytes.
///
/// # Errors
/// Returns `VfsDirtyError::IntentListFull` when the per-inode intent list
/// reaches its capacity (8192 entries). The error propagates out of the
/// driver's `AddressSpaceOps` callback to the Core-side write path, which
/// performs the writeback-and-retry (see "Intent list overflow policy"
/// below) — the driver itself never initiates writeback.
pub fn vfs_dirty_extent_reserve(
    sb_dev: DevId,
    inode_id: InodeId,
    file_offset: u64,
    len: u64,
) -> Result<DirtyExtentToken, VfsDirtyError>;

/// Phase 2: Commit a dirty extent with its physical block address.
///
/// Called by the filesystem's writeback path **after** it has allocated
/// the physical blocks for a dirty range (CoW allocation for Btrfs/XFS
/// reflink, or extent-tree lookup for ext4/XFS non-reflink). Binds the
/// physical block address to the previously reserved logical intent.
///
/// After `vfs_dirty_extent_commit()` returns, UmkaOS Core has a complete
/// record of the dirty extent's physical location. If the VFS crashes
/// between commit and actual I/O completion, Core can flush the extent
/// directly via the block layer.
///
/// # Parameters
/// - `token`: The token returned by `vfs_dirty_extent_reserve()` for
///   this extent. Tokens are single-use; committing the same token
///   twice is a kernel bug (caught by debug-mode assertion).
/// - `block_addr`: Physical block address assigned by the filesystem's
///   writeback allocator.
/// - `block_len`: Length of the physical block range in bytes. May
///   differ from the logical `len` if the filesystem compresses or
///   coalesces extents.
///
/// # Errors
/// Returns `VfsDirtyError::InvalidToken` if the token has already been
/// committed or was never issued.
pub fn vfs_dirty_extent_commit(
    token: DirtyExtentToken,
    block_addr: PhysBlockAddr,
    block_len: u64,
) -> Result<(), VfsDirtyError>;

/// Atomic reserve+commit for non-CoW filesystems.
///
/// Convenience function for filesystems that know the block address at
/// dirty time (ext4 non-delayed-alloc, FAT, exFAT). Equivalent to
/// calling `vfs_dirty_extent_reserve()` followed immediately by
/// `vfs_dirty_extent_commit()`, but avoids the overhead of a separate
/// token round-trip.
///
/// CoW filesystems MUST NOT use this function — they must use the
/// two-phase protocol because the block address is not available at
/// dirty time.
pub fn vfs_dirty_extent_reserve_and_commit(
    sb_dev: DevId,
    inode_id: InodeId,
    file_offset: u64,
    len: u64,
    block_addr: PhysBlockAddr,
    block_len: u64,
) -> Result<(), VfsDirtyError>;

/// Abort a previously reserved dirty extent. Called when writeback fails
/// after `vfs_dirty_extent_reserve()` but before
/// `vfs_dirty_extent_commit()` — e.g., block allocation failure, I/O
/// error during journal write, or filesystem shutdown. Releases the
/// reserved intent list entry, decrementing `nr_reserved` and freeing
/// the token. The token is consumed (single-use, same as commit).
///
/// **Retry policy**: After 3 consecutive writeback failures for the same
/// extent (tracked per-inode by a `(file_offset, len)` → `fail_count`
/// map in the intent list), the filesystem marks the extent permanently
/// dirty and logs an FMA error:
/// `"writeback abort: extent [offset, offset+len) on inode {id} failed 3 times"`.
/// The permanently-dirty extent remains in the intent list until the
/// inode is evicted or the filesystem is unmounted, ensuring crash
/// recovery can still identify it.
pub fn vfs_dirty_extent_abort(
    token: DirtyExtentToken,
) -> Result<(), VfsDirtyError>;

/// Acknowledge that a dirty extent has been successfully flushed to
/// stable storage. Removes the extent from Core's dirty extent log.
/// Two sanctioned callers: (1) the VFS driver, after receiving I/O
/// completion for the writeback (the normal path); (2) Core's crash
/// recovery bypass (U10b, [Section 14.1](#virtual-filesystem-layer--dirty-page-handling-on-vfs-crash)
/// step 4), after its direct block write of a committed extent reaches
/// stable storage — this is the ONLY terminal removal for
/// crash-flushed entries; there is no "crash-flushed" entry state.
pub fn vfs_flush_extent_complete(
    sb_dev: DevId,
    inode_id: InodeId,
    file_offset: u64,
    len: u64,
) -> Result<(), VfsDirtyError>;

/// Opaque token binding a reserved dirty extent to its commit.
/// Issued by `vfs_dirty_extent_reserve()`, consumed by
/// `vfs_dirty_extent_commit()` or `vfs_dirty_extent_abort()`. Encodes the
/// full `(sb_dev, inode_id)` list key — so commit/abort route to the
/// correct per-superblock intent list without any ambient context — plus
/// the file offset and a monotonic per-list sequence number for
/// double-commit detection.
///
/// `#[repr(C)]`: the token crosses the VFS ring (Core ↔ filesystem driver)
/// by value, so its layout must be stable across separately compiled
/// binaries.
///
/// Size: 32 bytes (inode_id: u64 + file_offset: u64 + seq: u64 +
/// sb_dev: u32 + explicit pad: 4).
#[repr(C)]
#[derive(Clone, Copy)]
pub struct DirtyExtentToken {
    pub inode_id: InodeId,
    pub file_offset: u64,
    pub seq: u64,
    pub sb_dev: DevId,
    /// Explicit trailing padding to u64 alignment (offset 28).
    pub _pad: [u8; 4],
}
const_assert!(core::mem::size_of::<DirtyExtentToken>() == 32);

/// Error type for dirty extent operations.
pub enum VfsDirtyError {
    /// The per-inode intent list is full (8192 entries). Trigger writeback
    /// to drain completed intents before retrying. Equivalent to EBUSY.
    IntentListFull,
    /// The token has already been committed or was never issued.
    InvalidToken,
    /// Other VFS error (invalid inode ID, etc.).
    Other(VfsError),
}

Dirty extent intent list — UmkaOS Core maintains a per-inode writeback intent list in core memory (not in VFS domain memory), anchored in the two-level Core-owned index:

/// Global dirty-intent index, owned by Core (Tier 0). Two levels: the outer
/// XArray is keyed by superblock device (`DevId` — the same key space as
/// `SUPER_BLOCK_MAP` below); the inner per-superblock XArray is keyed by
/// inode number. This is the Core-side anchor for every `DirtyIntentList`:
/// crash recovery iterates it directly, with no dependency on VFS-domain
/// state and no inode-cache walk.
///
/// Lifecycle: the per-superblock `SbDirtyIntents` entry is created at mount
/// (step 4a, alongside the `SUPER_BLOCK_MAP` registration) and removed at
/// umount after all its lists have drained. Individual `DirtyIntentList`
/// objects are created lazily on the first `vfs_dirty_extent_reserve()` for
/// an inode and removed when empty at inode eviction (see the allocation
/// note in `DirtyIntentList`).
pub static DIRTY_INTENT_INDEX: BootOnceCell<XArray<Arc<SbDirtyIntents>>> =
    BootOnceCell::new();

/// Lock level for `DirtyIntentList::intent_lock`. Leaf spinlock between
/// `PAGE_LOCK(180)` and `PTL(185)`: acquirable while holding the page lock
/// (the `set_page_dirty()` → `dirty_extent()` path) and above
/// `XA_LOCK(178)` (index insertion); nothing is acquired under it. The
/// master lock-ordering table
/// ([Section 3.4](03-concurrency.md#cumulative-performance-budget--lock-ordering)) carries the
/// authoritative row.
pub const DIRTY_INTENT_LOCK: u32 = 182;

/// Per-superblock dirty-intent table: inode number → intent list.
pub struct SbDirtyIntents {
    /// Intent lists keyed by inode number (`InodeId.0`). XArray per the
    /// integer-key collection policy: RCU lookup on the reserve/commit
    /// path; `XA_LOCK(178)` for insert/remove. An inserting reserve takes
    /// `XA_LOCK`, publishes the list, drops `XA_LOCK`, THEN takes the
    /// list's `intent_lock` — strictly ascending (178 < DIRTY_INTENT_LOCK).
    pub by_ino: XArray<Arc<DirtyIntentList>>,
}

/// Per-inode writeback intent list. Maintained by UmkaOS Core in its own
/// memory domain, surviving VFS crashes.
///
/// Each entry tracks a dirty file range through its lifecycle:
/// Reserved (logical intent only) → Committed (physical address bound) →
/// Complete (flushed to stable storage, entry removed).
///
/// **Locking**: all access — structural modification (reserve, commit,
/// abort, flush-complete) AND the writeback thread's committed-extent
/// snapshot — runs under the embedded `intent_lock` (level
/// `DIRTY_INTENT_LOCK`, between `PAGE_LOCK(180)` and `PTL(185)`; master
/// table row: [Section 3.4](03-concurrency.md#cumulative-performance-budget--lock-ordering)). Hold
/// times are bounded: ring push/pop and a scan-and-copy into a caller
/// `ArrayVec` — no I/O, no allocation, no nested ordered lock under it
/// (leaf). `i_rwsem` plays NO role in intent-list protection: callers of
/// the API may hold the page lock and/or `i_rwsem` in either mode without
/// affecting the list's own serialization. (An earlier revision protected
/// the list with `i_rwsem`, which made the overflow-recovery protocol
/// self-deadlock — see "Intent list overflow policy" below.)
pub struct DirtyIntentList {
    /// Guards `entries` and `next_seq`. Leaf spinlock, level
    /// `DIRTY_INTENT_LOCK` (182): acquirable while holding `PAGE_LOCK(180)`
    /// (the `set_page_dirty()` → `dirty_extent()` path) and above
    /// `XA_LOCK(178)` (index insertion). Never held across a blocking call.
    pub intent_lock: SpinLock<(), DIRTY_INTENT_LOCK>,
    /// Ring buffer of intent entries. Capacity: 8192 per inode.
    /// At 80 bytes per entry (inode_id(8) + file_offset(8) + len(8) +
    /// block_addr(16) + block_len(8) + seq(8) + sb_dev(4) + pad(4) +
    /// block_dev(16) = 80), worst case is ~640 KB per heavily-dirtied
    /// inode — acceptable for a production server. Most inodes have
    /// <100 entries at any given time.
    ///
    /// **Allocation**: `DirtyIntentList` is allocated from the
    /// `dirty_intent_slab` (a dedicated slab cache, object size = sizeof
    /// `DirtyIntentList`, created at boot Phase 2.4). Allocation occurs
    /// lazily on the first `vfs_dirty_extent_reserve()` for each inode.
    /// The slab is GC'd when idle inodes are evicted from the inode cache.
    pub entries: BoundedRing<DirtyIntentEntry, 8192>,
    /// Monotonic sequence counter for token generation.
    pub next_seq: u64,
}

/// Physical block address on a block device. Newtype around `u64` (plain u64,
/// no `NonZero` — `Option<PhysBlockAddr>` is 16 bytes: 8-byte discriminant +
/// 8-byte payload, with no niche optimization).
pub struct PhysBlockAddr(pub u64);

/// A single dirty extent intent entry.
// Kernel-internal, not KABI: contains Option<Arc<dyn>>, never crosses a compilation
// boundary. #[repr(C)] is required to make the const_assert deterministic across
// compiler versions — without it the compiler may reorder fields.
#[repr(C)]
pub struct DirtyIntentEntry {
    /// Stable inode identifier. Redundant during normal operation (the list
    /// is per-inode) but needed during crash recovery log replay, where entries
    /// may be iterated without per-inode context.
    pub inode_id: InodeId,
    /// File offset of the dirty range (bytes).
    pub file_offset: u64,
    /// Length of the dirty range (bytes).
    pub len: u64,
    /// Physical block address. `None` if Phase 1 only (reserved but not
    /// yet committed). `Some(addr)` after Phase 2 commit.
    pub block_addr: Option<PhysBlockAddr>,
    /// Physical block length. Only valid when `block_addr` is `Some`.
    pub block_len: u64,
    /// Sequence number (matches `DirtyExtentToken.seq`).
    pub seq: u64,
    /// Device ID of the superblock that owns this inode.
    ///
    /// This is the key that Core (Tier 0) uses to look up the block device
    /// during crash recovery, via the global device registry
    /// (`DEVICE_REGISTRY: XArray<Arc<dyn BlockDeviceOps>>`, keyed by `DevId`).
    /// When a Tier 1 VFS driver crashes, Core iterates dirty intent entries
    /// and uses `sb_dev` to resolve the target block device — without needing
    /// any VFS-domain state. This is strictly more reliable than using
    /// `block_dev` alone because `block_dev` is `None` for Phase 1 entries
    /// (deferred allocation) and for NFS, whereas `sb_dev` is always set for
    /// local filesystems and enables Core to match intent entries to the
    /// correct `SUPER_BLOCK_MAP` entry for filesystem journal replay.
    ///
    /// Always equal to the owning superblock's device identity
    /// (`SuperBlock.s_dev`), populated from the `sb_dev` parameter of
    /// `vfs_dirty_extent_reserve()` / `reserve_and_commit()`. Diskless
    /// filesystems (NFS, CIFS, tmpfs) carry the anonymous device number
    /// allocated at mount (see `SuperBlock.s_dev`), so the key is unique
    /// for them too. "No local block device" is signaled by
    /// `block_dev == None` together with the absence of a `DEVICE_REGISTRY`
    /// entry for the anonymous number — crash recovery for network
    /// filesystems uses the network reconnection path instead.
    pub sb_dev: DevId,
    /// Reference to the block device that owns the physical blocks.
    /// Required by crash recovery: when a Tier 1 VFS driver crashes, Core
    /// must issue direct block writes via the block layer (bypassing VFS)
    /// for all committed extents. Without this reference, Core would need
    /// to resolve the block device from the superblock — but the VFS driver
    /// holding the superblock may be the one that crashed.
    ///
    /// Set to `None` for network filesystems (NFS) where `block_addr` is an
    /// opaque server-side commit token, not a local block address.
    ///
    /// **Redundancy with `sb_dev`**: `block_dev` provides the fast path —
    /// Core can issue I/O immediately without a registry lookup. `sb_dev`
    /// provides the fallback — if `block_dev` is `None` (Phase 1 entry),
    /// Core uses `sb_dev` to locate the block device via `DEVICE_REGISTRY`
    /// and then uses the filesystem's block allocator (from the superblock)
    /// to determine whether the extent can be committed or must be discarded.
    /// The `block_dev` reference is valid during crash recovery because the
    /// block device driver is in a separate domain from the (isolated) VFS
    /// driver, so a VFS crash's blast radius does not include it. The
    /// load-bearing property is domain separation, not any particular tier: it
    /// holds whether the block driver is in domain 0 (Tier 0 — the case on
    /// arches without fast in-kernel isolation) or in its own isolated domain,
    /// as long as that domain differs from the VFS driver's. If
    /// the block device driver crashes simultaneously (a different, independently
    /// handled crash event), dirty intent entries referencing that block device
    /// are skipped with an FMA warning.
    pub block_dev: Option<Arc<dyn BlockDeviceOps>>,
}
// Verify DirtyIntentEntry size matches the capacity analysis (80 bytes per entry,
// 8192 entries = ~640 KB worst case per heavily-dirtied inode). If this assertion
// fails, update the capacity analysis in DirtyIntentList.entries comment above.
//
// Note: DirtyIntentEntry is kernel-internal (never crosses a compilation boundary —
// only Core crash-recovery code touches it). The `#[repr(C)]` attribute ensures
// deterministic layout for the const_assert below, NOT for KABI compatibility.
// `Option<Arc<dyn BlockDeviceOps>>` relies on Rust's niche optimization for
// `Arc<T>` (null pointer → None). This is guaranteed by the language for `Arc`/`Box`
// and verified at compile time by this const_assert. If a future rustc version
// changes the representation, the const_assert will fail and the struct must be
// reworked (e.g., raw pointer + validity flag).
//
// Field breakdown (64-bit target):
//   inode_id: InodeId(u64) = 8
//   file_offset: u64 = 8
//   len: u64 = 8
//   block_addr: Option<PhysBlockAddr> = 16 (u64 discriminant + u64 payload; no niche — PhysBlockAddr wraps plain u64)
//   block_len: u64 = 8
//   seq: u64 = 8
//   sb_dev: DevId(u32) = 4
//   block_dev: Option<Arc<dyn BlockDeviceOps>> = 16 (fat pointer: data_ptr + vtable_ptr)
//   padding for alignment = 4 (after sb_dev, before block_dev's 8-byte alignment)
//   TOTAL = 80 bytes
const_assert!(core::mem::size_of::<DirtyIntentEntry>() == 80);

Intent list overflow policy: vfs_dirty_extent_reserve() returns IntentListFull when the per-inode intent list reaches 8192 entries. The VFS driver must not proceed with the dirtying operation when IntentListFull is returned; it propagates the error out of its AddressSpaceOps callback (dirty_extent() via write_end()), and the Core-side write path — not the driver — performs the drain-and-retry:

1. page_cache_write_iter() (and any other Core path that dirties file pages) receives the propagated IntentListFull, calls writeback_single_inode(inode, WritebackSyncMode::Wait) to flush committed extents — each flush completion frees an intent slot via vfs_flush_extent_complete() — and then retries the failed page ONCE.
2. This is legal while i_rwsem is held EXCLUSIVE (the write loop holds it for its entire duration — see page_cache_write_iter() Step 1): writeback_single_inode() never acquires i_rwsem — its per-inode serialization is the I_WRITEBACK CAS (Section 4.6) — and the intent list is protected by DIRTY_INTENT_LOCK, not i_rwsem, so no lock is re-entered. (An earlier revision documented the intent list as i_rwsem-protected and directed the DRIVER to call writeback_single_inode(): a self-deadlock on the non-reentrant rwsem already held exclusive by the write loop, and a tier violation — writeback_single_inode() is a Core function that an isolated Tier 1/2 driver cannot call directly.)
3. If the retry fails again, the write stops: page_cache_write_iter() returns the short count when written > 0, else -EIO. The unresponsive-driver watchdog below is the backstop for a driver that has stopped completing flushes.

Dirtying contexts that cannot sleep must fail the operation instead of draining — writeback-and-retry is only legal in process context.

This is a deliberate design choice that differs from Linux's approach: UmkaOS never silently discards safety information. The backpressure ensures that on any VFS crash, Core has a complete record of all outstanding dirty extents and can accurately flag inconsistent data — no dirty extent is ever "forgotten."

If the VFS driver is unresponsive (not calling vfs_flush_extent_complete() for

5 seconds), Core treats all entries in the intent list as dirty and initiates VFS driver restart — the backpressure prevents intent list overflow from masking a stuck VFS driver.

Per-filesystem type usage patterns:

Filesystem	Write model	Phase 1 (reserve)	Phase 2 (commit)	Notes
ext4 (non-delayed-alloc)	In-place	reserve_and_commit()	N/A (atomic)	Block address known from extent tree at dirty time
ext4 (delayed-alloc)	Deferred	reserve() at dirty time	commit() during writeback after ext4_map_blocks() allocates	Delayed allocation defers block assignment
XFS (non-reflink)	In-place	reserve_and_commit()	N/A (atomic)	BMBT lookup gives block address at dirty time
XFS (reflink)	CoW	reserve() at dirty time	commit() during writeback after CoW fork allocates new blocks	Shared extents require new block allocation
Btrfs	Redirect-on-Write	reserve() at dirty time	commit() during writeback after extent allocator assigns new tree location	All writes redirect; old blocks freed at transaction commit. Crash semantics: reserved-only intents (Phase 1 without Phase 2) are volatile — on VFS crash, they are discarded because no physical blocks were allocated. The CoW filesystem's on-disk tree remains consistent because uncommitted writes never modified the on-disk tree.
FAT/exFAT	In-place	reserve_and_commit()	N/A (atomic)	Cluster chain gives block address at dirty time
NFS	Network	reserve() at dirty time	commit() after NFS WRITE RPC completes with server-assigned stable storage	Block address is the server's opaque commit token

Core-owned superblock registry for crash recovery bypass:

/// Global superblock map, owned by Core (Tier 0). Maps device IDs to
/// superblock references so that crash recovery can locate the block
/// device and filesystem metadata without going through the crashed VFS
/// driver.
///
/// Keyed by `DevId` (integer key → XArray per collection policy).
/// Populated during `mount()`: Core registers the superblock before handing
/// control to the VFS driver. Removed during `umount()` after the VFS driver
/// has cleanly shut down. Internal / synthetic mounts that never appear in a
/// user mount namespace — the pipefs, sockfs, and anon-inode pseudo-filesystems
/// — register their superblock here at internal-mount setup and remove it at
/// teardown on the SAME path, so the synthetic inodes they host (pipe, socket,
/// and `InodeFlags::ANON_INODE` inodes) are enumerable through `all_superblocks()` ×
/// `inode_cache` like any other inode. This universal registration is what lets
/// the Shadow-and-Migrate layout walk reach every live inode via a registered
/// superblock (universal cache-membership invariant,
/// [Section 14.1](#virtual-filesystem-layer--inode-cache-icache)).
///
/// This map is **not** used on the normal I/O path — it serves crash recovery
/// (superblock lookup for a crashed VFS driver) and the Shadow-and-Migrate
/// layout-migration enumeration walk (above), both off the per-syscall hot path.
/// Normal path resolution goes through the VFS mount tree.
pub static SUPER_BLOCK_MAP: BootOnceCell<XArray<Arc<SuperBlock>>> = BootOnceCell::new();

/// Snapshot every registered superblock as a list of pinned `Arc`s.
///
/// Iterates `SUPER_BLOCK_MAP` under an RCU read section, cloning each `Arc`
/// out so the returned iterator is valid after the RCU section closes (each
/// yielded `Arc` keeps its `SuperBlock` alive independently). If the map has
/// not been initialised yet (pre-`mount()` boot), the iterator is empty.
///
/// Warm/cold path only (CPU hotplug ring reassignment, umount-all on
/// shutdown, admin enumeration) — never a per-syscall hot path. The bounded
/// `Vec` holds one `Arc` per mounted filesystem (count bounded by mounts).
///
/// **No allocation inside the RCU read section**: `Vec` growth may enter
/// direct reclaim and sleep, which would stall RCU grace periods
/// system-wide. The snapshot therefore runs in two passes — a count pass
/// under RCU (no allocation), a `reserve()` OUTSIDE any RCU section, then
/// a fill pass under RCU that pushes only within the pre-reserved capacity
/// (`Vec::push` below capacity never allocates). A concurrent mount burst
/// that outgrows the reservation (+slack) aborts the fill pass and retries
/// with a fresh count — bounded in practice, since mounts are warm-path
/// rare relative to this cold enumeration. O(mounts) per pass.
pub fn all_superblocks() -> impl Iterator<Item = Arc<SuperBlock>> {
    let mut out: Vec<Arc<SuperBlock>> = Vec::new();
    if let Some(map) = SUPER_BLOCK_MAP.get() {
        loop {
            // Pass 1: count under RCU — no allocation.
            let mut n: usize = 0;
            {
                let _rcu = rcu_read_lock();
                map.for_each(|_dev, _sb| n += 1);
            }
            // Reserve OUTSIDE the RCU section (may sleep in reclaim).
            // +8 slack absorbs mounts racing between the two passes.
            out.clear();
            out.reserve(n + 8);
            // Pass 2: fill under RCU, never exceeding capacity.
            let mut overflowed = false;
            {
                let _rcu = rcu_read_lock();
                map.for_each(|_dev, sb| {
                    if out.len() < out.capacity() {
                        out.push(Arc::clone(sb)); // within capacity: no alloc
                    } else {
                        overflowed = true; // mount burst outran the slack
                    }
                });
            }
            if !overflowed {
                break;
            }
            // Retry with a fresh count (rare: >8 mounts landed mid-snapshot).
        }
    }
    out.into_iter()
}

On crash recovery, Core uses SUPER_BLOCK_MAP to resolve the superblock for the crashed filesystem, then iterates its inodes' dirty intent lists. Each DirtyIntentEntry carries its own block_dev reference, enabling Core to issue direct block writes without the VFS driver's cooperation.

Crash recovery sequence:

Dirty page flush during VFS crash uses the committed dirty extent records stored in Core memory (via vfs_dirty_extent_commit()). These records contain physical block addresses that survive the VFS crash. The file-to-inode-to-superblock-to-device mapping is NOT needed — the dirty extent protocol captures the physical location at commit time specifically to enable crash recovery without VFS state.

When a Tier 1 VFS driver crashes while UmkaOS Core detects pending dirty intents:

1. Iterate the dirty intent lists for all inodes on the crashed filesystem directly from DIRTY_INTENT_INDEX.get(sb_dev) — the index is Core-resident, so no superblock dereference or inode-cache walk is needed to find the lists. (SUPER_BLOCK_MAP.get(sb_dev) still resolves the superblock where journal-replay coordination requires it.)
2. Committed extents (Phase 2 complete — block_addr is Some):
3. For each committed extent in ascending seq order (oldest committed first): Issue a direct block write via the block layer (bypassing VFS), using entry.block_dev for the device handle and block_addr/block_len from the intent entry. Wait for write completion.
4. Why oldest-first: committed extents can overlap — a range re-dirtied and re-committed before the earlier flush's vfs_flush_extent_complete() removed the first entry leaves two committed entries for the same logical range. Flushing in ascending seq order makes the most recently committed data land LAST, so the final on-disk state is the newest data (last-writer-wins). For CoW filesystems the two commits target different physical blocks and only the newest is referenced by the on-disk tree, so ordering is additionally harmless there. (An earlier revision said "newest first, to preserve journal ordering" — backwards for overlaps, and there is no journal-ordering constraint here: these direct block-layer writes bypass the filesystem journal entirely. Their ordering against journal replay is provided by the recovery sequence itself: this bypass flush runs at Step U10b, strictly before the Step U14(e) journal replay, Section 14.3.)
5. Reserved-only extents (Phase 1 only — block_addr is None):
6. These extents have dirty pages in the page cache but no physical block assignment. Core cannot flush them without knowing the block address.
7. Crash-stranded writeback requeue (the normative page transition — "potentially inconsistent" is not a page state and named no flag operation): the writeback submitter cleared DIRTY and set WRITEBACK on every dispatched page at dispatch time (Section 4.6 step 2), so a dispatched Phase-1 page — a CoW/delayed-allocation page whose block commit would have happened in the crashed provider — is PageFlags::WRITEBACK, NOT dirty, at crash, with no completion coming, and invisible to the U15 DIRTY-list re-write. For each page of the extent's range still under WRITEBACK (and likewise for any committed extent whose step-2 direct write FAILED), run the shared ERROR epilogue writeback_page_epilogue(page, -EIO) (Section 15.2): it re-dirties the page (restoring both nr_dirty counters on the 0→1 edge), records the error in the mapping's ErrSeq (wb_err.set_err), clears WRITEBACK (decrementing both writeback counters), and wakes waiters. A parked fsync phase-2 waiter wakes and truthfully reports EIO; the re-dirtied data is re-written by U15 through the reloaded driver, and the filesystem's own journal/log makes it consistent on replay (same as a hard power-off scenario). Undispatched pages of the extent are simply still DIRTY — U15 handles them; no transition is needed here. The intent entry itself STAYS in the list (it still describes dirty data for any subsequent crash).
8. After all committed extents are flushed: run the shared writeback_page_epilogue(page, 0) (Section 15.2) on each successfully-flushed page — clearing WRITEBACK AND calling page.wake_waiters(). Clearing the flag ALONE does not wake an already-parked wait_on_page_writeback() sleeper (wait_event re-evaluates its predicate only on a wakeup, Section 14.4); the explicit wake_waiters() inside the epilogue — not the bare flag clear — is the ordered wake for filemap_write_and_wait_range phase-2 (fsync) waiters blocked mid-crash. Then REMOVE each fully-flushed committed entry from the intent list via the defined terminal API vfs_flush_extent_complete(entry.sb_dev, entry.inode_id, entry.file_offset, entry.len) — this crash bypass is that API's second sanctioned caller (see its declaration above). There is no "crash-flushed" entry state: an earlier revision marked entries so, but DirtyIntentEntry has no such field and nothing consumed it — the flushed entries would have occupied their bounded DirtyIntentList rings until inode eviction (a resource leak that compounds across crashes). Then continue with driver reload.
9. Any dirty pages NOT covered by any intent entry (neither reserved nor committed) are still plain DIRTY — only dispatch moves a page to WRITEBACK, and every dispatched page is intent-covered (intents are reserved at dirtying time). U15's DIRTY-list re-write flushes them through the reloaded driver; journal replay bounds any inconsistency exactly as after a power-off.

Design rationale: This is better than Linux's approach (which silently loses dirty pages when a kernel module crashes) while being simpler than running a full WAL in UmkaOS Core. The pre-registration overhead is one lightweight ring-buffer push per dirtied file region — negligible for writeback-dominated workloads.

Performance implications and mitigation:

The Core → VFS domain switch costs ~23 cycles for the bare WRPKRU instruction (x86-64 MPK). The full domain crossing — including argument marshaling via the inter-domain ring buffer and cache effects — is ~30-35 cycles per crossing. This overhead is amortized by:

1. Page Cache in Core: The Page Cache (Section 4.4) lives in Core, not VFS. Cached file reads/writes hit the Page Cache directly with zero domain switches. Only cache misses (actual I/O) cross into VFS.

2. Batching: Multiple file operations within a single syscall (e.g., readv, io_uring batches) amortize the domain switch over many operations.

3. Dentry cache hit rate: The dentry cache (in VFS) has >99% hit rate for typical workloads. Path resolution is fast, and the domain switch cost is dominated by the actual I/O latency (microseconds vs nanoseconds).

Measured overhead: For a 4KB NVMe read (~10μs device latency), the additional domain switches (Core → VFS → FS driver) add ~70 cycles (~30ns total), i.e. ~0.3% of the device-latency-bound operation — and that residual is already paid down by the compensating savings in this same read path, listed above: page-cache hits stay in Core with ZERO domain switches (only true misses cross into VFS), readv/io_uring batches amortize one crossing over many operations, and the >99% dentry-cache hit rate keeps path resolution in-domain. The isolation cost is compensated within the read path, not charged against a 5% allowance (the 5% figure is the failure threshold, not a budget to spend — Section 1.3).

Metadata-heavy workloads: Individual metadata syscalls (stat, readdir, open/close) pay higher per-call overhead because the operation base cost is lower (~200-500ns vs ~10μs for I/O). A single stat() on x86-64 incurs ~46 cycles (~18ns) for the Core → VFS → Core round-trip, which is ~3.6-9% per call. This is the design tradeoff for VFS crash containment: the dentry/inode cache is MANAGED by the VFS domain — lookup structures (hash table, LRU) and the code that walks them run in-domain, so metadata operations cross the boundary — enabling cache rebuild on VFS crash recovery. (Dentry instance STORAGE is Nucleus tracked memory, Section 14.1: the slots are Core-owned and enumerable, which is what makes post-crash slot reclamation possible — Section 14.1.) Two amortization mechanisms reduce this cost to ~0.3-0.5% effective overhead for the dominant readdir+stat access pattern and ~0.05% per stat for io_uring batch workloads; see Section 14.1 below. For per-architecture raw overhead figures, see Section 3.4.

14.1.1.1.3 Open File Descriptor Recovery: Generation Refresh

Crash recovery step f (Section 14.1 intro above) is lazy: no fd is touched during the recovery window itself. Instead, every surviving fd is revalidated on its first post-recovery operation by the protocol below. This keeps recovery latency independent of the number of open file descriptors, avoids walking every task's FdTable under load, and charges the revalidation cost (one FileOps::open() round-trip, microseconds) only to fds that are actually used again.

What survives, what dies. The OpenFile and everything Core-resident in it survive the crash: f_pos, f_flags, f_mode, f_cred, f_wb_err, ra_state, the pinned inode/dentry/mount references, and the inode's file lock records (InodeLocks, Section 14.14) — file locks are VFS bookkeeping, never materialized in the filesystem driver, so no lock re-acquisition is needed. (After a VFS-MODULE crash — as opposed to a provider-only crash — the mount reference survives as an orphan and is rebound to the reconstructed Mount instance by vfs_rebind_mount() during this same revalidation; see Section 14.1.) What dies with the domain is the driver's per-open state: the private_data token returned by FileOps::open() (journal handles, delegation IDs, driver context) and any lease/oplock/delegation the driver held (CIFS oplocks, NFS delegations) — for those, recovery is semantically a lease break; the revalidated open is lease-less until the driver's normal protocol traffic re-acquires one.

Trigger. The VFS dispatch path already compares open_generation against sb.driver_generation before select_ring() (vfs_check_open_generation(), Section 14.3). After recovery bumps the generation at Step U10a, that check fails for every surviving fd. Instead of returning -ENOTCONN to the caller, the dispatch wrapper routes the mismatch into vfs_revalidate_open_file():

/// Revalidate an `OpenFile` whose `open_generation` is stale after a driver
/// crash recovery. Re-runs `FileOps::open()` against the reloaded driver
/// instance to regenerate the per-open driver state, then refreshes
/// `open_generation`. Warm path: runs at most once per fd per crash.
///
/// Returns `Ok(())` when the fd is usable at the current generation (this
/// call revalidated it, or a racing thread already did). Errors:
/// - `Err(EIO)` — the fd is permanently dead (latched in `reopen_errno`):
///   the backing inode no longer exists after journal replay
///   (unlinked-but-open), or the driver refused the re-open.
/// - `Err(ENXIO)` — a second crash recovery is in progress; NOT latched.
///   The caller (dispatch wrapper) sleeps on `sb.recovery_wait` and retries
///   from the top, exactly as for a boundary ENXIO from `select_ring()`.
/// - `Err(ENOMEM)` — transient allocation failure; NOT latched. Returned to
///   the caller; the next operation retries revalidation.
fn vfs_revalidate_open_file(file: &OpenFile) -> Result<(), KernelError> {
    // Dead-fd fast reject (latched by a previous failed revalidation).
    if file.reopen_errno.load(Ordering::Acquire) != 0 {
        return Err(KernelError::EIO);
    }
    // O_PATH fds carry no driver open state (FileOps::open was never
    // material to them) — refresh the generation without a driver call.
    // They DO hold a mount pin, so the mount rebind still runs (under the
    // same lock that serializes it for regular fds).
    let cur = file.inode.i_sb.driver_generation.load(Ordering::Acquire);
    if file.f_flags.load(Ordering::Relaxed) & O_PATH != 0 {
        let _g = file.revalidate_lock.lock();
        vfs_rebind_mount(file)?;
        file.open_generation.store(cur, Ordering::Release);
        return Ok(());
    }
    // Serialize racing threads sharing this OpenFile (dup/fork/CLONE_FILES).
    let _g = file.revalidate_lock.lock();
    // Double-check under the lock: a racing thread may have finished.
    let cur = file.inode.i_sb.driver_generation.load(Ordering::Acquire);
    if file.open_generation.load(Ordering::Acquire) == cur {
        return Ok(());
    }
    if file.reopen_errno.load(Ordering::Acquire) != 0 {
        return Err(KernelError::EIO);
    }
    // Rebind the mount reference FIRST: after a VFS-module crash the tree
    // was rebuilt with fresh Mount instances, and this fd's pin must move
    // to the reconstructed mount (or the fd must die if its mount did not
    // survive). No-op for provider-only recoveries (epoch unchanged). See
    // "Open-file pins after reconstruction" in
    // [Section 14.1](#virtual-filesystem-layer--shadow-mount-registry-and-mount-tree-reconstruction).
    vfs_rebind_mount(file)?;
    // Re-run the driver open hook on the NEW instance. The OLD private
    // token is discarded WITHOUT FileOps::release(): it references per-open
    // state in the crashed instance's heap, which Step U13 already freed
    // wholesale — releasing it against the new instance would hand the new
    // driver a token it never issued.
    match file.f_ops.open(
        InodeId(file.inode.i_ino),
        OpenFlags::from_bits_truncate(file.f_flags.load(Ordering::Relaxed)),
    ) {
        Ok(outcome) => {
            // Only `outcome.private` is consumed. The re-open's
            // `OpenOutcome::data_inode` is DISCARDED: the data-inode binding
            // is per-open and immutable once the description is published
            // (see the `OpenFile::data_inode` field doc), and this fd was
            // published long before the crash. Recovery regenerates the
            // driver's per-open state, not the fd's data identity — the
            // inode objects themselves are Nucleus-preserved across the
            // driver reload, so the existing binding is still the right one.
            // Reconcile cached attributes with the post-journal-replay
            // on-disk state (an uncommitted size extension may have been
            // rolled back; the disk is authoritative after replay). Pages
            // beyond the reconciled i_size are truncated from the cache.
            // The getattr result is NOT droppable: the reconciliation
            // (and its truncation — "data rolled back by journal replay
            // must not be resurrected from cache") is declared mandatory,
            // and this function is `vfs_apply_reconciled_attr`'s only
            // caller. If the generation were stored despite a getattr
            // failure, the dispatch-path check would never route this fd
            // back into revalidation, and its stale i_size, timestamps,
            // flags, and cached pages (including pages beyond a
            // rolled-back extension) would stay authoritative to Core
            // indefinitely.
            match file.inode.i_op.getattr(InodeId(file.inode.i_ino)) {
                Ok(attr) => {
                    vfs_apply_reconciled_attr(&file.inode, attr);
                    // Publish the fresh token, THEN the generation
                    // (Release pairs with the dispatch path's Acquire):
                    // the fd becomes usable only fully revalidated.
                    file.private_data.store(outcome.private as *mut (), Ordering::Release);
                    file.open_generation.store(cur, Ordering::Release);
                    Ok(())
                }
                Err(errno) => {
                    // The re-open SUCCEEDED against the LIVE instance, so
                    // the fresh token is released properly before erroring
                    // — unlike the discarded pre-crash token, this one
                    // references live driver per-open state; dropping it
                    // unreleased would leak that state until unmount. A
                    // release failure is ignorable: the fd is erroring out
                    // either way, and a dead-again driver's state is freed
                    // wholesale by the next recovery's U13.
                    let _ = file.f_ops.release(InodeId(file.inode.i_ino), outcome.private);
                    // The generation is NOT stored: a transport-cause
                    // failure re-enters revalidation on the next operation;
                    // a provider-cause failure latches the fd dead — both
                    // via the shared failure-cause resolution below.
                    Err(vfs_resolve_reopen_failure(file, cur, errno))
                }
            }
        }
        Err(errno) => Err(vfs_resolve_reopen_failure(file, cur, errno)),
    }
}

/// Resolve the TYPED failure cause of a failed revalidation step (the
/// `FileOps::open()` re-open, or the mandatory post-replay `getattr`
/// reconciliation) at the Core-side dispatch wrapper, and apply its
/// disposition. Trait signatures are UNCHANGED — providers return only an
/// errno; the cause is distinguished HERE, from ring/recovery state the
/// wrapper can see (the same visibility the boundary-ENXIO arm relies on).
///
/// A SECOND crash landing while the failed call was in flight re-bumps
/// `sb.driver_generation` PAST the `cur` sampled under `revalidate_lock`
/// (U10a bumps the generation strictly before `VFSRS_ACTIVE`), so a
/// generation change means the already-submitted call was DRAINED by that
/// recovery — a transport loss (`LostByRecovery`, the `-EIO` in-flight
/// drain of [Section 14.2](#vfs-ring-buffer-protocol)), NOT a driver verdict — even
/// though the trait flattened it to an errno.
fn vfs_resolve_reopen_failure(file: &OpenFile, cur: u64, errno: Errno) -> KernelError {
    let cause = if file.inode.i_sb.driver_generation.load(Ordering::Acquire) != cur {
        // Generation advanced past the `cur` sampled under
        // `revalidate_lock`: our already-submitted call was DRAINED by
        // a second recovery — a transport loss, not a driver verdict.
        VfsFailureCause::LostByRecovery
    } else if errno == Errno::ENXIO {
        // `select_ring()` rejected the call at the ring boundary
        // because the ring set went non-ACTIVE (a second recovery
        // re-entered before submission, not yet reflected in
        // `driver_generation`) — a transport-detected instance-generation
        // boundary, never a provider verdict. `ENXIO` in this position is
        // reserved for the transport boundary.
        VfsFailureCause::GenerationBoundary
    } else {
        // A genuine provider verdict on the reloaded instance.
        VfsFailureCause::Provider(errno)
    };
    match cause {
        // Both non-latching transport causes take the sleep-and-retry
        // arm: `KernelError::ENXIO` is the retry sentinel the dispatch
        // loop catches (it sleeps on `sb.recovery_wait` and retries
        // against the next generation). Userspace still sees EIO if all
        // retries are exhausted; the syscall adapter maps both to EIO —
        // the userspace contract is unchanged.
        VfsFailureCause::LostByRecovery => KernelError::ENXIO,
        VfsFailureCause::GenerationBoundary => KernelError::ENXIO,
        // A provider-RETURNED errno ALWAYS latches the fd dead — NO
        // per-errno exceptions (ENOENT: inode vanished in journal replay;
        // driver-refused re-open; a provider getattr failure after a
        // successful re-open; any other provider errno is equally a
        // terminal driver verdict). All subsequent operations return EIO
        // until close(2), matching recovery step f's contract for
        // unlinked-but-open files.
        VfsFailureCause::Provider(_) => {
            file.reopen_errno.store(-(Errno::EIO as i32), Ordering::Release);
            KernelError::EIO
        }
    }
}

/// Apply a post-recovery `InodeAttr` snapshot to the Core-resident `Inode`:
/// update `i_size` (Release) and the timestamp triple (under `INODE_LOCK`)
/// from the post-journal-replay on-disk state, refresh the attribute-flag
/// mirror (`i_flags.store(fs_flags_to_inode_flags(attr.fs_flags).bits(), Release)` —
/// a journal replay may have rolled back a pre-crash `chattr`; see
/// [Section 14.1](#virtual-filesystem-layer--inode-attribute-flags)), then truncate
/// page-cache pages beyond the reconciled size via
/// `truncate_inode_pages_range(&inode.i_mapping, attr.size, u64::MAX)` —
/// data rolled back by journal replay must not be resurrected from cache.
/// Called only from `vfs_revalidate_open_file()`; idempotent (a racing
/// revalidation of a second fd on the same inode applies the same snapshot).
fn vfs_apply_reconciled_attr(inode: &Inode, attr: InodeAttr);

/// Look up the CURRENT owning instance for `(ns_id, mount_id)` in the
/// Shadow Mount Registry. Returns a clone of the shadow entry's owning
/// `Arc<Mount>`, or `None` if the pair is not attached (never attached
/// through the registry, removed by a committed umount, its namespace
/// destroyed, or lost in reconstruction). Core-side, `registry_lock` taken
/// internally (warm path).
fn shadow_mount_lookup(ns_id: u64, mount_id: u64) -> Option<Arc<Mount>>;

/// Move this fd's mount pin to the post-reconstruction `Mount` instance.
/// Caller MUST hold `file.revalidate_lock` (serializes the epoch check,
/// the `mnt_count` transfer, and the cell swap against racing fd clones).
/// Idempotent: at most one rebind per crash epoch per `OpenFile`.
fn vfs_rebind_mount(file: &OpenFile) -> Result<(), KernelError> {
    let epoch = SHADOW_MOUNT_REGISTRY.crash_epoch.load(Ordering::Acquire);
    // Shape-1 recovery (or already rebound): the mount tree was not
    // rebuilt since this fd's last bind — the existing reference is the
    // live instance, even when it is a lazy-unmounted/detached mount that
    // is legitimately absent from the registry.
    if file.mount_epoch.load(Ordering::Acquire) == epoch {
        return Ok(());
    }
    match shadow_mount_lookup(file.mount_ns_id, file.mount_id) {
        Some(live) => {
            // Transfer the pin: the reconstructed instance starts with no
            // knowledge of pre-crash open files. The orphan's own
            // mnt_count is untrusted post-crash garbage and is never read.
            live.mnt_count.fetch_add(1, Ordering::Relaxed);
            // Publish the new reference; the orphan Arc held by the cell
            // is dropped after a grace period (RcuCell store semantics),
            // releasing the old instance if this was its last reference.
            file.mount.store(live);
            file.mount_epoch.store(epoch, Ordering::Release);
            Ok(())
        }
        None => {
            // The mount did not survive reconstruction: lazy-unmounted or
            // detached pre-crash, or its subtree failed to rebuild. Latch
            // the fd dead — matching the documented rule that such mounts
            // do not survive a VFS-module crash.
            file.reopen_errno.store(-(Errno::EIO as i32), Ordering::Release);
            Err(KernelError::EIO)
        }
    }
}

Dispatch composition. The dispatch wrapper's pre-select_ring() sequence becomes:

loop {
    match vfs_check_open_generation(file) {         // Acquire loads
        Ok(())        => {}                         // fast path: one compare
        Err(ENOTCONN) => match vfs_revalidate_open_file(file) {
            Ok(())      => {}                       // fd refreshed
            // Second recovery in progress: same treatment as a boundary
            // ENXIO — sleep until resume_fn's wake, then retry everything.
            Err(ENXIO)  => { sb.recovery_wait.wait_interruptible()?; continue; }
            Err(e)      => return Err(e),           // EIO (dead fd) / ENOMEM
        },
        Err(e)        => return Err(e),
    }
    match select_ring(ring_set) {
        Ok(ring)     => break ring,                 // proceed with dispatch
        Err(ENXIO)   => sb.recovery_wait.wait_interruptible()?, // step b; retry
        Err(e)       => return Err(e),
    }
}

-ENOTCONN therefore never reaches userspace for a regular file fd whose superblock recovered — the fd is transparently refreshed. (The vfs_check_open_generation() doc in Section 14.3 — "userspace must close and re-open the file" — describes the pre-refresh transport behavior; this protocol is the layer above it that absorbs the ENOTCONN.)

Concurrency and second crashes. revalidate_lock (a sleeping mutex; the re-open sleeps on the ring) serializes racing threads that share the OpenFile; losers re-check the generation under the lock and return immediately. A second crash landing mid-revalidation has TWO sub-cases, and BOTH propagate without latching (the dispatch wrapper sleeps on sb.recovery_wait and the retry revalidates against the third-generation instance): - Before submission — select_ring() rejects the re-open at the boundary with a transport-level ENXIO; the GenerationBoundary arm handles it. - After submission — the re-open was already in flight when the second crash's Step U6/U7 sweep drained it with -EIO (the in-flight drain contract, Section 14.2). Without discrimination this -EIO would hit the latching catch-all and kill the fd. It is instead resolved as VfsFailureCause::LostByRecovery at the dispatch wrapper — the wrapper observes that sb.driver_generation advanced past the cur it sampled under revalidate_lock, so the errno is a transport loss, not a driver verdict — and takes the same no-latch, sleep-and-retry arm as the boundary ENXIO. Only a genuine provider-returned errno (a real FileOps::open() result) reaches the latching catch-all. The generation is re-sampled inside the lock, so a revalidation can never store a stale generation over a newer one: driver_generation is bumped only at Step U10a, strictly before VFSRS_ACTIVE (Step U17), and the re-open cannot complete successfully while the ring set is not ACTIVE.

Hot-path cost. Zero new cost: the per-operation generation compare already existed; open_generation becoming AtomicU64 changes a plain load into an Acquire load of the same width — the same instruction class on all eight architectures (and this path already performs an Acquire load of driver_generation). The revalidation body runs only on generation mismatch, i.e., at most once per fd per crash.

14.1.1.2 Metadata Access Amortization

Metadata-heavy workloads (find, package managers, ls -la, container image unpacking) are dominated by the readdir+stat pattern: the application reads a directory, then immediately stats every entry. Without amortization, each stat() incurs a full Core-to-VFS domain crossing (~18ns on x86-64 MPK, ~32-64ns on AArch64 POE). This section specifies two complementary mechanisms that eliminate most of those crossings, plus a per-filesystem policy framework that controls when prefetch is safe.

14.1.1.2.1 Mechanism 1: Readdir-Plus Prefetch with Per-Task Buffer

When a filesystem's readdir implementation returns directory entries, the VFS also collects statx metadata for each returned entry. The inode is already resolved from the dentry lookup during readdir — fetching its attributes is essentially free for local filesystems (the inode struct is already cache-hot). The VFS writes the metadata into a per-task prefetch buffer allocated in Core memory.

Subsequent stat() / statx() calls on the same directory's entries check the per-task buffer before crossing into the VFS domain. On hit, stat() returns immediately with zero domain crossings.

Key design constraints:

• Per-task buffer, not a global cache. The buffer is private to each task. No locking, no cross-task contention, no cache coherence traffic. Allocated on first readdir() for a given directory fd, freed when the directory fd is closed or the task exits.
• Bounded size: 128 entries x ~320 bytes = ~40KB per task. This exceeds L1 data cache capacity on some architectures (ARMv7 Cortex-A15: 32KB L1D; Cortex-A72/A76: 32KB-48KB L1D). On these targets the scan spills to L2 (~10-15 cycles per access vs ~4 cycles for L1), adding ~640-1280 cycles worst-case on a full miss path. On x86-64 (32-48KB L1D typical) the buffer may fit depending on core implementation. The sequential access pattern means the hardware prefetcher keeps up regardless — L2 prefetch on ARM delivers ~6-8 cycles per line, acceptable for the miss path (which is cold: a readdir-stat pattern that misses has already paid a domain crossing). Miss path cost: 128 iterations x (DevId compare + InodeId compare + Relaxed atomic load) = ~384-640 cycles on L1 hit, ~640-1280 cycles with L2 spills. This is paid only when the target inode is NOT in the prefetch buffer (the common case after readdir IS a hit). Covers the vast majority of directories (median directory size in real workloads is 20-60 entries). For directories larger than 128 entries, only the most recently returned batch is buffered; earlier entries that were already stat'd remain valid, later entries fall through to the normal VFS path.
• Keyed by (sb_dev, ino) — stable identifiers that survive VFS crash/evolution. No path strings, no dentry pointers, no VFS-internal state.
• Generation counter per entry. VFS increments the inode's generation counter on any metadata mutation (setattr, truncate, write that changes mtime, etc.). Core checks the generation before returning a prefetch hit. Stale entries produce a miss and fall through to the normal VFS path.
• VFS epoch counter for crash/evolution invalidation. The buffer records the VFS epoch at fill time. If the VFS is replaced (crash recovery or live evolution), the global VFS_EPOCH counter is incremented and all buffers become stale in O(1).
• Core memory (Tier 0) — the buffer itself is not in the VFS domain. It survives VFS crash without corruption.

Data structures:

/// One statx timestamp. Binary-compatible with Linux `struct
/// statx_timestamp` (`include/uapi/linux/stat.h`): 16 bytes, identical on
/// 32- and 64-bit ABIs. Native ints (userspace↔kernel, not a wire struct).
#[repr(C)]
pub struct StatxTimestamp {
    /// Seconds since the epoch.
    pub tv_sec: i64,
    /// Nanoseconds within the second (0..=999_999_999).
    pub tv_nsec: u32,
    /// Reserved; must be zero.
    pub __reserved: i32,
}
const_assert!(size_of::<StatxTimestamp>() == 16);

/// statx(2) result buffer — binary-compatible with Linux `struct statx`
/// (`include/uapi/linux/stat.h`, torvalds/linux master), copied verbatim to
/// userspace. Fixed 256-byte layout, IDENTICAL on all eight architectures
/// (every field is a fixed-width int or `StatxTimestamp`; no pointer/`usize`
/// members), so a single `const_assert!` covers 32- and 64-bit targets.
///
/// Native ints, NOT `Le`/`Be`: this is a userspace↔kernel ABI struct, never
/// a cross-node or on-disk wire format. Reserved fields (`__spare*`) are
/// zeroed by the filler. Field names and order match Linux exactly so a
/// glibc `statx()` sees the expected offsets.
#[repr(C)]
pub struct StatxBuf {
    pub stx_mask: u32,                      // 0x00
    pub stx_blksize: u32,                   // 0x04
    pub stx_attributes: u64,                // 0x08
    pub stx_nlink: u32,                     // 0x10
    pub stx_uid: u32,                       // 0x14
    pub stx_gid: u32,                       // 0x18
    pub stx_mode: u16,                      // 0x1C
    pub __spare0: [u16; 1],                 // 0x1E
    pub stx_ino: u64,                       // 0x20
    pub stx_size: u64,                      // 0x28
    pub stx_blocks: u64,                    // 0x30
    pub stx_attributes_mask: u64,           // 0x38
    pub stx_atime: StatxTimestamp,          // 0x40
    pub stx_btime: StatxTimestamp,          // 0x50
    pub stx_ctime: StatxTimestamp,          // 0x60
    pub stx_mtime: StatxTimestamp,          // 0x70
    pub stx_rdev_major: u32,                // 0x80
    pub stx_rdev_minor: u32,                // 0x84
    pub stx_dev_major: u32,                 // 0x88
    pub stx_dev_minor: u32,                 // 0x8C
    pub stx_mnt_id: u64,                    // 0x90
    pub stx_dio_mem_align: u32,             // 0x98
    pub stx_dio_offset_align: u32,          // 0x9C
    pub stx_subvol: u64,                    // 0xA0
    pub stx_atomic_write_unit_min: u32,     // 0xA8
    pub stx_atomic_write_unit_max: u32,     // 0xAC
    pub stx_atomic_write_segments_max: u32, // 0xB0
    pub stx_dio_read_offset_align: u32,     // 0xB4
    pub stx_atomic_write_unit_max_opt: u32, // 0xB8
    pub __spare2: [u32; 1],                 // 0xBC
    pub __spare3: [u64; 8],                 // 0xC0..0x100
}
const_assert!(size_of::<StatxBuf>() == 256);
const_assert!(align_of::<StatxBuf>() == 8);

/// Single prefetch entry. Aligned to cache line to avoid false sharing
/// when the VFS writer and the Core reader access adjacent entries.
///
/// Total size: 320 bytes (64-byte aligned).
/// - DevId (4 bytes) + InodeId (8 bytes) + generation (8 bytes)
///   + StatxBuf (256 bytes) + valid (1 byte) + padding (43 bytes) = 320.
#[repr(C, align(64))]
pub struct PrefetchEntry {
    /// Superblock device ID. Together with `ino`, forms the unique key.
    pub sb_dev: DevId,
    /// Explicit padding: DevId is 4 bytes, InodeId requires 8-byte alignment.
    pub _pad0: [u8; 4],
    /// Inode number within the filesystem identified by `sb_dev`.
    pub ino: InodeId,
    /// Inode generation counter at the time this entry was filled.
    /// VFS increments the inode's generation on any metadata mutation.
    /// Core compares this against the current inode generation before
    /// returning the entry. Mismatch → stale → fall through to VFS.
    ///
    /// Memory ordering: stored with Release by VFS (during readdir fill),
    /// loaded with Acquire by Core (during stat fast path).
    pub generation: AtomicU64,
    /// Cached statx result. Layout matches `struct statx` from Linux
    /// (256 bytes, binary compatible with the userspace ABI).
    pub stx: StatxBuf,
    /// Entry validity flag. 0 = invalid, 1 = valid. AtomicU8 instead of
    /// AtomicBool: this is cross-domain shared memory (VFS driver domain
    /// writes, Core reads). A non-0/1 value from a corrupted cross-domain
    /// writer would cause UB with AtomicBool's validity invariant.
    pub valid: AtomicU8,
    /// Explicit trailing padding: AtomicU8 ends at offset 281; align(64)
    /// rounds struct size to 320 (next multiple of 64). 320 - 281 = 39.
    pub _pad_tail: [u8; 39],
}
// Layout with align(64): DevId(4) + _pad0(4) + InodeId(8) + AtomicU64(8) +
// StatxBuf(256) + AtomicU8(1) + _pad_tail(39) = 320 bytes (5 × 64-byte cache lines).
const_assert!(size_of::<PrefetchEntry>() == 320);

/// Per-task readdir prefetch buffer. Allocated in Core memory (Tier 0).
///
/// **Exactly ONE buffer exists per task** (`Task.prefetch_buf:
/// Option<Box<TaskPrefetchBuf>>`), reused across directories: each
/// `readdir()` on a prefetching filesystem REFILLS the buffer for that
/// directory fd (`dir_fd` records which one it currently serves; a
/// readdir on a different fd resets the buffer first). This matches the
/// readdir+stat access pattern — `ls -l`-style consumers stat the
/// entries of the directory they just listed, not of directories listed
/// earlier — and it is what the fast path implements: `sys_statx_fast_path`
/// reads the single `task.prefetch_buf`. (An earlier revision said "one
/// buffer per open directory fd" — that contradicted the fast path AND
/// made aggregate memory proportional to open directory fds, an
/// unprivileged Core-memory DoS: 10^5 directory fds × ~41KB ≈ 4 GiB.
/// The per-task model caps the cost at one buffer per task.)
///
/// **Lifecycle and accounting**:
/// - Lazily allocated (single ~41KB Core allocation, warm path) on the
///   task's FIRST `readdir()` of a filesystem whose prefetch policy
///   enables prefetch — tasks that never readdir pay nothing.
/// - Charged to the task's memory cgroup as kernel memory (memcg kmem,
///   [Section 17.2](17-containers.md#control-groups)), so container limits bound the aggregate.
/// - Freed at task exit, and eagerly freed when the `dir_fd` it
///   currently serves is closed (the contents are useless then).
/// - Aggregate bound: ≤ 1 buffer × live tasks. A task already costs
///   ≥ 32KB of unswappable kernel memory (kernel stack + Task struct +
///   page tables), so the buffer adds a bounded constant factor — task
///   count is the existing, already-enforced limit (RLIMIT_NPROC,
///   pids cgroup, memcg). No dedicated shrinker: walking the task list
///   to reclaim ~41KB per task is poor reclaim value, and the memcg
///   charge already gives per-container enforcement.
///
/// **Collection policy**: Hot path (per-syscall lookup). `entries` is
/// a fixed-capacity `ArrayVec` — no heap allocation on the hot path.
/// The 128-entry limit bounds the buffer to ~41KB.
pub struct TaskPrefetchBuf {
    /// Prefetch entries, indexed by insertion order. Lookup is linear
    /// scan over at most 128 entries (~40KB — exceeds L1 on ARMv7/some
    /// AArch64; spills to L2 with ~3-5 extra cycles/access on those
    /// targets). For the readdir+stat pattern, entries are accessed in
    /// insertion order (sequential scan), so linear search has optimal
    /// prefetch behavior regardless of L1/L2 residency.
    entries: ArrayVec<PrefetchEntry, 128>,
    /// Directory fd this buffer is currently filled for. A `readdir()`
    /// on a different directory fd resets the buffer (clears `entries`)
    /// before refilling; closing this fd frees the buffer.
    dir_fd: i32,
    /// VFS epoch at fill time. If the global `VFS_EPOCH` has advanced
    /// (crash or live evolution), the entire buffer is stale and must
    /// be discarded. This is an O(1) invalidation mechanism.
    vfs_epoch: u64,
}

/// Global VFS epoch counter. Incremented on VFS crash recovery or
/// live evolution. All `TaskPrefetchBuf` instances whose `vfs_epoch`
/// differs from this value are stale.
///
/// Stored as AtomicU64 in Core memory. Incremented with Release
/// ordering; read with Acquire ordering in the stat fast path.
///
/// **Longevity**: u64 counter incremented only on VFS crash or
/// evolution events. At one event per second (vastly exceeding any
/// realistic crash rate), this counter lasts ~584 billion years.
pub static VFS_EPOCH: AtomicU64 = AtomicU64::new(0);

stat() fast path (Core-side, before any domain crossing):

/// Attempt to serve a statx() call from the per-task prefetch buffer.
/// Returns `Some(stx)` on hit (zero domain crossings), `None` on miss
/// (caller falls through to the normal VFS domain crossing path).
///
/// This function runs entirely in Core (Tier 0). No locks, no domain
/// crossings, no ring buffer interaction. The only synchronization is
/// atomic loads on the VFS epoch and per-entry generation counters.
///
/// # Hot path classification
///
/// This is called on every `stat()` / `statx()` / `fstat()` /
/// `newfstatat()` syscall when the task has an active prefetch buffer.
/// Must be O(1) amortized with no heap allocation.
fn sys_statx_fast_path(task: &Task, dentry_dev: DevId, dentry_ino: InodeId) -> Option<StatxBuf> {
    let buf = task.prefetch_buf.as_ref()?;
    // Check VFS epoch — if VFS was replaced, entire buffer is stale.
    if buf.vfs_epoch != VFS_EPOCH.load(Acquire) {
        return None;
    }
    // Linear scan over at most 128 entries. Sequential access pattern
    // means the prefetcher keeps up; worst case is 128 × 320B = 40KB
    // which exceeds L1 on some architectures (ARMv7 32KB L1D, some
    // AArch64 32-48KB L1D) — L2 spill adds ~3-5 cycles/access on
    // those targets. Acceptable: miss path is cold (already paying a
    // domain crossing on fallthrough).
    let entry = buf.entries.iter().find(|e| {
        e.valid.load(Relaxed) && e.sb_dev == dentry_dev && e.ino == dentry_ino
    })?;
    // Check generation — if inode was mutated since readdir, entry is stale.
    let gen = entry.generation.load(Acquire);
    if gen != inode_current_generation(dentry_dev, dentry_ino) {
        return None;
    }
    Some(entry.stx)
}

Generation counter update path (VFS-side):

When the VFS processes any inode-mutating operation (SetAttr, Truncate, Write that updates mtime/ctime, Link, Unlink, Rename), it increments the inode's generation counter. This is a single AtomicU64::fetch_add(1, Release) on the inode struct — the inode is already locked for the mutation, so this adds zero contention. The generation counter is stored in the inode struct itself (in VFS memory), and the Core stat fast path reads it via a shared-memory mapping (the inode's generation field is in a page mapped read-only into Core's domain). No ring buffer crossing is needed for the generation check.

The read and bump accessors resolve the inode through the two-level XArray lookup (superblock by DevId, then inode by number) — both O(1), lockless under RCU, no allocation:

/// Read an inode's current metadata-mutation generation
/// (`Inode.i_meta_generation`) for stat/statx prefetch staleness validation.
///
/// Resolves `(dev, ino)` via `SUPER_BLOCK_MAP` then the superblock's icache.
/// Returns `u64::MAX` — a value a live counter (starts at 0, monotonically
/// increments) can never hold within the uptime target — if the superblock
/// or inode is no longer cached, so a prefetch entry referencing an evicted
/// inode always registers as stale (fast-path miss → fall through to VFS).
///
/// Hot path (every `stat`/`statx`/`fstat`/`newfstatat` fast-path probe):
/// O(1), lockless, no heap allocation.
fn inode_current_generation(dev: DevId, ino: InodeId) -> u64 {
    let _rcu = rcu_read_lock();
    let Some(sb) = SUPER_BLOCK_MAP.get().and_then(|m| m.load(dev.raw as u64)) else {
        return u64::MAX; // superblock unmounted — cached entry is stale
    };
    match sb.inode_cache.load(ino.0) {
        Some(inode) => inode.i_meta_generation.load(Ordering::Acquire),
        None => u64::MAX, // inode evicted — cached entry is stale
    }
}

/// Increment an inode's metadata-mutation generation, invalidating any
/// stat/statx prefetch entry that snapshotted the previous value.
///
/// Called from every VFS inode-mutating path (SetAttr/Truncate/Write
/// mtime-ctime/Link/Unlink/Rename) and from `vfs_invalidate_prefetch()` on
/// network/cluster cache-invalidation events. No-op if the inode is not
/// cached (nothing could hold a matching prefetch entry). Warm/cold path.
fn inode_bump_generation(dev: DevId, ino: InodeId) {
    let _rcu = rcu_read_lock();
    let Some(sb) = SUPER_BLOCK_MAP.get().and_then(|m| m.load(dev.raw as u64)) else {
        return;
    };
    if let Some(inode) = sb.inode_cache.load(ino.0) {
        inode.i_meta_generation.fetch_add(1, Ordering::Release);
    }
}

Cross-domain memory ordering invariant: This is a shared-memory cross-domain access pattern — the VFS (Tier 1) writes the generation counter with Release, and Core (Tier 0) reads it with Acquire. On x86-64 (TSO), these translate to plain loads/stores with no additional fences. On ARM/AArch64 and RISC-V (weak memory models), the Acquire load emits the appropriate barrier instruction (LDAR on AArch64, fence r,rw on RISC-V) to ensure the stat fields are observed consistently with the generation counter. This ordering MUST NOT be downgraded to Relaxed in future maintenance — doing so would allow Core to observe a new generation counter but stale stat fields, returning incorrect metadata to userspace.

Readdir fill path (VFS-side):

During readdir() processing, after the filesystem driver returns each batch of directory entries (via the VfsResponse for ReadDir), the VFS first ensures the task's single prefetch buffer is ready: if task.prefetch_buf is None, allocate it (memcg-kmem-charged, see the TaskPrefetchBuf lifecycle above); if buf.dir_fd differs from the directory fd being read, clear entries and set dir_fd to the new fd (the buffer serves one directory at a time). Then, for each returned entry:

1. Looks up the inode in the inode cache (already resolved during readdir).
2. Copies the inode's current statx attributes into a PrefetchEntry.
3. Stores the current inode generation counter.
4. Writes the entry into the task's TaskPrefetchBuf via the shared-memory mapping.

This piggybacks on work already being done — the inode is cache-hot from the readdir lookup. The additional cost is ~50-80 cycles per entry (one memcpy of 256 bytes for the StatxBuf + two atomic stores). For a typical directory of 50 entries, this is ~2,500-4,000 cycles total — less than the cost of a single domain crossing.

14.1.1.2.2 Mechanism 2: io_uring Statx Coalescing

When the io_uring submission queue contains multiple consecutive IORING_OP_STATX entries, the VFS dispatcher coalesces them into a single domain crossing. Instead of N crossings for N stat requests, one crossing processes all N.

Detection and dispatch:

The io_uring dispatch loop (Section 19.3) already processes SQEs in batches. When the dispatcher encounters an IORING_OP_STATX SQE, it peeks ahead in the submission queue for consecutive IORING_OP_STATX entries, collecting up to 64 into a single batch. The batch is sent to the VFS as a single VfsRequest::StatxBatch over the ring buffer.

/// Batched statx request. Sent as a single VfsRequest when the io_uring
/// dispatcher detects consecutive IORING_OP_STATX SQEs.
///
/// The VFS resolves all paths in a single domain stay and writes all
/// results back in a single VfsResponse::StatxBatchResult.
pub struct StatxBatchArgs {
    /// Number of statx requests in this batch (1..=64).
    pub count: u8,
    /// DMA buffer handle containing an array of `StatxBatchEntry` structs.
    /// The buffer is allocated from the io_uring's pre-registered buffer
    /// pool when available, or from the shared DMA pool otherwise.
    pub entries_buf: DmaBufferHandle,
}

/// Single entry within a StatxBatch request.
#[repr(C)]
pub struct StatxBatchEntry {
    /// Directory fd for path resolution (AT_FDCWD or an open directory).
    pub dirfd: i32,
    /// AT_* flags (AT_SYMLINK_NOFOLLOW, AT_EMPTY_PATH, etc.).
    pub flags: u32,
    /// STATX_* request mask.
    pub mask: u32,
    /// Path string offset within the DMA buffer's string region.
    pub path_offset: u32,
    /// Path string length in bytes.
    pub path_len: u16,
    /// Padding to 4-byte alignment for array element stride. Without this
    /// pad, `path_len: u16` at offset 18 leaves the struct at 20 bytes but
    /// with 2 bytes of implicit tail padding for `u32`-aligned array access.
    /// Making it explicit ensures no uninitialized bytes leak across userspace.
    pub _pad: [u8; 2],
}
const_assert!(size_of::<StatxBatchEntry>() == 20);

/// Batched statx response. One result per entry in the request.
/// C-compatible layout: fixed array with explicit count. Neither
/// `ArrayVec` nor `Result<T, E>` have stable repr(C) layout.
#[repr(C)]
pub struct StatxBatchResult {
    /// Number of valid entries in `results`.
    pub count: u8,
    /// Explicit padding: count(u8, offset 1) to results[0] (align 8 from StatxBuf).
    /// 7 bytes: 1 + 7 = 8. CLAUDE.md rule 11.
    pub _pad: [u8; 7],
    /// Per-entry results. Index corresponds to the request entry index.
    pub results: [StatxBatchResultEntry; 64],
}

/// Single entry in a batched statx response.
#[repr(C)]
pub struct StatxBatchResultEntry {
    /// 0 = success (stx is valid), negative = errno (stx is zeroed).
    pub error: i32,
    /// Explicit padding: error(i32, offset 0+4=4) to stx (align 8 from StatxBuf).
    /// 4 bytes. CLAUDE.md rule 11.
    pub _pad: [u8; 4],
    /// Valid only when `error == 0`.
    pub stx: StatxBuf,
}
// Layout: error(4) + _pad(4) + StatxBuf(256) = 264 bytes. All padding explicit.
const_assert!(size_of::<StatxBatchResultEntry>() == 264);
// StatxBatchResult: count(1) + _pad(7) + 64 × 264 = 16904 bytes. All padding explicit.
const_assert!(size_of::<StatxBatchResult>() == 16904);

Key design properties:

• Transparent to userspace. Applications submit individual IORING_OP_STATX SQEs as usual. The coalescing is entirely internal to the kernel's io_uring dispatch path. Each SQE still gets its own CQE with the correct user_data and result code.
• No additional memory overhead. The batch uses the existing ring buffer and DMA buffer infrastructure. The StatxBatchEntry array is written into a DMA buffer that is already allocated for the io_uring ring.
• Works with all filesystem types. Each stat within the batch goes through normal VFS path resolution and filesystem locking. The coalescing only eliminates the domain crossing overhead, not any per-file locking. Filesystems with Never prefetch policy still benefit from crossing amortization.
• Interaction with readdir-plus prefetch. Before sending a StatxBatch to the VFS, the dispatcher checks each entry against the task's TaskPrefetchBuf. Entries that hit the prefetch buffer are resolved immediately and removed from the batch. Only cache-miss entries cross into the VFS domain. In the best case (all 64 entries hit the prefetch buffer), zero domain crossings occur.
• Ring protocol extension. StatxBatch is added as VfsOpcode::StatxBatch = 70 in the VFS ring protocol (Section 14.2). The response uses VfsOpcode::StatxBatchResult = 71. These opcodes are only generated by the io_uring coalescing path; they are never exposed to filesystem drivers directly (the VFS dispatches individual Getattr calls internally for each entry in the batch).

14.1.1.2.3 Prefetch Policy Framework

Each filesystem declares its readdir prefetch policy via a method on the FileSystemOps trait (Section 14.1):

/// Policy controlling whether the VFS prefetches statx metadata during
/// readdir for this filesystem.
///
/// The default implementation returns `Always`, which is correct for all
/// single-node local filesystems. Network and cluster filesystems must
/// override this to return the appropriate policy.
pub enum ReaddirPrefetchPolicy {
    /// Always prefetch. Data is authoritative — no external consistency
    /// concerns. The VFS fills the per-task prefetch buffer on every
    /// readdir, and stat() uses the buffer unconditionally (subject to
    /// generation counter freshness).
    ///
    /// Appropriate for: ext4, XFS, btrfs, tmpfs, procfs, sysfs, debugfs.
    Always,

    /// Prefetch, but layer on top of the filesystem's existing attribute
    /// cache. The prefetch buffer entries are valid only as long as the
    /// filesystem's own cache considers them valid. When the filesystem
    /// invalidates its cache (e.g., NFS delegation recall, CIFS oplock
    /// break, FUSE attr_timeout expiry), it calls
    /// `vfs_invalidate_prefetch(sb_dev, ino)` which bumps the generation
    /// counter on any matching prefetch entry. No additional locking is
    /// needed — the prefetch mechanism layers on top of whatever
    /// consistency protocol the filesystem already implements.
    ///
    /// Appropriate for: NFS, CIFS/SMB, FUSE, UmkaOS peerfs.
    CacheAware,

    /// Never prefetch. Each stat() acquires its own consistency token
    /// (e.g., DLM glock) for linearizability. The domain crossing cost
    /// (~18ns on x86-64) is negligible compared to the distributed lock
    /// round-trip (~50-500us), so prefetch elimination provides no
    /// measurable benefit and would violate the consistency model.
    ///
    /// Appropriate for: GFS2, OCFS2, UmkaOS DLM-based cluster FS.
    Never,
}

Per-filesystem policy table:

Filesystem Type	Policy	Rationale
ext4, XFS, btrfs, tmpfs	Always	Single-node, no external consistency concerns. Inode data is authoritative.
procfs, sysfs, debugfs	Always	Synthetic FS. Metadata is kernel-generated and stable within a readdir window.
NFS (v3/v4)	CacheAware	Layers on NFS actimeo/delegation cache. CB_RECALL bumps generation counter.
CIFS/SMB	CacheAware	Layers on oplock/lease cache. Lease break bumps generation counter.
FUSE	CacheAware	Layers on FUSE entry_timeout/attr_timeout. Unified regardless of whether the daemon supports FUSE_READDIRPLUS.
UmkaOS peerfs (distributed)	CacheAware	Peer protocol metadata push notifications bump generation counter (Section 5.1).
GFS2, OCFS2	Never	DLM linearizability required. ~18ns crossing is 0.004% of ~50-500us glock round-trip.
UmkaOS DLM-based cluster FS	Never	Same as GFS2/OCFS2 — DLM consistency model requires per-stat lock acquisition.
Overlayfs	Inherits	Upper layer: Always (local, mutable). Lower layers: read-only, so Always (immutable data is trivially consistent).

14.1.1.2.4 Network/Cluster Filesystem Invalidation Integration

The CacheAware policy integrates with each filesystem's existing cache invalidation mechanism through a single Core-side callback:

/// Invalidate every cached prefetch entry for the given (sb_dev, ino) pair.
/// Called by filesystem cache invalidation handlers (NFS CB_RECALL,
/// CIFS lease break, FUSE NOTIFY_INVAL_INODE, peerfs MetadataInvalidate
/// ([Section 5.1](05-distributed.md#distributed-kernel-architecture--metadata-invalidation-wire-message))).
///
/// Invalidation is O(1) and lazy: it bumps the inode's metadata-mutation
/// generation counter (`i_meta_generation`) via `inode_bump_generation()`.
/// No task list is walked and no per-entry iteration occurs — every prefetch
/// entry still holding the pre-bump generation simply fails its freshness
/// check on the next `stat()` fast path and falls through to the VFS.
///
/// # Performance
///
/// Cold path — called only on cache invalidation events, which are
/// infrequent relative to stat() calls. A single atomic increment; the cost
/// is independent of the number of tasks or cached prefetch entries.
pub fn vfs_invalidate_prefetch(sb_dev: DevId, ino: InodeId) {
    // Bump the generation counter on the inode. Any prefetch entry
    // holding the old generation will fail the freshness check on the
    // next stat() fast path and fall through to the VFS.
    inode_bump_generation(sb_dev, ino);
}

Per-filesystem invalidation triggers:

• NFS: When the NFS client receives CB_RECALL (NFSv4 delegation return) or detects actimeo expiry (NFSv3/v4 attribute timeout), it calls vfs_invalidate_prefetch(sb_dev, ino). Next stat() sees generation mismatch, crosses to VFS, and the NFS client re-fetches attributes from the server.

• CIFS/SMB: When the CIFS client receives an oplock break or lease break notification from the SMB server, it calls vfs_invalidate_prefetch(sb_dev, ino).

• FUSE: When attr_timeout expires or the FUSE daemon sends FUSE_NOTIFY_INVAL_INODE, the FUSE client calls vfs_invalidate_prefetch(sb_dev, ino).

• UmkaOS peerfs: The peer protocol MetadataInvalidate message (Section 5.1) triggers vfs_invalidate_prefetch(). This is tighter than NFS because the peer node pushes invalidations proactively (not just on delegation recall), reducing the stale-data window.

Why this is better than existing approaches:

1. Eliminates the domain crossing. NFS READDIRPLUS only eliminates the network round-trip for attribute fetches; the kernel-side VFS domain crossing still occurs for each stat(). Our readdir-plus prefetch eliminates both.
2. Unified across all remote filesystems. NFS, FUSE, CIFS, and peerfs all use the same Core-side prefetch buffer with the same generation-counter invalidation. No per-filesystem prefetch implementation is needed.
3. Generation-counter invalidation is more precise than time-based expiry. NFS actimeo is a blunt timeout; our generation counter reflects actual inode mutations. The result is fewer false invalidations and a higher effective hit rate.

14.1.1.2.5 Live Evolution Interaction

• The prefetch buffer is in Core memory (Tier 0) and survives VFS live evolution (Section 13.18) unchanged.
• On VFS evolution: Core increments VFS_EPOCH (single fetch_add(1, Release)). All prefetch buffers become stale in O(1) — no per-task or per-entry iteration.
• The new VFS instance exports fresh inode generation counters. The first readdir() after evolution refills the buffer with current data.
• No data from the old VFS instance leaks through — the epoch check catches everything before any stale entry is returned to userspace.

14.1.1.2.6 Crash Recovery Interaction

• On VFS crash: Core increments VFS_EPOCH (same mechanism as evolution). All prefetch buffers are invalidated atomically.
• Buffer memory is in Core (Tier 0) and cannot be corrupted by a VFS crash.
• Subsequent stat() calls miss the buffer, fall through to the VFS domain crossing, and trigger VFS restart via the normal crash recovery path (Section 11.9).
• After VFS recovery completes, the next readdir() refills the buffer. The transient period between crash and buffer refill uses the unoptimized path (full domain crossing per stat), which is correct but slower.

14.1.1.2.7 Amortized Performance Budget

With both mechanisms active, the effective metadata overhead for common access patterns:

Access Pattern	Mechanism	Effective Overhead (x86-64 MPK)	Domain Crossings
Single stat() (no prefetch)	None	~3.6-9% per call (~18ns / 200-500ns base)	1 round-trip
readdir + stat (Always/CacheAware FS)	Readdir-plus prefetch	~0.3-0.5% effective	1 crossing for readdir, 0 for stat hits (~95% hit rate)
io_uring batch of 64 IORING_OP_STATX	Statx coalescing	~0.05% per stat	1 crossing for 64 stats
io_uring batch + prefetch buffer	Both	~0.01% per stat (best case)	0 crossings on full prefetch hit
readdir + stat (Never-policy FS)	None (DLM overhead dominates)	~3.6-9% (negligible vs ~50-500us DLM)	1 round-trip per stat

Assumptions: 95% prefetch buffer hit rate for readdir+stat pattern (based on: median directory has <128 entries, stat() calls follow readdir in program order, inode mutation between readdir and stat is rare). Hit rate degrades for directories >128 entries (only the last batch is buffered) and for workloads that interleave mutations with stat.

Phase assignment: Readdir-plus prefetch is Phase 2 (required for metadata-heavy workload performance targets). io_uring statx coalescing is Phase 3 (optimization; the system is correct without it).

14.1.2 VFS Architecture

Responsibilities: path resolution, dentry caching, inode management, mount tree traversal, and permission checks (delegated to Core's capability system via the inter-domain ring buffer).

14.1.2.1 Nucleus / Evolvable Classification

Every VFS component is classified per the replaceability model (Section 13.18) along the data / code boundary — the line the Nucleus/Evolvable split is actually drawn at: data survives a live swap; code is replaced by it. Three categories appear below:

• Nucleus-owned data — instances live in the generic tracked allocator (Section 13.18) (e.g. Dentry, Mount, MountNamespace). They survive an Evolvable swap in place, and their layout is not frozen: it evolves via Shadow-and-Migrate. What survives is the instances and their identity, not a fixed field arrangement.
• Nucleus-preserved data — state kept in place across a swap (superblock freeze counters, ring memory, dirty-intent lists, errseq values) that the new image inherits and resumes from.
• Evolvable code and policy — every algorithm that reads or mutates the above, hot-swapped via EvolvableComponent without rebooting: the whole VFS module image (path walk, dcache lookup, mount-tree algorithms, writeback, readahead engine) plus the independently swappable policy vtables.

No VFS code is Nucleus. Correctness of the code is enforced by Nucleus invariant checkers (Section 13.18) that validate a proposed new image before it commits — not by freezing the code (freezing code is worse for 50-year correctness: a bug in it could not be fixed without a full rebuild). The "Classification" column below therefore names where a component's data lives (or, for the Evolvable rows, that it is code/policy); it is never a claim that the code cannot change.

Component	Classification	Rationale
Dentry cache (dcache hash table, Dentry struct, LRU list)	Nucleus-owned data	Dentry instances live in the generic tracked allocator ("Dentry Allocation — Nucleus Tracked Storage" below), so they survive an Evolvable swap in place and their layout evolves via Shadow-and-Migrate. All dcache code — RCU-walk, lookup, the LRU shrinker — is Evolvable. A dentry is not reclaimed while RCU readers may hold references (rcu_synchronize before free); that is a data-lifecycle rule, not a code-freeze claim.
Inode cache (per-superblock XArray, Inode struct, AddressSpace)	Nucleus-preserved data	Inode metadata (permissions, size, link count) is preserved across a swap so the new image resumes against the same objects; its layout is migratable via Shadow-and-Migrate — the migration walk enumerates live Inodes through each superblock's inode_cache XArray (the same index iget uses), NOT a per-type tracked registry (see "Layout migration of Inode and SuperBlock" below) — never "immutable". The generation-counter protocol for prefetch invalidation is a property of the preserved data; the accessors (iget/iput, the generation bump) are Evolvable code.
Mount table (MountNamespace, Mount struct, mount hash table)	Nucleus-owned data	Mount / MountNamespace instances are tracked (Section 14.6); the propagation graph and mount hash are preserved data that a new image inherits. All mount-tree algorithms — copy_tree, propagation, pivot_root, hash lookup — are Evolvable code, validated (not frozen) across evolution.
SuperBlock (per-mount filesystem state, SbWriters, freeze FSM)	Nucleus-preserved data	The superblock state — SbWriters counters, the freeze-state field, the error-behavior mode — is preserved in place across a swap and inherited by the new image; its layout is migratable via Shadow-and-Migrate — the migration walk enumerates live SuperBlocks through the global SUPER_BLOCK_MAP (all_superblocks()), NOT a per-type tracked registry (see "Layout migration of Inode and SuperBlock" below). The freeze-FSM transition code is Evolvable, not frozen; a swap replaces it along with the rest of the VFS image.
VFS ring protocol (VfsRingSet, VfsRingPair, request/response format)	Nucleus-preserved data + versioned KABI wire	The ring memory is preserved and inherited by a new VFS Evolvable (Section 14.3). The ring format (opcodes, response matching) is a versioned KABI wire contract: it must remain compatible across live evolution per the KABI versioning rules, not because the code is frozen. Producer/consumer code is IDL-generated Evolvable.
Dirty extent protocol (DirtyIntentList, reserve/commit/abort API)	Nucleus-preserved data	The DirtyIntentList contents are preserved data — crash recovery depends on the data surviving a swap. The reserve/commit/abort code is Evolvable; token single-use and overflow backpressure are its invariants (enforced in code, not by freezing it).
ErrSeq (writeback error tracking)	Nucleus-preserved data	The errseq value (an atomic u64 in its containing structs) is preserved data. The packing/advance code is Evolvable; the POSIX one-shot reporting contract is an ABI obligation of the algorithm, not a reason to freeze it.
Path resolution algorithm (RCU-walk, ref-walk fallback, symlink follow budget, mount crossing)	Evolvable (protected by the vfs_path_walk_safety invariant checker)	Runtime code: the per-syscall walk is VFS module code, replaced with the VFS Evolvable image. Its safety properties — the 40-follow symlink budget → ELOOP; mount-crossing confinement (BENEATH / NO_XDEV / IN_ROOT, .. clamping at root); RCU-walk seqcount revalidation before any result is trusted; and the deny-never-cached cached_perm rule — are preserved across live evolution by the vfs_path_walk_safety checker (defined below), not by freezing the code. Its op_check enforcement site is the symlink-follow step (Section 14.1).
Readahead window sizing (sequential detection, window growth/shrink)	Evolvable	Policy decision: the heuristic for when to grow or shrink the readahead window is a tuning knob, not a correctness property. ML can improve it. The readahead engine (page pre-allocation, I/O submission) is Evolvable module code, replaced with the whole VFS image; the window-sizing policy is an independently hot-swappable policy vtable.
Writeback scheduling (BDI dirty page selection, inode writeback ordering)	Evolvable	Policy decision: which dirty inodes to write back first, how to interleave sequential and random I/O, and when to trigger background writeback are heuristic choices. The writeback infrastructure (bio submission, completion tracking, writeback_lock) is Evolvable module code; the ordering policy is an independently hot-swappable vtable.
Dirty page throttling (balance_dirty_pages pause duration, dirty ratio)	Evolvable	Policy decision: the bandwidth-proportional throttling algorithm and dirty ratio thresholds are tunable via sysctl and ML. The throttling mechanism (task sleep, PerCpuCounter for dirty page counts) is Evolvable module code; the throttling policy is an independently hot-swappable vtable.
Dentry LRU eviction policy (which unused dentries to reclaim first)	Evolvable	Policy decision: LRU ordering and shrinker batch size are heuristics. The LRU linkage is Nucleus-owned data (it lives in tracked Dentry storage); the eviction policy is Evolvable.
Readdir-plus prefetch policy (ReaddirPrefetchPolicy per-filesystem)	Evolvable	Policy decision: whether to prefetch statx metadata during readdir is a per-filesystem heuristic. The prefetch buffers (TaskPrefetchBuf, PrefetchEntry) are preserved per-task kernel-domain data (their Core/Tier 0 placement is a tier statement, orthogonal to Nucleus/Evolvable); the prefetch code is Evolvable.
Doorbell coalescing policy (batch size, timeout thresholds)	Evolvable	Policy decision: the coalescing batch size and timeout are ML-tunable parameters. The CoalescedDoorbell bitmask is preserved data; the coalescing code is Evolvable.

Swap mechanics: When the VFS Evolvable is live-replaced (Section 13.18), the Nucleus-owned and Nucleus-preserved data (dcache, inode cache, mount table, ring set, dirty intent lists) are kept in place. The new Evolvable inherits them and resumes operation. Instances of the migration-tracked VFS types — Dentry ("Dentry Allocation — Nucleus Tracked Storage" in this section), Mount and MountNamespace (Section 14.6) — live in Nucleus tracked storage (Section 13.18), which is what makes both in-place preservation across an Evolvable swap and Shadow-and-Migrate layout evolution possible for those types. Inode and SuperBlock are preserved data too, but are NOT tracked — their Shadow-and-Migrate walk uses their existing global indexes rather than a tracked registry ("Layout migration of Inode and SuperBlock" below). An Evolvable swap replaces the entire VFS image — path resolution, dcache lookup, mount-tree algorithms, writeback, and the readahead engine all come from the new image; its safety-relevant code is validated before commit by the VFS invariant checkers (vfs_path_walk_safety, below). The policy vtables (readahead sizing, writeback scheduling, dirty throttling, LRU eviction) are additionally hot-swappable on their own, independent of a full image swap. This is why the Nucleus/Evolvable boundary is drawn at the data / code line: data survives the swap, code is replaced.

Layout migration of Inode and SuperBlock: unlike Dentry/Mount/ MountNamespace, Inode and SuperBlock are NOT tracked-allocated — they carry no per-type Nucleus instance registry. The tracked registry exists solely to enumerate live instances for the Shadow-and-Migrate walk (Section 13.18: "a registry of live instances per type"); Inode and SuperBlock already have enumerable global indexes, so the migration walk uses those and the hot inode-allocation path pays no per-instance registry cost:

• SuperBlock is enumerated through the global SUPER_BLOCK_MAP (all_superblocks(), above) — every mounted superblock is registered there at mount (step 4a) and removed at umount, so the walk sees exactly the live set.
• Inode is enumerated through each SuperBlock's inode_cache (XArray<u64, Arc<Inode>>, the same index iget/inode_cache_lookup use): the walk iterates all_superblocks(), then each superblock's inode_cache. This substitute is complete because of the live ⇒ enumerable invariant: every live Inode instance — regular filesystem inode, pipefs/sockfs synthetic inode, or InodeFlags::ANON_INODE anon-inode fd — is reachable via SUPER_BLOCK_MAP × inode_cache, guaranteed by the universal insertion requirement that every inode inserts into its superblock's inode_cache at creation (Section 14.1, Invariants: "Universal cache membership"). Layout liveness therefore follows cache membership, not a filesystem's discretion to hash: an inode is present in its inode_cache for the whole of its lifetime, which is exactly the window in which its layout could need migrating. No live inode — not even an open pipe, a socket, or an anon-inode fd — escapes the walk.

Concurrent allocation during the migration window. The Shadow-and-Migrate walk for these two types runs inside the VFS Evolvable swap window, after the component's Phase A' quiescence has queued all VFS operations (Section 13.18) and before operations resume in the new image. iget/iput, inode_cache_insert/removal, and mount/SUPER_BLOCK_MAP registration are all Evolvable VFS-component operations (the accessor rows above), so quiescence excludes them: no Inode or SuperBlock is created, cached, or freed concurrently with the walk. Operations that resume AFTER the swap run in the NEW image and therefore allocate the new layout — the same "new instances get the new layout" guarantee the tracked allocator provides, obtained here from component quiescence (because inode/superblock allocation is itself a quiesced VFS-Evolvable operation) rather than a per-alloc_tracked epoch. This is why an enumerable global index is a sufficient substitute for the tracked registry here but not for Dentry (the dcache is a hash table with no flat live-instance index, so Dentry must be tracked to be enumerable at all).

Quiesce/locking rules for the walk. Both indexes are walked under an RCU read section — SUPER_BLOCK_MAP via all_superblocks() (count/reserve/fill passes, cloning each Arc<SuperBlock> under RCU) and each inode_cache via its RCU XArray load — inside the quiesced window. Quiescence excludes the writers (mount/umount, inode_cache_insert and removal) for the walk's duration, and the migration writer is the sole layout mutator, so the walk observes a stable membership with no writer contention.

14.1.2.2 Nucleus Invariant Checker: Path-Walk Safety

The path-resolution algorithm is Evolvable code (row above); its safety is enforced not by freezing it but by a Nucleus invariant checker that validates every proposed new VFS image before the swap commits. vfs_path_walk_safety fits the generic checker framework (Section 13.18) — one CheckerDescriptor bundling the related walk-safety properties (INV-1..5 below), with one operation-time predicate — exactly as MergeInvariantChecker does for the elevator merge invariant. It is the checker half of the Nucleus checker discipline: (a) the descriptor data block registered below, and (b) the enforcement call-site, the shared symlink-follow step in → Evolvable code at Section 14.1. Phase A validation is automatic via evolution_check_all (no wiring).

Invariants verified (Phase A property tests):

• INV-1 Symlink follow budget / cycle termination — resolution follows at most 40 symlinks total, then ELOOP (the Linux ABI contract, Section 14.1). Boundary-exact: a 40-deep chain resolves; a 41-deep chain returns ELOOP; a symlink cycle (a→b→a) terminates in bounded lookups with ELOOP (the budget bounds the lookup count by construction — there is no "visited set").
• INV-2 Mount-crossing confinement — a mount-point node resolves into the mounted sub-tree root (covered content never visible); NO_XDEV crossing → EXDEV; BENEATH ..-escape above root → EXDEV; .. at root clamps to root; IN_ROOT treats the dirfd as root.
• INV-3 RCU-walk revalidation (TOCTOU) — an RCU-walk component step whose seqcount is bumped between lookup and validation must yield retry / ref-walk fallback, never a successful stale result.
• INV-4 Terminal-symlink & NO_SYMLINKS semantics — FOLLOW cleared → a terminal symlink is returned, not followed; NO_SYMLINKS → ELOOP on any symlink component.
• INV-5 Permission-cache soundness — the cached_perm hit rule (Section 14.1): a tag mismatch is a miss, a requested mode not in the granted set is a miss, and denials are never cached or served — evaluated as a pure function of the proposed image.

Descriptor and registration (the (a) data block):

// umka-vfs/src/walk_invariants.rs — Evolvable (VFS module)

static PATH_WALK_CHECKER_ID: BootOnceCell<CheckerId> = BootOnceCell::new();

/// Symlink follow budget — the `ELOOP` ABI contract (Linux `MAXSYMLINKS`
/// parity). Owned by the checker: `check_symlink_budget` (op_check) and the
/// Phase A boundary tests both read it, so the invariant has a single source
/// of truth. See the `symlink_follows` counter at
/// [Section 14.1](#virtual-filesystem-layer--path-resolution).
const MAX_SYMLINK_FOLLOWS: u32 = 40;

#[module_init]
fn vfs_register_walk_checker() {
    let descriptor = CheckerDescriptor {
        checker_id: CheckerId::PLACEHOLDER, // Filled in by Nucleus on register
        subsystem: SubsystemTag::Vfs,       // appended variant = 7
        name: "vfs_path_walk_safety",
        capture_fn: capture_walk_contract,
        verify_fn: verify_walk_safety,
        op_check_fn: Some(check_symlink_budget),
    };
    let id = register_invariant_checker(descriptor)
        .expect("path-walk checker registration");
    PATH_WALK_CHECKER_ID.set(id).expect("path-walk checker id");
}

Capture — snapshots the contract constants and generation state the invariants constrain, owning its data (no live borrows, per the CheckerState::new contract, Section 13.18):

/// Snapshot of the walk's ABI-contract parameters, captured from the OLD
/// (trusted) image so Phase A verifies the NEW image against the contract the
/// running kernel actually enforces. Owns its data — `'static`, no live borrows.
struct WalkContractSnapshot {
    max_symlink_follows: u32,  // 40
    path_max: u32,             // 4096
    name_max: u32,             // 255
    policy_gen: u64,           // LSM_REGISTRY.policy_generation at capture
}

unsafe fn capture_walk_contract() -> CheckerState {
    CheckerState::new(WalkContractSnapshot {
        max_symlink_follows: MAX_SYMLINK_FOLLOWS,
        path_max: PATH_MAX as u32,
        name_max: NAME_MAX as u32,
        policy_gen: LSM_REGISTRY.policy_generation.load(Ordering::Acquire),
    })
}

Verify — property tests INV-1..5 against the proposed image through the VFS harness surface (below), inside Phase A's panic catcher:

unsafe fn verify_walk_safety(
    captured: &CheckerState,
    new_image: &NewImageOps,
) -> Result<(), CheckerError> {
    let contract = captured.downcast_ref::<WalkContractSnapshot>()
        .ok_or(CheckerError::CaptureType)?;

    // INV-1: a cycle a->b->a terminates with ELOOP within bounded lookups; a
    //        40-deep chain resolves; a 41-deep chain -> ELOOP (boundary-exact
    //        against contract.max_symlink_follows).
    // INV-2: a mount-marker node resolves into the sub-fixture root; NO_XDEV ->
    //        EXDEV; a BENEATH ".."-escape -> EXDEV; ".." at root clamps; IN_ROOT
    //        treats the dirfd as root.
    // INV-4: FOLLOW cleared returns the terminal symlink node; NO_SYMLINKS ->
    //        ELOOP on any symlink component.
    //        (INV-1/2/4 driven through new_image.vfs_walk_fixture(..))
    // INV-3: new_image.vfs_rcu_step_seq_bumped(..) must return Retry, never
    //        Accepted (a stale result trusted under a bumped seqcount).
    // INV-5: new_image.vfs_cached_perm_decision(entry, tag, requested) — pure
    //        hit-rule probes: tag mismatch => Miss; requested not-subset-of
    //        granted => Miss; empty => Miss; and no probe may return a Deny
    //        sourced from the cache (there is no Deny verdict by construction).
    // Each failing probe returns Err(CheckerError::InvariantViolated("<reason>")).
    let _ = contract;
    Ok(())
}

Operation-time enforcement — the op_check predicate for INV-1 at the symlink-follow step ("the checker IS the invariant", same wiring as IoQueue::insert_merged, Section 13.18). The cost lands only on symlink follows (not per component): one predicted indirect call (~1-cycle dispatch, Section 13.18) against a step that already reads the target string — negative-overhead compliant.

/// `subject_ptr`: `*const u32` — the walk state's symlink-follow counter (the
///   number of symlinks followed so far in this lookup).
/// `_aux`: unused; callers pass `core::ptr::null()`.
/// Returns false when following one more symlink would exceed the budget; the
/// walk (RCU-walk and ref-walk share the follow step) maps false to `ELOOP`.
unsafe fn check_symlink_budget(subject_ptr: *const u8, _aux: *const u8) -> bool {
    let follows = *(subject_ptr as *const u32);
    follows < MAX_SYMLINK_FOLLOWS
}

Harness surface (NewImageOps, form 1 — hand-written impl block, per the two-form harness rule Section 13.18): the VFS domain authors its harness ops here, driving the proposed image's real walk core (parameterized over WalkProvider, Section 14.1) with a checker-owned synthetic tree (WalkFixture) that replaces the live dcache/mount-hash. The fixture is plain Evolvable-side heap data owned by the verify_fn call — NOT tracked storage, NOT the live dcache; Phase A must not pollute the Nucleus type registries.

/// Bounded synthetic dentry tree the walk-safety checker drives the proposed
/// image over. Node `FixtureNodeId(0)` is the fixture root. Plain
/// Evolvable-side heap data, owned by the verify call.
pub struct WalkFixture {
    pub nodes: ArrayVec<FixtureNode, MAX_FIXTURE_NODES>,
}

/// Index into `WalkFixture::nodes`; the `WalkProvider::Node` type the harness
/// provider binds ([Section 14.1](#virtual-filesystem-layer--path-resolution)).
#[derive(Copy, Clone, PartialEq, Eq)]
pub struct FixtureNodeId(pub u32);

/// One synthetic node: its kind plus its named child edges.
pub struct FixtureNode {
    pub kind: FixtureNodeKind,
    pub children: ArrayVec<FixtureEdge, MAX_FIXTURE_FANOUT>,
}

/// A named edge from a directory node to a child node.
pub struct FixtureEdge {
    pub name: ArrayVec<u8, FIXTURE_NAME_MAX>,
    pub child: FixtureNodeId,
}

/// Kind of a synthetic node. `Symlink` carries its target bytes; `MountPoint`
/// carries the sub-tree root the walk crosses into.
pub enum FixtureNodeKind {
    Dir,
    File,
    Symlink { target: ArrayVec<u8, FIXTURE_PATH_MAX> },
    MountPoint { sub_root: FixtureNodeId },
}

// Bounds on the synthetic fixture — small: property tests need only enough
// depth to exceed MAX_SYMLINK_FOLLOWS and enough breadth for the
// mount/`..`/BENEATH cases. Not a runtime limit — this is checker scaffolding.
const MAX_FIXTURE_NODES: usize = 128;
const MAX_FIXTURE_FANOUT: usize = 16;
const FIXTURE_NAME_MAX: usize = 255;   // POSIX NAME_MAX
const FIXTURE_PATH_MAX: usize = 4096;  // POSIX PATH_MAX

/// Harness `WalkProvider` ([Section 14.1](#virtual-filesystem-layer--path-resolution))
/// backing the walk core with the synthetic `WalkFixture` instead of the live
/// dcache/mount-hash. `with_seq_bump` drives INV-3 by signalling a seqcount
/// bump on the next lookup (as if a concurrent rename raced the RCU step).
pub struct FixtureWalkProvider<'f> {
    fixture: &'f WalkFixture,
    root: FixtureNodeId,
    bump_next: bool,
}

impl<'f> FixtureWalkProvider<'f> {
    pub fn new(fixture: &'f WalkFixture, root: FixtureNodeId) -> Self {
        Self { fixture, root, bump_next: false }
    }
    pub fn with_seq_bump(fixture: &'f WalkFixture, root: FixtureNodeId) -> Self {
        Self { fixture, root, bump_next: true }
    }
}

impl<'f> WalkProvider for FixtureWalkProvider<'f> {
    type Node = FixtureNodeId;

    fn root(&self) -> FixtureNodeId { self.root }

    fn lookup(&self, dir: FixtureNodeId, name: &[u8])
        -> Result<WalkStep<FixtureNodeId>, Errno>
    {
        let node = &self.fixture.nodes[dir.0 as usize];
        for edge in node.children.iter() {
            if edge.name.as_slice() == name {
                let child = &self.fixture.nodes[edge.child.0 as usize];
                let kind = match child.kind {
                    FixtureNodeKind::Dir               => WalkNodeKind::Dir,
                    FixtureNodeKind::File              => WalkNodeKind::File,
                    FixtureNodeKind::Symlink { .. }    => WalkNodeKind::Symlink,
                    FixtureNodeKind::MountPoint { .. } => WalkNodeKind::MountPoint,
                };
                return Ok(WalkStep { node: edge.child, kind, seq_valid: !self.bump_next });
            }
        }
        Err(Errno::ENOENT)
    }

    fn read_link(&self, node: FixtureNodeId, out: &mut [u8]) -> Result<usize, Errno> {
        match &self.fixture.nodes[node.0 as usize].kind {
            FixtureNodeKind::Symlink { target } => {
                let n = core::cmp::min(target.len(), out.len());
                out[..n].copy_from_slice(&target[..n]);
                Ok(n)
            }
            _ => Err(Errno::EINVAL),
        }
    }
}

/// VFS domain harness surface — the block domain's `try_merge` /
/// `drain_barrier_queue` analogue ([Section 15.2](15-storage.md#block-io-and-volume-management)). Each
/// method dispatches through the proposed-image proxy to the NEW image's
/// walk core; all are pure (allocate nothing global, mutate nothing outside
/// the fixture) so Phase A stays side-effect free.
impl NewImageOps {
    /// Drive the proposed image's full component-walk loop over a checker-owned
    /// synthetic tree. The fixture provider replaces dcache/mount-hash lookups;
    /// component parsing, the symlink stack and follow budget, `..`/root
    /// clamping, `BENEATH`/`NO_XDEV`/`IN_ROOT` policing, and terminal-`FOLLOW`
    /// handling are the proposed image's real walk code (INV-1/2/4).
    pub fn vfs_walk_fixture(&self, fixture: &WalkFixture, start: FixtureNodeId,
                            path: &[u8], flags: LookupFlags)
        -> Result<FixtureNodeId, Errno>
    {
        let provider = FixtureWalkProvider::new(fixture, start);
        let mut walk = WalkState::new();
        walk_components(&provider, &mut walk, start, path, flags)
    }

    /// Drive ONE RCU-walk component step with a seqcount bump injected by the
    /// fixture between lookup and validation. A correct image refuses the stale
    /// result and signals retry (`Err(EAGAIN)`); an image that returns a node
    /// anyway has trusted a stale result (INV-3 violation).
    pub fn vfs_rcu_step_seq_bumped(&self, fixture: &WalkFixture,
                                   at: FixtureNodeId, name: &[u8])
        -> RcuStepVerdict
    {
        let provider = FixtureWalkProvider::with_seq_bump(fixture, at);
        match rcu_walk_step(&provider, at, name) {
            Err(Errno::EAGAIN) => RcuStepVerdict::Retry,    // correct: seq bump -> retry/ref-walk
            Ok(_)              => RcuStepVerdict::Accepted,  // INV-3 violation: stale result trusted
            Err(_)             => RcuStepVerdict::Retry,     // any other refusal is also safe
        }
    }

    /// Evaluate the proposed image's `cached_perm` hit rule as a pure function
    /// (encoding per [Section 14.1](#virtual-filesystem-layer--path-resolution)). Returns a
    /// `Hit(granted_rwx)` only on a live entry whose tag matches and whose
    /// granted set covers the request; everything else is `Miss`. There is no
    /// `Deny` verdict — denials are always authoritative (INV-5).
    pub fn vfs_cached_perm_decision(&self, entry: u64, tag52: u64, requested: u8)
        -> CachedPermDecision
    {
        if entry == 0 {
            return CachedPermDecision::Miss;         // empty / invalidated
        }
        if entry >> 12 != tag52 {
            return CachedPermDecision::Miss;         // tag mismatch (subject changed)
        }
        let granted = (entry & 0b111) as u8;         // bits [2:0]: validated rwx
        if requested & !granted != 0 {
            return CachedPermDecision::Miss;         // requested not-subset-of granted
        }
        CachedPermDecision::Hit(granted)             // never a cached Deny
    }
}

/// Outcome of `vfs_rcu_step_seq_bumped`: a correct image returns `Retry`;
/// `Accepted` (trusting a stale result under a bumped seqcount) is an INV-3
/// violation the checker rejects.
pub enum RcuStepVerdict { Retry, Accepted }

/// Outcome of `vfs_cached_perm_decision`: `Hit(granted_rwx)` or `Miss`. The
/// cache never yields a denial (INV-5: denials are always authoritative), so
/// there is deliberately no `Deny` variant.
pub enum CachedPermDecision { Hit(u8), Miss }

Filesystem drivers register as VFS backends. The VFS never interprets on-disk format directly — it delegates all storage operations through three trait interfaces:

Foundational VFS types (used throughout this chapter):

/// Opaque filesystem inode identifier. Unique within a single SuperBlock.
///
/// Inode 0 is never valid (used as the null sentinel in `AtomicOption`).
/// Inode 1 is conventionally the root directory inode.
/// The u64 width accommodates all known filesystem inode spaces (ext4 uses
/// u32 internally but promotes to u64 for future-proofing; Btrfs and ZFS
/// use u64 natively).
///
/// `InodeId` is filesystem-private: the same u64 value in two different
/// `SuperBlock` instances refers to different inodes.
#[derive(Copy, Clone, Debug, PartialEq, Eq, Hash)]
#[repr(transparent)]
pub struct InodeId(pub u64);

impl From<u64> for InodeId { fn from(v: u64) -> Self { InodeId(v) } }
impl From<InodeId> for u64  { fn from(id: InodeId) -> u64 { id.0 } }

/// Opaque VFS pipe identifier. Each `pipe(2)` / `pipe2(2)` call produces a
/// unique `PipeId` for internal tracking (waitqueue association, splice
/// routing, and PipeBuffer lifetime management). Not visible to userspace.
#[derive(Copy, Clone, Debug, PartialEq, Eq, Hash)]
pub struct PipeId(pub u64);

/// Memory protection flags for `FileOps::mmap()`.
///
/// Bitfield matching Linux `PROT_*` constants from `<sys/mman.h>`.
/// Passed by the VMM to the filesystem's mmap callback so it can validate
/// or adjust protections (e.g., deny PROT_WRITE for read-only mounts,
/// deny PROT_EXEC for noexec mounts).
///
/// These are the userspace-facing PROT_* values, NOT the kernel-internal
/// VM_* flags. The VMM converts between MmapProt and VmFlags via
/// `prot_flags_to_vm_flags()` ([Section 4.8](04-memory.md#virtual-memory-manager)).
pub struct MmapProt(u32);

impl MmapProt {
    pub const NONE:  MmapProt = MmapProt(0x0);
    pub const READ:  MmapProt = MmapProt(0x1); // PROT_READ
    pub const WRITE: MmapProt = MmapProt(0x2); // PROT_WRITE
    pub const EXEC:  MmapProt = MmapProt(0x4); // PROT_EXEC

    pub fn contains(&self, flag: MmapProt) -> bool {
        self.0 & flag.0 == flag.0
    }
}

/// Result type returned by `FileOps::mmap()`.
///
/// On success, the filesystem returns `MmapResult` describing any
/// adjustments it made to the mapping. The VMM applies these adjustments
/// to the VMA after the callback returns.
///
/// For the Tier 0 in-kernel direct-call path (where `f_ops.mmap(f, &mut vma)`
/// modifies the VMA directly), `MmapResult::Ok` is returned after the VMA
/// has been modified in place. For the KABI ring transport (Tier 1/2), the
/// decomposed return struct carries the adjusted fields back to the VMM.
pub struct MmapResult {
    /// Adjusted vm_flags (the filesystem may set VM_IO, clear VM_MAYWRITE, etc.).
    /// If the filesystem did not modify flags, this equals the input vm_flags
    /// (the input arrives as the `vm_flags` argument of the decomposed
    /// `mmap()`, so the callback is round-trip symmetric: receive vm_flags,
    /// return the adjusted vm_flags).
    pub vm_flags: u64,
    /// Filesystem-specific VmOperations handle (opaque u64 for KABI transport).
    /// The VMM sets `vma.vm_ops` from this value. Zero means no custom vm_ops.
    pub vm_ops_handle: u64,
}

impl MmapResult {
    /// Apply the filesystem's adjustments to a same-domain `Vma`. Used by
    /// the default `FileOps::mmap_direct()` implementation and by the
    /// caller-side ring stub after a cross-domain `mmap()` completes.
    pub fn apply_to_vma(&self, vma: &mut Vma) {
        vma.vm_flags = VmFlags::from_bits_truncate(self.vm_flags);
        if let Some(ops) = vm_ops_registry_lookup(self.vm_ops_handle) {
            vma.vm_ops = Some(ops);
        }
    }
}

/// Resolve an opaque `vm_ops_handle` (from `MmapResult`) to the vtable it
/// names. Handles are registered by filesystems/drivers at module load
/// (cold path, XArray keyed by handle); `0` and unknown handles resolve
/// to `None` (no custom vm_ops installed).
pub fn vm_ops_registry_lookup(handle: u64) -> Option<&'static dyn VmOperations>;

/// Canonical I/O error type for VFS, filesystem, block, and network I/O
/// paths. **This is the single spec-wide definition** — every
/// `Result<_, IoError>` in Ch 13–16 (device classes, VFS, storage,
/// networking) and in the sysapi I/O traits refers to this type. Do not
/// define new local `IoError` types. (One pre-existing name collision:
/// the probe-level hardware error enum in
/// [Section 12.7](12-kabi.md#kabi-service-dependency-resolution) is a distinct, module-local
/// type used only inside `ProbeError::Io` — it is NOT this type.)
///
/// **Design: newtype over `Errno`, not a semantic enum.** An audit of all
/// ~90 construction/consumption sites across the spec found that callers
/// only ever (a) construct from a known errno and (b) propagate with `?`
/// or extract the errno at a boundary (`Errno::from_io` at the syscall
/// boundary, `e.errno()` in writeback). No call site matches on semantic
/// variants, so a variant enum (Device/Fs/Timeout/…) would add a
/// classification burden at every construction site and a lossy
/// variant→errno mapping at the boundary — with no consumer. The newtype
/// keeps construction O(0), conversion a field read, and the type `Copy`,
/// 4 bytes, register-passed.
///
/// **Why a distinct type at all** (instead of using `Errno` directly):
/// the I/O traits (`FileOps`, `AddressSpaceOps`, `NetDeviceOps`, …) are
/// KABI-dispatched and their error domain is deliberately narrower than
/// the full syscall errno space — `IoError` marks values that originated
/// below the VFS dispatch layer, and the type distinction forces the
/// exactly-once conversion at the syscall boundary
/// (`Errno::from_io`, [Section 19.1](19-sysapi.md#syscall-interface)) instead of ad-hoc integer
/// casts scattered through handlers.
///
/// **KABI/wire representation**: `IoError` itself never crosses a
/// compilation or domain boundary as a struct — it is kernel-internal,
/// so no `#[repr(C)]` is required. At ring-transport boundaries the
/// error is encoded as a **negated Linux errno** in the response status
/// (`VfsResponseWire.status < 0`, see `VfsResponse` below; same encoding
/// as `Bio.status: AtomicI32`, [Section 15.2](15-storage.md#block-io-and-volume-management)).
/// `to_neg_errno()` / `from_neg_errno()` are the canonical converters
/// for that encoding.
///
/// **Naming convention for shorthands**: associated consts are named
/// exactly after the Linux errno macro (`IoError::EIO`, never
/// `IoError::Io` or `IoError::TimedOut`). `IoError::EIO` ≡
/// `IoError::new(Errno::EIO)`. Values are Linux-exact, verified against
/// `torvalds/linux` master `include/uapi/asm-generic/errno-base.h` and
/// `errno.h`. Note: `ENOTSUP` is a userspace alias for `EOPNOTSUPP`
/// (95) — kernel code uses `EOPNOTSUPP` only.
#[derive(Copy, Clone, PartialEq, Eq, Debug)]
#[repr(transparent)] // zero-cost over Errno; kernel-internal, NOT a KABI layout guarantee
pub struct IoError(Errno);

impl IoError {
    /// Wrap a known errno. The canonical constructor; `const` so it can
    /// initialize statics and the associated consts below.
    pub const fn new(errno: Errno) -> IoError { IoError(errno) }

    /// Construct from a raw **positive** Linux errno (e.g., `5` for EIO).
    /// Valid range is `1..=4095` (`MAX_ERRNO`, Linux `include/linux/err.h`).
    /// Out-of-range values map to `EIO` (defensive: an I/O error path must
    /// never panic); debug builds assert.
    pub fn from_raw(raw: i32) -> IoError;

    /// Construct from the **negated-errno** encoding used by ring response
    /// status words and `Bio.status` (e.g., `-5` for EIO). `status` must be
    /// in `-4095..=-1`; equivalent to `from_raw(-status)`.
    pub fn from_neg_errno(status: i32) -> IoError;

    /// The wrapped errno. Used by `Errno::from_io` at the syscall boundary
    /// and by writeback error folding (`mapping_set_error`).
    pub const fn errno(self) -> Errno { self.0 }

    /// Raw positive errno value (e.g., `5`).
    pub const fn to_raw(self) -> i32 { self.0 as i32 }

    /// Negated-errno encoding for ring status words and bio completion
    /// (e.g., `-5`). Inverse of `from_neg_errno`.
    pub const fn to_neg_errno(self) -> i32 { -(self.0 as i32) }

    // Associated consts for every errno used as an I/O error in this spec.
    // One const per Linux errno macro, same name, Linux-exact value
    // (asm-generic values apply on all 8 supported architectures for this
    // set). Extend this list as needed — always E-prefixed, never semantic.
    pub const EIO:          IoError = IoError::new(Errno::EIO);          // 5
    pub const EBADF:        IoError = IoError::new(Errno::EBADF);        // 9
    pub const EAGAIN:       IoError = IoError::new(Errno::EAGAIN);       // 11
    pub const ENOMEM:       IoError = IoError::new(Errno::ENOMEM);       // 12
    pub const EACCES:       IoError = IoError::new(Errno::EACCES);       // 13
    pub const EBUSY:        IoError = IoError::new(Errno::EBUSY);        // 16
    pub const ENODEV:       IoError = IoError::new(Errno::ENODEV);       // 19
    pub const EINVAL:       IoError = IoError::new(Errno::EINVAL);       // 22
    pub const EFBIG:        IoError = IoError::new(Errno::EFBIG);        // 27
    pub const ENOSPC:       IoError = IoError::new(Errno::ENOSPC);       // 28
    pub const ENOSYS:       IoError = IoError::new(Errno::ENOSYS);       // 38
    pub const EMSGSIZE:     IoError = IoError::new(Errno::EMSGSIZE);     // 90
    pub const EOPNOTSUPP:   IoError = IoError::new(Errno::EOPNOTSUPP);   // 95
    pub const ENOBUFS:      IoError = IoError::new(Errno::ENOBUFS);      // 105
    pub const ETIMEDOUT:    IoError = IoError::new(Errno::ETIMEDOUT);    // 110
    pub const EHOSTUNREACH: IoError = IoError::new(Errno::EHOSTUNREACH); // 113
}

/// Response envelope for cross-domain VFS ring buffer calls.
///
/// This is the kernel-internal typed representation. The wire-level
/// representation on the ring buffer is `VfsResponseWire`
/// ([Section 14.2](#vfs-ring-buffer-protocol)), which uses a single `i64 status`
/// field for compact encoding. The VFS dispatch layer converts between
/// the two: `status >= 0` → `Ok(status)`, `status < 0 && status !=
/// i64::MIN` → `Err(status as i32)`, `status == i64::MIN` → `Pending`.
/// The `Err` payload is `IoError::to_neg_errno()` (see `IoError` above).
#[derive(Debug)]
pub enum VfsResponse {
    /// Success, possibly with a return value (e.g., byte count for read/write).
    Ok(i64),
    /// Error code (negated Linux errno, e.g., `-ENOENT`).
    Err(i32),
    /// Asynchronous completion pending; caller must wait on the completion ring.
    Pending,
}

/// Filesystem-level operations (mount, unmount, statfs).
/// Implemented once per filesystem type (ext4, XFS, btrfs, ZFS, tmpfs, etc.).
pub trait FileSystemOps: Send + Sync {
    /// Mount a filesystem from the given source device with flags and options.
    fn mount(&self, source: &str, flags: MountFlags, data: &[u8]) -> Result<SuperBlock>;

    /// Unmount a previously mounted filesystem.
    fn unmount(&self, sb: &SuperBlock) -> Result<()>;

    /// Force-unmount: abort in-flight I/O with EIO. Called when umount2()
    /// is invoked with MNT_FORCE. Not all filesystems support this — return
    /// ENOSYS if unsupported. NFS uses this for stale server recovery.
    fn force_umount(&self, sb: &SuperBlock) -> Result<()>;

    /// Return filesystem statistics (total/free/available blocks and inodes).
    fn statfs(&self, sb: &SuperBlock) -> Result<StatFs>;

    /// Flush all dirty data and metadata for this filesystem to stable storage.
    /// Backend for syncfs(2) and the filesystem-level portion of sync(2).
    fn sync_fs(&self, sb: &SuperBlock, wait: bool) -> Result<()>;

    /// Remount with changed flags/options (e.g., `mount -o remount,ro`).
    fn remount(&self, sb: &SuperBlock, flags: MountFlags, data: &[u8]) -> Result<()>;

    /// Freeze the filesystem for a consistent snapshot. All pending writes are
    /// flushed and new writes block until thaw. Used by LVM snapshots, device-mapper,
    /// and backup tools via FIFREEZE ioctl.
    fn freeze(&self, sb: &SuperBlock) -> Result<()>;

    /// Thaw a previously frozen filesystem, allowing writes to resume.
    fn thaw(&self, sb: &SuperBlock) -> Result<()>;

    /// Format filesystem-specific mount options for /proc/mounts output.
    fn show_options(&self, sb: &SuperBlock, buf: &mut [u8]) -> Result<usize>;

    /// Declare the filesystem's write mode. Called once at mount time and cached
    /// by the VFS in `SuperBlock.write_mode`. Informs writeback scheduling, page
    /// cache sharing, and free space accounting.
    /// See [Section 14.4](#vfs-fsync-and-cow--copy-on-write-and-redirect-on-write-infrastructure)
    /// for the `WriteMode` enum and design rationale.
    /// Default: `WriteMode::InPlace` (traditional overwrite semantics).
    fn write_mode(&self) -> WriteMode {
        WriteMode::InPlace
    }

    /// Declare this filesystem's readdir prefetch policy. Called once at mount
    /// time and cached by the VFS with the superblock; consulted by the
    /// readdir-plus statx prefetch path
    /// ([Section 14.1](#virtual-filesystem-layer--prefetch-policy-framework)).
    /// The default is `Always`, correct for all single-node local filesystems
    /// (ext4, XFS, btrfs, tmpfs, procfs, sysfs). Network and cluster
    /// filesystems override it to return `CacheAware` (NFS, CIFS, FUSE,
    /// peerfs) or `Never` (GFS2, OCFS2, DLM-based cluster FS). See the
    /// per-filesystem policy table under the prefetch framework for the full
    /// mapping.
    fn readdir_prefetch_policy(&self) -> ReaddirPrefetchPolicy {
        ReaddirPrefetchPolicy::Always
    }
}

/// `renameat2(2)` flags. Bit values match Linux `include/uapi/linux/fs.h`
/// exactly (the userspace ABI is passed through unchanged). `RENAME_EXCHANGE`
/// is mutually exclusive with both `RENAME_NOREPLACE` and `RENAME_WHITEOUT`
/// (either combination is rejected with `EINVAL`, matching Linux
/// `do_renameat2()` in `fs/namei.c`).
bitflags! {
    pub struct RenameFlags: u32 {
        /// Fail with `EEXIST` if the destination already exists.
        const RENAME_NOREPLACE = 0x0000_0001;
        /// Atomically exchange source and destination (both must exist).
        const RENAME_EXCHANGE  = 0x0000_0002;
        /// Move source to destination and atomically leave a whiteout at the
        /// source's old name (overlayfs copy-up). Incompatible with
        /// `RENAME_EXCHANGE`.
        const RENAME_WHITEOUT  = 0x0000_0004;
    }
}

/// `fallocate(2)` mode flags. Bit values match Linux
/// `include/uapi/linux/falloc.h` exactly. `mode == 0` (no bits) is the
/// default allocate-and-extend behaviour.
bitflags! {
    pub struct FallocateMode: u32 {
        /// Do not extend the file size past the allocated range.
        const FALLOC_FL_KEEP_SIZE      = 0x01;
        /// Deallocate the range (create a hole); requires `KEEP_SIZE`.
        const FALLOC_FL_PUNCH_HOLE     = 0x02;
        /// Expose stale extents without zeroing (privileged).
        const FALLOC_FL_NO_HIDE_STALE  = 0x04;
        /// Collapse (remove) the range, shifting later data down.
        const FALLOC_FL_COLLAPSE_RANGE = 0x08;
        /// Convert the range to zeros, preallocating as needed.
        const FALLOC_FL_ZERO_RANGE     = 0x10;
        /// Insert a hole, shifting later data up.
        const FALLOC_FL_INSERT_RANGE   = 0x20;
        /// Unshare shared (reflinked/CoW) extents in the range.
        const FALLOC_FL_UNSHARE_RANGE  = 0x40;
        /// Write zeroes to the range, marking extents written.
        const FALLOC_FL_WRITE_ZEROES   = 0x80;
    }
}

/// Directory-entry file type. Discriminants match Linux `DT_*`
/// (`getdents64(2)` `d_type`) exactly; carried by value in the `readdir`
/// callback and serialized into the readdir response. Explicit `#[repr(u8)]`
/// keeps the wire discriminant stable across compilers.
#[repr(u8)]
pub enum FileType {
    Unknown = 0,
    Fifo = 1,
    Chr = 2,
    Dir = 4,
    Blk = 6,
    Reg = 8,
    Lnk = 10,
    Sock = 12,
}

/// `llseek(2)`/`lseek(2)` reference point. Discriminants match Linux
/// `SEEK_*` (`include/uapi/linux/fs.h`) exactly. `Data`/`Hole` are the
/// sparse-file extensions dispatched to `FileOps::llseek`. Explicit
/// `#[repr(u32)]` keeps the ring discriminant stable.
#[repr(u32)]
pub enum SeekWhence {
    /// Absolute offset from the start of the file.
    Set = 0,
    /// Relative to the current file position.
    Cur = 1,
    /// Relative to the end of the file.
    End = 2,
    /// Next data-bearing offset at or after `offset` (`SEEK_DATA`).
    Data = 3,
    /// Next hole at or after `offset` (`SEEK_HOLE`).
    Hole = 4,
}

/// Inode (directory structure) operations.
/// Handles namespace operations: lookup, create, link, unlink, rename.
///
/// Note: `OsStr` is a kernel-defined type (NOT `std::ffi::OsStr`, which is
/// unavailable in `no_std`). It is a dynamically-sized type (DST) wrapping
/// `[u8]`, representing filenames that may contain arbitrary non-UTF-8 bytes
/// (Linux filenames are byte strings, not Unicode). Defined in
/// `umka-vfs/src/types.rs`:
///   `pub struct OsStr([u8]);`
/// As a DST, `OsStr` cannot be used by value — it is always behind a
/// reference (`&OsStr`) or `Box<OsStr>`. `&OsStr` is a fat pointer
/// (pointer + length), analogous to `&[u8]` but carrying the semantic
/// intent of "filesystem name component." Conversion from `&str` is
/// infallible (UTF-8 is a valid byte sequence); conversion TO `&str`
/// returns `Result` (may fail on non-UTF-8 filenames).
pub trait InodeOps: Send + Sync {
    /// Look up a child entry by name within a parent directory.
    fn lookup(&self, parent: InodeId, name: &OsStr) -> Result<InodeId>;

    /// Create a regular file in the given directory.
    fn create(&self, parent: InodeId, name: &OsStr, mode: FileMode) -> Result<InodeId>;

    /// Create a subdirectory.
    fn mkdir(&self, parent: InodeId, name: &OsStr, mode: FileMode) -> Result<InodeId>;

    /// Create a hard link: new entry `new_name` in `new_parent` pointing to `inode`.
    fn link(&self, inode: InodeId, new_parent: InodeId, new_name: &OsStr) -> Result<()>;

    /// Create a symbolic link containing `target` at `parent/name`.
    fn symlink(&self, parent: InodeId, name: &OsStr, target: &OsStr) -> Result<InodeId>;

    /// Read the target of a symbolic link.
    fn readlink(&self, inode: InodeId, buf: &mut [u8]) -> Result<usize>;

    /// Create a device special file (block/char device, FIFO, or socket).
    fn mknod(&self, parent: InodeId, name: &OsStr, mode: FileMode, dev: DevId) -> Result<InodeId>;

    /// Remove a directory entry (unlink for files, rmdir for empty directories).
    fn unlink(&self, parent: InodeId, name: &OsStr) -> Result<()>;

    /// Remove an empty directory. Separate from unlink for POSIX semantics:
    /// `unlink()` on a directory returns EISDIR; `rmdir()` on a file returns ENOTDIR.
    fn rmdir(&self, parent: InodeId, name: &OsStr) -> Result<()>;

    /// Rename/move a directory entry, possibly across directories.
    /// `flags` supports RENAME_NOREPLACE, RENAME_EXCHANGE, and RENAME_WHITEOUT
    /// (Linux renameat2 semantics, required for overlayfs).
    fn rename(
        &self,
        old_parent: InodeId, old_name: &OsStr,
        new_parent: InodeId, new_name: &OsStr,
        flags: RenameFlags,
    ) -> Result<()>;

    /// Get inode attributes (size, mode, timestamps, link count).
    fn getattr(&self, inode: InodeId) -> Result<InodeAttr>;

    /// Set inode attributes (chmod, chown, utimes).
    fn setattr(&self, inode: InodeId, attr: &SetAttr) -> Result<()>;

    /// Truncate a byte range within a file, deallocating the corresponding
    /// on-disk blocks (extent tree updates, journal entries, COW handling).
    /// Used by hole-punch (`FALLOC_FL_PUNCH_HOLE`) and range-discard
    /// operations. The VFS calls this after evicting the affected pages
    /// from the page cache; the filesystem is responsible only for the
    /// on-disk state. `start` and `end` are byte offsets (inclusive start,
    /// exclusive end; `end == u64::MAX` means "to end of file").
    fn truncate_range(&self, inode: InodeId, start: u64, end: u64) -> Result<(), IoError>;

    /// List extended attributes on an inode.
    fn listxattr(&self, inode: InodeId, buf: &mut [u8]) -> Result<usize>;

    /// Get an extended attribute value.
    fn getxattr(&self, inode: InodeId, name: &OsStr, buf: &mut [u8]) -> Result<usize>;

    /// Set an extended attribute value.
    fn setxattr(&self, inode: InodeId, name: &OsStr, value: &[u8], flags: XattrFlags)
        -> Result<()>;

    /// Remove an extended attribute.
    fn removexattr(&self, inode: InodeId, name: &OsStr) -> Result<()>;

    /// Read the inode's persistent attribute flags (`FS_*_FL` space) —
    /// the `FS_IOC_GETFLAGS`/`lsattr(1)` backend, called via
    /// `vfs_fileattr_get()` ([Section 14.1](#virtual-filesystem-layer--inode-attribute-flags)).
    /// Default: `ENOTTY` — the Linux-parity outcome for a filesystem
    /// without persistent flags (missing `->fileattr_get` yields
    /// `ENOIOCTLCMD` → `ENOTTY` in Linux `fs/file_attr.c`).
    fn fileattr_get(&self, inode: InodeId) -> Result<FileAttr> {
        let _ = inode;
        Err(Errno::ENOTTY)
    }

    /// Persist a new attribute-flag word (`FS_IOC_SETFLAGS`/`chattr(1)`
    /// backend). ALL permission checks (owner-or-CAP_FOWNER,
    /// CAP_LINUX_IMMUTABLE gate, EROFS, freeze) already ran Core-side in
    /// `vfs_fileattr_set()`; the driver validates only filesystem
    /// support — flag bits it cannot persist are rejected with
    /// `EOPNOTSUPP`, leaving on-disk state unchanged (per-filesystem
    /// modifiable masks are wider than the generic
    /// `FS_FL_USER_MODIFIABLE`; Linux parity). Nonzero `_reserved` words
    /// are rejected with `EINVAL`. When `Inode::is_dirsync()` semantics
    /// apply to the change, the driver commits it synchronously.
    /// Default: `ENOTTY` (same rationale as `fileattr_get`). A driver that
    /// overrides this MUST also return `true` from `supports_fileattr()`.
    fn fileattr_set(&self, inode: InodeId, attr: &FileAttr) -> Result<()> {
        let _ = (inode, attr);
        Err(Errno::ENOTTY)
    }

    /// Whether this filesystem exposes the persistent `FS_*_FL` attribute
    /// interface (`chattr`/`lsattr`, `FS_IOC_GET`/`SETFLAGS`). A driver that
    /// overrides `fileattr_get`/`fileattr_set` MUST also return `true` here;
    /// the default `false` matches a filesystem with no persistent flags.
    /// `vfs_fileattr_set()` probes this BEFORE the owner check, so
    /// `FS_IOC_SETFLAGS` on an unsupported filesystem returns `ENOTTY` even
    /// to a non-owner — Linux `fs/file_attr.c vfs_fileattr_set()` gates on
    /// the presence of the SETTER (`!inode->i_op->fileattr_set` ⇒
    /// `ENOIOCTLCMD` ⇒ `ENOTTY`) ahead of `inode_owner_or_capable()`.
    fn supports_fileattr(&self) -> bool {
        false
    }

    /// Flush inode metadata to stable storage. Called by
    /// `vfs_fsync_metadata()` for O_SYNC/O_DSYNC writes when the inode's
    /// on-disk metadata must be updated (timestamps, size, block map).
    ///
    /// `sync_mode`: `WriteSyncMode::Sync` (wait for I/O completion;
    /// Linux `WB_SYNC_ALL`) or `WriteSyncMode::Async` (schedule I/O but do not
    /// wait — Linux `WB_SYNC_NONE`). O_SYNC always uses `Sync`.
    fn write_inode(&self, ino: InodeId, sync_mode: WriteSyncMode) -> Result<()>;
}

/// Synchronization mode for `InodeOps::write_inode()` inode-metadata flushes.
///
/// Two-state by design: an inode metadata flush either waits for the write
/// to reach stable media or it does not. This is deliberately DISTINCT from
/// `WritebackSyncMode` ([Section 4.6](04-memory.md#writeback-subsystem)), which governs page-data
/// writeback requests and carries a third state (`Normal`) for bandwidth-
/// tracked periodic writeback — a concept that does not apply to single-inode
/// metadata flushes.
///
/// Linux equivalents: `Sync` = `WB_SYNC_ALL`, `Async` = `WB_SYNC_NONE`
/// (Linux `include/linux/writeback.h`, `enum writeback_sync_modes`, as
/// Linux consumes these modes in `write_inode_now()` / `->write_inode`).
#[derive(Clone, Copy, PartialEq, Eq, Debug)]
pub enum WriteSyncMode {
    /// Schedule the metadata write but do not wait for completion.
    /// Used by background inode writeback (Linux `WB_SYNC_NONE`).
    Async,
    /// Block until the inode metadata is on stable media, including any
    /// journal commit and device cache flush the filesystem requires
    /// (Linux `WB_SYNC_ALL`). Used by `fsync()`, O_SYNC/O_DSYNC, `sync(2)`.
    Sync,
}

/// Validated userspace pointer wrapper for writing data to userspace.
///
/// `UserSliceMut` represents a region of userspace memory that the kernel has
/// validated for write access. It ensures that:
/// 1. The pointer range `[ptr, ptr + len)` lies entirely within the task's
///    user address space (below `TASK_SIZE`, not in kernel address space).
/// 2. The pages are mapped writable (or will be demand-faulted on copy).
///
/// **Construction**: Created by `UserSliceMut::new(ptr, len)` which performs
/// the address range validation. This is called early in the syscall path
/// (before any I/O) so that an invalid buffer is rejected with `EFAULT`
/// before work is done.
///
/// **Copy path**: `copy_to_user(dst: &mut UserSliceMut, src: &[u8])` copies
/// kernel data into the validated userspace region. The copy handles:
/// - Page faults: if a destination page is not resident, the fault handler
///   allocates and maps it (demand paging), then retries the copy.
/// - Partial copies: if a fault cannot be resolved (e.g., SIGBUS on a
///   mapped-but-uncommittable page), the copy returns the number of bytes
///   successfully copied. The caller (VFS read dispatch) returns a short
///   read to userspace.
/// - SMAP/PAN enforcement: on architectures with Supervisor Mode Access
///   Prevention (x86 SMAP, ARM PAN), the copy temporarily enables user
///   access via `stac`/`clac` (x86) or `uaccess_enable`/`uaccess_disable`
///   (ARM). The access window is scoped to the copy operation.
///
/// **Advance semantics**: After each `copy_to_user()` call, the internal
/// pointer advances by the number of bytes written and `remaining()` decreases
/// accordingly. This allows iterative filling (e.g., page-by-page copy from
/// the page cache in `page_cache_read_iter()`).
///
/// **Thread safety**: `UserSliceMut` is `!Send` and `!Sync` — it is valid
/// only for the current task's address space on the current CPU. It must not
/// be stored beyond the syscall lifetime.
pub struct UserSliceMut {
    /// Validated userspace destination pointer. Guaranteed to be below
    /// `TASK_SIZE` at construction time.
    ptr: *mut u8,
    /// Remaining bytes available for writing.
    len: usize,
}

impl UserSliceMut {
    /// Create a validated userspace write buffer.
    ///
    /// Returns `EFAULT` if `ptr + len` overflows or exceeds `TASK_SIZE`.
    pub fn new(ptr: *mut u8, len: usize) -> Result<Self, Errno>;

    /// Number of bytes remaining in the buffer.
    pub fn remaining(&self) -> usize;

    /// Copy `src` into the userspace buffer, advancing the internal pointer.
    /// Returns the number of bytes actually copied (may be less than
    /// `src.len()` if a page fault cannot be resolved).
    pub fn write(&mut self, src: &[u8]) -> Result<usize, Errno>;
}

/// Validated userspace pointer wrapper for reading data from userspace.
///
/// Analogous to `UserSliceMut` but for kernel reads from user memory.
/// `copy_from_user(dst: &mut [u8], src: &UserSlice)` copies userspace data
/// into a kernel buffer with the same fault-handling and SMAP/PAN semantics
/// as `UserSliceMut`.
pub struct UserSlice {
    /// Validated userspace source pointer. Guaranteed to be below
    /// `TASK_SIZE` at construction time.
    ptr: *const u8,
    /// Remaining bytes available for reading.
    len: usize,
}

impl UserSlice {
    /// Create a validated userspace read buffer.
    ///
    /// Returns `EFAULT` if `ptr + len` overflows or exceeds `TASK_SIZE`.
    pub fn new(ptr: *const u8, len: usize) -> Result<Self, Errno>;

    /// Number of bytes remaining in the buffer.
    pub fn remaining(&self) -> usize;

    /// Copy data from the userspace buffer into `dst`, advancing the
    /// internal pointer. Returns the number of bytes actually copied.
    pub fn read(&mut self, dst: &mut [u8]) -> Result<usize, Errno>;

    /// Random-access copy from this userspace slice starting at internal
    /// offset `src_off`, writing `len` bytes to kernel address `dst`.
    /// Returns the number of bytes successfully copied (may be less than
    /// `len` on page fault or if the request would over-read the slice).
    ///
    /// Unlike `read()`, this method does NOT advance the internal position
    /// counter — it is intended for page-by-page write loops that maintain
    /// their own `written` byte counter (e.g., `page_cache_write_iter`).
    ///
    /// # Safety
    /// `dst` must point to a kernel-mapped page for at least `len` bytes.
    /// The caller must hold the page lock for the duration of the copy.
    pub fn read_at(&self, src_off: usize, dst: *mut u8, len: usize) -> usize;
}

/// Everything a filesystem resolves for ONE open, returned as a unit by
/// `FileOps::open()`.
///
/// Kernel-internal, NOT KABI: an isolated filesystem never sees this struct.
/// The ring stubs encode both fields into the response wire and reconstruct
/// the struct Core-side ([Section 14.2](#vfs-ring-buffer-protocol)), so a same-domain
/// and a cross-domain open produce the identical binding.
pub struct OpenOutcome {
    /// The filesystem-private per-open token. Stored in
    /// `OpenFile::private_data` and handed back to every later `FileOps`
    /// call on this description.
    pub private: u64,

    /// `Some` ONLY for stacking filesystems: the resolved real inode —
    /// upper if present, else lower, AFTER any copy-up this open performed
    /// — whose embedded `AddressSpace` backs all data I/O for the
    /// description being built. `None` means the data lives in the opened
    /// inode itself, which is the case for every non-stacking filesystem.
    ///
    /// `open_and_install` writes it into `OpenFile::data_inode` before the
    /// descriptor is published; the binding is immutable from then on (see
    /// that field's doc).
    pub data_inode: Option<Arc<Inode>>,
}

/// File data operations (open, read, write, sync, allocate, close).
pub trait FileOps: Send + Sync {
    /// Called when a file is opened. Allows the filesystem to initialize per-open
    /// state (NFS delegation, device state, lock state). Returns the
    /// filesystem-private context value stored in the file descriptor, together
    /// with the data-inode binding a stacking filesystem resolved for this open
    /// (`None` for every non-stacking filesystem) — one call, one atomic open
    /// contract, so no second round trip is needed when the filesystem is in
    /// another domain.
    fn open(&self, inode: InodeId, flags: OpenFlags) -> Result<OpenOutcome>;

    /// Called when the last file descriptor referencing this open file is closed.
    /// Filesystem releases per-open state (flock release-on-close, NFS delegation
    /// return, device cleanup). `private` is the value returned by `open()`.
    fn release(&self, inode: InodeId, private: u64) -> Result<()>;

    /// Called on EVERY `close(2)` of a descriptor referencing this open file
    /// (before the file's refcount is decremented), NOT only on the last-
    /// reference `release()`. `closer` is the process performing the close.
    /// Linux `f_op->flush`. The distinction matters for cleanup that must run
    /// on each owner-close even while dup'd/fork-inherited descriptors keep the
    /// file alive — mqueuefs overrides it for `mq_notify` owner deregistration
    /// ([Section 17.3](17-containers.md#posix-ipc--mqnotify-ownership-and-exit-cleanup)). The fd-close path
    /// invokes it as `file_ops.flush(file, closer)` on each close. Default:
    /// no-op, so no existing filesystem impl changes.
    fn flush(&self, file: &OpenFile, closer: &Process) -> Result<()> {
        let _ = (file, closer);
        Ok(())
    }

    /// Read data from a file. `file` provides the OpenFile context (f_pos,
    /// f_flags, filesystem-private state). `offset` is read-write: the
    /// implementation advances it by the number of bytes read (supporting
    /// both pread with caller-supplied offset and read with f_pos).
    /// `buf` is a user-space slice descriptor for safe copy-to-user.
    fn read(
        &self,
        file: &OpenFile,
        buf: &mut UserSliceMut,
        offset: &mut i64,
    ) -> Result<usize, IoError>;

    /// Write data to a file. Same conventions as `read()`: `offset` is
    /// advanced by the number of bytes written.
    fn write(
        &self,
        file: &OpenFile,
        buf: &UserSlice,
        offset: &mut i64,
    ) -> Result<usize, IoError>;

    /// Truncate a file to the specified size. This is separate from setattr
    /// because truncation is a complex operation on many filesystems: it must
    /// free blocks/extents, update extent trees, handle COW (ZFS/btrfs),
    /// interact with snapshots, and flush in-progress writes beyond the new
    /// size. The VFS calls truncate after updating the in-memory inode size.
    /// `private` is the filesystem-private context value returned by `open()`.
    fn truncate(&self, inode: InodeId, private: u64, new_size: u64) -> Result<()>;

    /// Flush file data (and optionally metadata) to stable storage.
    /// `private` is the filesystem-private context value returned by `open()`.
    /// For DSM-managed pages: fsync waits for both local writeback completion
    /// AND DSM PutAck receipt ([Section 6.12](06-dsm.md#dsm-subscriber-controlled-caching--fsync-semantics)).
    /// The VFS fsync path calls `dsm_sync_pages(inode, start, end)` after
    /// `filemap_write_and_wait_range()` to ensure DSM coherence.
    fn fsync(&self, inode: InodeId, private: u64, start: u64, end: u64, datasync: u8) -> Result<()>;

    /// Pre-allocate or punch holes in file storage. `private` is the
    /// filesystem-private context value returned by `open()`.
    fn fallocate(&self, inode: InodeId, private: u64, offset: u64, len: u64, mode: FallocateMode) -> Result<()>;

    /// Read directory entries. Returns entries starting from `offset` (an opaque
    /// cookie, not a byte position). The callback is invoked for each entry; it
    /// returns `false` to stop iteration (buffer full). This is the backend for
    /// `getdents64(2)`. `private` is the filesystem-private context value
    /// returned by `open()`.
    fn readdir(
        &self,
        inode: InodeId,
        private: u64,
        offset: u64,
        emit: &mut dyn FnMut(InodeId, u64, FileType, &OsStr) -> bool,
    ) -> Result<()>;

    /// Seek to a data or hole region (SEEK_DATA / SEEK_HOLE, lseek(2)).
    /// Filesystems that do not support sparse files return the file size for
    /// SEEK_DATA at any offset, and ENXIO for SEEK_HOLE at any offset.
    /// `private` is the filesystem-private context value returned by `open()`.
    fn llseek(&self, inode: InodeId, private: u64, offset: i64, whence: SeekWhence) -> Result<u64>;

    /// Map a file region into a process address space. The VFS calls this to
    /// obtain the page frame list; the actual page table manipulation is done
    /// by Core (Section 4.1). Filesystems that do not support mmap (e.g.,
    /// procfs, sysfs) return ENODEV. `private` is the filesystem-private
    /// context value returned by `open()`.
    ///
    /// This decomposed form (plain-data arguments, no `&mut Vma`) is the
    /// ring-transport representation: it is what `kabi_call!` serializes
    /// when the filesystem runs in a different domain. Same-domain callers
    /// use `mmap_direct()` below, whose default implementation delegates
    /// here.
    ///
    /// `vm_flags`: the kernel-internal `VmFlags` bits of the in-progress
    /// VMA — includes the mapping type (`VM_SHARED` set for `MAP_SHARED`,
    /// clear for `MAP_PRIVATE`), the `VM_MAY*` mask, and the PROT-derived
    /// `VM_READ`/`VM_WRITE`/`VM_EXEC`. This is the SAME encoding
    /// `MmapResult.vm_flags` returns; the filesystem returns the (possibly
    /// adjusted) value. Passing the full `vm_flags` (a superset of the
    /// PROT bits) gives a cross-domain filesystem the same decision power
    /// as a same-domain one (deny `PROT_WRITE` on a RO mount, reject
    /// `MAP_SHARED` under DAX constraints, clear `VM_MAYWRITE`).
    fn mmap(&self, inode: InodeId, private: u64, offset: u64, len: usize, vm_flags: u64) -> Result<MmapResult>;

    /// In-kernel direct-call mmap variant. Called by `establish_mapping()` (Step 8,
    /// [Section 4.8](04-memory.md#virtual-memory-manager--establishmapping-mmap-syscall-implementation))
    /// when the filesystem shares the caller's domain, with the
    /// freshly-allocated, not-yet-inserted VMA. The filesystem may:
    /// - install its `VmOperations` vtable (`vma.vm_ops = Some(&EXT4_FILE_VM_OPS)`),
    /// - install an intrinsic custom-fault backing (`vma.custom_fault`,
    ///   e.g., DAX/pmem or a cross-domain backing registration),
    /// - adjust `vma.vm_flags` in place (e.g., clear `VM_MAYWRITE` for
    ///   DAX constraints, set `VM_IO` / `VM_NORESERVE`).
    ///
    /// This mirrors Linux's in-kernel `f_op->mmap(file, vma)` calling
    /// convention. For a cross-domain filesystem the `&mut Vma` cannot
    /// cross the ring — the binding layer passes `vma.vm_flags.bits() as
    /// u64` to the decomposed `mmap()` above and applies the returned
    /// `MmapResult` (vm_ops selection, flag adjustments) to the VMA on the
    /// caller's side.
    ///
    /// # Contract
    /// - The VMA is not yet visible to any other thread (not in the maple
    ///   tree, not in `i_mmap`); no locking is required to mutate it.
    /// - On `Err`, the implementation must leave NO dangling per-VMA
    ///   resources except those released by `vm_ops.close()` — establish_mapping's
    ///   error path calls `close()` (if `vm_ops` was installed) and then
    ///   frees the VMA.
    ///
    /// # Default implementation
    /// Delegates to the decomposed `mmap()` and applies its `MmapResult`
    /// to `vma` — filesystems only override this when they need direct
    /// access to the `Vma` (DAX registration, driver mappings).
    fn mmap_direct(&self, file: &OpenFile, vma: &mut Vma) -> Result<(), Errno> {
        // Data plane: the mapping is established over the RESOLVED data
        // inode (`OpenFile::data_inode`), which equals `file.inode` for
        // every non-stacking filesystem.
        let res = self.mmap(
            InodeId(file.data_inode.i_ino),
            file.private_data.load(Relaxed) as u64,
            vma.vm_pgoff * PAGE_SIZE as u64,
            vma.len(),
            vma.vm_flags.bits() as u64,
        )?;
        res.apply_to_vma(vma);
        Ok(())
    }

    /// Handle a filesystem-specific ioctl. The VFS dispatches generic ioctls
    /// (FIOCLEX, FIONREAD, etc.) itself; only unrecognized ioctls reach the
    /// filesystem driver. Returns ENOTTY for unsupported ioctls. `private` is
    /// the filesystem-private context value returned by `open()`.
    fn ioctl(&self, inode: InodeId, private: u64, cmd: u32, arg: u64) -> Result<i64>;

    /// Splice data between a file and a pipe without copying through userspace.
    /// Backend for splice(2), sendfile(2), and copy_file_range(2). Filesystems
    /// that do not implement this get a generic page-cache-based fallback
    /// provided by the VFS. `private` is the filesystem-private context value
    /// returned by `open()`.
    fn splice_read(
        &self,
        inode: InodeId,
        private: u64,
        offset: u64,
        pipe: PipeId,
        len: usize,
    ) -> Result<usize>;

    /// Splice data from a pipe into a file without copying through userspace.
    /// Reverse direction of splice_read: pipe is the data source, file is the
    /// destination. Backend for splice(2) write direction and vmsplice(2).
    /// Filesystems that do not implement this get a generic page-cache-based
    /// fallback provided by the VFS. `private` is the filesystem-private
    /// context value returned by `open()`.
    fn splice_write(
        &self,
        pipe: PipeId,
        inode: InodeId,
        private: u64,
        offset: u64,
        len: usize,
    ) -> Result<usize>;

    /// Remap a file range: create shared extent references between files.
    /// Backend for FICLONE, FICLONERANGE, and FIDEDUPERANGE ioctls, and the
    /// server-side copy path of copy_file_range(2). Source and destination
    /// must be on the same filesystem.
    ///
    /// `flags` controls behavior (see `RemapFlags` in
    /// [Section 14.4](#vfs-fsync-and-cow--copy-on-write-and-redirect-on-write-infrastructure)):
    /// - `REMAP_FILE_DEDUP`: only remap if source and destination byte ranges
    ///   are identical (deduplication mode; byte-by-byte comparison first).
    /// - `REMAP_FILE_CAN_SHORTEN`: caller accepts a shorter remap than
    ///   requested (e.g., if source extent ends before `len` bytes).
    ///
    /// Returns the number of bytes actually remapped. Filesystems that do not
    /// support reflinks return `EOPNOTSUPP`. The VFS generic layer handles
    /// permission checks, file size validation, and lock ordering before
    /// dispatching to this method.
    fn remap_file_range(
        &self,
        src_inode: InodeId,
        src_private: u64,
        src_offset: u64,
        dst_inode: InodeId,
        dst_private: u64,
        dst_offset: u64,
        len: u64,
        flags: RemapFlags,
    ) -> Result<u64> {
        Err(Errno::EOPNOTSUPP)
    }

    /// Poll for readiness events (POLLIN, POLLOUT, POLLERR, etc.).
    ///
    /// Called by `poll(2)`, `select(2)`, and `epoll_ctl(EPOLL_CTL_ADD)` to:
    /// 1. Register the caller's wait entry on the file's internal WaitQueue(s)
    ///    via `poll_wait()`, so the caller is woken when readiness changes.
    /// 2. Return the current readiness mask (which events are ready *right now*).
    ///
    /// `pt` is `Some(&mut PollTable)` on the first call (registration pass) and
    /// `None` on subsequent re-polls after wakeup (just check readiness, don't
    /// re-register). Regular files always return `EPOLLIN | EPOLLOUT | EPOLLRDNORM
    /// | EPOLLWRNORM` — they are always ready. Special files (pipes, sockets,
    /// eventfd, signalfd, timerfd, pidfd) check their internal state and call
    /// `poll_wait()` on the appropriate WaitQueue(s).
    ///
    /// `private` is the filesystem-private context value returned by `open()`.
    fn poll(
        &self,
        inode: InodeId,
        private: u64,
        events: PollEvents,
        pt: Option<&mut PollTable>,
    ) -> Result<PollEvents>;
}

/// Poll callback registration table.
///
/// Passed to `FileOps::poll()` on the first call. The file implementation calls
/// `poll_wait(wq, pt)` for each WaitQueue that can change the file's readiness.
/// The `PollTable` records which wait queues were registered so that the polling
/// infrastructure (epoll, poll, select) can install wakeup callbacks.
///
/// **Lifecycle**: allocated on the caller's stack (for poll/select) or embedded
/// in the `EpollItem` (for epoll). The `queue_proc` function pointer is the
/// mechanism that installs the actual `WaitQueueEntry`:
/// - For `poll(2)` / `select(2)`: installs a one-shot entry that wakes the
///   calling task.
/// - For `epoll_ctl(EPOLL_CTL_ADD)`: installs a persistent entry whose wakeup
///   function is `ep_poll_callback` ([Section 19.1](19-sysapi.md#syscall-interface--epoll-primary)).
pub struct PollTable {
    /// Callback invoked by `poll_wait()`. Installs a `WaitQueueEntry` on the
    /// given `WaitQueueHead`. The `key` parameter carries the events mask so
    /// the wakeup callback can filter spurious wakes.
    pub queue_proc: fn(wq: &WaitQueueHead, pt: &mut PollTable, key: PollEvents),

    /// Opaque pointer to the polling infrastructure's private state.
    /// For epoll: points to the `EpollItem` that owns this poll table entry.
    /// For poll/select: points to the per-fd poll state on the caller's stack.
    /// SAFETY: For poll/select: points to caller-stack-allocated poll state,
    /// valid for the duration of the poll syscall. For epoll: points to the
    /// owning EpollItem, valid for the EpollItem's lifetime. The queue_proc
    /// callback must cast to the correct type.
    pub private: *mut (),

    /// Events the caller is interested in. Set by the polling infrastructure
    /// before calling `FileOps::poll()`. The file implementation may use this
    /// to avoid registering on wait queues that cannot produce requested events.
    pub events: PollEvents,
}

/// Register a wait queue with the poll table.
///
/// Called by `FileOps::poll()` implementations to tell the polling infrastructure
/// "wake me when this wait queue fires." The `PollTable` installs a
/// `WaitQueueEntry` on `wq` with the appropriate wakeup function.
///
/// If `pt` is `None` (re-poll after wakeup), this is a no-op — the entry is
/// already installed from the first call.
///
/// **Cost**: one `WaitQueueEntry` insertion per wait queue per monitored fd.
/// Most files have one wait queue; sockets may have two (read + write).
///
/// ```rust
/// fn poll_wait(wq: &WaitQueueHead, pt: Option<&mut PollTable>) {
///     if let Some(pt) = pt {
///         (pt.queue_proc)(wq, pt, pt.events);
///     }
/// }
/// ```
pub fn poll_wait(wq: &WaitQueueHead, pt: Option<&mut PollTable>);

/// Dentry (directory entry) lifecycle operations.
/// Most filesystems use the default VFS implementations. Only network and
/// clustered filesystems need custom implementations (primarily d_revalidate).
pub trait DentryOps: Send + Sync {
    /// Revalidate a cached dentry. Called before using a cached dentry to verify
    /// it is still valid. Returns true if the dentry is still valid, false if
    /// the VFS should discard it and perform a fresh lookup.
    /// Default: always returns true (local filesystems).
    /// Network FS: checks with the server. Clustered FS: checks DLM lease (Section 15.12.6).
    fn d_revalidate(&self, parent: InodeId, name: &OsStr) -> Result<bool> {
        Ok(true)
    }

    /// Automount trigger hook. Called by `follow_automount()` during path
    /// resolution when the walked dentry's inode carries
    /// `InodeFlags::AUTOMOUNT` and the lookup does not carry
    /// `LookupFlags::NO_AUTOMOUNT`
    /// ([Section 14.10](#autofs-kernel-automount-trigger--vfs-integration)).
    ///
    /// Precondition: REF-walk only — the walk is downgraded from RCU-walk
    /// before this is invoked; the method may sleep (autofs waits for its
    /// daemon).
    ///
    /// Returns:
    ///   - `Ok(None)` — nothing for the VFS to attach: the trigger resolved
    ///     out-of-band (the autofs daemon performed the mount, so the dentry
    ///     is now `DCACHE_MOUNTED`) or a racing walker already mounted it.
    ///     The walk re-checks `DCACHE_MOUNTED` and crosses the mounted
    ///     subtree.
    ///   - `Ok(Some(mnt))` — a kernel-constructed submount (e.g. a
    ///     network-filesystem referral) that the VFS must attach at this
    ///     position via `attach_mount()`
    ///     ([Section 14.6](#mount-tree-data-structures-and-operations--attachmount-attach-a-constructed-mount-into-the-tree)).
    ///     The producer sets policy flags (e.g. `MountFlags::MNT_SHRINKABLE`)
    ///     on the `Mount` before returning it.
    ///   - `Err(e)` — path resolution fails with `e`.
    ///
    /// Default: `Ok(None)` — a filesystem with no trigger semantics never
    /// reaches this hook anyway, because the inode flag gates the call.
    fn d_automount(&self, at: &MountDentry) -> Result<Option<Arc<Mount>>> {
        Ok(None)
    }

    /// Custom name comparison. Called during lookup to compare a dentry name
    /// with a search name. Used by case-insensitive filesystems (e.g., VFAT,
    /// CIFS with case folding, ext4 with casefold feature).
    /// Default: byte-exact comparison.
    fn d_compare(&self, name: &OsStr, search: &OsStr) -> bool {
        name == search
    }

    /// Returns a custom hash for this dentry name, or `None` to use the
    /// VFS default (SipHash-1-3 with per-superblock key from `SuperBlock.hash_key`).
    /// Must be consistent with d_compare: if two names are equal per d_compare,
    /// they must produce the same hash.
    ///
    /// The VFS lookup layer calls `d_hash()` and checks the return value.
    /// If `None`, the VFS uses its own SipHash-1-3 with the per-superblock
    /// random key directly, without requiring filesystem involvement. This
    /// matches Linux's pattern where `d_hash` is only invoked when
    /// Linux invokes `d_hash` only when `dentry->d_op->d_hash` is non-NULL.
    ///
    /// Filesystems with custom hash requirements (e.g., case-insensitive)
    /// override this to return `Some(hash_value)` using their own algorithm —
    /// they never see the SipHash key. The per-superblock key is managed by
    /// the VFS, not exposed to filesystem implementations.
    fn d_hash(&self, name: &OsStr) -> Option<u64> {
        None
    }

    /// Called when a dentry's reference count drops to zero (dentry enters
    /// the unused LRU list). Filesystem can veto caching by returning false.
    fn d_delete(&self, inode: InodeId, name: &OsStr) -> bool {
        true // default: allow LRU caching
    }

    /// Called when a dentry is finally freed from the cache.
    fn d_release(&self, inode: InodeId, name: &OsStr) {}
}

/// Kernel-internal inode attribute structure. Contains all fields exposed by
/// Linux statx(2). The SysAPI layer translates to the userspace struct statx
/// layout (different field ordering, padding, and encoding).
pub struct InodeAttr {
    /// Bitmask of valid fields (STATX_* flags). Filesystems set only
    /// the bits for fields they actually populate.
    pub mask: u32,

    pub mode: u32,        // File type and permissions. u32 for internal storage
                          // convenience and future extensibility. Only bits [15:0]
                          // are defined (identical to Linux umode_t). Bits [31:16]
                          // are reserved and must be zero. The SysAPI translation
                          // to userspace statx truncates to u16.
    pub nlink: u32,       // Hard link count
    pub uid: u32,         // Owner UID
    pub gid: u32,         // Group GID
    pub ino: u64,         // Inode number
    pub size: u64,        // File size in bytes
    pub blocks: u64,      // 512-byte blocks allocated
    pub blksize: u32,     // Preferred I/O block size

    // Timestamps with nanosecond precision
    pub atime_sec: i64,   // Last access
    pub atime_nsec: u32,
    pub mtime_sec: i64,   // Last modification
    pub mtime_nsec: u32,
    pub ctime_sec: i64,   // Last status change
    pub ctime_nsec: u32,
    pub btime_sec: i64,   // Creation time (birth time)
    pub btime_nsec: u32,

    /// Device ID (for device special files: char/block). Uses the `DevId` type
    /// ([Section 14.5](#device-node-framework)) with Linux-compatible MKDEV encoding:
    /// `(major << 20) | (minor & 0xFFFFF)`. Major occupies bits 31:20 (12 bits,
    /// 0-4095), minor occupies bits 19:0 (20 bits, 0-1048575). The SysAPI layer
    /// ([Section 19.1](19-sysapi.md#syscall-interface)) splits `DevId` into separate
    /// `stx_rdev_major`/`stx_rdev_minor` u32 fields for `statx()` responses using
    /// `dev_id.major()` and `dev_id.minor()`.
    pub rdev: DevId,
    /// Device ID of the filesystem containing this inode. Same `DevId` encoding
    /// as `rdev`. The SysAPI layer splits into `stx_dev_major`/`stx_dev_minor`
    /// for `statx()` responses.
    pub dev: DevId,
    pub mount_id: u64,    // Mount identifier (STATX_MNT_ID, since Linux 5.8)
    pub attributes: u64,  // File attributes (STATX_ATTR_* flags)
    pub attributes_mask: u64, // Supported attributes mask

    // Direct I/O alignment (STATX_DIOALIGN, since Linux 6.1)
    pub dio_mem_align: u32,    // Required alignment for DIO memory buffers
    pub dio_offset_align: u32, // Required alignment for DIO file offsets

    // Subvolume identifier (STATX_SUBVOL, since Linux 6.10; btrfs, bcachefs)
    pub subvol: u64,

    // Atomic write limits (STATX_WRITE_ATOMIC, since Linux 6.11)
    pub atomic_write_unit_min: u32,  // Min atomic write size (power-of-2)
    pub atomic_write_unit_max: u32,  // Max atomic write size (power-of-2)
    pub atomic_write_segments_max: u32, // Max segments in atomic write
    pub atomic_write_unit_max_opt: u32, // Optimal max atomic write size (STATX_WRITE_ATOMIC, since Linux 6.13)

    // Direct I/O read alignment (STATX_DIO_READ_ALIGN, since Linux 6.14)
    pub dio_read_offset_align: u32,  // DIO read offset alignment (0 = use dio_offset_align)

    /// Persistent inode attribute flags (`FS_*_FL` user-flag space — NOT
    /// the kernel-internal `InodeFlags` space, NOT statx `attributes`). Always
    /// populated by the driver: the on-disk flag word for filesystems
    /// with persistent flags, `0` otherwise (enforcement-neutral). The
    /// VFS instantiation path derives `Inode::i_flags` from this via
    /// `fs_flags_to_inode_flags()` ([Section 14.1](#virtual-filesystem-layer--inode-attribute-flags));
    /// `vfs_apply_reconciled_attr()` re-derives it after crash recovery.
    /// Not a statx field — never copied to userspace by the SysAPI
    /// translation.
    pub fs_flags: u32,
}

Linux comparison: Linux's VFS uses struct super_operations, struct inode_operations, struct file_operations, and struct dentry_operations — C structs of function pointers (Linux's file_operations alone has 30+ methods). UmkaOS's trait-based design serves the same purpose but with Rust's safety guarantees: a filesystem that forgets to implement fsync is a compile-time error, not a null pointer dereference at runtime. The trait methods above cover the operations needed for POSIX compatibility, including remap_file_range() for reflink/clone/dedup (see Section 14.4). Rarely-used operations (e.g., fiemap) are handled by generic VFS fallback code that calls the core read/write/fallocate methods.

14.1.2.3 File Handle Export (ExportOps)

The ExportOps trait is implemented by filesystems that support persistent file handles — opaque tokens that identify an inode across server reboots and path renames. Required for:

• NFS server (clients hold file handles that survive server restart)
• CRIU checkpoint/restore (open_by_handle_at reopens files by handle)
• Backup software (rsync --no-implied-dirs, backup agents)

/// File system export operations. Optional — implement only if the filesystem
/// supports persistent, path-independent file handles.
///
/// A file handle is a short opaque byte string (max 128 bytes) that uniquely
/// identifies an inode within a filesystem instance. The handle must survive:
/// - Server reboots (handle encodes stable inode ID + generation counter)
/// - Directory renames (handle does not encode path)
/// - Mount point changes (handle is filesystem-relative, not global)
pub trait ExportOps: Send + Sync {
    /// Encode an inode into a file handle.
    ///
    /// Returns the handle bytes written and a filesystem-defined `fh_type` code
    /// (passed back to `decode_fh`; used to distinguish handle formats).
    ///
    /// # Typical encoding
    /// ext4:  [ inode_number: u32, generation: u32 ] → 8 bytes, fh_type=1
    /// XFS:   [ ino: u64, gen: u32, parent_ino: u64, parent_gen: u32 ] → 24 bytes, fh_type=1
    /// Btrfs: [ objectid: u64, root_objectid: u64, gen: u64 ] → 24 bytes, fh_type=1
    ///
    /// Returns `Err(EOVERFLOW)` if `max_bytes` is too small for this filesystem's handle.
    fn encode_fh(
        &self,
        inode: &Inode,
        handle: &mut [u8; 128],
        max_bytes: usize,
        /// If true, include parent inode info to enable NFS reconnect after server reboot.
        connectable: bool,
    ) -> Result<(usize, u8), VfsError>; // (bytes_written, fh_type)

    /// Decode a file handle back to an inode reference.
    ///
    /// Called by `open_by_handle_at`. Must look up the inode using the filesystem's
    /// internal handle format without path traversal.
    ///
    /// Returns `Err(ESTALE)` if the inode no longer exists or the generation counter
    /// does not match (inode number reused after deletion).
    fn decode_fh(
        &self,
        handle: &[u8],
        fh_type: u8,
    ) -> Result<Arc<Inode>, VfsError>;

    /// Get the parent directory inode of an inode (for NFS reconnect after reboot).
    ///
    /// Returns `Err(EACCES)` if the filesystem cannot determine the parent without a
    /// full tree walk (e.g., hardlinks with multiple parents).
    fn get_parent(&self, inode: &Inode) -> Result<Arc<Inode>, VfsError>;

    /// Get the directory entry name for `child` within `parent`.
    ///
    /// Used by the NFS server to reconstruct paths for client caches.
    /// Returns the byte length of the name written into `name_buf`.
    /// Returns `Err(ENOENT)` if no entry for `child` is found in `parent`.
    fn get_name(
        &self,
        parent: &Inode,
        child: &Inode,
        name_buf: &mut [u8; 256],
    ) -> Result<usize, VfsError>;
}

/// Kernel-side file handle: wraps the opaque handle bytes with metadata.
/// Matches the layout of Linux's `struct file_handle` for syscall ABI compatibility.
#[repr(C)]
pub struct FileHandle {
    /// Byte length of the handle data (the populated prefix of `f_handle`).
    pub handle_bytes: u32,
    /// Filesystem-defined type code (passed back verbatim to `ExportOps::decode_fh`).
    pub handle_type: i32,
    /// Opaque handle data (filesystem-defined encoding, up to 128 bytes).
    pub f_handle: [u8; 128],
}
const_assert!(size_of::<FileHandle>() == 136);

name_to_handle_at(2) implementation:

name_to_handle_at(dirfd, pathname, handle, mount_id, flags):

1. Resolve pathname to an inode (using normal path resolution with dirfd as the base;
   AT_EMPTY_PATH allows operating on dirfd itself without a pathname component).
2. Retrieve the inode's superblock.
3. Check that the superblock implements ExportOps. Return ENOTSUP if not.
4. Call superblock.export_ops.encode_fh(inode, handle.f_handle, handle.handle_bytes,
   connectable=true).
5. Write back handle_bytes and handle_type into the userspace handle struct.
6. Write the mount's numeric ID to *mount_id. Mount IDs are assigned at mount time
   via a monotonic counter (Section 14.2.3 MountNode.mnt_id).
7. Return 0 on success; EOVERFLOW if the handle buffer is too small.

open_by_handle_at(2) implementation:

open_by_handle_at(mount_fd, handle, flags):

1. Requires CAP_DAC_READ_SEARCH. This syscall bypasses normal path-based access checks
   by design — it is intended for root-equivalent processes such as NFS servers and
   backup agents. Return EPERM if the capability is absent.
2. Resolve mount_fd to identify which filesystem the handle belongs to:
   fdget(mount_fd) → extract the file's MountDentry → use that mount's superblock.
   mount_fd must be an open fd on any file or directory within the target filesystem
   (typically the mountpoint itself, e.g., `fd = open("/mnt")`). If mount_fd is
   AT_FDCWD, the current working directory's mount is used.
3. Retrieve the mount's superblock (from the MountDentry resolved in step 2).
4. Check that the superblock implements ExportOps. Return ENOTSUP if not.
5. Call superblock.export_ops.decode_fh(handle.f_handle, handle.handle_type) → Arc<Inode>.
6. If Err(ESTALE): the inode was deleted or the generation counter does not match
   (inode number reused). Return ESTALE.
7. Perform a DAC check and LSM check on the inode using the caller's credentials.
8. Allocate a new OpenFile wrapping the inode. The open file description does not
   carry a path — the inode is accessed directly without directory traversal.
9. Return the new file descriptor number.

Security note: open_by_handle_at intentionally skips directory execute-permission
checks along the path to the inode (the path is not known at this point). This is
the documented and expected behavior for NFS server use. CAP_DAC_READ_SEARCH is the
required guard.

14.1.2.4 Core VFS Data Structures

The VFS layer operates on four fundamental data structures: dentries (directory entries), inodes (index nodes), superblocks (mounted filesystem state), and open files (open file handles). All four are defined in this section.

14.1.2.4.1.1 OpenFile (Open File Description)

/// An open file description — the kernel-internal object backing one or more
/// file descriptors. Created by `open(2)`, `openat(2)`, `socket(2)`, `pipe(2)`,
/// `accept(2)`, etc. Multiple file descriptors can reference the same `OpenFile`
/// via `dup(2)` or `fork(2)`.
///
/// **Lifecycle**: Allocated at open time. Reference-counted (`Arc<OpenFile>`).
/// The `FdTable` holds `Arc<OpenFile>` entries. When the last fd referencing
/// this open file is closed (refcount drops to zero), `FileOps::release()` is
/// called and the `OpenFile` is freed.
///
/// **Concurrency**: Most fields are immutable after creation (`inode`, `dentry`,
/// `mount`, `f_ops`, `f_cred`, `f_mode`). Mutable fields use atomic operations:
/// - `f_pos`: `AtomicI64` — updated by `read()`/`write()`/`lseek()`. `pread()`
///   and `pwrite()` do not touch `f_pos`. Access is mediated by `fdget_pos()`:
///
///   **`fdget_pos()` protocol** (f_pos serialization):
///   The VFS read/write dispatch path looks the file up with `fdget()`
///   (see "File Descriptor Lookup — Lockless Fast Paths") and calls
///   `fdget_pos(&fd_guard)` on the result instead of touching `f_pos`
///   directly. `fdget_pos()` returns an `FdPosGuard` that provides
///   exclusive `&mut i64` access to the file position:
///
///   - **Lock-free fast path**: taken only when NO other execution
///     context can concurrently operate on this open file description,
///     which requires BOTH of:
///     1. the calling task's `FdTable` is unshared — `fdget()` returned
///        `FdGuard::Borrowed` (no `CLONE_FILES` sibling can reach the
///        same fd), AND
///     2. the `OpenFile` has exactly one owning reference — the fd
///        table's own (no `dup(2)`, no `fork(2)` table copy, no
///        in-flight `SCM_RIGHTS`), observed at lookup time and recorded
///        as `FdGuard::Borrowed.pos_exclusive`.
///     Then `fdget_pos()` loads `f_pos` into a local `i64`, returns
///     `&mut` to it, and stores it back on drop. No mutex, no atomic
///     RMW. This is the common case for most file descriptors.
///
///     The `OpenFile` reference count ALONE is NOT a sufficient
///     discriminator: two `CLONE_FILES` threads share ONE `FdTable`
///     holding ONE `Arc<OpenFile>`, so both would observe refcount == 1
///     while racing `read(2)` on the same description — two independent
///     local-copy write-backs would lose position updates (a POSIX
///     atomicity violation). Condition 1 excludes exactly that case.
///
///   - **Locked slow path**: in every other case (`FdGuard::Owned`, or
///     `pos_exclusive == false`), `fdget_pos()` acquires `f_pos_lock`
///     (a per-OpenFile `Mutex<()>`) before loading the position into the
///     local copy. This serializes concurrent `read()`/`write()` calls
///     that share the open file description, meeting the POSIX atomic
///     position-update requirement. The mutex is released when the
///     `FdPosGuard` is dropped, after the write-back.
///
///   - **Files without a seek cursor**: the whole protocol applies only
///     to files with `FMODE_ATOMIC_POS` (regular files and directories).
///     Pipes, sockets, and character streams never take `f_pos_lock` —
///     their position is meaningless and their reads/writes are
///     stream-ordered by their own internal locks.
///
///   - **Directories**: always take the locked slow path, even with
///     `pos_exclusive` — the `readdir` cursor is also advanced by
///     kernel-internal iterators, matching Linux's unconditional
///     `iterate_shared` arm of `file_needs_f_pos_lock()`.
///
///   - **`pread()`/`pwrite()` bypass**: These syscalls use a caller-supplied
///     offset and never call `fdget_pos()` — plain `fdget()` suffices.
///     No f_pos serialization is needed because the caller-supplied offset
///     is on the stack.
///
///   **Linux parity** (`fs/file.c`, verified against `torvalds/linux`
///   master): Linux `__fget_light()` borrows without a refcount bump only when
///   `atomic_read_acquire(&files->count) == 1`; when the fd TABLE is
///   shared, `fdget()` takes a real file reference, so
///   `file_needs_f_pos_lock()`'s
///   Linux test `__file_ref_read_raw(&file->f_ref) != FILE_REF_ONEREF`
///   observes ≥ 2 and `fdget_pos()` takes `f_pos_lock`. Linux's
///   effective discriminator is therefore fd-table sharing PLUS
///   description sharing — the same two conditions as above, and the
///   same set of executions takes each path.
/// - `f_flags`: `AtomicU32` — modified by `fcntl(F_SETFL)` for `O_APPEND`,
///   `O_NONBLOCK`, `O_ASYNC`, `O_DIRECT`. Read-only flags (`O_RDONLY`,
///   `O_RDWR`, `O_CREAT`, `O_EXCL`) are set at open time and never change.
/// - `f_wb_err`: `WbErrSnapshot` (`AtomicU64` on 64-bit, `AtomicU32` on
///   32-bit) — writeback error snapshot. Initialized from
///   `AddressSpace::wb_err.sample()` at open time. Compared at `fsync()`
///   time against `AddressSpace::wb_err` via
///   `check_and_advance(&self.f_wb_err)` to detect new errors. Atomic
///   because dup'd/forked fds share this OpenFile across threads.
/// - `private_data`: `AtomicPtr` — set by `FileOps::open()` and read by
///   subsequent operations. Rewritten (Release) only by
///   `vfs_revalidate_open_file()` after a driver crash recovery, which
///   re-runs `FileOps::open()` on the reloaded instance.
/// - `open_generation`: `AtomicU64` — the driver generation this fd is
///   valid for; refreshed by the same revalidation protocol
///   ([Section 14.1](#virtual-filesystem-layer--open-file-descriptor-recovery-generation-refresh)).
///   `reopen_errno`/`revalidate_lock` support that protocol.
///
/// **Relationship to FdTable**: The `FdTable` (in [Section 8.1](08-process.md#process-and-task-management))
/// maps integer file descriptors (0, 1, 2, ...) to `Arc<OpenFile>`. `dup(2)`
/// creates a new fd pointing to the same `Arc<OpenFile>`. `fork()` copies the
/// `FdTable`, incrementing the `Arc` refcount for each entry.
pub struct OpenFile {
    /// Inode backing this open file. For regular files, directories, symlinks,
    /// and device nodes, this is the filesystem inode. For pipes and sockets,
    /// this is a synthetic inode from the pipefs/sockfs pseudo-filesystem. Like
    /// every inode, that synthetic inode is inserted into its pipefs/sockfs
    /// superblock's `inode_cache` at creation (setting `I_HASHED`) and removed
    /// at destruction — the universal cache-membership invariant
    /// ([Section 14.1](#virtual-filesystem-layer--inode-cache-icache), Invariants) admits no
    /// unhashed inode class, so a pipe/socket inode is enumerable via
    /// `SUPER_BLOCK_MAP` × `inode_cache` for the whole of its life exactly like a
    /// disk inode (which the Shadow-and-Migrate layout walk relies on).
    pub inode: Arc<Inode>,

    /// Dentry that was used to open this file. A strong `Arc<Dentry>` so the
    /// dentry is pinned for the lifetime of the open file — this prevents the
    /// dentry from being evicted while the file is open, which is necessary for
    /// `/proc/[pid]/fd/N` readlink (renders the path from this
    /// dentry) and for `fd_to_mount_dentry()` (which `Arc::clone`s this field
    /// into a `MountDentry`). A weak `DentryRef` lookup key cannot pin, so the
    /// open file description holds the strong reference directly.
    pub dentry: Arc<Dentry>,

    /// Resolved data inode: the inode whose embedded `AddressSpace`
    /// (`i_mapping`), size, and address-space operations back all data I/O
    /// for this open file description.
    ///
    /// For ordinary filesystems this is the same `Arc` as `inode`. Stacking
    /// filesystems (overlayfs,
    /// [Section 14.8](#overlayfs-union-filesystem-for-containers)) bind it at open time
    /// to the backing real inode — upper if present, else lower — AFTER any
    /// copy-up their open path performs.
    ///
    /// **Immutable after the `OpenFile` is published.** The binding is
    /// per-open: an fd opened before a later copy-up keeps its resolved lower
    /// inode, which is contract-faithful because that lower data was
    /// identical at copy-up time, and opens for write trigger copy-up first
    /// and therefore always bind upper. Immutability is load-bearing: no
    /// atomics, no swap protocol, and no reader/writer hazard on the per-I/O
    /// fast path. A future filesystem that genuinely needs live rebinding
    /// must design that explicitly rather than inherit a mutable slot.
    ///
    /// Set during `open_and_install` while the `OpenFile` is still
    /// exclusively owned; a stacking filesystem's `FileOps::open()` RETURNS
    /// the binding in `OpenOutcome::data_inode`, and `open_and_install`
    /// writes it before publication.
    ///
    /// **Which field to use**: every generic VFS site that derives the DATA
    /// mapping, data size, or address-space ops uses `data_inode`. Identity
    /// and metadata sites — permission checks, `stat`, `/proc/[pid]/fd`,
    /// dentry pinning — use `inode`/`dentry`.
    pub data_inode: Arc<Inode>,

    /// Mount instance through which this file was opened. Pinned for the
    /// lifetime of the open file — this prevents `umount` from proceeding
    /// while files are open on the filesystem (umount checks `mnt_count`).
    /// `RcuCell`, not a plain `Arc`: after a VFS-module crash the mount
    /// tree is rebuilt with fresh `Mount` instances, and
    /// `vfs_revalidate_open_file()` REBINDS this field to the reconstructed
    /// instance (under `revalidate_lock`; readers load via RCU) — see
    /// [Section 14.1](#virtual-filesystem-layer--shadow-mount-registry-and-mount-tree-reconstruction),
    /// "Open-file pins after reconstruction". Outside recovery the cell is
    /// never restored.
    pub mount: RcuCell<Arc<Mount>>,

    /// Core-written copies of the opened mount's identity, set once when
    /// the `OpenFile` is constructed and never read back from `Mount`
    /// payload memory (which is driver-domain-writable and untrusted after
    /// a crash). These are the trusted rebind keys for the shadow-registry
    /// lookup in `vfs_revalidate_open_file()`.
    pub mount_id: u64,
    /// Owning mount namespace of `mount_id` (`MountNamespace.ns_id`).
    pub mount_ns_id: u64,
    /// Snapshot of `ShadowMountRegistry.crash_epoch` at the last (re)bind
    /// of `mount`. Written at open and by `vfs_rebind_mount()` under
    /// `revalidate_lock`; atomic for `&self` interior mutability.
    pub mount_epoch: AtomicU64,

    /// File operations vtable. Set at open time from the inode's `i_fop`
    /// (regular files, directories) or the device driver's registered
    /// `FileOps` (character/block devices). Immutable after creation.
    pub f_ops: &'static dyn FileOps,

    /// Current file position (seek offset). Updated by `read()`, `write()`,
    /// and `lseek()`. Not used by `pread()`/`pwrite()` (which take an
    /// explicit offset). Initialized to 0 for ALL opens, including
    /// `O_APPEND` — `open(2)` never presets the position (Linux parity:
    /// Linux `do_dentry_open()` leaves `f_pos` at 0). The write path seeks to
    /// EOF under exclusive `i_rwsem` before every `O_APPEND` write
    /// regardless of the stored position (`page_cache_write_iter()`
    /// Step 3), which is what makes append atomic; a position preset at
    /// open time would be stale by the first write anyway.
    pub f_pos: AtomicI64,

    /// Mutex protecting `f_pos` for shared open file descriptions.
    /// Acquired by `fdget_pos()` unless the description is provably
    /// private to the calling context — fd table unshared AND exactly
    /// one owning reference AND not a directory (see the `fdget_pos()`
    /// protocol above). Private descriptors (the common case) never
    /// touch this mutex. Never taken for files without
    /// `FMODE_ATOMIC_POS`. This matches Linux's
    /// `struct file::f_pos_lock` mutex and the conditions of
    /// `file_needs_f_pos_lock()` (`fs/file.c`).
    pub f_pos_lock: Mutex<()>,

    /// Open flags. Lower bits contain the access mode (O_RDONLY=0, O_WRONLY=1,
    /// O_RDWR=2). Upper bits contain status flags (O_APPEND, O_NONBLOCK,
    /// O_ASYNC, O_DIRECT, O_NOATIME, O_CLOEXEC). Status flags may be modified
    /// by `fcntl(F_SETFL)`; access mode bits are immutable after open.
    ///
    /// **`F_SETFL` substrate gates** (Linux `fs/fcntl.c setfl()` parity;
    /// [Section 14.1](#virtual-filesystem-layer--inode-attribute-flags)): toggling
    /// `O_APPEND` in either direction on an `InodeFlags::APPEND` inode fails with
    /// EPERM when O_APPEND is toggled on an `InodeFlags::APPEND` inode, and
    /// newly setting `O_NOATIME` requires the caller to be the file
    /// owner or `CAP_FOWNER`-capable (`EPERM` otherwise).
    pub f_flags: AtomicU32,

    /// File mode derived from open flags. Bitflags indicating which operations
    /// are permitted on this open file. Set at open time and immutable.
    /// Checked by the VFS before dispatching to `FileOps` methods.
    pub f_mode: FileMode,

    /// Credentials captured at open time. Used for permission checks that
    /// occur after open (e.g., writeback, async I/O completion) where the
    /// original opener's credentials must be used, not the current task's.
    /// Immutable after creation.
    pub f_cred: Arc<Credentials>,

    /// Writeback error snapshot (atomic). Initialized as
    /// `WbErrSnapshot::new(mapping.wb_err.sample())` at open time. At
    /// `fsync()` time, compared against the current `AddressSpace::wb_err`
    /// via `check_and_advance(&self.f_wb_err)` — if a new error occurred
    /// since this fd was opened (or since the last `fsync()`), `fsync()`
    /// returns the error. The snapshot is CAS-advanced so the error is
    /// reported exactly once per open file description.
    ///
    /// **Why atomic**: an `OpenFile` is a SHARED open file description —
    /// `dup(2)` and `fork(2)` hand the same `Arc<OpenFile>` to multiple
    /// threads/processes, and two of them can call `fsync()` concurrently.
    /// A plain integer here would be a data race, and would be unwritable
    /// through the `&OpenFile` the fsync path holds (no `&mut` through an
    /// `Arc`). The CAS in `check_and_advance()` guarantees exactly one of
    /// two racing fsyncs reports a given error generation.
    ///
    /// Type is `WbErrSnapshot` (defined in [Section 14.4](#vfs-fsync-and-cow)) —
    /// `AtomicU64` on 64-bit targets, `AtomicU32` on 32-bit targets — to
    /// match the `since` parameter of `ErrSeq::check_and_advance` on all
    /// eight supported architectures.
    pub f_wb_err: WbErrSnapshot,

    /// Readahead state for this open file. Tracks sequential access detection,
    /// the current readahead window size, and the last readahead position.
    /// Used by `filemap_get_pages()` and the readahead engine
    /// ([Section 4.4](04-memory.md#page-cache--readahead-engine)) to decide how many pages to
    /// prefetch. Each open file has independent readahead state — two
    /// processes reading the same file at different positions maintain
    /// separate readahead windows.
    pub ra_state: Mutex<FileRaState>,

    /// Filesystem-private data. Set by `FileOps::open()` to store per-open
    /// state (e.g., ext4 journal handle, NFS delegation ID, device driver
    /// context). The VFS passes this value (as `private: u64`) to all
    /// subsequent `FileOps` method calls. Cleared by `FileOps::release()`.
    /// Null for `O_PATH` descriptions — `FileOps::open()` never ran
    /// (see `open_and_install` step 1).
    pub private_data: AtomicPtr<()>,

    /// Driver generation this open file is currently valid for. Set to
    /// `sb.driver_generation.load(Acquire)` during `open()`; **refreshed**
    /// by `vfs_revalidate_open_file()`
    /// ([Section 14.1](#virtual-filesystem-layer--open-file-descriptor-recovery-generation-refresh))
    /// after a crash recovery bumps `sb.driver_generation` at Step U10a.
    /// Atomic BECAUSE of that refresh: revalidation runs on whichever
    /// thread first touches the fd after recovery, concurrently with other
    /// threads' dispatch-path loads on the same shared `OpenFile`
    /// (dup/fork/CLONE_FILES). Stored with `Release` (after the new
    /// `private_data` is published), loaded with `Acquire`.
    ///
    /// The generation check is in the VFS dispatch path (before
    /// `select_ring()`); a mismatch routes into the revalidation slow path
    /// instead of surfacing `ENOTCONN`:
    /// ```rust
    /// if file.open_generation.load(Acquire)
    ///     != file.inode.i_sb.driver_generation.load(Acquire) {
    ///     vfs_revalidate_open_file(file)?; // transparent fd refresh
    /// }
    /// ```
    pub open_generation: AtomicU64,

    /// Serializes `vfs_revalidate_open_file()` for threads sharing this
    /// open file description. Sleeping mutex (the re-open crosses the ring).
    /// Warm path only: contended at most once per fd per driver crash;
    /// never touched on the normal dispatch fast path.
    pub revalidate_lock: Mutex<()>,

    /// Dead-fd latch. `0` = healthy; otherwise the negative errno latched
    /// by a permanently failed revalidation (backing inode gone after
    /// journal replay, or driver refused the re-open) — every subsequent
    /// operation on this fd returns `EIO` until `close(2)`. Written once
    /// (Release) under `revalidate_lock`; read (Acquire) by the
    /// revalidation fast-reject. Transient failures (`ENOMEM`, boundary
    /// `ENXIO` from a second crash) are never latched.
    pub reopen_errno: AtomicI32,
}

/// RAII wrapper that owns a file descriptor and closes it on `Drop`.
/// Analogous to `std::os::fd::OwnedFd`. Ownership is unique: neither `Copy`
/// nor `Clone`; duplication is explicit (`dup`).
///
/// **Applicability**: only for a kernel-held descriptor whose *number* still
/// has meaning at close time — i.e. one closed while the owning task and its
/// fd table are still alive (see INV-FD1 below). Kernel state that must
/// outlive the owning process's fd table holds the OBJECT instead
/// (`Arc<EventFd>`, `Arc<OpenFile>`, …): that is why the exit-cleanup
/// `NotifyEventFd` action stores `Arc<EventFd>` and not an fd
/// ([Section 8.1](08-process.md#process-and-task-management--umkaos-process-exit-cleanup-tokens)) —
/// its `Weak<FdTable>` upgrade would fail exactly when the action fires.
/// No in-spec consumer holds an `OwnedFd` today; the type is the declared
/// shape for one that later needs it.
pub struct OwnedFd {
    /// The owned descriptor number within `files`.
    fd: i32,
    /// The fd table the descriptor belongs to. `Weak` so this wrapper does not
    /// keep a dead process's table alive: if the table is already gone, the fd
    /// was closed with it and Drop is a no-op.
    files: Weak<FdTable>,
}

impl Drop for OwnedFd {
    fn drop(&mut self) {
        // Close the fd against its owning table if that table is still alive.
        if let Some(files) = self.files.upgrade() {
            files.close_fd(self.fd);
        }
    }
}

/// RAII guard returned by `fdget_pos()` proving the caller holds the `f_pos`
/// serialization right for an `OpenFile` (see the `fdget_pos()` protocol on
/// `OpenFile::f_pos` above). It provides exclusive `&mut i64` access to the
/// file position and, on the locked slow path, owns the `f_pos_lock`
/// mutex guard; both the working copy write-back and the lock release happen
/// on `Drop`.
///
/// Obtained from a caller-held `FdGuard` via `fdget_pos(&fd_guard)` (see
/// "File Descriptor Lookup — Lockless Fast Paths") — the two-step
/// composition is what lets this guard borrow the `OpenFile` without a
/// reference-count bump of its own: the `FdGuard` outlives it and keeps
/// the description reachable in both its `Borrowed` and `Owned` forms.
///
/// The guard doubles as a **lock witness**: kernel APIs that may only run
/// while `read()`/`write()` are excluded on a given fd — e.g.
/// `replace_fops()` ([Section 14.5](#device-node-framework--file-operations-replacement-replacefops))
/// — take `&FdPosGuard` so the requirement is unforgeable at the type level.
/// Kernel-internal; never crosses a KABI or userspace boundary.
pub struct FdPosGuard<'f> {
    /// The open file whose position this guard serializes. Borrowed from
    /// the caller's `FdGuard` (via its `Deref`), never from the fd table
    /// directly.
    file: &'f OpenFile,
    /// Working copy of `f_pos`. Callers mutate this; it is stored back into
    /// `file.f_pos` (`AtomicI64`, `Release`) on `Drop`.
    pos: i64,
    /// Held only on the locked slow path (shared fd table, shared
    /// description, or directory — see the `fdget_pos()` protocol),
    /// where `f_pos_lock` serializes concurrent position updates. `None`
    /// on the lock-free fast path, which takes no mutex.
    _lock: Option<MutexGuard<'f, ()>>,
}

bitflags! {
    /// File mode flags — derived from open flags at open time. These indicate
    /// which operations the VFS permits on this open file description.
    /// Immutable after open.
    ///
    /// These are internal VFS flags (not directly visible to userspace). They
    /// are derived from the `O_*` flags passed to `open(2)`:
    /// - `O_RDONLY` (0) → `FMODE_READ`
    /// - `O_WRONLY` (1) → `FMODE_WRITE`
    /// - `O_RDWR` (2) → `FMODE_READ | FMODE_WRITE`
    ///
    /// Additional flags are set based on the file type and filesystem
    /// capabilities.
    pub struct FileMode: u32 {
        /// Read operations permitted (`read`, `pread`, `readv`, `mmap PROT_READ`).
        const FMODE_READ    = 0x0001;
        /// Write operations permitted (`write`, `pwrite`, `writev`, `mmap PROT_WRITE`).
        const FMODE_WRITE   = 0x0002;
        /// `lseek` is meaningful. Set for regular files and block devices.
        /// Not set for pipes, sockets, and some character devices.
        const FMODE_LSEEK   = 0x0004;
        /// `pread` is supported (implies the file has a stable notion of offset).
        /// Set for regular files and block devices. Not set for pipes or sockets.
        const FMODE_PREAD   = 0x0008;
        /// `pwrite` is supported.
        const FMODE_PWRITE  = 0x0010;
        /// Execute permission was checked at open time (implies `O_PATH` was not
        /// used and the file's execute bit was verified). Used by `execveat(2)`
        /// with `AT_EMPTY_PATH` to avoid a redundant permission check.
        const FMODE_EXEC    = 0x0020;
        /// File does not contribute to filesystem busy state. Set for files
        /// opened with `O_PATH` (which are just path references, not real opens).
        const FMODE_PATH    = 0x0040;
        /// Direct I/O mode. Set when `O_DIRECT` is in effect and the filesystem
        /// supports it. The VFS bypasses the page cache for read/write.
        const FMODE_DIRECT  = 0x0080;
        /// `f_pos` updates must be POSIX-atomic: `fdget_pos()` applies its
        /// serialization protocol (see `OpenFile::f_pos`). Set at open time
        /// for regular files and directories only (Linux `FMODE_ATOMIC_POS`,
        /// Linux `fs/open.c do_dentry_open()`: `S_ISREG || S_ISDIR`). Never set
        /// for pipes, sockets, or character devices — their reads and
        /// writes are position-less, so `f_pos_lock` is never taken.
        const FMODE_ATOMIC_POS = 0x0100;
    }
}

14.1.2.4.1.2 Dentry (Directory Cache Entry)

/// Directory cache entry — represents a single component in a pathname.
///
/// Dentries form a tree that mirrors the filesystem namespace. Each dentry
/// caches the result of a directory lookup: the mapping from a name to an
/// inode. The dentry cache (dcache) is the primary mechanism for avoiding
/// repeated directory lookups on hot paths.
///
/// **Lifecycle**: Created by `dentry_alloc()` on a dcache miss (before
/// `InodeOps::lookup()` runs). Cached in the dcache hash table (keyed by
/// parent + name). Freed when the reference count drops to zero AND the
/// dentry is evicted from the LRU. Negative dentries (name exists but no
/// inode) are also cached to avoid repeated failed lookups; they are full
/// `Dentry` instances with `d_inode = None` and share the same allocation
/// lifecycle.
///
/// **Allocation**: `Dentry` is a migration-tracked type
/// ([Section 13.18](13-device-classes.md#live-kernel-evolution--generic-tracked-allocator)). `dentry_alloc()`
/// allocates from Nucleus tracked storage via `alloc_tracked::<Dentry>()`
/// and bridges to `Arc<Dentry>` via `Arc::from_tracked` (refcounts in the
/// slot side header; the payload slot is pure `Dentry`). Eviction unhashes
/// the dentry and releases the dcache's owning `Arc` from an `rcu_call`
/// callback; the final `Arc` drop invokes `free_tracked::<Dentry>()`. See
/// "Dentry Allocation — Nucleus Tracked Storage" below for the full
/// protocol, instance budget, and shrinker interaction.
///
/// **Concurrency**: Dentries are RCU-protected for lockless path resolution
/// (RCU-walk mode, Section 14.1.3). Mutations (create, unlink, rename)
/// acquire the parent dentry's `d_lock` spinlock.
///
/// `#[repr(C)]` on `Dentry` is for deterministic field ordering (cache line
/// layout control), not for cross-compilation-unit ABI stability. Inner types
/// (`DentryName`, `RcuCell<..>`, `IntrusiveList<..>`) retain Rust-default layout.
/// Tier 1 drivers never receive raw `Dentry` pointers — all access is through
/// the VFS ring protocol by inode number.
// kernel-internal, not KABI — no const_assert (contains Rust-layout inner types).
#[repr(C)]
pub struct Dentry {
    /// The name of this directory entry (the final component, not the full path).
    /// Inline for short names (<=32 bytes); heap-allocated for longer names.
    /// Immutable after creation (renames create a new dentry).
    pub d_name: DentryName,

    /// Inode that this dentry points to. `None` for negative dentries
    /// (cached "does not exist" results). Set once when lookup or create instantiates the dentry
    /// after a successful lookup or create. Protected by RCU for readers;
    /// `d_lock` for writers.
    pub d_inode: RcuCell<Option<Arc<Inode>>>,

    /// Parent dentry. The root dentry's parent is itself.
    /// Protected by RCU (for RCU-walk path resolution).
    pub d_parent: RcuCell<Arc<Dentry>>,

    /// Hash table linkage for dcache lookup (keyed by parent + name hash).
    pub d_hash: HashListNode,

    /// Children list (subdirectories and files in this directory).
    /// Only meaningful for directory dentries. Protected by `d_lock`.
    pub d_children: IntrusiveList<Dentry>,

    /// Sibling linkage (entry in parent's `d_children` list).
    pub d_sibling: IntrusiveListNode,

    /// Per-dentry spinlock. Protects `d_children`, `d_inode` mutations,
    /// and `d_flags` updates. Lock level: DENTRY_LOCK (level 190,
    /// [Section 3.4](03-concurrency.md#cumulative-performance-budget--lock-ordering)).
    pub d_lock: SpinLock<(), DENTRY_LOCK>,

    /// Dentry flags (DCACHE_MOUNTED, DCACHE_NEGATIVE, etc.).
    pub d_flags: AtomicU32,

    /// Cross-namespace mount refcount. Counts how many mount namespaces have
    /// a mount at this dentry. Incremented in `mount_filesystem()`, decremented in
    /// `do_umount()`. `DCACHE_MOUNTED` is cleared only when this reaches 0.
    ///
    /// **Why needed**: A single dentry can be a mount point in multiple
    /// namespaces simultaneously (e.g., "/" is mounted in every namespace
    /// that cloned the mount tree). Without this refcount, `do_umount()` in
    /// one namespace would clear `DCACHE_MOUNTED` and break path resolution
    /// in all other namespaces that still have mounts at this dentry.
    ///
    /// **Protocol**:
    /// - `attach_mount()` mountpoint accounting: `dentry.d_mount_refcount.fetch_add(1, Relaxed)`
    ///   THEN `dentry.d_flags.fetch_or(DCACHE_MOUNTED, Release)`
    ///   ([Section 14.6](#mount-tree-data-structures-and-operations--attachmount-attach-a-constructed-mount-into-the-tree)).
    /// - `do_umount()` step 9:
    ///   `if dentry.d_mount_refcount.fetch_sub(1, AcqRel) == 1 {`
    ///   `    dentry.d_flags.fetch_and(!DCACHE_MOUNTED, Release);`
    ///   `}`
    /// - Same pattern in `do_umount_tree()` step 3d, `do_move_mount()` step 5c.
    ///
    /// u32: bounded by mount_max × max_namespaces. Even with 100K mounts ×
    /// 100K namespaces, the per-dentry count is bounded by namespace count
    /// (~100K), well within u32 range.
    pub d_mount_refcount: AtomicU32,

    /// Reference count. Dentries with refcount > 0 are pinned (in use).
    /// Dentries with refcount == 0 are on the LRU and may be evicted
    /// under memory pressure.
    /// u32: bounded by max_files sysctl (default 8M). At max_files=8M
    /// concurrent references to a single dentry, u32 provides ~536x
    /// headroom. AtomicU64 rejected: hot-path refcount, 2x width penalty
    /// on ILP32 architectures (ARMv7, PPC32).
    pub d_refcount: AtomicU32,

    /// Cached traverse-permission entry for fast path resolution (see
    /// "Capability checks" under Path Resolution for the full design).
    /// Layout: bits [63:12] = 52-bit subject tag (credential
    /// permission-identity hash ⊕ spread LSM policy generation),
    /// bit [3] = VALID, bits [2:0] = union of slow-path-validated rwx
    /// grants. 0 = empty/invalidated. AtomicU64 (not U32): the subject
    /// tag must be wide enough that a colliding credential cannot be
    /// minted (see the collision analysis in the Path Resolution
    /// section); 32-bit targets pay a doubleword atomic load here, the
    /// same cost profile as `Inode.i_meta_generation` on the stat fast
    /// path.
    pub cached_perm: AtomicU64,

    /// Superblock this dentry belongs to.
    pub d_sb: Arc<SuperBlock>,

    /// Filesystem-specific dentry operations (d_revalidate, d_release, etc.).
    /// Set by the filesystem during lookup. NULL for simple filesystems.
    pub d_ops: Option<&'static dyn DentryOps>,

    /// RCU head for deferred freeing.
    pub d_rcu: RcuHead,

    /// LRU list linkage for dcache reclaim.
    pub d_lru: IntrusiveListNode,

    /// Mount point generation counter. Incremented when a filesystem is
    /// mounted or unmounted on this dentry. Used by RCU-walk to detect
    /// mount table changes during lockless traversal. This is a generation
    /// counter protocol, not a Linux-style seqcount (no even/odd semantics).
    ///
    /// Reader protocol: (1) sample d_mount_seq with Acquire, (2) lookup in
    /// mount hash table, (3) sample d_mount_seq again with Acquire, (4) if
    /// values differ, retry from step 1.
    pub d_mount_seq: AtomicU32,
}

/// Short name inline buffer size. Names <=32 bytes are stored inline
/// in the dentry (no heap allocation). Covers >99% of real filenames.
pub const DENTRY_INLINE_NAME_LEN: usize = 32;

/// Maximum dentry name length (POSIX NAME_MAX).
pub const DENTRY_MAX_NAME_LEN: usize = 255;

/// Dentry name: inline for short names, heap-allocated for long names.
/// The Heap variant stores names up to DENTRY_MAX_NAME_LEN bytes; the
/// bound is enforced by dentry_alloc() which validates name.len() <= NAME_MAX
/// before construction. debug_assert!(name.len() <= DENTRY_MAX_NAME_LEN)
/// in the Heap constructor provides defense-in-depth.
pub enum DentryName {
    Inline { buf: [u8; DENTRY_INLINE_NAME_LEN], len: u8 },
    Heap { ptr: Box<[u8]> },
}

14.1.2.4.1.3 AddressSpace (Page Cache Mapping)

bitflags! {
    /// Mapping-level state bits stored in `AddressSpace::flags`. Bit
    /// positions are stable — other spec text cites them by number.
    pub struct AddressSpaceFlags: u32 {
        /// Pages must not be reclaimed under memory pressure (e.g. ramfs,
        /// tmpfs locked pages).
        const UNEVICTABLE = 0x01;
        /// Pages are balloon-inflated and may be reclaimed by the balloon
        /// driver at any time.
        const BALLOON     = 0x02;
        /// A writeback error occurred; subsequent `fsync` calls must return
        /// `-EIO` until the bit is cleared. Writeback-error quick-check bit —
        /// see the dual-error-reporting rule on `AddressSpace::flags`.
        const EIO         = 0x04;
        /// A writeback error occurred because no space remains on the
        /// device. Writeback-error quick-check bit — see the
        /// dual-error-reporting rule on `AddressSpace::flags`.
        const ENOSPC      = 0x08;
        /// Direct Access mapping: file data lives in persistent memory and
        /// is mapped directly with no page cache. Set once during inode
        /// instantiation per [Section 15.16](15-storage.md#persistent-memory) (`-o dax=always`, or
        /// `-o dax=inode` + `FS_DAX_FL`), immutable for the inode lifetime.
        /// When set: `page_cache` is `None`, the fault path dispatches to
        /// `dax_iomap_fault()`, and `writepages`/`writepage` are never
        /// called. This is the mapping-level authority; `InodeFlags::DAX`
        /// mirrors it at instantiation.
        const DAX         = 0x10;
    }
}

/// VFS-layer page cache wrapper for one inode. Wraps a `PageCache`
/// ([Section 4.4](04-memory.md#page-cache)) with VFS-layer writeback
/// coordination, error tracking, and filesystem-specific operations.
///
/// Each inode for a regular file or block device has exactly one
/// `AddressSpace`. Directories and symlinks typically do not use
/// `AddressSpace` unless the filesystem maps their data through the page
/// cache (e.g., directories in ext4 are page-cache-backed).
///
/// **Storage**: `AddressSpace` is embedded directly inside `Inode`
/// (field `i_mapping`). No separate allocation is needed on the fast
/// path.
///
/// **Concurrency**:
/// - `page_cache`: `Option<PageCache>` — `Some` for normal files, `None` for
///   DAX files (`AddressSpaceFlags::DAX` set). When `Some`, the inner XArray provides RCU-safe
///   lock-free reads and per-instance `xa_lock` for writers. See
///   [Section 4.4](04-memory.md#page-cache) for the full concurrency model.
///   All code paths that access `page_cache` must check `is_some()` first;
///   DAX paths bypass the page cache entirely.
/// - `page_cache.nr_pages`, `page_cache.nr_dirty`, `nrwriteback`: independent
///   atomic counters; no lock needed for individual increments/decrements
///   (`nr_pages` and `nr_dirty` only exist when `page_cache` is `Some`).
/// - Writeback mutual exclusion: the single authority for "a writeback
///   agent owns this inode" is the `I_WRITEBACK` bit in `Inode::i_state`,
///   acquired and released exclusively by the `writeback_single_inode()`
///   CAS protocol ([Section 4.6](04-memory.md#writeback-subsystem)). This mirrors Linux, where
///   the `I_SYNC` bit in `inode->i_state` is the one writeback-ownership
///   flag (`fs/fs-writeback.c` `writeback_single_inode()` /
///   Linux `inode_wait_for_writeback()`). `writeback_lock` below is NOT a
///   second authority — see its field doc. Reclaim and eviction test
///   `i_state & I_WRITEBACK` directly (one atomic load); there is no
///   separate "writeback in progress" boolean.
pub struct AddressSpace {
    /// Back-pointer to the owning inode. `Weak` to avoid a reference
    /// cycle (Inode → AddressSpace → Inode).
    pub host: Weak<Inode>,

    /// Page storage backend — XArray with RCU-safe lock-free reads and
    /// per-instance `xa_lock` for writers. Defined in [Section 4.4](04-memory.md#page-cache).
    /// `page_cache.nr_pages` and `page_cache.nr_dirty` are the canonical
    /// page/dirty counters (no separate copies here — use accessors).
    /// None for DAX-capable filesystems that map persistent memory directly.
    pub page_cache: Option<PageCache>,

    /// Number of pages currently under active writeback I/O. A page is
    /// counted here from the moment writeback I/O is submitted until the
    /// I/O completion handler clears the `PageFlags::WRITEBACK` flag.
    pub nrwriteback: AtomicU64,

    /// Writeback error sequence counter. Updated on I/O errors via
    /// `ErrSeq::set_err(errno)`. Each open file descriptor snapshots
    /// `wb_err` at open time (`file.f_wb_err`); `fsync()` compares the
    /// snapshot to detect new errors. See [Section 14.4](#vfs-fsync-and-cow).
    pub wb_err: ErrSeq,

    /// Interior-mutability container for the per-inode writeback cursor
    /// (`WritebackState`). NOT a serialization authority: at most one
    /// concurrent writeback agent is permitted per `AddressSpace` (to
    /// avoid seek amplification on rotational storage and to simplify
    /// error propagation), and that mutual exclusion is provided SOLELY
    /// by the `I_WRITEBACK` CAS in `writeback_single_inode()`
    /// ([Section 4.6](04-memory.md#writeback-subsystem)).
    ///
    /// **Protocol**: (1) win the `I_DIRTY* → I_WRITEBACK` CAS on
    /// `inode.i_state`; (2) lock `writeback_lock` to read/advance the
    /// cursor; (3) submit I/O; (4) unlock; (5) clear `I_WRITEBACK`
    /// (`Release`) and wake `inode_waitqueue()` sleepers. Because step 1
    /// admits exactly one agent, this Mutex is uncontended by invariant
    /// (`debug_assert!(try_lock succeeds)` is legal); it exists to give
    /// the agent mutable access to `WritebackState` through `&Inode`.
    ///
    /// Per-inode scope only: multiple inodes on the same backing device
    /// write back concurrently — `BdiWriteback`
    /// ([Section 4.6](04-memory.md#writeback-subsystem--writeback-thread-organization))
    /// coordinates device-level I/O scheduling across inodes, not
    /// per-inode serialization. Two agents owning `I_WRITEBACK` on two
    /// different inodes may both submit bios to the same block device —
    /// correct and desirable for throughput.
    pub writeback_lock: Mutex<WritebackState>,

    /// Sequence counter for truncation-fault coordination.
    ///
    /// Replaces Linux's `mapping->invalidate_lock` (rwsem, added v5.15,
    /// commit 730633f0b7f9) with a lockless seqcount. Writers (truncate,
    /// hole-punch, collapse-range) bracket page cache mutations with
    /// `invalidate_begin()` / `invalidate_end()` while holding
    /// `I_RWSEM(write)`. Readers (page fault) call `read_begin()` before
    /// page cache lookup and `read_check()` after PTE installation -- two
    /// atomic loads, no lock acquired.
    ///
    /// The seqcount eliminates the ONLY lock ordering exception that was
    /// previously required in the page fault path (`VMA_LOCK(105)` ->
    /// the former level-90 invalidation lock violated descending-level order). With
    /// `InvalidateSeq`, the fault path lock chain is strictly ascending:
    /// `VMA_LOCK(105, read)` -> `PAGE_LOCK(180)` -> `PTL(185)`.
    ///
    /// See [Section 4.8](04-memory.md#virtual-memory-manager--invalidateseq-lockless-truncation-fault-coordination)
    /// for the full struct definition, memory ordering table, and edge
    /// case analysis.
    pub invalidate_seq: InvalidateSeq,

    /// Filesystem-provided callbacks for page cache operations.
    /// Rebind-mutable binding cell (`AspaceOpsBinding`, defined with the
    /// bind-time resolution in this section): written exactly by two
    /// writers — `AddressSpace` instantiation (seeded from
    /// `Mount::resolve_aspace_ops()`) and the domain REBIND protocol's
    /// re-resolution under its traffic quiesce (tier promotion/demotion,
    /// crash-recovery re-bind — the ruled rebind model). Readers
    /// auto-deref (`mapping.ops.read_page(...)`) at zero cost; no reader
    /// executes concurrently with a rebind write — the quiesce's resume
    /// edge publishes the new binding. (An earlier revision declared this
    /// `&'static dyn AddressSpaceOps`, "never changes" — which made the
    /// mandated rebind re-resolution unimplementable and could not point
    /// at the mount-owned `RingMountAddressSpaceOps` at all.)
    pub ops: AspaceOpsBinding,

    /// Mapping-level state bits. Typed as `AddressSpaceFlags` (defined
    /// above); storage is an `AtomicU32`, accessed only through the
    /// `flags_load` / `flags_fetch_or` / `flags_fetch_and` helpers so no
    /// call site handles a bare bit mask.
    ///
    /// **Dual error reporting**: the `AddressSpaceFlags::EIO`/`ENOSPC` bits
    /// and `wb_err` (ErrSeq) serve complementary purposes. The bits provide a
    /// quick boolean "any error occurred?" check used by `sync_file_range()`
    /// and the writeback scanner. The ErrSeq counter provides per-fd error
    /// tracking so that multiple concurrent `fsync()` callers each see the
    /// error exactly once. Both are set atomically in `writeback_end_io()`.
    /// This dual mechanism matches Linux 4.13+ semantics (commit 5660e13d).
    ///
    /// **DAX consequences**: when `AddressSpaceFlags::DAX` is set,
    /// `page_cache` is `None` — no `PageCache` is allocated for DAX files.
    /// Direct-access files use CPU load/store through the DAX mapping
    /// ([Section 15.16](15-storage.md#persistent-memory--design-dax-direct-access-integration)),
    /// bypassing the page cache entirely. This saves ~256 bytes per DAX
    /// inode (the `PageCache` struct including its embedded XArray root,
    /// counters, and xa_lock).
    pub flags: AtomicU32,

    /// DAX error generation counter. Only meaningful when `flags` contains
    /// `AddressSpaceFlags::DAX`. DAX files bypass the page cache, so the standard `wb_err`
    /// mechanism (which tracks writeback I/O errors on page cache pages)
    /// does not apply. Instead, hardware-detected errors on persistent
    /// memory (MCE on x86, SEA on ARM64) are recorded here.
    ///
    /// Error propagation for DAX files:
    /// - MCE/SEA → `SIGBUS` to the accessing process (immediate, via the
    ///   page fault / machine-check handler).
    /// - MCE/SEA → increment `dax_err` generation (for deferred `fsync`
    ///   reporting).
    /// - `fsync()` on a DAX file: compare `file.f_dax_err` with
    ///   `mapping.dax_err`. If generations differ, return `-EIO`. This is
    ///   the same generation-counter protocol used by `wb_err` for non-DAX
    ///   files ([Section 4.6](04-memory.md#writeback-subsystem)),
    ///   but applied to DAX hardware errors instead of writeback I/O errors.
    /// - `f_dax_err` is snapshotted at `open()` time, identical to `f_wb_err`.
    ///
    /// For non-DAX files (`AddressSpaceFlags::DAX` clear), this field is unused (reads as 0).
    pub dax_err: AtomicU32,

    /// Interval tree of file-backed VMAs mapping this file. Used for
    /// reverse mapping: truncation, writeback, page migration, and KSM
    /// need to find all VMAs mapping a given file offset range. This is
    /// the UmkaOS equivalent of Linux's `address_space.i_mmap` (`rb_root_cached`
    /// interval tree) protected by `i_mmap_rwsem`.
    ///
    /// The `RwLock` protects concurrent insert/remove during mmap/munmap
    /// (writers) vs. read during truncation/writeback/rmap walks (readers).
    /// Lock level: `I_MMAP_LOCK`, level 106
    /// ([Section 3.4](03-concurrency.md#cumulative-performance-budget--lock-ordering)) — acquired
    /// under `VM_LOCK(100, write)` (establish_mapping/vma_merge) or additionally
    /// under `VMA_LOCK(105, write)` (destroy_mapping per-VMA teardown), and by
    /// truncate under `I_RWSEM(80)`. Readers (rmap walks) acquire
    /// `i_mmap.read()` independently, then `PTL(185)` — all ascending.
    ///
    /// `IntervalTree<VmaRef>` stores `(start_pgoff, end_pgoff, VmaRef)`
    /// tuples, node storage intrusive in the entries (no allocation under
    /// the lock). Access goes THROUGH the guard — the guard derefs to the
    /// tree. Lookup: `let imap = mapping.i_mmap.read();
    /// imap.query(pgoff_start, pgoff_end)` returns all VMAs whose file
    /// offset range overlaps `[pgoff_start, pgoff_end)`. (There is no
    /// `.tree` field on the `RwLock` — `i_mmap.tree.query(...)` does not
    /// compile.)
    pub i_mmap: RwLock<IntervalTree<VmaRef>>,
}

impl AddressSpace {
    /// Typed read of the mapping-state bits.
    #[inline]
    pub fn flags_load(&self, order: Ordering) -> AddressSpaceFlags {
        AddressSpaceFlags::from_bits_truncate(self.flags.load(order))
    }

    /// Atomically set the given bits; returns the previous value.
    #[inline]
    pub fn flags_fetch_or(&self, flags: AddressSpaceFlags, order: Ordering)
        -> AddressSpaceFlags
    {
        AddressSpaceFlags::from_bits_truncate(self.flags.fetch_or(flags.bits(), order))
    }

    /// Atomically mask the bits with `flags` (clearing every bit not in it);
    /// returns the previous value. Callers pass the complement they wish to
    /// retain, e.g.
    /// `mapping.flags_fetch_and(!AddressSpaceFlags::EIO, Release)`.
    #[inline]
    pub fn flags_fetch_and(&self, flags: AddressSpaceFlags, order: Ordering)
        -> AddressSpaceFlags
    {
        AddressSpaceFlags::from_bits_truncate(self.flags.fetch_and(flags.bits(), order))
    }
}

/// DAX File Handling
///
/// DAX (Direct Access) files on persistent memory bypass the page cache
/// entirely. When a filesystem is mounted with `-o dax` on a persistent
/// memory device, every inode's `AddressSpace` has `AddressSpaceFlags::DAX` set and
/// `page_cache` is `None`.
///
/// **Memory savings**: Skipping `PageCache` allocation saves ~256 bytes per
/// DAX inode (XArray root node, `nr_pages`/`nr_dirty` counters, `xa_lock`,
/// internal bookkeeping). On a persistent memory filesystem with millions
/// of small files, this is significant.
///
/// **Error tracking**: DAX files cannot use the standard `wb_err` writeback
/// error mechanism because there are no page cache pages and no writeback
/// I/O. Instead, hardware memory errors (MCE on x86-64, Synchronous
/// External Abort on AArch64) are tracked via `AddressSpace::dax_err`:
///
///   1. Hardware detects uncorrectable error on persistent memory address.
///   2. MCE/SEA handler delivers `SIGBUS` (`BUS_MCEERR_AR` for synchronous,
///      `BUS_MCEERR_AO` for asynchronous) to the process whose access
///      triggered the fault. This is immediate — the process is notified
///      before `fsync` is ever called.
///   3. MCE/SEA handler increments `mapping.dax_err` (AtomicU32 generation
///      counter, same wrap-around protocol as `ErrSeq`).
///   4. On `fsync()`: the VFS compares `file.f_dax_err` (snapshotted at
///      `open()`) with `mapping.dax_err`. If they differ, `fsync` returns
///      `-EIO` and advances `file.f_dax_err` to the current generation
///      (so the error is reported exactly once per fd, matching `wb_err`
///      semantics).
///
/// **Dirty page throttling**: `balance_dirty_pages()` excludes DAX files.
/// DAX writes go directly to persistent memory via CPU store instructions
/// — there are no dirty page cache pages to throttle. Write bandwidth is
/// bounded by the persistent memory device's write throughput, not by the
/// kernel's dirty page ratio. The `writeback_lock` and `nrwriteback`
/// fields (and the `I_WRITEBACK` state bit) are unused for DAX inodes.
///
/// **Page fault path**: When a DAX file is faulted, the VFS calls
/// `dax_iomap_fault()` (not `file_backed_fault()`). This maps the persistent
/// memory physical address directly into the process's page table — no
/// page allocation, no page cache insertion, no copy. For huge page faults
/// (PMD-level, 2 MiB on x86-64), `dax_iomap_pmd_fault()` maps a single
/// PMD entry covering the entire 2 MiB region.

/// Per-inode writeback cursor embedded inside `AddressSpace::writeback_lock`.
///
/// Protected by `AddressSpace::writeback_lock`. Single-agent access is
/// guaranteed upstream by the `I_WRITEBACK` CAS (see the `writeback_lock`
/// field doc) — the Mutex is the interior-mutability container, not the
/// exclusion mechanism. The fields track progress so that the next
/// writeback pass resumes where the previous one left off.
pub struct WritebackState {
    /// Next page index to examine during writeback. The writeback agent
    /// advances this forward as pages are submitted for I/O. Wraps to 0
    /// after reaching the last page, implementing a cyclic scan
    /// consistent with the kernel's periodic writeback policy.
    pub writeback_index: u64,

    /// Accumulated bytes of dirty data at the time writeback started.
    /// Used to limit how much data a single writeback pass writes, so
    /// that a continuous dirty stream does not starve readers.
    pub dirty_bytes: u64,
}

/// Filesystem callbacks invoked by the VFS page cache layer.
///
/// Each filesystem that participates in the page cache provides a
/// static `AddressSpaceOps` implementation. The VFS calls these methods
/// when it needs to populate the cache (read miss), flush dirty pages
/// (writeback), or decide whether a page can be dropped (reclaim).
///
/// **Object safety**: all methods take `&self` on the ops vtable plus
/// explicit `AddressSpace`/`Page` references. The vtable itself is
/// `'static`, `Send`, and `Sync`.
pub trait AddressSpaceOps: Send + Sync {
    /// Read one page (identified by `index`, a page-aligned file offset
    /// divided by `PAGE_SIZE`) from the backing store into the page
    /// cache. Fill the already-allocated and cache-inserted `page` with
    /// data from backing store. The page has already been allocated,
    /// locked (`PageFlags::LOCKED`), and inserted into the page cache
    /// XArray by the caller (`filemap_get_pages`). The filesystem must
    /// initiate the I/O to populate the page contents.
    ///
    /// **Contract**: Implementations MUST NOT allocate a new page or
    /// overwrite the page cache XArray slot. The caller owns the slot;
    /// overwriting it orphans the locked page and deadlocks concurrent
    /// readers waiting on `PageFlags::LOCKED`.
    ///
    /// The `page` parameter is a bare `&Page` (not `&Arc<Page>`): `Page`
    /// is a memmap frame descriptor and is never `Arc`-allocated. Liveness
    /// of the frame across the call is guaranteed by the caller's
    /// `FillLease` (or the Core in-flight entry the lease moved into) —
    /// the fill lifecycle owns the allocation reference and the page LOCK
    /// for the duration of the fill ([Section 4.4](04-memory.md#page-cache)).
    ///
    /// **Synchronous-fill contract**: `Ok(())` means the page data is FULLY
    /// VALID — the fill completed synchronously before return (e.g. via
    /// `bio_submit_and_wait`). The dispatch seam (`dispatch_read_page`, below)
    /// discharges the fill obligation (`FillCompletion::complete_ok`) on
    /// `Ok(())`; `Err(e)` leaves the obligation with the seam for its error
    /// policy. A provider that fills ASYNCHRONOUSLY (submits I/O and returns
    /// BEFORE the data lands) MUST NOT signal completion through `read_page` —
    /// returning `Ok(())` early would publish `UPTODATE` before the data
    /// arrives. Such providers override `read_page_async` (below), which takes
    /// ownership of the `FillCompletion` so the completion path discharges it
    /// exactly once.
    ///
    /// Called with no locks held. The implementation may block.
    fn read_page(
        &self,
        mapping: &AddressSpace,
        index: u64,
        page: &Page,
    ) -> Result<(), IoError>;

    /// **Optional** asynchronous-fill variant for providers that submit the
    /// fill I/O and return WITHOUT waiting for the data (a block-backed
    /// filesystem issuing an asynchronous `Bio`, or a cross-domain provider
    /// submitting a `ReadPage` on the VFS ring). Ownership of the linear
    /// `FillCompletion` obligation MOVES into this call: the provider stows it
    /// on its async carrier — `PageFillBioCtx.completion` for the same-domain
    /// `Bio` path ([Section 4.4](04-memory.md#page-cache)), or `VfsInflightEntry.fill` for the
    /// cross-domain ring path ([Section 14.2](#vfs-ring-buffer-protocol)) — and the I/O
    /// completion worker (or the ring response drain) discharges it exactly
    /// once, publishing `UPTODATE`-or-`ERROR` and unlocking from completion
    /// context.
    ///
    /// - `Ok(())`: the fill was ENROLLED on an async carrier that now owns the
    ///   obligation. The page is still `LOCKED`/not-`UPTODATE`; the completion
    ///   publishes its terminal state later. The caller must NOT touch `fill`
    ///   again.
    /// - `Err((fill, e))`: submission FAILED before any enrollment (e.g. the
    ///   inode was freed, a bio/context slab allocation failed, or the ring
    ///   boundary rejected the request). Ownership of the still-undischarged
    ///   `fill` returns to the caller.
    ///
    /// Providers that fill synchronously do NOT override this — the default
    /// returns the obligation with `EOPNOTSUPP`, and the dispatch seam falls
    /// back to the synchronous `read_page` path. NEVER called directly by
    /// filemap/fault code — only through `dispatch_read_page` (below), which
    /// owns the fall-back decision.
    fn read_page_async(
        &self,
        _mapping: &AddressSpace,
        _index: u64,
        _page: &Page,
        fill: FillCompletion,
    ) -> Result<(), (FillCompletion, IoError)> {
        Err((fill, IoError::new(Errno::EOPNOTSUPP)))
    }

    /// The mount-scoped DMA grant table backing this `AddressSpace`'s
    /// cross-domain demand fills, or `None` for a same-domain provider (which
    /// grants nothing — its fills land via a `Bio` in the shared domain).
    /// `RingMountAddressSpaceOps`
    /// ([Section 14.2](#vfs-ring-buffer-protocol--ringmount-addressspaceops-provider))
    /// overrides this to return `Some(&self.grant)`; the ring response drain
    /// (`vfs_complete_read_pages`, [Section 14.2](#vfs-ring-buffer-protocol)) calls it to
    /// revoke a page's grant BEFORE publishing the page readable. Default:
    /// `None`.
    fn dma_grant_table(&self) -> Option<&FsDmaGrantTable> {
        None
    }

    /// **Example: ext4 synchronous `read_page()` flow**
    ///
    /// When a file-backed page fault reaches the dispatch seam
    /// (`dispatch_read_page`, below) and ext4 is a sync-only provider, the seam
    /// calls `read_page()`:
    ///
    /// 1. `ext4_read_page(mapping, pgoff, page)`:
    ///    a. Map logical block: `ext4_map_blocks(inode, pgoff)` → translates file offset
    ///       to physical block number via the extent tree.
    ///    b. Build Bio: `Bio::new_read(bdev, phys_block, page)`.
    ///    c. Submit AND WAIT: `bio_submit_and_wait(bio)` blocks until the device
    ///       DMA has landed the file data into `page`.
    ///    d. Do NOT unlock the page or publish `UPTODATE`. On `Ok(())` the
    ///       caller-side dispatch seam's `FillCompletion::complete_ok()` owns BOTH
    ///       the unlock and the single `UPTODATE` publish (the synchronous-fill
    ///       contract on `read_page` above). The page is still LOCKED and
    ///       not-`UPTODATE` when `read_page` returns.
    ///    e. Return `Ok(())` — the page holds the file data; the seam publishes it.
    ///
    /// The readahead engine ([Section 4.4](04-memory.md#page-cache--readahead-engine)) may
    /// batch multiple pages into a single Bio with scatter-gather, submitting them
    /// via `AddressSpaceOps::readahead()` instead of individual `read_page()` calls.
    /// A provider that fills ASYNCHRONOUSLY overrides `read_page_async` instead
    /// (above), moving the obligation onto its completion carrier so the
    /// completion — not `read_page` — publishes the terminal page state.

    /// Read multiple pages as a batch for readahead. Receives the readahead
    /// window from the readahead engine ([Section 4.4](04-memory.md#page-cache--readahead-engine)).
    /// Implementations should submit I/O for all requested pages in a single
    /// Bio batch. Filesystems that do not implement this method fall back to
    /// per-page fills through the dispatch seam (`dispatch_read_page`), which
    /// drives each page's async `read_page_async` or synchronous `read_page`
    /// arm — raw `read_page` is never invoked directly.
    /// Default: returns `EOPNOTSUPP` (the readahead engine falls back to the
    /// per-page dispatch seam).
    fn readahead(
        &self,
        mapping: &AddressSpace,
        ra: &ReadaheadControl,
    ) -> Result<(), IoError> {
        Err(IoError::new(Errno::EOPNOTSUPP))
    }

    /// Physical location of the extent backing page `index` of `mapping`, for
    /// the shared-extent page cache (reflink/CoW/RoW filesystems). Consulted by
    /// `page_cache_get_or_fill()`'s miss path via `phys_extent_cache_pin()`
    /// BEFORE allocating a frame ([Section 4.4](04-memory.md#page-cache)), so a reflinked file read
    /// OR mmap fault hits an extent already resident under another inode instead
    /// of re-reading it. Returns `None` for holes, unallocated ranges, and
    /// inline data. `index` is a page index (byte offset / `PAGE_SIZE`).
    ///
    /// Default: `None` — non-sharing filesystems (ext4 without reflinks, tmpfs,
    /// ramfs) inherit the default and pay one branch on the miss path only.
    /// CoW/RoW filesystems (XFS / ext4 with reflinks, btrfs) implement it; the
    /// `PHYS_EXTENT_CACHE`, shared-frame refcount, and insertion rules — and the
    /// `phys_extent_cache_pin()` body that consumes this hook — live in
    /// [Section 14.4](#vfs-fsync-and-cow).
    fn extent_phys_addr(&self, mapping: &AddressSpace, index: u64) -> Option<PhysExtent> {
        None
    }

    /// Write a single dirty page to the backing store. `wbc` carries
    /// writeback control parameters (sync mode, range limits, number
    /// of pages already written in this pass).
    ///
    /// **Submitter-owned state — the implementation touches none of it**: the
    /// writeback submitter that calls `writepage` (the per-page fallback in
    /// `writeback_inode_pages()` and the fsync per-page fallback,
    /// [Section 4.6](04-memory.md#writeback-subsystem), [Section 14.4](#vfs-fsync-and-cow)) performs the ENTIRE
    /// submission-side transition under the page lock BEFORE this call: it clears
    /// `PageFlags::DIRTY` on the frame (H1: `Page.flags` is the sole DIRTY authority) and
    /// the XArray dirty tag, decrements `nr_dirty` on that 1→0 edge (both the
    /// per-inode `PageCache.nr_dirty` and the per-BDI `BdiWriteback.nr_dirty`),
    /// sets `PageFlags::WRITEBACK`, and increments the writeback counters (`nrwriteback`
    /// + per-BDI `nr_writeback`). The implementation MUST NOT touch `PageFlags::DIRTY`,
    /// `PageFlags::WRITEBACK`, `nr_dirty`, `nrwriteback`, or `nr_writeback` — it only
    /// submits the I/O. The completion epilogue
    /// ([Section 15.2](15-storage.md#block-io-and-volume-management--writeback-io-completion-callback))
    /// clears `PageFlags::WRITEBACK` and decrements the writeback counters but NEVER
    /// touches `PageFlags::DIRTY` or `nr_dirty` (those were edge-guarded at submission).
    /// Clearing `PageFlags::DIRTY` or decrementing `nr_dirty` here would double-count
    /// against the submitter and corrupt the throttle.
    ///
    /// Called with no locks held. The implementation may block.
    fn writepage(
        &self,
        mapping: &AddressSpace,
        page: &Page,
        wbc: &WritebackControl,
    ) -> Result<(), IoError>;

    /// Write multiple dirty pages to the backing store in a single batch.
    /// Called by the writeback subsystem ([Section 4.6](04-memory.md#writeback-subsystem)) instead of
    /// iterating `writepage()` one page at a time. The filesystem should submit
    /// all dirty pages in the address space (subject to `wbc` constraints) as
    /// coalesced Bio requests for maximum throughput.
    ///
    /// # Returns
    /// - `Ok(n)`: Number of pages successfully submitted for writeback.
    /// - `Err(IoError)`: Fatal error; writeback aborted for this inode.
    ///
    /// Default: returns `EOPNOTSUPP` (writeback layer falls back to per-page
    /// `writepage()` calls). Filesystems that support extent-based I/O (ext4,
    /// XFS, btrfs) should implement this for 5-10x writeback throughput vs.
    /// per-page writepage on rotational media.
    ///
    /// **All-or-nothing contract**: `writepages()` is a writeback SUBMITTER — it
    /// owns the full submission-side state transition for every page it hands off
    /// to I/O, exactly as the per-page `writepage()` submitter does. For each such
    /// page, under the page lock: clear `PageFlags::DIRTY` on the frame (H1: `Page.flags`
    /// is the sole DIRTY authority) and the XArray dirty tag, decrement `nr_dirty`
    /// on that 1→0 edge (both `PageCache.nr_dirty` and per-BDI
    /// `BdiWriteback.nr_dirty`, edge-guarded on the RMW old value), set
    /// `PageFlags::WRITEBACK`, and increment the writeback counters (`mapping.nrwriteback`
    /// + per-BDI `BdiWriteback.nr_writeback`) — all paired in the same code path.
    /// On error it MUST NOT leave any page it already submitted in a
    /// half-accounted state: either roll the WHOLE transition back for pages that
    /// were tracked but not actually dispatched (re-set `PageFlags::DIRTY` + the dirty
    /// tag, re-increment `nr_dirty`, clear `PageFlags::WRITEBACK`, decrement the writeback
    /// counters), or ensure every page for which `PageFlags::WRITEBACK` remains set has a
    /// matching bio in flight (so the ordinary completion epilogue clears it and
    /// decrements the writeback counters exactly once). Callers that fall back to
    /// per-page `writepage()` after a `writepages()` error (e.g.
    /// [Section 14.4](#vfs-fsync-and-cow)) rely on this: a page is either (a) untouched by
    /// the failed `writepages()` call — safe to submit via the per-page fallback,
    /// which re-runs the confirm-step and its own edge-guarded accounting — or
    /// (b) already in flight under `writepages()`'s own bio — the fallback MUST
    /// NOT re-submit it or re-touch any counter (the Phase-1 fsync fallback skips
    /// pages that already have `PageFlags::WRITEBACK` set for exactly this reason), since
    /// that double-counts the pairwise accounting across the batch and per-page
    /// paths and can underflow `nrwriteback` / `nr_writeback` (u64 wrap) when the
    /// completion epilogue decrements twice for one page. This matches Linux's
    /// Linux `do_writepages()` expects `->writepages` to be all-or-nothing.
    fn writepages(
        &self,
        mapping: &AddressSpace,
        wbc: &WritebackControl,
    ) -> Result<u64, IoError> {
        Err(IoError::new(Errno::EOPNOTSUPP))
    }

    /// Verify data integrity of a page populated through a non-standard
    /// path (RDMA fetch, DSM migration, decompression). Filesystems that
    /// store per-page checksums (btrfs, ext4 metadata, ZFS) implement this
    /// to catch silent corruption from paths that bypass the standard block
    /// I/O checksum pipeline.
    ///
    /// Called by the DSM page fetch path ([Section 6.11](06-dsm.md#dsm-distributed-page-cache))
    /// after RDMA-fetching a page from a remote peer, before setting
    /// `PageFlags::UPTODATE`. If verification fails, the fetched page is
    /// discarded and the DSM falls back to storage I/O.
    ///
    /// Default: returns `Ok(true)` — the page is accepted without
    /// verification (appropriate for filesystems without per-page
    /// checksums, e.g., tmpfs, ext2, NFS).
    fn verify_page(
        &self,
        mapping: &AddressSpace,
        index: u64,
        page: &Page,
    ) -> Result<bool, IoError> {
        Ok(true)
    }

    /// Called by the page reclaimer immediately before a clean page is
    /// removed from the cache. The filesystem may decline eviction by
    /// returning `false` (e.g., because it has pinned the page for
    /// journalling). Returning `true` grants permission to evict.
    ///
    /// Must not block; must not acquire locks that might sleep.
    fn releasepage(&self, page: &Page) -> bool;

    /// Called by `page_cache_write_iter()` before writing user data into
    /// a page. The filesystem prepares the page for writing:
    ///
    /// - **ext4**: starts a JBD2 journal handle (`journal_start()`), allocates
    ///   blocks for delayed allocation, reads the page from disk if the write
    ///   is partial (does not cover the entire page).
    /// - **XFS**: creates a delayed allocation extent reservation.
    /// - **tmpfs**: allocates a swap-backed page.
    /// - **Default (simple filesystems)**: allocates a clean page from the
    ///   page cache if not already present, zeroing unwritten portions.
    ///
    /// The returned page reference is locked (`PageFlags::LOCKED` set).
    /// The caller (`page_cache_write_iter`) copies user data into the
    /// page between `write_begin` and `write_end`.
    ///
    /// On error (e.g., `ENOSPC` from block allocation), the write is aborted
    /// and the page is released without modification.
    ///
    /// See [Section 15.6](15-storage.md#filesystem-ext4) for ext4's implementation.
    fn write_begin(
        &self,
        mapping: &AddressSpace,
        pos: u64,
        len: usize,
        flags: u32,
    ) -> Result<PageRef, IoError>;

    /// Called by `page_cache_write_iter()` after writing user data into
    /// the page returned by `write_begin()`. The filesystem commits the
    /// write:
    ///
    /// - **ext4**: marks buffer heads dirty, stops the JBD2 journal handle
    ///   (`journal_stop()`), updates `i_size` if the write extended the file.
    /// - **XFS**: marks the page dirty, updates extent state.
    /// - **Default (simple filesystems)**: marks the page dirty via
    ///   `set_page_dirty()`.
    ///
    /// `copied` is the number of bytes actually copied by the write (may be
    /// less than `len` for a short copy from user memory). The filesystem
    /// must handle partial writes correctly (e.g., by not advancing `i_size`
    /// past the last successfully written byte).
    ///
    /// The page is still locked on entry; the filesystem may unlock it
    /// before returning.
    ///
    /// **Tier boundary for `set_page_dirty()`**: For Tier 1 filesystems
    /// (ext4, XFS, Btrfs), `write_end()` is invoked via the KABI ring:
    /// the Tier 0 VFS dispatches a `WriteEnd` command to the filesystem's
    /// domain, the filesystem processes it and returns a response. The
    /// response includes a `dirty: bool` flag indicating whether the page
    /// should be marked dirty. The **Tier 0 VFS ring consumer** -- not the
    /// Tier 1 filesystem -- calls `set_page_dirty()` upon receiving a
    /// response with `dirty == true`. This keeps all page cache metadata
    /// operations (`set_page_dirty()`, `nr_dirty` counters, BDI dirty
    /// list) in Tier 0, avoiding cross-domain direct calls from Tier 1.
    ///
    /// For Tier 0 filesystems (tmpfs, ramfs -- statically linked), the
    /// `write_end()` callback runs in the same domain and calls
    /// `set_page_dirty()` directly. No ring dispatch is needed.
    ///
    /// This is the same pattern used for block I/O completion (Tier 1
    /// NVMe driver signals via outbound ring, Tier 0 consumer calls
    /// `bio_complete()`) -- see [Section 12.8](12-kabi.md#kabi-domain-runtime) and
    /// [Section 15.19](15-storage.md#nvme-driver-architecture).
    ///
    /// **Return value**: the number of bytes successfully committed to the
    /// page cache, in the range `[0, copied]`. On full success returns
    /// `copied`. On partial commit (e.g., the filesystem could not allocate
    /// journal space for the full range), returns a prefix byte count; the
    /// page remains dirty for the uncommitted suffix, which will be
    /// re-written on the next writeback cycle.
    ///
    /// **Must NOT exceed `copied`.** If the filesystem cannot commit any
    /// bytes (e.g., journal_stop() failed), it returns `0` and leaves the
    /// page clean (rolls back any dirty marking it performed). The caller
    /// (`page_cache_write_iter`) stops the write loop when the return
    /// value is less than `copied` and reports the short count to
    /// userspace. No data is silently lost: the partial page either stays
    /// dirty for later writeback or was never marked dirty.
    fn write_end(
        &self,
        mapping: &AddressSpace,
        pos: u64,
        len: usize,
        copied: usize,
        page: PageRef,
    ) -> Result<usize, IoError>;

    /// Called by the page cache when a page is first dirtied. Allows the
    /// filesystem to register the affected block extent for crash recovery
    /// journaling BEFORE the page is modified.
    ///
    /// Filesystems with journaling (ext4, btrfs, XFS) implement this to
    /// record dirty extents in their journal. Filesystems without journaling
    /// (tmpfs, ramfs, NFS) leave this as the default no-op.
    ///
    /// Called from `set_page_dirty()` with the page locked. The `offset`
    /// and `len` arguments describe the byte range within the file that
    /// will be dirtied (typically `page_offset` and `PAGE_SIZE`, but
    /// sub-page dirty tracking for large folios may pass smaller ranges).
    ///
    /// **Interaction with two-phase dirty extent protocol**: For Tier 1
    /// VFS drivers running in an isolated domain, the `dirty_extent()`
    /// callback calls `vfs_dirty_extent_reserve()` to register the
    /// logical intent in Core's dirty intent list
    /// ([Section 14.1](#virtual-filesystem-layer)). For in-place filesystems that
    /// know the block address at dirty time (ext4 non-delayed-alloc),
    /// the callback may use `vfs_dirty_extent_reserve_and_commit()` to
    /// atomically reserve and bind the physical address. CoW filesystems
    /// (Btrfs, XFS reflink) call only `vfs_dirty_extent_reserve()` here
    /// and defer `vfs_dirty_extent_commit()` to the writeback path after
    /// block allocation. For Tier 0 filesystems (statically linked, e.g.,
    /// tmpfs), the callback can directly update internal journal
    /// structures without crossing a domain boundary.
    ///
    /// **In-place filesystem implementation pattern** (ext4 example):
    /// 1. `dirty_extent()` is called with the byte range `[offset, offset+len)`.
    /// 2. The filesystem maps the byte range to physical block extents via
    ///    its extent tree.
    /// 3. Calls `vfs_dirty_extent_reserve_and_commit(sb_dev, inode_id, offset,
    ///    len, block_addr, block_len)` — atomic reserve+commit since the block
    ///    address is known.
    /// 4. The filesystem writes a journal descriptor block recording the
    ///    physical extents that are about to be modified.
    /// 5. Only after the journal descriptor is committed (or at least
    ///    queued for commit) does `dirty_extent()` return `Ok(())`.
    /// 6. The caller (`set_page_dirty()`) then sets `PageFlags::DIRTY`
    ///    on the page.
    ///
    /// **CoW filesystem implementation pattern** (Btrfs example):
    /// 1. `dirty_extent()` is called with the byte range `[offset, offset+len)`.
    /// 2. Calls `vfs_dirty_extent_reserve(sb_dev, inode_id, offset, len)` —
    ///    Phase 1 only. No block address is available yet (CoW allocates at
    ///    writeback).
    /// 3. Returns `Ok(())` with the `DirtyExtentToken` stored in the inode's
    ///    per-extent pending-commit table (filesystem-private state).
    /// 4. During writeback, the filesystem allocates new blocks via its
    ///    extent allocator, then calls `vfs_dirty_extent_commit(token,
    ///    block_addr, block_len)` — Phase 2.
    /// 5. After I/O completion, calls `vfs_flush_extent_complete()`.
    ///
    /// This ordering guarantee ensures that on crash recovery, Core has a
    /// record of every dirty extent — no data modification happens without
    /// a corresponding intent entry.
    fn dirty_extent(
        &self,
        _mapping: &AddressSpace,
        _offset: u64,
        _len: u64,
    ) -> Result<(), IoError> {
        // Default: no-op. Appropriate for in-memory filesystems (tmpfs,
        // ramfs) and network filesystems (NFS, which has its own write
        // delegation protocol).
        Ok(())
    }

    /// Returns the direct-I/O implementation for this address space,
    /// if the filesystem supports bypassing the page cache (e.g., for
    /// `O_DIRECT` opens). Returns `None` if direct I/O is not supported;
    /// the VFS will then fall back to the page-cache path.
    fn direct_io(&self) -> Option<&dyn DirectIoOps> {
        None
    }
}

14.1.2.4.1.4 Generic File Operations (VFS-to-Page-Cache Bridge)

The VFS provides generic implementations of file read/write that bridge FileOps calls to the page cache. Most filesystem types delegate their FileOps::read() and FileOps::write() to these generic functions, only providing the AddressSpaceOps callbacks for cache miss I/O.

Isolation domain: filemap_get_pages() runs in Tier 0 (Core domain). The page cache XArray is Core memory. VFS dispatches the read request to Core via the KABI ring; Core's filemap_get_pages() accesses the page cache directly. The ~23-46 cycle domain crossing happens at the VFS-to-Core ring boundary.

/// Maximum number of pages fetched in a single readahead or
/// `filemap_get_pages()` call. 32 pages is a 128 KiB readahead window at a
/// 4 KiB page size — large enough to cover a full media-scan/streaming
/// readahead burst, small enough that the read path uses a fixed-capacity
/// `ArrayVec<PagePin, MAX_READAHEAD_PAGES>` with no hot-path heap
/// allocation. (Same window size as Linux `VM_READAHEAD_PAGES`,
/// `SZ_128K / PAGE_SIZE`, `include/linux/pagemap.h` — a factual parity note,
/// not the reason for the value.)
pub const MAX_READAHEAD_PAGES: usize = 32;

/// Core-VFS dispatch seam for one demand page fill. It is the single bridge
/// from faulter/filemap context — which holds the linear `FillCompletion`
/// obligation — to the provider machinery that actually completes the fill.
/// The obligation MOVES across the transport boundary so it is discharged
/// EXACTLY ONCE by whoever completes the I/O; the caller never publishes page
/// content state (`UPTODATE`/`ERROR`) itself. It is the hitherto-missing caller
/// of the async fill contracts ([Section 4.4](04-memory.md#page-cache), [Section 14.2](#vfs-ring-buffer-protocol)).
///
/// Transport is bound at mount time by `Mount::resolve_aspace_ops` (below,
/// tier-agnostic — the caller never learns the provider's tier or domain). The
/// same-domain and cross-domain arms below are two RESOLUTIONS of that ONE
/// bind-time decision, not two caller-visible paths: `dispatch_read_page`
/// always calls `mapping.ops.read_page_async` and never branches on tier. The
/// bound provider's ops select the arm:
/// - **Same-domain, async-capable provider** (block-backed FS overriding
///   `read_page_async`): the obligation moves into `PageFillBioCtx.completion`
///   ([Section 4.4](04-memory.md#page-cache)); the `blk-io` completion worker discharges it.
/// - **Cross-domain provider** (`RingMountAddressSpaceOps`, the ONE generic
///   ring-mount provider,
///   [Section 14.2](#vfs-ring-buffer-protocol--ringmount-addressspaceops-provider)): its
///   `read_page_async` grants the page, enrolls the obligation on the
///   request's `VfsInflightEntry.fill` ([Section 14.2](#vfs-ring-buffer-protocol)), and
///   submits a `ReadPage` on the VFS ring; the ring response drain
///   (`vfs_complete_read_pages`) revokes the grant and discharges it.
/// - **Same-domain, sync-only provider** (only `read_page`): the unchanged
///   trait fills the page synchronously; on `Ok(())` (page data fully valid)
///   the seam discharges `fill.complete_ok()` itself.
///
/// Both async arms are realized by the provider's `read_page_async` override;
/// the seam is transport-agnostic and does not name the ring — the async
/// carrier (`Bio` vs ring `VfsInflightEntry`) is the provider's choice, bound
/// at mount. This is exactly the Unified Domain Model idiom: the dispatch,
/// bound at bind time, owns enrollment.
///
/// # Ownership (single-owner at every program point)
/// - `Ok(())`: the obligation was ACCEPTED — already discharged (sync fill) or
///   enrolled on an async carrier that now owns the discharge. The caller must
///   NOT touch `fill` again.
/// - `Err((fill, e))`: submission failed BEFORE any enrollment; ownership of
///   the still-undischarged `fill` returns to the caller, which runs its error
///   policy (`complete_err` / short-read). Move semantics make double-discharge
///   unrepresentable: on every path `fill` is either consumed here or handed
///   back in the `Err` payload, never both.
///
/// Warm path (one per demand miss); no heap allocation of its own — the async
/// carrier's slab object is the only allocation, already accounted by the
/// fill-path budget ([Section 4.4](04-memory.md#page-cache)).
pub fn dispatch_read_page(
    mapping: &AddressSpace,
    index: u64,
    pin: &PagePin,
    fill: FillCompletion,
) -> Result<(), (FillCompletion, IoError)> {
    // Prefer the async-capable op. The obligation MOVES in; on `Ok(())` it has
    // been enrolled on the provider's async carrier (same-domain `Bio` ctx, or
    // the cross-domain ring's `VfsInflightEntry.fill`) and the seam is done.
    // The default `read_page_async` returns the obligation with `EOPNOTSUPP`;
    // that (and only that) sentinel means "sync-only provider" — fall back to
    // the synchronous `read_page` and discharge the obligation inline. Any
    // OTHER errno is a genuine async submit failure before enrollment: the
    // obligation is returned to the caller unchanged.
    match mapping.ops.read_page_async(mapping, index, pin.page(), fill) {
        Ok(()) => Ok(()),
        Err((fill, e)) if e.errno() == Errno::EOPNOTSUPP => {
            match mapping.ops.read_page(mapping, index, pin.page()) {
                Ok(()) => {
                    // Synchronous fill complete: page data is fully valid.
                    // Publish UPTODATE, unlock, wake — the single terminal.
                    fill.complete_ok();
                    Ok(())
                }
                // Sync submit failed before any publish — return the obligation.
                Err(e2) => Err((fill, e2)),
            }
        }
        Err((fill, e)) => Err((fill, e)),
    }
}

// ---------------------------------------------------------------------------
// Bind-time AddressSpaceOps resolution
// ---------------------------------------------------------------------------
// Which `AddressSpaceOps` a mount's inodes use is a BIND-TIME transport
// decision — the `kabi_call!` bind-time pattern lifted to ops assignment,
// tier-agnostic. Resolved once at mount bind and re-resolved at every domain
// REBIND under the module-lifecycle rebind quiesce ([Section 13.18](13-device-classes.md#live-kernel-evolution));
// the result is assigned to `mapping.ops` when each `AddressSpace` is
// instantiated (inode setup). There is exactly ONE cross-domain provider — the
// generic `RingMountAddressSpaceOps` ([Section 14.2](#vfs-ring-buffer-protocol)) — never a
// per-FS ring impl.

/// Rebind-mutable holder of an `AddressSpace`'s resolved `AddressSpaceOps`
/// binding. Auto-derefs to the ops trait object, so every call site
/// (`mapping.ops.read_page(...)`, `mapping.ops.dma_grant_table()`) reads
/// through it unchanged at zero cost — one pointer load, no refcount, no
/// lock.
///
/// **Write discipline (the ONLY two writers)**:
///  1. `AddressSpace` instantiation (inode setup): seeded from
///     `Mount::resolve_aspace_ops()`.
///  2. The domain rebind protocol ([Section 13.18](13-device-classes.md#live-kernel-evolution)): re-runs
///     `resolve_aspace_ops()` under the rebind traffic quiesce and calls
///     `rebind()` on every live `AddressSpace` of the mount BEFORE
///     traffic resumes.
/// No reader executes concurrently with a write: the quiesce excludes
/// in-flight operations, and its resume edge (Release/Acquire on the
/// quiesce state) publishes the new pointer — so the non-atomic
/// fat-pointer cell can never be read torn. That protocol-carried
/// exclusion is the `unsafe impl Sync` justification.
///
/// **Pointee lifetime**: the pointee is either a module's native
/// `AddressSpaceOps` (a `'static` vtable instance) or the mount's OWNED
/// `RingMountAddressSpaceOps` (`Mount.ring_aspace_ops`,
/// [Section 14.6](#mount-tree-data-structures-and-operations)); the owner frees a
/// displaced instance only AFTER the rebind quiesce completes — at which
/// point no binding points at it and no reader borrows through it.
pub struct AspaceOpsBinding {
    /// The current resolution. Raw pointer, not `&'static`: the
    /// cross-domain provider is mount-owned, not leaked — a 50-year
    /// uptime with repeated tier moves must not leak one provider
    /// instance per rebind.
    cell: UnsafeCell<*const dyn AddressSpaceOps>,
}

// SAFETY: writes are confined to instantiation and the rebind quiesce (no
// concurrent reader or second writer — write discipline above); reads are
// plain loads of a pointer stable for the whole traffic epoch.
unsafe impl Sync for AspaceOpsBinding {}

impl AspaceOpsBinding {
    /// Seed the binding at `AddressSpace` instantiation.
    pub fn new(ops: &dyn AddressSpaceOps) -> Self {
        Self { cell: UnsafeCell::new(ops as *const dyn AddressSpaceOps) }
    }

    /// Replace the binding. Caller MUST be the rebind protocol, under its
    /// traffic quiesce (write discipline above).
    pub fn rebind(&self, ops: &dyn AddressSpaceOps) {
        // SAFETY: the rebind quiesce excludes every concurrent reader
        // and any second writer.
        unsafe { *self.cell.get() = ops as *const dyn AddressSpaceOps; }
    }
}

impl core::ops::Deref for AspaceOpsBinding {
    type Target = dyn AddressSpaceOps;
    fn deref(&self) -> &Self::Target {
        // SAFETY: the pointee outlives the traffic epoch this read
        // occurs in (pointee-lifetime rule above).
        unsafe { &**self.cell.get() }
    }
}

impl Mount {
    /// Resolve this mount's `AddressSpaceOps` at bind time (and re-run at each
    /// rebind). `mapping.ops` is assigned from this resolution at `AddressSpace`
    /// instantiation. The caller of `dispatch_read_page` never learns the tier;
    /// the same-domain and cross-domain arms are two RESOLUTIONS of this one
    /// bind decision, and runtime tier promotion/demotion rides the existing
    /// rebind machinery.
    ///
    ///  - Same domain as Core: the module's native `AddressSpaceOps` impl
    ///    (direct calls, `Bio`-carrier style — the same-domain async fill arm).
    ///  - Cross domain: the mount's `RingMountAddressSpaceOps`
    ///    ([Section 14.2](#vfs-ring-buffer-protocol--ringmount-addressspaceops-provider)),
    ///    constructed at bind from the mount's ring transport handle
    ///    (`SuperBlock.ring_set`) and an `FsDmaGrantTable` established as part
    ///    of the KABI bind-grant. One instance per ring mount, OWNED by the
    ///    mount in `Mount.ring_aspace_ops`
    ///    ([Section 14.6](#mount-tree-data-structures-and-operations)) — which is why
    ///    the return type borrows from `self` rather than claiming
    ///    `'static`: only the native arm's module vtable instance is
    ///    `'static`; the ring instance lives exactly as long as its owning
    ///    slot, per the `AspaceOpsBinding` pointee-lifetime rule above.
    ///
    /// Rebind re-resolution: on domain rebind (tier promotion/demotion, crash
    /// recovery re-bind) the rebind protocol re-runs this under its quiesce and
    /// repoints every live `AddressSpace`'s binding via
    /// `mapping.ops.rebind(..)` before traffic resumes, so a mount that
    /// moves same-domain/cross-domain switches
    /// between native ops and `RingMountAddressSpaceOps` with no caller-visible
    /// change; a displaced `ring_aspace_ops` instance is dropped only after
    /// the quiesce completes. The concrete re-invocation hook lives in the
    /// rebind protocol ([Section 13.18](13-device-classes.md#live-kernel-evolution)).
    fn resolve_aspace_ops(&self) -> &dyn AddressSpaceOps;
}

/// Read pages from the page cache for a file read operation.
/// This is the generic implementation used by most filesystem types.
/// Equivalent to Linux's `filemap_get_pages()` + `generic_file_read_iter()`.
///
/// Returns pages covering the requested range `[pgoff, pgoff + nr_pages)`.
/// Pages not in cache are fetched via the fill dispatch seam
/// (`dispatch_read_page()`), which moves the fill obligation to the provider.
///
/// **Concurrency**: Called with no inode locks held. Multiple threads may
/// call this concurrently on the same `AddressSpace`; the page cache XArray
/// ([Section 4.4](04-memory.md#page-cache)) provides internal synchronization.
///
/// **Concurrent reader deduplication (lock-or-find protocol)**:
/// When a cache miss occurs, multiple threads may race to populate the same
/// page index. Exactly one thread performs I/O while all others wait for the
/// result; this is surfaced as the typed `PageLookup::OwnFill(FillLease)`
/// vs `PageLookup::Hit(PagePin)` outcome of `page_cache_get_or_fill()`
/// ([Section 4.4](04-memory.md#page-cache)) — the `OwnFill` winner owns the linear fill lease (the
/// allocation reference plus the page LOCK) and MUST consume it exactly once
/// (submit-async / complete_ok / complete_err), while `Hit` waiters
/// `wait_on_page_locked()` and then observe UPTODATE/ERROR. The steps below
/// describe the underlying frame-refcount protocol that the lease/pin
/// lifecycle implements:
///
/// 1. **Cache probe**: `pc.pages.load(idx)` — if found, return (cache hit).
/// 2. **Allocate**: Allocate a new page, set `PageFlags::LOCKED` atomically.
/// 3. **Atomic insert**: `pc.pages.try_store(idx, page)` — attempts a
///    compare-and-swap insertion into the XArray slot.
/// 4. **Lost race**: If `try_store` returns an existing page (a concurrent
///    reader won the race and inserted first), drop our freshly allocated
///    page, then wait for the existing page's `PageFlags::LOCKED` to be
///    cleared (the winner is performing I/O). Once unlocked, the existing
///    page contains valid data — return it.
/// 5. **Won race**: If `try_store` succeeds (our page is now in the cache),
///    call `read_page()` to fill the page from the backing store. On I/O
///    completion, clear `PageFlags::LOCKED` and wake all waiters sleeping
///    on this page's lock (step 4 above). Return the filled page.
///
/// This protocol prevents duplicate I/O: at most one `read_page()` call is
/// issued per page index, regardless of the number of concurrent readers.
/// The cost of the losing path is one wasted page allocation (returned to
/// the buddy allocator immediately) plus a sleep on the page lock — no I/O.
///
/// **Readahead integration**: Before the lock-or-find path, this function
/// checks the readahead state (`FileRaState` on the `OpenFile`) and may
/// trigger `AddressSpaceOps::readahead()` to batch-fetch a window of pages
/// in a single I/O. The readahead engine ([Section 4.4](04-memory.md#page-cache--readahead-engine))
/// determines the window size based on sequential access detection.
/// Readahead-populated pages are inserted via the same `try_store` protocol,
/// so concurrent readahead and fault-driven reads do not duplicate I/O.
///
/// **Error handling (short-read semantics)**: If `read_page()` fails for
/// any page in the range, the function clears `PageFlags::LOCKED` on the
/// failed page (waking waiters), sets `PageFlags::ERROR` to signal the
/// failure, and removes the failed page from the cache via
/// `pc.pages.erase(idx)`. If pages were successfully fetched in earlier
/// iterations, they are returned as a short read (the caller receives
/// fewer pages than requested — not an error). Only if *no* pages were
/// successfully fetched does the function return `Err`. This matches
/// POSIX read semantics: a successful partial transfer is reported as a
/// short read, not an error. Waiters sleeping on a page that fails I/O
/// are woken and observe `PageFlags::ERROR`, causing them to return `EIO`.
pub fn filemap_get_pages(
    mapping: &AddressSpace,
    pgoff: u64,
    nr_pages: u32,
    ra_state: &mut FileRaState,
) -> Result<ArrayVec<PagePin, MAX_READAHEAD_PAGES>, IoError> {
    let mut pages = ArrayVec::new();
    let pc = mapping.page_cache.as_ref().ok_or(IoError::new(Errno::EINVAL))?;
    for i in 0..nr_pages as u64 {
        let idx = pgoff + i;

        // Step 1: Cache probe with RCU + speculative refcount.
        // The XArray load returns a PageRef valid only under RCU read lock.
        // We must bump the refcount before releasing RCU to prevent the
        // page reclaimer from freeing the page between lookup and use.
        // This matches Linux's folio_try_get_rcu() pattern in filemap_get_pages().
        {
            let rcu = rcu_read_lock();
            if let Some(entry) = pc.pages.load(idx) {
                // `PageEntry::try_pin` ([Section 4.4](04-memory.md#page-cache)) is the lifecycle escape
                // from the RCU section: an increment-if-nonzero on the frame plus
                // a same-slot revalidation, yielding a non-Copy `PagePin` that
                // owns the escaped reference. `None` — the frame is being freed
                // (false increment) or the slot was replaced under us — falls
                // through to the miss path (allocate, or find a replacement after
                // reclaim completes). Content flags (ACCESSED/UPTODATE/…) live on
                // the physical `Page.flags`, reached via `pin.page()`.
                if let Some(pin) = entry.try_pin(&rcu, pc, idx) {
                    drop(rcu); // Release RCU after the pin is stable.
                    // Fast path is UPTODATE-only: a cached page that is not yet
                    // UPTODATE (an in-flight readahead fill) or is ERROR-marked
                    // must NOT be handed to the reader here. It falls through to
                    // the slow path, whose `Hit` arm waits on the page lock and
                    // re-checks UPTODATE/ERROR per the demand-read contract
                    // ([Section 4.4](04-memory.md#page-cache)). On the fall-through `pin` drops at the
                    // end of this block, releasing the speculative reference.
                    if pin.page().flags_load(Acquire).contains(PageFlags::UPTODATE) {
                        // Cache hit — mark referenced for LRU aging, then keep the
                        // owning `PagePin` in the returned vec (RAII: its `Drop`
                        // releases the reference after the reader copies out,
                        // [Section 4.4](04-memory.md#page-cache)).
                        pin.page().flags_fetch_or(PageFlags::ACCESSED, Relaxed);
                        pages.push(pin);
                        continue;
                    }
                }
            }
        }

        // Cache miss — DSM cooperative cache check (three-stage filter).
        //
        // Design: never add latency to the common case. Most misses are
        // local-only (no remote node has the page). The three stages
        // progressively filter out unnecessary RDMA lookups:
        //
        //   Stage 1: Bloom filter (~15-30ns, 3-5 cache lines).
        //            Per-peer counting Bloom filters ([Section 6.11](06-dsm.md#dsm-distributed-page-cache))
        //            are exchanged lazily via DSM heartbeat piggyback. A negative
        //            result means "definitely not cached remotely" — skip RDMA.
        //            Eliminates ~90-95% of remote lookups with zero I/O.
        //
        //   Stage 2: Sequential access rejection (~5ns, single branch).
        //            Sequential readahead streams rarely benefit from cooperative
        //            caching — the same sequential stream is unlikely to be cached
        //            on a peer. If FileRaState indicates sequential pattern (the
        //            readahead engine already tracks this), skip RDMA even if
        //            bloom says "maybe". Eliminates another ~3-5% of lookups.
        //
        //   Stage 3: Speculative parallel issue (RDMA + NVMe simultaneously).
        //            For the remaining ~2-5% of random-access misses where bloom
        //            says "maybe", fire BOTH the RDMA cooperative lookup AND the
        //            local NVMe readahead in parallel. First completion wins;
        //            the loser is cancelled. RDMA (~2-5μs) typically beats NVMe
        //            (~10-100μs) when the remote cache is warm, so we get a real
        //            speedup. When the remote cache is cold, NVMe completes
        //            normally — no added latency.
        //
        // Net effect: zero overhead for ~95% of misses; ~5-10μs speedup for
        // the ~2-5% where remote cache is warm; never slower than local-only.
        if mapping.host.i_sb.s_flags.load(Relaxed) & MS_DSM_COOPERATIVE != 0 {
            let file_id = DsmFileId::from_inode(&mapping.host);

            // Stage 1: Bloom filter — fast local rejection.
            let bloom_hit = dsm_bloom_probe(&file_id, idx);

            if bloom_hit {
                // Stage 2: Sequential access rejection.
                let is_sequential = ra_state.prev_pos != 0
                    && idx == ra_state.prev_pos + 1;

                if !is_sequential {
                    // Stage 3: Speculative parallel issue.
                    // Fire RDMA cooperative lookup. Simultaneously, fall through
                    // to readahead below (the NVMe path). The RDMA result is
                    // checked after readahead submission — if RDMA completed
                    // first, use the remote page and cancel the local I/O.
                    let rdma_fut = dsm_cooperative_cache_lookup_async(
                        &file_id, idx,
                    );

                    // Fall through to readahead (NVMe path starts here).
                    ra_state.start = idx;
                    page_cache_readahead(mapping, ra_state, nr_pages - i as u32);

                    // Check RDMA result — did the remote cache beat NVMe?
                    //
                    // The RDMA-won page is a freestanding frame (allocated
                    // by the probe, filled by the peer's RDMA Write). It
                    // MUST go through the SAME page-cache insertion
                    // discipline (`try_store`) as any other miss fill —
                    // returning it to the caller without inserting would let
                    // the concurrent readahead insert ITS page at the same
                    // index: two physical copies of one file page on one
                    // node, with writers dirtying whichever copy they happen
                    // to hold (coherence hole), and the `verify_page()`
                    // integrity contract silently bypassed.
                    if let Some(page) = rdma_fut.try_complete() {
                        // (a) Verify BEFORE the page becomes visible: RDMA
                        // data carries no filesystem-level integrity
                        // guarantee ([Section 6.11](06-dsm.md#dsm-distributed-page-cache)). On
                        // Ok(false) or Err: release the probe frame's
                        // allocation reference (a bare `drop` of the `Copy`
                        // `PageRef` does NOT free it) and fall through to the
                        // local path — readahead I/O is already in flight. The
                        // DSM probe layer logs the FmaEvent against the remote
                        // node.
                        match mapping.ops.verify_page(mapping, idx, &page) {
                            Ok(true) => {
                                // (b) Publish flags BEFORE insertion so no
                                // concurrent reader can observe an in-cache
                                // page that is neither LOCKED nor UPTODATE.
                                page.flags_fetch_or(
                                    PageFlags::UPTODATE | PageFlags::ACCESSED,
                                    Release,
                                );
                                // (c) Admit the RDMA-filled frame under the same
                                // `try_store` discipline as any miss fill
                                // (`PageEntry::from_ref` adopts the probe's
                                // allocation reference as the slot's,
                                // [Section 4.4](04-memory.md#page-cache)), then FALL THROUGH to the
                                // slow path below rather than pushing here: the
                                // RDMA branch never constructs a returnable pin
                                // itself. `page_cache_get_or_fill` re-probes this
                                // now-resident UPTODATE page, `Hit`s it (no I/O),
                                // and yields the owning `PagePin` — unifying pin
                                // acquisition (and its refcount) through one path.
                                match pc.pages.try_store(idx, PageEntry::from_ref(page)) {
                                    Ok(()) => {
                                        // Published: the slot owns the reference
                                        // (`from_ref` adopted it). Fall through.
                                    }
                                    Err(_existing) => {
                                        // A concurrent reader/readahead won the
                                        // slot first. WINNER RULE: the resident
                                        // page wins (it may have waiters parked
                                        // on its LOCKED bit). Release our
                                        // redundant probe frame (`page` still
                                        // owns the allocation ref — `from_ref`
                                        // took a `Copy`, and the losing store
                                        // adopted nothing) and fall through: the
                                        // slow path `Hit`s the winner, the same
                                        // shape as `page_cache_get_or_fill`'s own
                                        // lost store.
                                        page_put_rcu(page.page());
                                    }
                                }
                            }
                            Ok(false) | Err(_) => {
                                // Unverifiable frame: release the probe
                                // allocation reference and fall through to the
                                // local path below.
                                page_put_rcu(page.page());
                            }
                        }
                    }

                    // RDMA didn't complete yet, missed, or failed
                    // verification. NVMe readahead is already in flight —
                    // re-check cache below (normal path). The RDMA future is
                    // dropped (cancelled on drop).
                }
            }
        }

        // Cache miss — trigger readahead before attempting I/O.
        // The readahead engine ([Section 4.4](04-memory.md#page-cache--readahead-engine)) may
        // submit a larger I/O batch here to prefetch upcoming pages.
        // Pass the remaining page count (nr_pages - pages already fetched),
        // not the page index — page_cache_readahead uses FileRaState.start
        // (set by the caller) to determine *which* pages to read, and
        // nr_pages to bound the readahead window size.
        ra_state.start = idx;
        page_cache_readahead(mapping, ra_state, nr_pages - i as u32);

        // Fill the page through the single page-cache admission primitive
        // ([Section 4.4](04-memory.md#page-cache)). `page_cache_get_or_fill` folds the post-readahead
        // re-probe, the shared-extent (CoW/reflink) hook, allocation, the page
        // LOCK, the `try_store` publish, and the lost-insert retry into one call,
        // returning the linear pin/fill lifecycle outcome — the SAME idiom the
        // sibling reader `file_backed_fault` uses ([Section 4.4](04-memory.md#page-cache)).
        //
        // Inner loop: re-run the admission for THIS index when a joined `Hit`
        // page turns out to have been truncated/invalidated while locked
        // (unlocked with neither UPTODATE nor ERROR set) — the "restart lookup"
        // arm of the demand-read contract ([Section 4.4](04-memory.md#page-cache)). It converges: a
        // truncated page re-misses and this thread refills it (`OwnFill`).
        loop {
            let lookup = match page_cache_get_or_fill(mapping, idx, GFP_KERNEL) {
                Ok(l) => l,
                Err(_oom) => {
                    // OOM on the miss allocation — POSIX short-read semantics: if
                    // earlier iterations already collected pages, return them as a
                    // short read; ONLY a first-page failure (nothing collected) is
                    // an error. Every collected `PagePin` releases on drop in each
                    // path ([Section 4.4](04-memory.md#page-cache)), so no reference leaks on the short
                    // return.
                    if !pages.is_empty() {
                        return Ok(pages);
                    }
                    return Err(IoError::ENOMEM);
                }
            };
            match lookup {
                PageLookup::Hit(pin) => {
                    // Resident and pinned: readahead filled this index, a racing
                    // reader won the slot, or a shared extent was joined. Wait for
                    // any in-flight fill, then read CONTENT state on `Page.flags`
                    // (the single authority) per the demand-read contract.
                    wait_on_page_locked(pin.page());
                    let flags = pin.page().flags_load(Acquire);
                    if flags.contains(PageFlags::ERROR) {
                        // I/O failed. Short-read: return the pages collected so
                        // far, else EIO. `pin` drops here, releasing its ref.
                        if !pages.is_empty() {
                            return Ok(pages);
                        }
                        return Err(IoError::new(Errno::EIO));
                    }
                    if !flags.contains(PageFlags::UPTODATE) {
                        // Unlocked but neither UPTODATE nor ERROR: the page was
                        // truncated/invalidated out of the cache while locked.
                        // Restart the admission for this index ([Section 4.4](04-memory.md#page-cache)
                        // demand-read contract). `pin` drops here.
                        continue;
                    }
                    pin.page().flags_fetch_or(PageFlags::ACCESSED, Relaxed);
                    // Keep the owning `PagePin` in the returned vec (RAII: its
                    // `Drop` releases the reference after the reader copies out,
                    // [Section 4.4](04-memory.md#page-cache)).
                    pages.push(pin);
                    break;
                }
                PageLookup::OwnFill(lease) => {
                    // We won the fill race and own the linear obligation. Take the
                    // installable pin FIRST (`lease.pin()` — a plain `page_get`,
                    // legal because the lease holds the allocation reference), THEN
                    // extract the fill obligation so it is discharged EXACTLY ONCE
                    // regardless of how the read completes.
                    let pin = lease.pin();
                    let fc = lease.into_inflight();
                    // Hand the obligation to the fill dispatch seam, which MOVES
                    // it to the bound provider's machinery: a sync provider fills
                    // and the seam discharges `fc` inline; an async provider
                    // (same-domain `Bio` ctx, or cross-domain ring
                    // `VfsInflightEntry.fill`) enrolls `fc` so its completion
                    // discharges it exactly once. `read_page`'s signature is
                    // UNCHANGED — the obligation rides the dispatch seam, never the
                    // trait (same contract as `file_backed_fault`, [Section 4.4](04-memory.md#page-cache)).
                    match dispatch_read_page(mapping, idx, &pin, fc) {
                        // Obligation accepted (sync-discharged or async-enrolled);
                        // the wait below picks up the published terminal state.
                        Ok(()) => {}
                        Err((fc, e)) => {
                            // Submit failed before any enrollment: ownership of the
                            // undischarged obligation came back — discharge inline
                            // (publish ERROR, demand-erase the slot, unlock, wake,
                            // release the fill reference — the `complete_err` body).
                            // Then short-read: pages collected so far, or the
                            // originating errno. `pin` drops here, releasing the
                            // installable reference.
                            fc.complete_err(e.errno());
                            if !pages.is_empty() {
                                return Ok(pages);
                            }
                            return Err(e);
                        }
                    }
                    // Fill dispatched/completed: wait for LOCKED to clear, then
                    // read CONTENT state on `Page.flags`. This thread's fill (or
                    // the async completion) always publishes UPTODATE or ERROR, so
                    // no truncation re-check is needed on this arm.
                    wait_on_page_locked(pin.page());
                    if pin.page().flags_load(Acquire).contains(PageFlags::ERROR) {
                        // The fill completed with ERROR (this thread's synchronous
                        // fill, or the async completion): short-read, else EIO.
                        // `pin` drops here, releasing its reference.
                        if !pages.is_empty() {
                            return Ok(pages);
                        }
                        return Err(IoError::new(Errno::EIO));
                    }
                    pin.page().flags_fetch_or(PageFlags::ACCESSED, Relaxed);
                    pages.push(pin);
                    break;
                }
            }
        }
    }
    Ok(pages)
}

/// Generic file read iterator. Used by most filesystem `FileOps::read()`
/// implementations. Reads from the page cache, triggering readahead and
/// I/O as needed.
///
/// **Flow**:
/// 1. Compute the starting page offset and intra-page offset from `*offset`.
/// 2. Call `filemap_get_pages()` for the required page range.
/// 3. Copy data from the returned pages into `buf` via `copy_to_user()`.
/// 4. Advance `*offset` by the number of bytes read.
/// 5. Return the total bytes copied, or an error if no bytes were read.
///
/// **Short reads**: If the file ends mid-page (offset + len > i_size),
/// only the valid bytes are copied. This is not an error — the return
/// value reflects the actual bytes read, and `*offset` is advanced
/// accordingly.
///
/// **DAX bypass**: If the mapping's DAX bit is set, this function
/// is never called — DAX files use `dax_iomap_rw()` instead, which
/// maps persistent memory directly into the user's address space.
pub fn page_cache_read_iter(
    file: &OpenFile,
    buf: &mut UserSliceMut,
    offset: &mut i64,
) -> Result<usize, IoError> {
    // Data plane: the mapping and the size come from the RESOLVED data inode
    // (`data_inode`), which equals `inode` for every non-stacking filesystem.
    let mapping = &file.data_inode.i_mapping;
    let mut total = 0usize;

    // VFS-BUG-5 fix: i_size check before read loop. Without this, a read
    // past EOF would copy uninitialized page data to userspace (information
    // disclosure). POSIX: read() past EOF returns 0 bytes, not an error.
    let i_size = file.data_inode.i_size.load(Acquire) as i64;
    if *offset >= i_size {
        return Ok(0);
    }
    // Clamp the effective read length to not exceed i_size. This ensures
    // we never read beyond the file's logical end, even if page cache pages
    // exist beyond i_size (e.g., from a concurrent truncate race — the
    // truncate path clears those pages asynchronously).
    let max_readable = (i_size - *offset) as usize;
    let effective_remaining = min(buf.remaining(), max_readable);

    while total < effective_remaining {
        let pgoff = (*offset as u64) / PAGE_SIZE as u64;
        let intra = (*offset as usize) % PAGE_SIZE;
        let remaining = effective_remaining - total;
        let nr = min((remaining + PAGE_SIZE - 1) / PAGE_SIZE, MAX_READAHEAD_PAGES as usize);
        // Partial-transfer rule (POSIX read semantics, Linux
        // filemap_read: `return already_read ? already_read : error`):
        // an I/O error on a LATER batch, after earlier batches already
        // copied bytes to the user, reports the short count Ok(total) —
        // the error re-fires on the caller's NEXT read (which starts at
        // total == 0 and returns it as Err). Only a first-batch failure
        // with nothing copied is an error.
        let pages = match filemap_get_pages(mapping, pgoff, nr as u32, &mut file.ra_state.lock()) {
            Ok(p) => p,
            Err(_) if total > 0 => break,   // short read: Ok(total)
            Err(e) => return Err(e),        // nothing copied: real error
        };
        // `page` is an owning `&PagePin` ([Section 4.4](04-memory.md#page-cache)); the batch's frame
        // references are held across the copy-out below and released when `pages`
        // (the `ArrayVec<PagePin>`) drops at the end of this loop iteration — the
        // RAII discipline that replaced the leaking `ArrayVec<PageRef>`.
        for (i, page) in pages.iter().enumerate() {
            // intra-page offset: non-zero only for the first page of each
            // filemap_get_pages batch (the read may start mid-page). All
            // subsequent pages are read from offset 0.
            let page_intra = if i == 0 { intra } else { 0 };
            // Clamp to i_size: never copy beyond the file's logical end.
            let avail = min(
                min(PAGE_SIZE - page_intra, effective_remaining - total),
                buf.remaining(),
            );
            if avail == 0 { break; }
            // Same partial-transfer rule for a mid-transfer user fault
            // (EFAULT on a later page of a multi-page read): the bytes
            // already copied are reported; EFAULT surfaces on the next
            // call. `*offset` reflects exactly the bytes delivered.
            if let Err(e) = copy_to_user(buf, page.as_ptr().add(page_intra), avail) {
                if total > 0 {
                    return Ok(total);       // short read: partial success
                }
                return Err(e);              // first byte failed: EFAULT
            }
            *offset += avail as i64;
            total += avail;
        }
        if pages.len() < nr as usize { break; } // short read or EOF
    }
    Ok(total)
}

/// Wake every task blocked on this page's lock/uptodate transitions.
///
/// Counterpart to `unlock_page()` for paths that clear `LOCKED`/`ERROR`
/// directly (e.g. a failed read that must both flag the error AND release
/// sleepers before removing the page). Tasks park in `wait_on_page_locked()`
/// on the hashed page-wait table keyed by the page's frame address; this
/// wakes all of them so each re-checks `LOCKED`/`UPTODATE`/`ERROR` and
/// proceeds (retry, EIO, or success).
///
/// Idempotent and safe to call with no waiters (empty wake is a no-op).
/// Callers MUST have already published the flag change (`Release`) so woken
/// tasks observe the final page state.
fn wake_page_waiters(page: &Page) {
    // Hashed page-wait table (keyed by frame address); one bucket wake.
    page_waitqueue(page).wake_up_all();
}

Tier isolation note: The VFS read path dispatches copy_to_user via a KABI return-buffer mechanism when the filesystem runs in Tier 1 (hardware memory domain isolated). The Tier 1 filesystem populates a shared bounce buffer (mapped into the driver's isolation domain as writable), and Core (Tier 0) performs the actual copy_to_user() after the KABI ring response returns. This ensures the Tier 1 driver never directly accesses userspace memory — all user memory writes go through Tier 0 copy_to_user(), which validates the destination address against the task's address space. For Tier 0 filesystems (e.g., the root filesystem driver), the copy_to_user() call is inlined directly — no bounce buffer needed.

Read path walkthrough (numbered trace, analogous to write path):

1. sys_read() — extract fd, buf, count from SyscallContext; resolve OpenFile.
2. VFS dispatch — call file.ops.read_iter() (or page_cache_read_iter for regular files).
3. filemap_get_pages() — compute page offset, probe page cache XArray.
4. Page cache hit — mark PageFlags::ACCESSED, return cached page.
5. Page cache miss — trigger readahead (AddressSpaceOps::readahead()).
6. wait_on_page_locked() — sleep until I/O completion clears PageFlags::LOCKED.
7. copy_to_user() — copy page data to userspace buffer (bounce buffer if Tier 1).
8. Return total bytes read to userspace via syscall return register.

Cross-references: - Page cache structure and XArray: Section 4.4 - Readahead engine: Section 4.4 - DAX direct access path: Section 15.16

14.1.2.4.1.5 page_cache_write_iter() — Buffered Write Path

/// Generic buffered write implementation. Called from `FileOps::write()`
/// for regular file writes. Iterates over the write range page by page, using
/// the filesystem's `write_begin()`/`write_end()` callbacks for per-page
/// preparation and commit.
///
/// Equivalent to Linux's `generic_file_write_iter()` + `generic_perform_write()`.
///
/// # Steps
///
/// 0. Enter freeze protection: `sb_start_write(sb, SbFreezeLevel::Write)`
///    — blocks (interruptibly) while the filesystem is frozen at Write
///    level or beyond. The guard is held until the function returns,
///    covering the O_SYNC flush in step 6 (Linux `vfs_write()` keeps freeze protection
///    across `->write_iter`, including its durability flush). Freeze protection nests
///    OUTSIDE
///    `i_rwsem`: entered before step 1, released after everything.
/// 1. Acquire `i_rwsem` exclusive (`I_RWSEM`, level 80) — serializes
///    writers, excludes concurrent truncate, and makes the O_APPEND
///    position read + write atomic.
/// 2. RLIMIT_FSIZE check (truncate write to limit).
/// 3. O_APPEND: seek to i_size (atomic w.r.t. other writers — step 1).
/// 4. For each page in [pos, pos+count):
///    a. `write_begin(mapping, pos, len, flags)` — filesystem prepares page
///       (journal reservation, delayed allocation, partial page read-in).
///       Returns locked page reference.
///    b. `copy_from_user(page_addr + offset, buf, bytes)` — copy user data
///       into the page. Handles fault (short copy) via `bytes_copied`.
///    c. `write_end(mapping, pos, len, bytes_copied, page)` — filesystem
///       commits the write (mark dirty, update journal, update i_size).
///    d. `balance_dirty_pages_ratelimited(mapping)` — throttle the writer
///       if dirty page count exceeds the threshold, preventing memory
///       pressure from unbounded dirtying
///       ([Section 4.6](04-memory.md#writeback-subsystem--dirty-throttling-ratelimit)).
/// 5. Release `i_rwsem`, then update `mtime` and `ctime` timestamps.
///    Timestamps MUST be updated BEFORE any O_SYNC metadata flush (step 6)
///    so the flushed on-disk metadata carries the current mtime — flushing
///    first and stamping after would persist a stale mtime, defeating the
///    O_SYNC durability guarantee for the timestamp.
/// 6. O_SYNC/O_DSYNC: flush the written range, then metadata if required.
/// 7. Return total bytes written.
///
/// # Locking
///
/// `i_rwsem` (exclusive) is held from before the initial `i_size` read
/// until after the write loop completes — covering the RLIMIT check, the
/// O_APPEND seek, and every write_begin/write_end pair (which also keeps
/// the dirty-intent list updates for this write atomic w.r.t. concurrent
/// truncate). It is released BEFORE the O_SYNC flush: writeback needs no
/// exclusive inode access (matching Linux ext4, which drops `inode_lock`
/// before the durability flush), and holding it across storage I/O
/// would block concurrent readers for milliseconds. Lock chain stays
/// strictly ascending: `I_RWSEM(80)` → `PAGE_LOCK(180)` → `XA_LOCK(181)`
/// (the page is locked first, then the page-cache `xa_lock` is
/// taken under it — [Section 3.4](03-concurrency.md#cumulative-performance-budget--lock-ordering)).
///
/// # Error handling
///
/// **Partial-transfer rule (POSIX / Linux `generic_perform_write()`:
/// `return written ? written : status`)**: if `write_begin()` or
/// `write_end()` fails on an iteration AFTER earlier iterations already
/// committed bytes (`written > 0`), the function returns the SHORT COUNT
/// `Ok(written)` — never an error that would deny the committed bytes.
/// The error is not lost: the failing condition (ENOSPC, EIO, quota)
/// re-fires on the caller's NEXT write attempt, which then starts at
/// `written == 0` and returns it as `Err`. Only a failure on the FIRST
/// iteration (nothing committed) returns `Err` directly. Pages already
/// committed via `write_end()` remain dirty and will be written back
/// asynchronously — a partial write is not rolled back. Early exits
/// unwind correctly by construction: `write_guard` (i_rwsem) and
/// `_sb_write` (freeze protection, step 0) are RAII guards dropped on
/// every return path.
///
/// If `copy_from_user()` returns a short count (user page not present),
/// `write_end()` is called with the short `bytes_copied`. The filesystem
/// handles the partial page correctly (e.g., ext4 does not advance i_size
/// past the last successfully written byte).
///
/// The O_SYNC/O_DSYNC flush (step 6) is the deliberate exception: a
/// flush failure returns `Err` even though `written > 0`, because the
/// caller asked for durability and did not get it (Linux parity:
/// Linux `generic_write_sync()` returns the sync error, discarding the count).
fn page_cache_write_iter(
    file: &OpenFile,
    buf: &UserSlice,
    pos: &mut i64,
) -> Result<usize, IoError> {
    // WF-10 fix: field is i_mapping, not address_space.
    // WF-11 fix: data_inode is a field (Arc<Inode>), not a method call.
    // Data plane: the whole write path — freeze level, `i_rwsem`, `i_size`,
    // and the mapping — operates on the RESOLVED data inode, which equals
    // `file.inode` for every non-stacking filesystem.
    let inode = &*file.data_inode;
    let mapping = &inode.i_mapping;

    // Step 0: Freeze protection (Write level). Declared FIRST so it drops
    // LAST — the guard outlives i_rwsem and the O_SYNC flush. Blocks
    // until thaw if the filesystem is frozen; a fatal signal interrupts
    // the wait (EINTR, converted to the I/O error domain here).
    let _sb_write = sb_start_write(&inode.i_sb, SbFreezeLevel::Write)
        .map_err(IoError::new)?;

    // Step 1: Exclusive inode write lock (I_RWSEM, level 80). Serializes
    // writers, excludes truncate, and makes the O_APPEND i_size read +
    // write loop one atomic section (POSIX O_APPEND atomicity). Taken
    // BEFORE the i_size reads below — both the O_DSYNC capture and the
    // O_APPEND seek must observe a size no concurrent writer can change.
    // Dropped explicitly after the write loop (see the Locking section
    // of the doc comment).
    let write_guard = inode.i_rwsem.write();

    // Capture i_size before the write for the O_DSYNC metadata-flush
    // decision: if the write extends the file, metadata must be flushed.
    let old_i_size = inode.i_size.load(Acquire);

    // Step 2: RLIMIT_FSIZE — truncate write to file size limit.
    let rlimit_fsize = current_task().process.rlimits.limits[RLIMIT_FSIZE].soft;
    let count = if *pos as u64 + buf.len() as u64 > rlimit_fsize && rlimit_fsize != u64::MAX {
        signal_send(current_task(), SIGXFSZ);
        if *pos as u64 >= rlimit_fsize { return Err(IoError::EFBIG); }
        (rlimit_fsize - *pos as u64) as usize
    } else {
        buf.len()
    };

    // Step 3: O_APPEND — seek to end of file. Atomic w.r.t. other writers
    // because i_rwsem is held exclusively from before this read until the
    // write loop completes: no other writer can advance i_size between
    // this load and our write_end() calls.
    if file.f_flags.load(Relaxed) & O_APPEND != 0 {
        *pos = inode.i_size.load(Acquire) as i64;
    }

    // Step 4: Page-by-page write loop.
    let mut written: usize = 0;
    while written < count {
        let offset_in_page = (*pos as usize) % PAGE_SIZE;
        let bytes = core::cmp::min(PAGE_SIZE - offset_in_page, count - written);

        // 4a: Filesystem prepares page (alloc, journal, partial read-in).
        // Partial-transfer rule (see doc comment): a mid-loop failure
        // after committed bytes reports the short count, not the error —
        // the error surfaces on the caller's NEXT write.
        let page = match mapping.ops.write_begin(mapping, *pos as u64, bytes, 0) {
            Ok(p) => p,
            Err(_) if written > 0 => break, // short count: Ok(written)
            Err(e) => return Err(e),        // nothing committed: real error
        };

        // 4b: Copy user data from UserSlice into page.
        // UserSlice::read_to_page() performs copy_from_user internally,
        // handling SMAP/PAN page faults and returning bytes copied.
        // This is the correct API for user → kernel page copies.
        // page_address() (defined below) returns the page's direct-map
        // kernel VA; VirtAddr implements Add<usize>.
        let page_kaddr = page_address(&page) + offset_in_page;
        let bytes_copied = buf.read_at(written, page_kaddr, bytes);

        // 4c: Filesystem commits write (dirty, journal, i_size update).
        // WF-03 fix: use the return value — the filesystem may commit
        // fewer bytes than copied (block-aligned partial commit).
        // Same partial-transfer rule as 4a on failure.
        let committed = match mapping.ops.write_end(
            mapping, *pos as u64, bytes, bytes_copied, page,
        ) {
            Ok(c) => c,
            Err(_) if written > 0 => break, // short count: Ok(written)
            Err(e) => return Err(e),        // first iteration: real error
        };

        written += committed;
        *pos += committed as i64;

        if committed < bytes {
            break; // Short write — stop.
        }

        // 4d: Dirty page throttling
        // ([Section 4.6](04-memory.md#writeback-subsystem--dirty-throttling-ratelimit)).
        balance_dirty_pages_ratelimited(mapping);
    }

    // Step 5: Release i_rwsem — the write is committed to the page cache;
    // the O_SYNC flush below needs no exclusive inode access and must not
    // block readers for the duration of storage I/O.
    drop(write_guard);

    // Step 5 (cont.): Update timestamps — BEFORE the O_SYNC/O_DSYNC
    // metadata flush, so the flushed on-disk inode carries the new mtime.
    // (Stamping after the flush would persist a stale mtime, breaking the
    // O_SYNC guarantee for timestamp durability.)
    // The timestamp triple is a multi-field block behind the INODE_LOCK
    // spinlock (`i_times`) — see the Inode struct. mtime and ctime are
    // updated together under one lock hold so readers never observe
    // mtime-new/ctime-old.
    if written > 0 {
        let now = Timespec::from_ns(current_time_ns());
        {
            let mut times = inode.i_times.lock();
            times.i_mtime = now;
            times.i_ctime = now;
        }
    }

    // Step 6: O_SYNC / O_DSYNC / InodeFlags::SYNC — flush the written range before
    // returning so the caller blocks until data is on stable media.
    // `inode.is_sync()` (per-inode `chattr +S` or `mount -o sync` —
    // Linux `IS_SYNC()`, see the enforcement map in
    // [Section 14.1](#virtual-filesystem-layer--inode-attribute-flags)) forces full
    // O_SYNC semantics regardless of the open flags.
    // This reuses the normal writeback path; no duplicate write occurs
    // because filemap_write_and_wait_range() only flushes dirty pages
    // (already dirtied by write_end above), then waits for completion.
    if written > 0 {
        let sync_flags = file.f_flags.load(Relaxed) & (O_SYNC | O_DSYNC);
        if sync_flags != 0 || inode.is_sync() {
            let start_pos = *pos - written as i64;
            let end_pos = *pos - 1;
            filemap_write_and_wait_range(mapping, start_pos, end_pos)?;

            // O_SYNC (or InodeFlags::SYNC): data + all metadata must reach stable
            // storage.
            // O_DSYNC: metadata only if file size changed (data integrity
            // requires the updated i_size to be durable; timestamp updates
            // are NOT required for data recoverability per POSIX).
            // NOTE: `sync_flags & O_SYNC != 0` tests the composite value
            // (__O_SYNC | O_DSYNC) — O_DSYNC-only opens do NOT satisfy
            // it because the __O_SYNC bit is absent; this arm therefore
            // distinguishes O_SYNC from O_DSYNC via the __O_SYNC bit,
            // matching Linux's encoding.
            let needs_metadata = if sync_flags & (O_SYNC & !O_DSYNC) != 0
                || inode.is_sync()
            {
                true
            } else {
                *pos > old_i_size as i64  // file grew: i_size update must be durable
            };
            if needs_metadata {
                vfs_fsync_metadata(inode)?;
            }
        }
    }

    // Step 7: Return bytes written.
    Ok(written)
}

/// Return the kernel direct-map virtual address of a page's contents.
///
/// Every physical page frame is mapped in the kernel direct map (physmap /
/// PAGE_OFFSET region — [Section 4.15](04-memory.md#extended-memory-operations) describes the
/// region and its excision rules). The `Page` descriptor lives in the
/// contiguous memmap array, so its PFN is derived from its own address by
/// index arithmetic; the PFN's physical address is then translated through
/// the arch-specific direct-map offset.
///
/// All hardware specifics (direct-map base, per-arch layout) live in
/// `arch::current::mm` — this wrapper contains no architecture knowledge.
///
/// **Preconditions**: the page must be a regular RAM page present in the
/// direct map. Callers must not use this for pages excised from the direct
/// map ([Section 4.15](04-memory.md#extended-memory-operations)) — the write path never sees
/// those (excised pages are never in a file's page cache).
///
/// Hot path: pure arithmetic (memmap index → PFN → PA → VA), no locks,
/// no allocation. Equivalent to Linux `page_address()` for lowmem pages
/// (UmkaOS has no highmem: all 8 targets either are 64-bit or use LPAE-
/// style direct maps sized at boot).
pub fn page_address(page: &Page) -> VirtAddr {
    let pfn = page_to_pfn(page);                        // memmap index arithmetic (Pfn)
    let pa = pfn.to_phys();                             // frame physical address
    arch::current::mm::phys_to_virt(pa)                 // direct-map translation
}

/// Read file page `index` into the page cache and return a counted reference to
/// the up-to-date page. Used by in-kernel whole-page content readers — e.g. IMA
/// measurement ([Section 9.5](09-security.md#runtime-integrity-measurement)). Analogous to Linux
/// Linux `read_mapping_page()`.
///
/// Delegates to `filemap_get_pages()` for a single page: it probes the cache,
/// allocates and inserts on a miss, drives the filesystem `read_page()` op, and
/// waits for I/O completion (UPTODATE) internally. `filemap_get_pages` returns
/// owning `PagePin`s ([Section 4.4](04-memory.md#page-cache)); this is the ONE sanctioned
/// `PagePin::into_page_ref` site — it transfers the single filled pin's counted
/// reference into the returned `PageRef`, which the caller (e.g. IMA measurement)
/// releases EXACTLY ONCE with `page_put_rcu()` (the RCU-deferred release for
/// page-cache frames, [Section 4.2](04-memory.md#physical-memory-allocator)). If no page is returned
/// (short read at EOF), the empty `ArrayVec<PagePin>` drops with nothing to
/// release.
pub fn vfs_read_page(file: &OpenFile, index: u64) -> Result<PageRef, IoError> {
    // Data plane — resolved data inode (see `OpenFile::data_inode`).
    let mapping = &file.data_inode.i_mapping;
    let mut ra = FileRaState::default();
    let pages = filemap_get_pages(mapping, index, 1, &mut ra)?;
    pages.into_iter().next()
        .map(|pin| pin.into_page_ref())
        .ok_or_else(|| IoError::new(Errno::EIO))
}

/// Flush inode metadata (size, blocks, mode, uid/gid, timestamps) to stable
/// storage. Called by the O_SYNC branch of `page_cache_write_iter` and by
/// full `fsync()` when data writeback is already complete. Does NOT flush data
/// pages — the caller is expected to have already issued
/// `filemap_write_and_wait_range()`.
///
/// For journaling filesystems (ext4, XFS, btrfs), this commits the journal
/// transaction containing the inode update (equivalent to
/// forcing the journal through its transaction or covering log sequence). For non-journaling
/// filesystems, this writes the inode block and issues a cache flush.
///
/// Uses `WriteSyncMode::Sync` (Linux `WB_SYNC_ALL`) semantics: the call
/// blocks until the inode metadata is on stable media.
pub fn vfs_fsync_metadata(inode: &Inode) -> Result<(), IoError> {
    // The InodeOps interface addresses inodes by the InodeId newtype;
    // `Inode` stores the raw number as `i_ino: u64` — wrap explicitly.
    inode.i_op.write_inode(InodeId(inode.i_ino), WriteSyncMode::Sync)?;
    Ok(())
}

14.1.2.4.1.6 Inode (Index Node)

/// Kernel-internal timestamp: seconds + nanoseconds since the Unix epoch.
///
/// Multi-field by nature — a `Timespec` can NEVER live in a single atomic;
/// any `Timespec` shared across threads must sit behind a lock or seqlock
/// (see `Inode::i_times`). The userspace `struct timespec` (whose `tv_nsec`
/// is a C `long` — `KernelLong` on ILP32 targets) is a distinct ABI struct;
/// conversion happens at the syscall boundary (`stat`, `utimensat`).
#[derive(Clone, Copy, PartialEq, Eq, Debug)]
pub struct Timespec {
    /// Seconds since the epoch. i64: y2038-safe on all 8 architectures.
    pub tv_sec: i64,
    /// Nanoseconds within the second. Invariant: 0..=999_999_999.
    pub tv_nsec: u32,
}

impl Timespec {
    /// Convert a nanoseconds-since-epoch value (e.g. `current_time_ns()`)
    /// into a `Timespec`.
    pub fn from_ns(ns: u64) -> Self {
        Self {
            tv_sec: (ns / 1_000_000_000) as i64,
            tv_nsec: (ns % 1_000_000_000) as u32,
        }
    }
}

/// The inode timestamp triple, updated and read as one consistent block
/// under `Inode::i_times` (SpinLock, `INODE_LOCK` level 160). `Copy` so
/// readers (`stat`) can take a snapshot with a single short lock hold.
#[derive(Clone, Copy)]
pub struct InodeTimes {
    /// Last access time (reads). The update decision follows Linux
    /// `fs/inode.c atime_needs_update()` order: per-inode `InodeFlags::NOATIME`
    /// (`chattr +A`) is checked FIRST, then the superblock `MS_NOATIME` /
    /// read-only state, then mount-level `noatime`/`nodiratime`, then
    /// relatime. See the enforcement map in
    /// [Section 14.1](#virtual-filesystem-layer--inode-attribute-flags).
    pub i_atime: Timespec,
    /// Last data modification time (writes, truncate).
    pub i_mtime: Timespec,
    /// Last inode change time (writes, chmod, chown, link count changes).
    pub i_ctime: Timespec,
}

/// In-memory representation of a filesystem object (file, directory,
/// symlink, device, pipe, socket).
///
/// Each inode has a unique (superblock, inode_number) pair. The VFS
/// maintains an inode cache (icache) keyed by this pair to avoid
/// repeated disk reads.
///
/// **Lifecycle**: Created by `FileSystemOps::mount()` (root inode) or
/// `InodeOps::lookup()`/`InodeOps::create()` for other entries. Cached
/// in the icache. Freed when the last dentry referencing it is evicted
/// AND the on-disk link count drops to zero (unlinked).
///
/// **Concurrency**: Inode metadata is protected by `i_lock` (spinlock).
/// File data is protected by `i_rwsem` (read-write semaphore) — readers
/// (read, readdir) take shared; writers (write, truncate) take exclusive.
// kernel-internal, not KABI — no const_assert (contains Arc, RwLock, dyn traits).
#[repr(C)]
pub struct Inode {
    /// Inode number. Unique within a superblock. Assigned by the filesystem.
    pub i_ino: u64,

    /// File type and permission mode (S_IFREG, S_IFDIR, etc. | rwxrwxrwx).
    /// Interior-mutable: the type bits (`S_IFMT`) are fixed at instantiation,
    /// but the permission bits are updated by chmod/`setattr` through the
    /// shared `&Inode` (inodes are held as `Arc<Inode>`). `AtomicU32` (like
    /// `i_nlink`/`i_flags`): hot-path readers (file-type and permission checks)
    /// do one `Relaxed` load; the mutation path stores `Release` under
    /// `i_rwsem` write.
    pub i_mode: AtomicU32,

    /// Owner UID (kernel-internal representation, namespace-agnostic).
    /// Permission checks translate `i_uid.load(...)` through `mnt_userns`
    /// to translate between the filesystem's user namespace and the calling
    /// process's user namespace ([Section 17.1](17-containers.md#namespace-architecture)). `AtomicU32`
    /// for the same reason as `i_mode`: chown writes it through `&Inode`.
    pub i_uid: AtomicU32,

    /// Owner GID (kernel-internal representation, namespace-agnostic).
    /// Permission checks translate `i_gid.load(...)` through `mnt_userns`
    /// analogously. `AtomicU32` — chown writes it through `&Inode`.
    pub i_gid: AtomicU32,

    /// Kernel-internal attribute/behavior flags (`InodeFlags` bits —
    /// see "Inode Attribute Flags" below for the bit
    /// definitions, the `FS_*_FL` persistent space, the chattr/ioctl
    /// surface, and the normative per-flag enforcement map). Hot-path
    /// readers (`is_immutable()`, `is_append()`, `is_sync()`, …) perform
    /// one `Relaxed` load — never a lock, never a ring call. Written at
    /// inode instantiation (the `I_NEW` construction path) and thereafter
    /// only by `vfs_fileattr_set()` / `vfs_update_inode_flags()` under
    /// `i_rwsem` write (`Release` stores). Flag gates are deliberately
    /// NOT folded into the dentry `cached_perm` grants, so no flag change
    /// ever requires a permission-cache invalidation sweep.
    pub i_flags: AtomicU32,

    /// Hard link count. When this reaches 0 and no open file descriptors
    /// remain, the inode is freed (both in-memory and on-disk).
    pub i_nlink: AtomicU32,

    /// File size in bytes. AtomicI64 for compatibility with Linux loff_t
    /// semantics. For regular files and directories, i_size is always >= 0.
    /// For symlinks: length of the target path. Updated under `i_rwsem`.
    /// Consumers cast via `i_size.load(Acquire) as u64` after asserting
    /// non-negative: `debug_assert!(self.i_size.load(Acquire) >= 0)`.
    pub i_size: AtomicI64,

    /// Timestamps (seconds + nanoseconds since epoch), as one
    /// interior-mutable block.
    ///
    /// **Why a lock, not atomics**: a `Timespec` is multi-field
    /// (`tv_sec` + `tv_nsec`) and the three timestamps are updated
    /// together (write() sets mtime+ctime; chmod sets ctime) — no single
    /// atomic can hold one timestamp, let alone keep the triple mutually
    /// consistent. Writers reach the inode through shared references
    /// (`&Inode` via `Arc<Inode>`), so plain fields would be both a data
    /// race and unwritable — interior mutability is mandatory here.
    ///
    /// The block is protected by its own `SpinLock` at `INODE_LOCK`
    /// (level 160) — the level documented for inode attribute updates in
    /// the lock hierarchy ([Section 3.4](03-concurrency.md#cumulative-performance-budget--lock-ordering)). Data-in-lock
    /// (rather than a detached "protected by i_lock" convention) makes
    /// the protection compiler-enforced. Readers (`stat(2)`) take the
    /// same spinlock for a 48-byte copy — warm path, ~20-cycle hold; the
    /// alternative (lockless seqcount reads) is not needed at stat()
    /// frequency and Linux itself tolerates torn timestamp reads here,
    /// which UmkaOS chooses not to.
    ///
    /// Never nested inside `i_lock` (also `INODE_LOCK`): a code path
    /// takes one or the other, never both (same-level acquisition is a
    /// compile error under `Lock<T, LEVEL>`).
    pub i_times: SpinLock<InodeTimes, INODE_LOCK>,

    /// Block size for this inode's filesystem (typically 4096).
    pub i_blksize: u32,

    /// Number of 512-byte blocks allocated on disk.
    pub i_blocks: u64,

    /// Device number (major:minor) for device special files (S_IFBLK/S_IFCHR).
    /// Uses `DevId` type with Linux MKDEV encoding: `(major << 20) | minor`.
    /// See [Section 14.5](#device-node-framework) for encoding details.
    /// `DevId { raw: 0 }` for regular files.
    pub i_rdev: DevId,

    /// Generation number. Incremented when an inode is recycled (same i_ino
    /// reused for a new file). Used by NFS file handles to detect stale handles.
    /// Constrained to u32 by NFS file handle wire format (nfs_fh generation
    /// field). At 10K inode recycled/sec, wraps after ~5 days — but
    /// stale-handle collision requires matching SAME i_ino AND i_generation
    /// (1-in-4B chance). NFS clients detect wrap mismatch via ESTALE.
    /// Matches Linux i_generation behavior.
    pub i_generation: u32,

    /// Metadata-mutation generation counter for the stat/statx prefetch
    /// cache. DISTINCT from `i_generation` (NFS inode-recycle counter): this
    /// is bumped on EVERY metadata mutation of the SAME inode (SetAttr,
    /// Truncate, Write updating mtime/ctime, Link, Unlink, Rename) via
    /// `inode_bump_generation()`. The Core stat fast path
    /// (`sys_statx_fast_path`) compares a prefetch entry's stored value
    /// against `inode_current_generation()`; a mismatch means the cached
    /// `StatxBuf` is stale and the syscall falls through to the VFS domain.
    ///
    /// Written with `Release` (the inode is already locked for the
    /// mutation, so this adds zero contention); read with `Acquire` by
    /// Core. u64: at 10M mutations/sec on one inode it wraps after ~58,000
    /// years — no wrap handling needed.
    pub i_meta_generation: AtomicU64,

    /// Futex-owned inode sequence number — the storage backing
    /// `futex_seq()`
    /// ([Section 19.4](19-sysapi.md#futex-and-userspace-synchronization)), which forms the
    /// `{i_seq, pgoff}` shared-futex key. The VFS only DECLARES and
    /// zero-initializes this field; all assignment logic lives in the
    /// futex subsystem (mirroring Linux, where the counter and
    /// Linux `get_inode_sequence_number()` lives in `kernel/futex/core.c`).
    ///
    /// **Contract**: 0 = unassigned sentinel (an assigned sequence is
    /// never 0). Lazily CAS-installed on first futex use from the futex
    /// subsystem's global monotonic u64 counter. In-memory-only, NEVER
    /// persisted to disk: a NEW in-memory instance created after
    /// eviction/re-read of the same on-disk inode starts at 0 again and
    /// draws a FRESH sequence on its next futex use — the property that
    /// makes a stale futex key never falsely match a re-instantiated
    /// inode (no false positives in futex key matching). Initialized to
    /// 0 at every inode-instantiation site (the `I_NEW` construction
    /// path — see the `inode_cache_insert()` preconditions).
    pub i_seq: AtomicU64,

    /// Per-inode spinlock. Serializes inode refcount revival and teardown —
    /// the `i_refcount` 0->1 revival and `I_FREEING` transition
    /// (`inode_ref_acquire`/`iput`) and the LRU (un)linking done under it.
    /// Identity fields (`i_mode`/`i_uid`/`i_gid`) are self-synchronizing
    /// atomics; timestamps live in the `i_times` data-in-lock block above;
    /// `i_nlink` is atomic. Lock level: INODE_LOCK (level 160,
    /// [Section 3.4](03-concurrency.md#cumulative-performance-budget--lock-ordering)).
    pub i_lock: SpinLock<(), INODE_LOCK>,

    /// Read-write semaphore for file data. read()/readdir() take shared;
    /// write()/truncate() take exclusive.
    pub i_rwsem: RwLock<()>,

    /// Superblock this inode belongs to.
    pub i_sb: Arc<SuperBlock>,

    /// Inode operations (lookup, create, link, unlink, etc.).
    /// Set by the filesystem when the inode is created.
    pub i_op: &'static dyn InodeOps,

    /// File operations (read, write, mmap, ioctl, etc.).
    /// Set by the filesystem; used when opening this inode as a file.
    pub i_fop: &'static dyn FileOps,

    /// Filesystem-private data. Opaque pointer used by the filesystem
    /// driver to attach its own per-inode state (e.g., ext4_inode_info).
    /// SAFETY: Set to a filesystem-specific type (e.g., *mut Ext4InodeInfo)
    /// during inode initialization under I_NEW flag. The filesystem's
    /// evict_inode() method must cast back to the original type and free.
    /// Type safety is NOT enforced — callers must maintain the type
    /// invariant. Set once during inode init, read-only thereafter.
    /// Concurrent access is safe because i_private is immutable after
    /// I_NEW is cleared.
    ///
    /// `unsafe impl Send for Inode {}` — SAFETY: i_private is set once
    /// during inode initialization (under I_NEW). After I_NEW is cleared,
    /// i_private is read-only. All other fields of Inode are either atomic
    /// or protected by documented locks.
    /// `unsafe impl Sync for Inode {}` — same safety argument applies.
    pub i_private: *mut (),

    /// Per-inode LSM security blob. Allocated by `security_inode_alloc()`
    /// during `Inode::get()`; freed by `security_inode_free()` in `evict()`
    /// step 6a (both dispositions — this is in-memory teardown). See
    /// [Section 9.8](09-security.md#linux-security-module-framework) for LsmBlob definition and
    /// lifecycle.
    pub i_security: Option<NonNull<LsmBlob>>,

    /// Page cache address space for this inode's data.
    /// Contains the `PageCache` storage backend, writeback coordination,
    /// and error tracking. See the `AddressSpace` struct defined above.
    pub i_mapping: AddressSpace,

    /// Reference count. Managed by dentry references and open file handles.
    /// u32: bounded by concurrent references (max_files sysctl, default 8M).
    /// Same rationale as Dentry.d_refcount — hot-path, ILP32 penalty.
    pub i_refcount: AtomicU32,

    /// RCU callback head for deferred inode slab free. The inode struct
    /// cannot be freed immediately when the last reference is dropped
    /// because concurrent RCU readers (path lookup via `rcu_walk`,
    /// `inode_cache_lookup()` under `rcu_read_lock()`) may still hold pointers
    /// to this inode. Instead, evict() step 6e calls
    /// `call_rcu(&inode.i_rcu, inode_free_rcu)` which defers the slab
    /// free until all RCU readers have exited their critical sections.
    /// `inode_free_rcu` calls `inode_slab.free(inode)` to return the
    /// object to the inode slab cache.
    pub i_rcu: RcuHead,

    /// Dirty flag. Set when inode metadata has been modified in memory
    /// but not yet written to disk.
    pub i_state: AtomicU32,

    /// Inode cache membership flag is tracked in `i_state` (`I_HASHED` bit).
    /// Lookup is via per-superblock XArray (`SuperBlock.inode_cache`).
    /// No hash-table linkage node — XArray manages its own internal nodes.

    /// Superblock dirty inode list linkage.
    pub i_sb_list: IntrusiveListNode,

    // ---- Writeback integration ----

    /// Nanosecond timestamp when this inode was first dirtied (metadata or data).
    /// Set by `mark_inode_dirty()` on the first dirty transition (I_DIRTY_*
    /// flags going from 0 → non-zero). Used by the writeback subsystem to
    /// implement `dirty_expire_centisecs`: inodes dirtied longer than the
    /// threshold are prioritized for writeback. Zero when clean.
    pub dirtied_when: AtomicU64,

    /// Writeback association. Links this inode to a specific `BdiWriteback`
    /// instance (backing device writeback context; cgroup-v2 dirty-page
    /// attribution — [Section 4.6](04-memory.md#writeback-subsystem)). `None` (empty) for inodes
    /// on pseudo-filesystems (procfs, sysfs) that have no backing device
    /// and for inodes never dirtied.
    ///
    /// **Type**: `ArcSwapOption<BdiWriteback>`
    /// ([Section 3.1](03-concurrency.md#rust-ownership-for-lock-free-paths--arcswap-lock-free-atomic-arct-replacement))
    /// — the field is mutated after inode initialization through shared
    /// `&Inode` references (a plain `Option<Arc<..>>` would be unwritable
    /// and a data race), and readers must be able to hold the context
    /// across blocking I/O, so reads return an owned `Arc` clone.
    ///
    /// **Update protocol** (writers serialized by `i_lock`, `INODE_LOCK`
    /// level 160):
    /// - ATTACH — `inode_attach_wb()` ([Section 4.6](04-memory.md#writeback-subsystem)) on the
    ///   FIRST dirtying of the inode, double-checked under `i_lock`:
    ///   `if i_wb.is_none() { i_lock.lock(); if i_wb.is_none() {
    ///   i_wb.store(Some(wb)); } }`. Two racing first-dirtiers cannot
    ///   both attach; the attribution is the winner's cgroup.
    /// - READ — `i_wb.load_full()` returns an owned
    ///   `Option<Arc<BdiWriteback>>`; `None` falls back to the device's
    ///   root `BdiWriteback` (see `bdi_writeback_for()`,
    ///   [Section 4.6](04-memory.md#writeback-subsystem)).
    /// - DETACH — eviction step 6a stores `None` under `i_lock`. The
    ///   attribution is NEVER cleared merely because writeback completed:
    ///   it persists for the inode's in-memory lifetime (matching
    ///   `wb_remove_inode_from_lists()`'s contract that attribution "is
    ///   released separately when the inode is evicted"). Detach at
    ///   eviction releases the `Arc` so an offlined cgroup's writeback
    ///   context can drain and free — no accumulation over the 50-year
    ///   uptime budget. Precondition: the inode is off all wb dirty lists
    ///   (eviction step 1) before detach.
    pub i_wb: ArcSwapOption<BdiWriteback>,

    /// BDI dirty inode list linkage. Links this inode into the per-BDI
    /// dirty list (`BdiWriteback.b_dirty`, `b_io`, or `b_more_io`)
    /// maintained by the writeback subsystem. The writeback thread walks
    /// these lists to find inodes that need flushing. The node is unlinked
    /// when the inode is no longer dirty.
    pub wb_link: IntrusiveListNode,
}

impl Inode {
    /// Pin this inode against eviction by taking an owned reference
    /// (Linux `igrab`). Callers holding only a borrowed `&Inode` (e.g.
    /// while walking a BDI dirty list under `rcu_read_lock`) use this to
    /// obtain an `Arc<Inode>` they can hold across a blocking section.
    ///
    /// Returns `None` only when teardown is in progress (`I_FREEING` —
    /// see `inode_ref_acquire()` in the lifecycle state machine below).
    /// `I_WILL_FREE` (unlinked-but-open) inodes ARE grabbable: they are
    /// live, and the writeback path must be able to pin them to flush
    /// their remaining dirty pages.
    ///
    /// Unlike a lookup, `igrab()` never waits: a `FREEING` inode yields
    /// `None` and the caller skips it.
    pub fn igrab(&self) -> Option<Arc<Inode>> {
        match inode_ref_acquire(self) {
            // Memory-lifetime handle: the icache still holds the Arc
            // (unhashing happens only at evict() step 6, which cannot
            // run concurrently with the reference we just acquired).
            InodeAcquire::Acquired => self.i_sb.inode_cache.get(self.i_ino),
            InodeAcquire::Freeing => None,
        }
    }
}

14.1.2.4.1.7 Inode Attribute Flags

Pseudocode convention: Code in this subsection uses Rust syntax and follows Rust ownership, borrowing, and type rules. &self methods use interior mutability for mutation. Atomic fields use .store()/.load() with explicit Ordering. All #[repr(C)] structs have const_assert! size verification. See CLAUDE.md §Spec Pseudocode Quality Gates.

Per-inode behavior flags exist in TWO distinct spaces:

1. InodeFlags — the kernel-internal space (Inode::i_flags). A native UmkaOS bitflags! set whose bit layout is a private Core allocation that no external interface observes. This is the word every enforcement point reads. Some bits are kernel-managed lifecycle state (InodeFlags::DEAD, InodeFlags::SWAPFILE, InodeFlags::PRIVATE, InodeFlags::KERNEL_FILE, …) that never corresponds to a persistent user flag.
2. FS_*_FL — the persistent user space (include/uapi/linux/fs.h). This is the word chattr(1)/lsattr(1) exchange through FS_IOC_GETFLAGS/FS_IOC_SETFLAGS and the word filesystems persist on disk (ext4 i_flags, XFS di_flags-derived, Btrfs inode item flags). These values are a Linux-mandated external ABI, unrelated to the InodeFlags bit layout — the two spaces must never be mixed.

The filesystem driver owns the persistent FS_*_FL word; Core owns the in-memory InodeFlags mirror that enforcement reads. fs_flags_to_inode_flags() below is the one translation point between the two spaces.

// UmkaOS kernel-internal inode behavior flags — the bits of
// `Inode::i_flags`. This word is never exposed to userspace: it is a
// private Core allocation that enforcement points read directly. The
// persistent, user-visible attributes live in the separate `FS_*_FL`
// space below and are translated in through `fs_flags_to_inode_flags()`.
// Bit values are an internal allocation; nothing external observes them.
bitflags! {
    pub struct InodeFlags: u32 {
        /// Writes to this inode are synchronous (`chattr +S`). Enforcement
        /// uses `Inode::is_sync()`, which also honors the superblock-wide
        /// `MS_SYNCHRONOUS` mount flag (Linux `IS_SYNC()` parity).
        const SYNC        = 1 << 0;
        /// Do not update atime on access (`chattr +A`). Checked first in
        /// the atime-update decision (Linux `fs/inode.c
        /// atime_needs_update()` returns early for a no-atime inode).
        const NOATIME     = 1 << 1;
        /// Append-only file (`chattr +a`). See the enforcement map.
        const APPEND      = 1 << 2;
        /// Immutable file (`chattr +i`). See the enforcement map.
        const IMMUTABLE   = 1 << 3;
        /// Removed, but still open directory. Kernel-managed: set by the
        /// rmdir/rename-victim teardown on the victim directory inode; makes
        /// `IS_DEADDIR` checks in create/delete paths return `ENOENT`.
        const DEAD        = 1 << 4;
        /// Inode is not counted toward quota. Kernel-managed
        /// ([Section 14.15](#disk-quota-subsystem)).
        const NOQUOTA     = 1 << 5;
        /// Directory modifications are synchronous (`chattr +D`).
        /// Enforcement uses `Inode::is_dirsync()` (Linux `IS_DIRSYNC()`).
        const DIRSYNC     = 1 << 6;
        /// Do not update file c/mtime (filesystem-internal, e.g. XFS DMAPI).
        const NOCMTIME    = 1 << 7;
        /// Active swap file: do not truncate — swapon captured its block
        /// mappings. Kernel-managed by swapon/swapoff ([Section 4.13](04-memory.md#swap-subsystem)).
        const SWAPFILE    = 1 << 8;
        /// Filesystem-internal inode: LSM hooks skip it. Kernel-managed.
        const PRIVATE     = 1 << 9;
        /// Inode has an associated IMA structure
        /// ([Section 9.5](09-security.md#runtime-integrity-measurement)). Kernel-managed.
        const IMA         = 1 << 10;
        /// Automount/referral quasi-directory (autofs,
        /// [Section 14.10](#autofs-kernel-automount-trigger)). Reflected as
        /// `STATX_ATTR_AUTOMOUNT`.
        const AUTOMOUNT   = 1 << 11;
        /// No suid or xattr security attributes: write paths may skip the
        /// `should_remove_suid()` check until the next attribute change.
        /// Kernel-managed optimization bit.
        const NOSEC       = 1 << 12;
        /// Direct Access inode (persistent memory, no page cache). Kept in
        /// sync with `AddressSpaceFlags::DAX` at instantiation —
        /// `AddressSpaceFlags::DAX` remains the mapping-level authority
        /// ([Section 15.16](15-storage.md#persistent-memory--design-dax-direct-access-integration)).
        const DAX         = 1 << 13;
        /// Encrypted file (fscrypt, [Section 15.20](15-storage.md#fscrypt-file-level-encryption)).
        const ENCRYPTED   = 1 << 14;
        /// Casefolded (case-insensitive) directory.
        const CASEFOLD    = 1 << 15;
        /// fs-verity protected file.
        const VERITY      = 1 << 16;
        /// File is in use by the kernel (e.g. held open by a kernel
        /// subsystem); protects against tampering. Kernel-managed.
        const KERNEL_FILE = 1 << 17;
        /// Anonymous inode (anon_inode-class fds). Kernel-managed. Each anon-inode
        /// instance is inserted into the anon-inode pseudo-filesystem superblock's
        /// `inode_cache` at creation (setting `I_HASHED`) and removed at destruction,
        /// exactly like every other inode — the universal cache-membership invariant
        /// ([Section 14.1](#virtual-filesystem-layer--inode-cache-icache), Invariants) leaves no
        /// inode live-but-unhashed, so an `InodeFlags::ANON_INODE` fd's inode is
        /// enumerable via `SUPER_BLOCK_MAP` × `inode_cache` for the whole of its life
        /// (which the Shadow-and-Migrate layout walk depends on).
        const ANON_INODE  = 1 << 19;
    }
}

// Persistent user-visible inode flags — the FS_IOC_GETFLAGS/SETFLAGS and
// on-disk flag space. Values are the Linux userspace ABI, verified against
// torvalds/linux master `include/uapi/linux/fs.h`. UmkaOS must expose
// these identically or chattr/lsattr break.
pub const FS_SECRM_FL: u32        = 0x0000_0001; // Secure deletion
pub const FS_UNRM_FL: u32         = 0x0000_0002; // Undelete
pub const FS_COMPR_FL: u32        = 0x0000_0004; // Compress file
pub const FS_SYNC_FL: u32         = 0x0000_0008; // Synchronous updates
pub const FS_IMMUTABLE_FL: u32    = 0x0000_0010; // Immutable file
pub const FS_APPEND_FL: u32       = 0x0000_0020; // Writes may only append
pub const FS_NODUMP_FL: u32       = 0x0000_0040; // Do not dump file
pub const FS_NOATIME_FL: u32      = 0x0000_0080; // Do not update atime
pub const FS_DIRTY_FL: u32        = 0x0000_0100;
pub const FS_COMPRBLK_FL: u32     = 0x0000_0200; // Compressed clusters
pub const FS_NOCOMP_FL: u32       = 0x0000_0400; // Don't compress
pub const FS_ENCRYPT_FL: u32      = 0x0000_0800; // Encrypted file
pub const FS_BTREE_FL: u32        = 0x0000_1000; // btree format dir
pub const FS_INDEX_FL: u32        = 0x0000_1000; // hash-indexed directory
pub const FS_IMAGIC_FL: u32       = 0x0000_2000; // AFS directory
pub const FS_JOURNAL_DATA_FL: u32 = 0x0000_4000; // Reserved for ext3/4
pub const FS_NOTAIL_FL: u32       = 0x0000_8000; // No tail-merging
pub const FS_DIRSYNC_FL: u32      = 0x0001_0000; // dirsync (directories)
pub const FS_TOPDIR_FL: u32       = 0x0002_0000; // Top of dir hierarchies
pub const FS_HUGE_FILE_FL: u32    = 0x0004_0000; // Reserved for ext4
pub const FS_EXTENT_FL: u32       = 0x0008_0000; // Extents
pub const FS_VERITY_FL: u32       = 0x0010_0000; // Verity protected inode
pub const FS_EA_INODE_FL: u32     = 0x0020_0000; // Large-EA inode
pub const FS_EOFBLOCKS_FL: u32    = 0x0040_0000; // Reserved for ext4
pub const FS_NOCOW_FL: u32        = 0x0080_0000; // Do not CoW (btrfs)
pub const FS_DAX_FL: u32          = 0x0200_0000; // Inode is DAX
pub const FS_INLINE_DATA_FL: u32  = 0x1000_0000; // Reserved for ext4
pub const FS_PROJINHERIT_FL: u32  = 0x2000_0000; // Inherit parent projid
pub const FS_CASEFOLD_FL: u32     = 0x4000_0000; // Casefolded directory
pub const FS_RESERVED_FL: u32     = 0x8000_0000; // Reserved for ext2 lib

/// Generic user-visible / user-modifiable masks (Linux
/// `include/uapi/linux/fs.h`). These are the FLOOR, not a Core-enforced
/// ceiling: exactly as in Linux, each filesystem applies ITS OWN
/// modifiable mask (btrfs accepts `FS_NOCOW_FL`, ext4 accepts
/// `FS_CASEFOLD_FL`/`FS_DAX_FL`, … — all outside this generic mask), so
/// `vfs_fileattr_set()` passes the full word through and the DRIVER
/// rejects bits it does not support with `EOPNOTSUPP`.
pub const FS_FL_USER_VISIBLE: u32    = 0x0003_DFFF;
pub const FS_FL_USER_MODIFIABLE: u32 = 0x0003_80FF;

/// Translate a persistent `FS_*_FL` word into the kernel-internal
/// `InodeFlags` set cached in `Inode::i_flags`. This is the single
/// translation point between the two spaces, hoisted into the VFS so
/// every filesystem gets identical translation semantics. Persistent
/// bits without an `InodeFlags` equivalent (`FS_NODUMP_FL`,
/// `FS_NOCOW_FL`, `FS_PROJINHERIT_FL`, compression bits, …) have no
/// in-memory mirror: they are persistent-only state the driver reports
/// through `fileattr_get()` and, where applicable, the statx
/// `attributes` field. Warm/cold path (instantiation, chattr).
pub fn fs_flags_to_inode_flags(fs_flags: u32) -> InodeFlags {
    let mut f = InodeFlags::empty();
    if fs_flags & FS_SYNC_FL != 0      { f |= InodeFlags::SYNC; }
    if fs_flags & FS_NOATIME_FL != 0   { f |= InodeFlags::NOATIME; }
    if fs_flags & FS_APPEND_FL != 0    { f |= InodeFlags::APPEND; }
    if fs_flags & FS_IMMUTABLE_FL != 0 { f |= InodeFlags::IMMUTABLE; }
    if fs_flags & FS_DIRSYNC_FL != 0   { f |= InodeFlags::DIRSYNC; }
    if fs_flags & FS_ENCRYPT_FL != 0   { f |= InodeFlags::ENCRYPTED; }
    if fs_flags & FS_CASEFOLD_FL != 0  { f |= InodeFlags::CASEFOLD; }
    if fs_flags & FS_VERITY_FL != 0    { f |= InodeFlags::VERITY; }
    if fs_flags & FS_DAX_FL != 0       { f |= InodeFlags::DAX; }
    f
}

impl Inode {
    /// Linux `IS_IMMUTABLE()` parity. Hot path: one `Relaxed` load —
    /// the value gates a control decision only; no data is published
    /// through it, so no ordering is required (a concurrent `chattr`
    /// races exactly as it does in Linux — see "Enforcement raciness").
    pub fn is_immutable(&self) -> bool {
        InodeFlags::from_bits_retain(self.i_flags.load(Ordering::Relaxed))
            .contains(InodeFlags::IMMUTABLE)
    }

    /// Linux `IS_APPEND()` parity. Hot path: one `Relaxed` load.
    pub fn is_append(&self) -> bool {
        InodeFlags::from_bits_retain(self.i_flags.load(Ordering::Relaxed))
            .contains(InodeFlags::APPEND)
    }

    /// Linux `IS_SYNC()` parity: per-inode `InodeFlags::SYNC` OR
    /// superblock-wide `MS_SYNCHRONOUS` (`mount -o sync`).
    pub fn is_sync(&self) -> bool {
        InodeFlags::from_bits_retain(self.i_flags.load(Ordering::Relaxed))
            .contains(InodeFlags::SYNC)
            || self.i_sb.s_flags.load(Ordering::Relaxed)
                & MS_SYNCHRONOUS != 0
    }

    /// Linux `IS_DIRSYNC()` parity: directory modifications must reach
    /// stable storage before the operation returns when the superblock
    /// is `MS_SYNCHRONOUS`/`MS_DIRSYNC` or the directory inode carries
    /// `InodeFlags::SYNC`/`InodeFlags::DIRSYNC`.
    pub fn is_dirsync(&self) -> bool {
        InodeFlags::from_bits_retain(self.i_flags.load(Ordering::Relaxed))
            .intersects(InodeFlags::SYNC | InodeFlags::DIRSYNC)
            || self.i_sb.s_flags.load(Ordering::Relaxed)
                & (MS_SYNCHRONOUS | MS_DIRSYNC) != 0
    }

    /// Linux `IS_SWAPFILE()` parity.
    pub fn is_swapfile(&self) -> bool {
        InodeFlags::from_bits_retain(self.i_flags.load(Ordering::Relaxed))
            .contains(InodeFlags::SWAPFILE)
    }

    /// Linux `IS_DEADDIR()` parity — removed-but-open directory.
    pub fn is_dead_dir(&self) -> bool {
        InodeFlags::from_bits_retain(self.i_flags.load(Ordering::Relaxed))
            .contains(InodeFlags::DEAD)
    }
}

Lifecycle of the mirror. Inode::i_flags is written at exactly three points, all Core-side:

1. Instantiation (the I_NEW construction path — see the inode_cache_insert() preconditions): i_flags = fs_flags_to_inode_flags(attr.fs_flags).bits(), where attr.fs_flags is the persistent flag word the driver returned in the instantiating InodeAttr (field defined on the struct above; 0 for filesystems without persistent flags — enforcement-neutral). Plain store: the inode is not yet published.
2. vfs_fileattr_set() (below): authoritative read-back from the driver after a successful chattr, Release store under i_rwsem write.
3. vfs_update_inode_flags() (below): driver-initiated refresh when a network/cluster filesystem revalidates attributes (NFS attr-cache refresh, CIFS lease update, peerfs metadata push) — the same trigger class as vfs_invalidate_prefetch().

Kernel-managed bits (InodeFlags::DEAD, InodeFlags::SWAPFILE, InodeFlags::NOSEC, InodeFlags::IMA, InodeFlags::KERNEL_FILE, InodeFlags::PRIVATE, InodeFlags::ANON_INODE) are set/cleared by their owning subsystems with fetch_or/fetch_and (Release) — they never round-trip through FS_*_FL and are unreachable from the ioctl surface.

Enforcement raciness (Linux-exact): flag checks are point-in-time — an operation that passed its gate proceeds even if chattr +i lands mid-operation, exactly as in Linux. UmkaOS is in fact strictly stronger for the write path: vfs_fileattr_set() holds i_rwsem write, so it serializes against the entire buffered-write loop (page_cache_write_iter() Step 1), which Linux's inode_lock also does. No enforcement point may sleep-retry on a flag check.

Interaction with cached_perm: the dentry permission cache (Section 14.1, "Capability checks") caches DAC/LSM rwx grants ONLY. Immutable/append gates are evaluated as separate i_flags loads at each enforcement point, never stored in cached_perm — so chattr requires no permission-cache invalidation and a cached write grant can never override InodeFlags::IMMUTABLE.

The chattr/lsattr surface (FS_IOC_GETFLAGS / FS_IOC_SETFLAGS)

/// FS_IOC_GETFLAGS — `_IOR('f', 1, long)` — and FS_IOC_SETFLAGS —
/// `_IOW('f', 2, long)` (Linux `include/uapi/linux/fs.h`). The encoded
/// size is `sizeof(long)`, so the request VALUE differs by ABI width:
///
///   LP64  (x86-64, AArch64, RISC-V 64, PPC64LE, s390x, LoongArch64):
///     FS_IOC_GETFLAGS = 0x8008_6601, FS_IOC_SETFLAGS = 0x4008_6602
///   ILP32 (ARMv7, PPC32) — identical to Linux FS_IOC32_GETFLAGS/SETFLAGS
///   (`_IOR('f', 1, int)` / `_IOW('f', 2, int)`):
///     FS_IOC_GETFLAGS = 0x8004_6601, FS_IOC_SETFLAGS = 0x4004_6602
///
/// **Transfer-width quirk (Linux-exact)**: despite the `long` in the
/// ioctl encoding, Linux transfers a 32-bit `int` at the user pointer
/// (`fs/file_attr.c ioctl_getflags()/ioctl_setflags()` use
/// Linux's `unsigned int __user *argp`). UmkaOS matches: a 4-byte user access on
/// all eight architectures, for BOTH request values.
#[cfg(target_pointer_width = "64")]
pub const FS_IOC_GETFLAGS: u32 = 0x8008_6601; // _IOR('f', 1, long), LP64
#[cfg(target_pointer_width = "64")]
pub const FS_IOC_SETFLAGS: u32 = 0x4008_6602; // _IOW('f', 2, long), LP64
#[cfg(target_pointer_width = "32")]
pub const FS_IOC_GETFLAGS: u32 = 0x8004_6601; // _IOR('f', 1, long), ILP32
#[cfg(target_pointer_width = "32")]
pub const FS_IOC_SETFLAGS: u32 = 0x4004_6602; // _IOW('f', 2, long), ILP32

/// Persistent-attribute exchange record for `InodeOps::fileattr_get` /
/// `fileattr_set` — the UmkaOS analogue of Linux `struct file_kattr`
/// restricted to the flags surface. `#[repr(C)]`: crosses the VFS ring
/// by value (`VfsRequestArgs::FileattrGet`/`FileattrSet`,
/// [Section 14.2](#vfs-ring-buffer-protocol)).
#[repr(C)]
#[derive(Clone, Copy)]
pub struct FileAttr {
    /// `FS_*_FL` flag word (persistent user space, never `InodeFlags`).
    pub flags: u32,
    /// Reserved for the `FS_IOC_FSGETXATTR`/`FS_IOC_FSSETXATTR`
    /// extension (fsx_xflags, fsx_extsize, fsx_projid, fsx_cowextsize —
    /// Linux `struct fsxattr`). Must be zero; a driver receiving nonzero
    /// reserved words rejects with `EINVAL`. Sized so the record does
    /// not change size when that surface lands (KABI stability).
    pub _reserved: [u32; 7],
}
const_assert!(size_of::<FileAttr>() == 32);

Both ioctls are handled by the generic VFS ioctl dispatch (the same Core-side layer that handles FIOCLEX/FIONREAD/FICLONE — see FileOps::ioctl and Section 14.4); they are never forwarded raw to FileOps::ioctl (Linux parity: do_vfs_ioctl() handles them before f_op->unlocked_ioctl). The generic handler transfers the 32-bit word from/to userspace and calls:

/// FS_IOC_GETFLAGS backend. Returns the driver-authoritative persistent
/// flag word — NOT the Core mirror, which holds only the `InodeFlags`-mappable
/// subset. Cold path. Linux parity: `fs/file_attr.c vfs_fileattr_get()`
/// (missing `->fileattr_get` ⇒ `ENOIOCTLCMD` ⇒ `ENOTTY` to userspace —
/// here the trait default returns `ENOTTY` directly).
pub fn vfs_fileattr_get(file: &OpenFile) -> Result<u32, Errno> {
    // LSM hook `inode_file_getattr` (Linux `security_inode_file_getattr`)
    // runs here, before the driver call, via the standard LSM dispatch
    // ([Section 9.8](09-security.md#linux-security-module-framework)).
    let fa = file.inode.i_op.fileattr_get(InodeId(file.inode.i_ino))?;
    Ok(fa.flags)
}

/// FS_IOC_SETFLAGS backend — all permission logic is Core-side, so the
/// checks are identical at every deployment tier; only the final
/// persistence call crosses `kabi_call!`. Cold path. Linux parity:
/// `fs/file_attr.c ioctl_setflags()` + `vfs_fileattr_set()` +
/// `fileattr_set_prepare()`, verified against torvalds/linux master.
pub fn vfs_fileattr_set(file: &OpenFile, new_flags: u32) -> Result<(), Errno> {
    let inode = &*file.inode;
    let task = current_task();

    // 1. Write access to the filesystem FIRST (Linux: the caller —
    //    the ioctl flag-setting path acquires mount write access
    //    BEFORE `vfs_fileattr_set()`): EROFS on a read-only superblock or
    //    read-only mount; blocks (not EROFS) while frozen — the same
    //    Write-level freeze bracket as every write-class path.
    if inode.i_sb.s_flags.load(Ordering::Relaxed) & MS_RDONLY != 0 {
        return Err(Errno::EROFS);
    }
    {
        let guard = rcu_read_lock();
        let mnt = Arc::clone(&*file.mount.read(&guard));
        drop(guard);
        if mnt.flags.load(Ordering::Relaxed) & MNT_READONLY != 0 {
            return Err(Errno::EROFS);
        }
    }
    let _write = sb_start_write(&inode.i_sb, SbFreezeLevel::Write)?;

    // 2. Filesystem-support probe (Linux `vfs_fileattr_set()`:
    //    `!inode->i_op->fileattr_set` ⇒ `ENOIOCTLCMD`, surfaced as ENOTTY).
    //    Gates on the SETTER, lock-free and side-effect-free, and runs
    //    BEFORE the owner check — so SETFLAGS on a filesystem without
    //    persistent flags returns ENOTTY even to a non-owner.
    if !inode.i_op.supports_fileattr() {
        return Err(Errno::ENOTTY);
    }

    // 3. Owner-or-capable (Linux `inode_owner_or_capable()` → EPERM):
    //    fsuid must own the inode, or the caller holds CAP_FOWNER.
    if file.f_cred.fsuid != inode.i_uid.load(Ordering::Relaxed)
        && !has_cap(task, SystemCaps::CAP_FOWNER)
    {
        return Err(Errno::EPERM);
    }

    // 4. Serialize the read-modify-write against concurrent chattr,
    //    writes, and truncate (Linux `inode_lock(inode)` held across
    //    get + prepare + set).
    let _guard = inode.i_rwsem.write();

    // 5. Current persistent word from the driver (Linux `vfs_fileattr_get()`
    //    under the inode lock; establishes the compare baseline).
    let old = inode.i_op.fileattr_get(InodeId(inode.i_ino))?;

    // 6. Capability gate (Linux `fileattr_set_prepare()`): CHANGING
    //    FS_APPEND_FL or FS_IMMUTABLE_FL — either direction — requires
    //    CAP_LINUX_IMMUTABLE ([Section 9.2](09-security.md#permission-and-acl-model)).
    if (new_flags ^ old.flags) & (FS_APPEND_FL | FS_IMMUTABLE_FL) != 0
        && !has_cap(task, SystemCaps::CAP_LINUX_IMMUTABLE)
    {
        return Err(Errno::EPERM);
    }

    // 7. LSM hook `inode_file_setattr` (Linux
    //    `security_inode_file_setattr`) via the standard LSM dispatch.
    //
    // 8. Driver persists the new word. The DRIVER applies its own
    //    modifiable mask and rejects bits it does not support with
    //    EOPNOTSUPP, leaving on-disk state unchanged (Linux parity:
    //    per-fs masks like ext4's are wider than FS_FL_USER_MODIFIABLE;
    //    Core deliberately does not pre-mask).
    inode.i_op.fileattr_set(
        InodeId(inode.i_ino),
        &FileAttr { flags: new_flags, _reserved: [0; 7] },
    )?;

    // 9. Authoritative read-back → refresh the Core mirror (the driver
    //    may have adjusted the word), publish, and invalidate cached
    //    stat state. ctime advances (Linux: the filesystem stamps ctime
    //    in ->fileattr_set; UmkaOS Core owns the cached triple).
    let now_fa = inode.i_op.fileattr_get(InodeId(inode.i_ino))?;
    inode.i_flags.store(fs_flags_to_inode_flags(now_fa.flags).bits(), Ordering::Release);
    {
        let mut times = inode.i_times.lock();
        times.i_ctime = Timespec::from_ns(current_time_ns());
    }
    inode.i_meta_generation.fetch_add(1, Ordering::Release);
    // fsnotify: IN_ATTRIB to watchers (Linux `fsnotify_xattr()` parity;
    // [Section 14.13](#file-notification-system)).
    Ok(())
}

/// Driver-initiated mirror refresh for network/cluster filesystems whose
/// persistent flags can change remotely (NFS attr-cache revalidation,
/// CIFS lease update, peerfs MetadataInvalidate). Resolves `(dev, ino)`
/// through `SUPER_BLOCK_MAP` + the superblock icache exactly like
/// `inode_bump_generation()`; no-op when the inode is not cached. Takes
/// `i_rwsem` write for the store (same writer discipline as
/// `vfs_fileattr_set`), then bumps `i_meta_generation` so stale statx
/// prefetch entries miss. Warm/cold path (invalidation events only).
pub fn vfs_update_inode_flags(dev: DevId, ino: InodeId, fs_flags: u32);

statx reflection. The Core statx filler derives the generic STATX_ATTR_* bits from the mirror and merges them into the driver-supplied InodeAttr.attributes:

// statx(2) attribute bits (`include/uapi/linux/stat.h`, torvalds/linux
// master). NOTE: several values coincide with the corresponding FS_*_FL
// bits (IMMUTABLE 0x10, APPEND 0x20, NODUMP 0x40, COMPRESSED 0x04,
// ENCRYPTED 0x800, VERITY 0x100000) but STATX_ATTR_DAX (0x0020_0000)
// does NOT equal FS_DAX_FL (0x0200_0000) — never convert by copying the
// word.
pub const STATX_ATTR_COMPRESSED: u64   = 0x0000_0004;
pub const STATX_ATTR_IMMUTABLE: u64    = 0x0000_0010;
pub const STATX_ATTR_APPEND: u64       = 0x0000_0020;
pub const STATX_ATTR_NODUMP: u64       = 0x0000_0040;
pub const STATX_ATTR_ENCRYPTED: u64    = 0x0000_0800;
pub const STATX_ATTR_AUTOMOUNT: u64    = 0x0000_1000;
pub const STATX_ATTR_MOUNT_ROOT: u64   = 0x0000_2000;
pub const STATX_ATTR_VERITY: u64       = 0x0010_0000;
pub const STATX_ATTR_DAX: u64          = 0x0020_0000;
pub const STATX_ATTR_WRITE_ATOMIC: u64 = 0x0040_0000;

/// Map the kernel-internal flag word onto statx attribute bits. Called
/// by the statx filler, which ORs the result into `stx_attributes` and
/// the corresponding constant set into `stx_attributes_mask` — so the
/// readdir-plus prefetch entries ([Section 14.1](#virtual-filesystem-layer--metadata-access-amortization))
/// carry correct attribute bits with no extra work, and any flag change
/// invalidates them via the `i_meta_generation` bump. Bits without an
/// `InodeFlags` mirror (`NODUMP`, `COMPRESSED`, `MOUNT_ROOT`,
/// `WRITE_ATOMIC`) come from the driver's `InodeAttr.attributes` / the
/// mount layer and are merged, not overwritten.
pub fn inode_flags_to_statx_attrs(flags: InodeFlags) -> u64 {
    let mut a: u64 = 0;
    if flags.contains(InodeFlags::IMMUTABLE) { a |= STATX_ATTR_IMMUTABLE; }
    if flags.contains(InodeFlags::APPEND)    { a |= STATX_ATTR_APPEND; }
    if flags.contains(InodeFlags::ENCRYPTED) { a |= STATX_ATTR_ENCRYPTED; }
    if flags.contains(InodeFlags::VERITY)    { a |= STATX_ATTR_VERITY; }
    if flags.contains(InodeFlags::DAX)       { a |= STATX_ATTR_DAX; }
    if flags.contains(InodeFlags::AUTOMOUNT) { a |= STATX_ATTR_AUTOMOUNT; }
    a
}

Per-flag enforcement map (NORMATIVE). Every row is a binding requirement on the named enforcement point; an implementation missing any row diverges from the Linux ABI. Errnos and conditions are verified against torvalds/linux master (file anchors in the last column).

Flag	Operation	Rule	Linux anchor
InodeFlags::IMMUTABLE	open for write / O_TRUNC	EPERM — open_and_install step 3e (Section 14.1)	Linux fs/namei.c inode_permission() ("Nobody gets write access to an immutable file"), reached from may_open()
InodeFlags::IMMUTABLE	every MAY_WRITE inode-permission check (create/remove in a directory, write-intent probes)	EPERM, checked as a direct i_flags load alongside the DAC check — never served from cached_perm	fs/namei.c inode_permission()
InodeFlags::IMMUTABLE	setattr class: chmod, chown, utimes-with-explicit-times (SetAttr dispatch; size change is the separate truncate row below)	EPERM in the Core dispatch wrapper before the ring	Linux fs/attr.c may_setattr() (ATTR_MODE\|ATTR_UID\|ATTR_GID\|ATTR_TIMES_SET ⇒ IS_IMMUTABLE\|\|IS_APPEND → EPERM; ATTR_SIZE deliberately excluded)
InodeFlags::IMMUTABLE	truncate(2)/ftruncate(2)	EPERM (write-permission path + setattr rule above)	fs/open.c vfs_truncate()
InodeFlags::IMMUTABLE	unlink/rmdir/rename (as victim)	EPERM in the may-delete parity block of the Core dispatch	fs/namei.c may_delete_dentry()
InodeFlags::IMMUTABLE	link(2) (as link target)	EPERM ("A link to an append-only or immutable file cannot be created")	Linux fs/namei.c vfs_link()
InodeFlags::IMMUTABLE	setxattr/removexattr	EPERM before the ring dispatch (Section 14.16)	fs/xattr.c may_write_xattr()
InodeFlags::IMMUTABLE	fallocate(2), any mode	EPERM	fs/open.c vfs_fallocate()
InodeFlags::IMMUTABLE	shared-writable mmap	transitively impossible: no write-open can exist (FMODE_WRITE unobtainable), and existing write fds predate the flag — Linux-identical	—
InodeFlags::IMMUTABLE/InodeFlags::APPEND	changing either bit via FS_IOC_SETFLAGS	CAP_LINUX_IMMUTABLE or EPERM (vfs_fileattr_set step 6)	fs/file_attr.c fileattr_set_prepare()
InodeFlags::APPEND	open with write access but no O_APPEND; open with O_TRUNC	EPERM — open_and_install step 3f	Linux fs/namei.c may_open() ("An append-only file must be opened in append mode for writing")
InodeFlags::APPEND	fcntl(F_SETFL) toggling O_APPEND (either direction)	EPERM (see OpenFile::f_flags doc)	Linux fs/fcntl.c setfl(): ((arg ^ filp->f_flags) & O_APPEND) && IS_APPEND
InodeFlags::APPEND	truncate(2)/ftruncate(2)	EPERM	fs/open.c vfs_truncate() / do_ftruncate()
InodeFlags::APPEND	setattr class (chmod/chown/utimes-with-times)	EPERM (same wrapper row as InodeFlags::IMMUTABLE)	fs/attr.c
InodeFlags::APPEND	unlink/rmdir/rename (as victim)	EPERM	fs/namei.c may_delete_dentry()
InodeFlags::APPEND (on a DIRECTORY)	removing/renaming ANY entry out of an append-only directory	EPERM (IS_APPEND(dir) — creation inside remains legal)	Linux fs/namei.c may_delete_dentry()
InodeFlags::APPEND	link(2) (as link target)	EPERM	Linux fs/namei.c vfs_link()
InodeFlags::APPEND	setxattr/removexattr	EPERM	fs/xattr.c may_write_xattr()
InodeFlags::APPEND	fallocate(2) with any mode bit other than FALLOC_FL_KEEP_SIZE (pure preallocation stays legal)	EPERM	Linux fs/open.c vfs_fallocate(): (mode & ~FALLOC_FL_KEEP_SIZE) && IS_APPEND
InodeFlags::APPEND	mmap(MAP_SHARED) through a write-mode fd	EACCES in establish_mapping's file-backed validation (Section 4.8) — this row is normative for that step	Linux mm/mmap.c do_mmap() ("Make sure we don't allow writing to an append-only file")
InodeFlags::SYNC	every buffered/direct write on the inode	treated as O_SYNC: page_cache_write_iter() Step 6 flushes data + metadata before returning	include/linux/fs.h IS_SYNC()
InodeFlags::DIRSYNC	directory-mutating ops (create/link/unlink/mkdir/rmdir/rename/symlink/mknod)	the DRIVER commits the directory update (journal transaction / synchronous metadata write) before responding when is_dirsync() holds — driver-side contract, stated on the InodeOps methods' dispatch	include/linux/fs.h IS_DIRSYNC()
InodeFlags::NOATIME	every atime update decision	suppressed — checked FIRST, before mount MS_NOATIME/relatime logic (see InodeTimes::i_atime)	fs/inode.c atime_needs_update()
InodeFlags::NOATIME (FL)	setting FS_NOATIME_FL via chattr	owner-or-CAP_FOWNER (step 3 covers it — no extra capability, Linux parity)	fs/file_attr.c vfs_fileattr_set()
InodeFlags::SWAPFILE	unlink/rename (as victim)	EPERM	fs/namei.c may_delete_dentry()
InodeFlags::SWAPFILE	fallocate(2)	ETXTBSY	fs/open.c vfs_fallocate()
InodeFlags::SWAPFILE	set/clear	kernel-managed by swapon/swapoff only — never reachable from FS_IOC_SETFLAGS (fs_flags_to_inode_flags has no source bit)	mm/swapfile.c
InodeFlags::DEAD	create/lookup/delete inside a removed-but-open directory	ENOENT (is_dead_dir() in the may-create/may-delete parity checks); set on the victim inode when rmdir/rename removes a directory	fs/namei.c (IS_DEADDIR)
FS_NODUMP_FL	—	no kernel enforcement: persistent + statx-reflected only (STATX_ATTR_NODUMP, driver-supplied); consumed by dump(8)-class userspace	include/uapi/linux/fs.h

Bits not listed (compression bits, FS_NOCOW_FL, FS_PROJINHERIT_FL, InodeFlags::NOQUOTA, InodeFlags::PRIVATE, InodeFlags::IMA, InodeFlags::NOSEC, InodeFlags::KERNEL_FILE, InodeFlags::ANON_INODE, InodeFlags::ENCRYPTED, InodeFlags::CASEFOLD, InodeFlags::VERITY, InodeFlags::DAX) carry no VFS-generic enforcement row: their semantics belong to their owning subsystems (btrfs CoW, quota, IMA, fscrypt, fs-verity, casefold lookup, DAX/persistent-memory), which read the same substrate.

Tier note — how the flags travel. Inode (and therefore i_flags) is Core-resident state that no driver domain maps; the persistent FS_*_FL word is driver/on-disk state that Core never scrapes. The coherence rule is the same one i_seq and i_size follow: Core-side derived state is updated only through defined VFS entry points — instantiation (InodeAttr.fs_flags), vfs_fileattr_set() read-back, and vfs_update_inode_flags() — each of which works over kabi_call! at ANY deployment tier (co-located: direct call; isolated: ring round-trip via VfsOpcode::FileattrGet/FileattrSet, Section 14.2). Every ENFORCEMENT read is a local Core memory load — zero ring traffic, zero cycles added to paths where no flag is set beyond the one predicted branch. After a driver crash, the mirror survives in Core and remains enforceable throughout recovery; the lazy fd-revalidation pass (Section 14.1) reconciles i_size/timestamps via vfs_apply_reconciled_attr(), whose InodeAttr now also carries fs_flags — a journal-replay rollback of a chattr is folded back into the mirror at the same point.

14.1.2.4.1.8 Inode Lifecycle and Page Cache Teardown

This subsection specifies the interaction between inode reference counting, page cache lifetime, and the eviction sequence. These paths are critical for correctness: a missed writeback silently loses data; a missed page free leaks memory; a race between eviction and page fault corrupts the page cache.

Inode state flags (i_state: AtomicU32):

/// Inode state bitflags stored in `Inode::i_state`.
///
/// Multiple flags may be set simultaneously. All updates use atomic
/// CAS (compare-and-swap) on `i_state` — no separate lock is required
/// for flag manipulation, but metadata fields protected by `i_lock`
/// must still be accessed under that lock.
pub mod InodeStateFlags {
    /// Inode is newly allocated; filesystem has not yet filled its
    /// on-disk fields. Publishing initialization clears this flag and
    /// wakes waiters.
    pub const I_NEW: u32         = 1 << 0;

    /// Inode metadata (mode, uid, timestamps, etc.) is dirty. Set by
    /// `mark_inode_dirty(I_DIRTY_SYNC)`.
    pub const I_DIRTY_SYNC: u32  = 1 << 1;

    /// Inode has dirty data-bearing metadata (file size, block
    /// mappings) that `fdatasync` must flush. Set by
    /// `mark_inode_dirty(I_DIRTY_DATASYNC)`.
    pub const I_DIRTY_DATASYNC: u32 = 1 << 2;

    /// Inode has dirty pages in its AddressSpace. Set when the first
    /// page is dirtied; cleared when writeback drains all dirty pages.
    pub const I_DIRTY_PAGES: u32 = 1 << 3;

    /// Inode teardown (`evict()`) is in progress. While set,
    /// `mark_inode_dirty()` is a no-op (does not re-add the inode to
    /// dirty lists), and `inode_cache_lookup()` waits for
    /// teardown to complete (`wait_on_freeing_inode()`, defined in the
    /// lifecycle state machine below) instead of instantiating a second
    /// in-memory copy of the same `(sb, ino)`. Set ONLY under `i_lock`
    /// with `i_refcount == 0` observed under that lock (invariant
    /// INV-LC1 below); never cleared (the inode is freed).
    pub const I_FREEING: u32     = 1 << 4;

    /// Writeback of dirty timestamp fields is pending but not yet
    /// submitted. Used to coalesce frequent `atime` updates into a
    /// single writeback pass. `fdatasync` skips metadata flush when
    /// only this flag is set (timestamps are not data-relevant).
    pub const I_DIRTY_TIME: u32  = 1 << 5;

    /// Inode will be freed as soon as its reference count reaches
    /// zero. Set (under `i_lock`) when `i_nlink` drops to 0 while file
    /// descriptors still hold references (unlinked-but-open). The inode
    /// remains fully LIVE while this flag is set — reads, writes,
    /// writeback, and `igrab()` all proceed normally. At the final
    /// `inode_put()`, `I_WILL_FREE` is replaced by `I_FREEING` and the inode
    /// is evicted with `EvictDisposition::Delete`. (NOTE: this is a
    /// UmkaOS-defined long-lived state; Linux's `I_WILL_FREE` is a
    /// short window inside Linux `iput_final()` — same bit name, different
    /// lifetime.)
    pub const I_WILL_FREE: u32   = 1 << 6;

    /// Writeback in progress. Set by the writeback thread before issuing
    /// I/O for this inode's pages, cleared on writeback completion.
    /// Prevents concurrent writeback of the same inode (a second writeback
    /// request skips the inode while this flag is set). Also prevents the
    /// inode from being evicted from the inode cache during writeback.
    pub const I_WRITEBACK: u32   = 1 << 7;

    /// Inode is present in its superblock's inode cache XArray.
    /// Set by `inode_cache_insert()`, cleared when removed from the XArray
    /// (during eviction or `inode_cache_evict()`). Used by
    /// debug assertions to verify cache consistency.
    pub const I_HASHED: u32      = 1 << 8;

    /// Page cache of this inode was corrupted during a driver crash
    /// recovery cycle. Set by VFS ring crash recovery
    /// ([Section 14.3](#vfs-per-cpu-ring-extension)) when coherence checks fail on
    /// the inode's cached pages. While set, writeback skips this inode
    /// (writing corrupt data to disk would propagate the corruption).
    /// Applications reading from this inode receive `EIO`. Cleared
    /// only by unmount + fsck + remount, or by inode eviction.
    pub const I_PAGE_CACHE_CORRUPT: u32 = 1 << 9;

    /// Combined mask: any data or metadata is dirty (not including timestamps).
    pub const I_DIRTY: u32 = I_DIRTY_SYNC | I_DIRTY_DATASYNC | I_DIRTY_PAGES;

    /// Combined mask: any dirty state including timestamps.
    pub const I_DIRTY_ALL: u32 = I_DIRTY | I_DIRTY_TIME;
}

Inode lifecycle state machine:

Every in-memory inode is in exactly one lifecycle state. All transitions across the zero-reference boundary are serialized by the per-inode i_lock (INODE_LOCK, level 160) — it is the single serialization authority for the i_refcount 0↔1 transitions, the I_FREEING/I_WILL_FREE flags, and LRU-membership decisions. (Linux uses inode->i_lock identically — fs/inode.c locking rules include inode->i_state; Linux also places inode->i_hash and __iget() under that lock.)

State	Discriminant	Meaning
NEW	I_NEW set	Allocated; filesystem still filling on-disk fields. Lookups sleep on inode_waitqueue(); publishing initialization clears the flag and wakes them.
LIVE	i_refcount > 0, I_FREEING/I_WILL_FREE clear	Referenced by dentries, open files, or the writeback engine.
WILL_FREE	I_WILL_FREE set (i_nlink == 0, i_refcount > 0)	Unlinked but still open. Fully operational (reads/writes/writeback/igrab() all work). Evicted with EvictDisposition::Delete at the final inode_put().
CACHED_IDLE	i_refcount == 0, i_nlink > 0, on the LRU	Reclaimable cache entry. Revived by lookup (0→1 under i_lock) or reclaimed by the shrinker.
FREEING	I_FREEING set (i_refcount == 0, off LRU)	Teardown (evict()) in progress. Still present in sb.inode_cache until teardown completes — concurrent lookups WAIT (wait_on_freeing_inode()), never instantiate a second copy.
FREED	erased from sb.inode_cache; RCU-deferred slab free pending	Unreachable; memory returns to the slab after the grace period.

Reference-count discipline (i_refcount: AtomicU32):

References are held by: (1) the dentry cache — each dentry pointing at the inode; (2) open file descriptors — each OpenFile via its dentry; (3) the writeback engine — a temporary ref while the inode is being flushed; the page cache does NOT hold a ref (pages hold Weak<Inode> via AddressSpace::host and are torn down inside evict()).

// INV-LC1: I_FREEING is set only under i_lock, and only after observing
//          i_refcount == 0 under that same lock hold.
// INV-LC2: the 0 -> 1 refcount transition happens only under i_lock, and
//          only while I_FREEING is clear.
// INV-LC3: any decrement that can reach 0 goes through inode_put(), which takes
//          i_lock for the 1 -> 0 transition. Lock-free decrements are legal
//          only via the decrement-if-greater-than-one fast path below.
//
// Consequence (closes the lookup-vs-shrinker UAF race): a lock-free
// increment can never resurrect a FREEING inode. The lock-free fast path
// is increment-if-positive — a CAS loop that FAILS at i_refcount == 0 —
// and I_FREEING implies i_refcount == 0 (INV-LC1) with no lock-free
// 0 -> 1 possible (INV-LC2). This is the same shape as Linux's lockless
// Linux `igrab_from_hash()` fast path + `i_lock` slow path (fs/inode.c).

/// Acquire one reference on an inode found via RCU (icache XArray load,
/// BDI dirty-list walk, dcache pointer). Common case (i_refcount > 0):
/// one CAS, no lock. Slow case (i_refcount == 0, i.e. CACHED_IDLE or
/// FREEING): per-inode i_lock.
pub enum InodeAcquire {
    /// Reference acquired; i_refcount incremented.
    Acquired,
    /// The inode has I_FREEING set — teardown in progress. The caller
    /// either waits for teardown (`wait_on_freeing_inode()` + retry the
    /// lookup) or gives up (`igrab()` returns None).
    Freeing,
}

pub fn inode_ref_acquire(inode: &Inode) -> InodeAcquire {
    // Fast path: increment-if-positive (never resurrects zero).
    let mut c = inode.i_refcount.load(Ordering::Acquire);
    while c != 0 {
        match inode.i_refcount.compare_exchange_weak(
            c, c + 1, Ordering::AcqRel, Ordering::Acquire,
        ) {
            Ok(_) => return InodeAcquire::Acquired,
            Err(cur) => c = cur,
        }
    }
    // Slow path: i_refcount == 0 — CACHED_IDLE revival or FREEING.
    let guard = inode.i_lock.lock();
    if inode.i_state.load(Ordering::Acquire) & InodeStateFlags::I_FREEING != 0 {
        drop(guard);
        return InodeAcquire::Freeing;
    }
    // 0 -> 1 under i_lock with I_FREEING clear (INV-LC2). The count may
    // have been raised concurrently by another slow-path caller; a plain
    // fetch_add under the lock is correct either way.
    inode.i_refcount.fetch_add(1, Ordering::AcqRel);
    // Revival: if the inode sits on the LRU (CACHED_IDLE), unlink it.
    // Lock chain i_lock(160) -> ICACHE_LRU_LOCK(179) is ascending.
    inode_lru_remove_if_linked(inode);
    drop(guard);
    InodeAcquire::Acquired
}

/// Release one reference. Fast path: decrement while the observed count
/// is > 1 (CAS loop — cannot reach 0 lock-free, INV-LC3). Final path
/// (count == 1): take i_lock, drop to 0 under the lock, and decide the
/// inode's fate:
///
///   - `i_nlink == 0` (i.e. WILL_FREE, or a fresh unlink observed here),
///     or the superblock is shutting down (umount teardown):
///       set I_FREEING (clearing I_WILL_FREE), remove from the LRU if
///       linked, drop i_lock, then run
///       `evict(inode, disposition_from_nlink(inode))`.
///   - otherwise (`i_nlink > 0`, sb active): insert at the LRU tail
///     (CACHED_IDLE) and return — the inode stays cached for reuse.
pub fn inode_put(inode: &Inode);

/// Disposition of an inode leaving memory, computed ONCE — under the
/// same i_lock hold that sets I_FREEING — from `i_nlink`, and passed
/// explicitly to `InodeOps::evict_inode()`. The driver never re-derives
/// it.
///
/// `#[repr(u8)]`: crosses the VFS ring as a `u8` in
/// `VfsRequestArgs::EvictInode`.
#[repr(u8)]
pub enum EvictDisposition {
    /// `i_nlink > 0` — cache reclaim only. The file is LIVE on disk:
    /// in-memory teardown (writeback, page cache, driver private state)
    /// is performed; ON-DISK STATE IS UNTOUCHED.
    Release = 0,
    /// `i_nlink == 0` — the last link is gone and this was the last
    /// reference: full delete. In-memory teardown PLUS on-disk
    /// deallocation (blocks, extent tree, inode-table entry).
    Delete = 1,
}

/// `disposition_from_nlink(inode)`:
///   if inode.i_nlink.load(Acquire) == 0 { Delete } else { Release }
pub fn disposition_from_nlink(inode: &Inode) -> EvictDisposition;

Waiting on inode state transitions (inode_waitqueue()):

/// Hashed inode wait table — a fixed array of `WaitQueueHead` buckets
/// keyed by the inode's address, exactly like the hashed page-wait table
/// (`page_waitqueue()`, [Section 4.4](04-memory.md#page-cache)). One table serves ALL inode
/// state-bit waits: `I_NEW` (woken when initialization completes),
/// `I_WRITEBACK` (woken when `writeback_single_inode()` clears the bit —
/// the sleep side is `wait_on_inode_writeback()`,
/// [Section 4.6](04-memory.md#writeback-subsystem)), and `I_FREEING` teardown completion
/// (woken by `evict()` step 6c below). Hash collisions produce spurious
/// wakeups only — every sleeper re-checks its predicate. Linux
/// equivalent: `inode_bit_waitqueue()` on `i_state` bits (fs/inode.c).
pub fn inode_waitqueue(inode: &Inode) -> &'static WaitQueueHead;

/// Sleep until a FREEING inode's teardown completes, then let the caller
/// RETRY its lookup from scratch. Called by `inode_cache_lookup()` /
/// `find_inode()` when the looked-up inode has I_FREEING set. Linux
/// Linux equivalent: `__wait_on_freeing_inode()` (fs/inode.c).
///
/// Entry: caller holds the inode's i_lock (it just read I_FREEING under
/// it) and is inside the RCU read section of the XArray lookup.
///
///   1. wq = inode_waitqueue(inode); wq.prepare_to_wait_exclusive();
///      (register BEFORE the re-check — closes the lost-wakeup race.)
///   2. Re-check under i_lock: if I_FREEING is clear or I_HASHED is
///      clear (teardown already finished between the first read and
///      registration), drop i_lock, finish_wait(wq), and return — the
///      caller retries immediately.
///   3. Drop i_lock, exit the RCU read section, schedule().
///   4. finish_wait(wq). The old inode pointer is DEAD from this point:
///      the woken caller MUST NOT touch it (the RCU grace period may
///      have expired). It re-runs the whole lookup — which now misses
///      (the inode was erased in evict() step 6b) and reads a fresh
///      inode from POST-teardown disk state, or finds a new instance
///      created after the teardown.
///
/// No-lost-wakeup argument (the Linux `remove_inode_hash` argument):
/// the waker (evict step 6) clears I_HASHED and erases the XArray entry
/// UNDER i_lock, then wakes AFTER releasing it. Either the sleeper's
/// step-2 re-check (under i_lock) sees the erase and aborts, or it
/// registered before the wake — `wake_up_all()` finds it. Both sides
/// take i_lock, so no ordering window exists.
pub fn wait_on_freeing_inode(inode: &Inode /* i_lock held; consumed */);

Eviction (evict() — the sole teardown path):

evict() is the only path through which an inode and its page cache pages are freed. It runs with EvictDisposition::Delete from inode_put()'s final path when i_nlink == 0, and with EvictDisposition::Release from LRU reclaim (inode_cache_evict()) and unmount teardown of still-linked inodes. LRU reclaim of an i_nlink > 0 inode NEVER frees on-disk resources — the file is live; only its in-memory footprint is released.

evict(inode, disposition):

  Preconditions (established by the caller under i_lock):
  - I_FREEING is set; i_refcount == 0; the inode is off the LRU.
  - The inode is STILL PRESENT in sb.inode_cache. Concurrent lookups
    find it, see I_FREEING, and wait (wait_on_freeing_inode). It is
    unhashed only in step 6 — AFTER all flush I/O has landed — so a
    fresh in-memory instance can never be built from stale pre-flush
    disk state (no divergent (sb, ino) copies, matching Linux, where
    Linux `remove_inode_hash()` is likewise the LAST visible step of `evict()`).

  Step 1: Detach from writeback lists.
          Remove from the BDI dirty lists (b_dirty / b_io / b_more_io /
          b_dirty_time, under wb.list_lock) and from the superblock
          dirty-inode list. mark_inode_dirty() is already a no-op
          (I_FREEING), so the inode cannot re-enter them.

  Step 2: Wait for RUNNING writeback.
          Sleep on inode_waitqueue() until the I_WRITEBACK bit clears,
          then wait until inode.i_mapping.nrwriteback == 0. The
          writeback engine checks I_FREEING before starting new work on
          an inode, so this wait is bounded by one in-flight pass.

  Step 3: Flush residual dirty state (if any I_DIRTY_* flag or dirty
          pages remain — possible when the final inode_put() follows an
          unflushed write):
          writeback_single_inode(inode, WritebackSyncMode::Wait)
          Flushes ALL dirty pages via AddressSpaceOps::writepage()/
          writepages() and blocks until every writeback I/O completes
          (inode.i_mapping.nrwriteback == 0). If any writeback I/O
          fails, the error is recorded in AddressSpace::wb_err (ErrSeq).
          The pages are still freed in step 4 — errors are recorded,
          not retried.

  Step 4: truncate_inode_pages_range(&inode.i_mapping, 0, u64::MAX)
          If page_cache is None (DAX inode), skip this step entirely —
          there are no cached pages to tear down.
          This is the page cache teardown:
          a. Walk AddressSpace.page_cache (unwrapped XArray), removing all entries.
          b. For each page:
             - If PageFlags::WRITEBACK is set: wait for I/O completion
               (spin on the page's wait queue — this should be rare
               after steps 2-3 drained writeback, but handles races
               with async I/O completion).
             - If PageFlags::DIRTY is set: BUG — step 3 should have flushed
               all dirty pages. In release builds, log a warning and
               proceed (the page will be freed without writeback).
             - Remove the page from the LRU list if present.
             - Release the page frame back to the buddy allocator
               (Section 4.2).
          c. Set AddressSpace.page_cache.nr_pages = 0.
          d. Set AddressSpace.page_cache.nr_dirty = 0.
          Ordering: step 4 MUST NOT begin until steps 2-3 have fully
          completed (all writeback I/O acknowledged).

  Step 5: Filesystem teardown, gated by disposition:
          inode.i_op.evict_inode(InodeId(inode.i_ino), disposition)
          - Delete  (i_nlink == 0): free disk blocks (block bitmap /
            extent tree), remove the on-disk inode-table entry, commit
            a journal transaction if journaled, AND free the driver's
            per-inode in-memory state (i_private / per-ino map).
          - Release (i_nlink > 0): free the driver's per-inode
            IN-MEMORY state ONLY. On-disk state is NOT touched — the
            file's blocks and inode-table entry are live data that a
            later lookup re-reads.
          For KABI Tier 1/2 drivers, this is dispatched as
          VfsOpcode::EvictInode through the VFS ring buffer, with the
          disposition in the request args.

  Step 6: Unhash, wake, free:
          a. inode.i_wb.store(None) under i_lock — release the cgroup
             writeback attribution (see the i_wb field doc).
             security_inode_free(inode) releases the LSM blob
             ([Section 9.8](09-security.md#linux-security-module-framework)).
          b. Under i_lock: sb.inode_cache.erase(inode.i_ino) (the
             XArray's internal XA_LOCK(178) nests above INODE_LOCK(160)
             — ascending) and clear I_HASHED. From here the inode is
             unreachable through the icache.
          c. Drop i_lock, then inode_waitqueue(inode).wake_up_all() —
             freeing-waiters re-run their lookups, which now miss and
             read fresh POST-teardown state.
          d. Remove the inode from the superblock's inode list:
             sb.s_inode_list_lock.lock();
             sb.s_inodes.remove(&inode.i_sb_list);
             sb.s_inode_list_lock.unlock();
          e. Defer the slab free via RCU:
             call_rcu(&inode.i_rcu, inode_free_rcu)
             where inode_free_rcu returns the object to the inode slab
             cache. Required because concurrent RCU readers (rcu_walk
             path lookup, inode_cache_lookup under rcu_read_lock) may still
             hold pointers to this inode; a direct slab free would be a
             use-after-free.

InodeOps::evict_inode() method (extends the InodeOps trait defined above):

    /// Called during inode eviction (evict() step 5) after the VFS has
    /// flushed all dirty pages and torn down the page cache.
    ///
    /// `disposition` is computed by the VFS from `i_nlink` under the
    /// same `i_lock` hold that set `I_FREEING`:
    ///
    /// - `EvictDisposition::Delete` — `i_nlink == 0`. Free on-disk
    ///   resources (blocks, extent tree entries, inode bitmap bit),
    ///   commit any necessary journal transactions, and free per-inode
    ///   in-memory driver state.
    /// - `EvictDisposition::Release` — `i_nlink > 0` (LRU reclaim,
    ///   unmount of a still-linked inode). Free per-inode in-memory
    ///   driver state ONLY (`i_private` cast back to the concrete type
    ///   and freed; ring drivers drop their per-ino private entry).
    ///   The implementation MUST NOT touch on-disk allocation state:
    ///   the file is live and will be re-read from disk on the next
    ///   lookup. (Linux parity: `->evict_inode()` implementations gate
    ///   on-disk deletion on `inode->i_nlink == 0` — e.g.
    ///   `ext4_evict_inode()`; UmkaOS passes the decision explicitly
    ///   instead of having the driver re-derive it.)
    ///
    /// For pseudo-filesystems (tmpfs, procfs) with no on-disk state,
    /// both dispositions reduce to freeing in-memory state.
    ///
    /// Must not fail — cleanup is best-effort. If a journal commit
    /// fails, the filesystem marks itself as needing fsck (sets the
    /// error flag in the superblock) and returns.
    fn evict_inode(&self, ino: InodeId, disposition: EvictDisposition);

VfsOpcode::EvictInode (extends the VfsOpcode enum):

    /// `InodeOps::evict_inode`. Inode teardown — Release frees only the
    /// driver's in-memory per-inode state; Delete additionally frees
    /// on-disk resources. Sent after the VFS has completed page cache
    /// teardown (evict() step 4).
    EvictInode = 38,

With a corresponding VfsRequestArgs variant:

    /// `InodeOps::evict_inode`. The inode number is in
    /// `VfsRequest::ino`; `disposition` is the wire form of
    /// `EvictDisposition` (0 = Release, 1 = Delete — any other value is
    /// a protocol error, request rejected with EINVAL).
    EvictInode { disposition: u8 },

Truncate path (truncate_inode_pages_range(mapping, lstart, lend)):

Called by ftruncate(2) (shrinking a file), open(2) with O_TRUNC (handle_truncate()), unlink(2) (via eviction), fallocate(FALLOC_FL_PUNCH_HOLE), and fallocate(FALLOC_FL_COLLAPSE_RANGE). This is a partial teardown — only pages in the specified range are removed, unlike truncate_inode_pages_range(mapping, 0, u64::MAX), which removes all pages.

/// Remove the page-cache pages overlapping `[lstart, lend]` (byte
/// offsets, inclusive) from `mapping`, waiting out in-flight writeback
/// and zeroing the partial edge pages — the numbered step sequence is
/// specified in the block below. `lend == u64::MAX` means "through EOF"
/// (ftruncate / O_TRUNC). Precondition: caller holds `i_rwsem` exclusive.
pub fn truncate_inode_pages_range(mapping: &AddressSpace, lstart: u64, lend: u64);

truncate_inode_pages_range(mapping, lstart, lend):

  Precondition: caller holds inode.i_rwsem exclusive (write lock).
  This prevents concurrent page faults, reads, and writes from
  populating the range being truncated.

  1. Compute page-aligned range:
     start_index = lstart / PAGE_SIZE
     end_index   = lend / PAGE_SIZE  (or u64::MAX for "to end of file")

  2. If mapping.page_cache is None (DAX inode), skip steps 2–3 — DAX
     files have no cached pages. Proceed directly to step 4 (on-disk
     block deallocation).
     For each page in mapping.page_cache[start_index..=end_index]:

     a. Acquire page lock (set `PageFlags::LOCKED` via atomic CAS; sleep if
        already locked by another thread — e.g., readahead).

        RESIDENCY RE-CHECK (after the lock): acquiring the page lock can
        block, and while blocked a completion-context evictor holding no
        i_rwsem — a fill-error path (`FillCompletion::complete_err`, whose
        TRUNCATE COUPLING is documented in [Section 4.4](04-memory.md#page-cache)) or the
        readahead-error eviction path — can remove THIS page and let a
        different page (or nothing) occupy this slot. Re-verify the slot at
        this index still maps THIS page in THIS mapping
        (`mapping.page_cache.get(index)` under the now-held page lock)
        before running steps b–g and before counting it in step 3. If it no
        longer maps this page, unlock it and `continue` to the next index.

     b. If `PageFlags::DIRTY` is set:
        cancel_dirty_page(page):
        - Clear `PageFlags::DIRTY` on the page.
        - Decrement mapping.page_cache.nr_dirty.
        - Decrement the BDI (backing device info) dirty page counter.
        - The page is NOT written back — truncated data is discarded.

     c. If `PageFlags::WRITEBACK` is set:
        wait_on_page_writeback(page):
        - Sleep until the bio completion handler clears `PageFlags::WRITEBACK`.
        - This handles the race where writeback was submitted before
          truncate acquired i_rwsem but has not yet completed.

     d. Remove the page from mapping.page_cache (XArray delete).
        (No futex-key action is needed here: shared futex keys are
        `{i_seq, pgoff}` — keyed on the inode's futex sequence number
        (`Inode.i_seq`) and file offset, not on the physical frame —
        so freeing the backing page does not invalidate any key.
        Waiters on a truncated-away range simply remain blocked until
        a matching FUTEX_WAKE, exactly as in Linux; see
        [Section 19.4](19-sysapi.md#futex-and-userspace-synchronization--shared-futex-key-stability).)

     e. Remove the page from the LRU list (if present).

     f. Unlock the page (clear `PageFlags::LOCKED`).

     g. Release the page frame reference. If this is the last
        reference, the page is freed back to the buddy allocator
        (Section 4.2). If another mapping holds a reference (e.g.,
        a shared mmap), the page survives until that reference is
        dropped.

  3. Decrement mapping.page_cache.nr_pages by the count of removed
     pages (atomic subtract).

  4. Notify the filesystem for on-disk block deallocation:
     inode.i_op.truncate_range(ino, lstart, lend)
     The filesystem frees the corresponding disk blocks, updates
     extent trees, and journals the change. For KABI Tier 1/2
     drivers this is dispatched as the existing VfsOpcode::Truncate
     with the range encoded in the size field.

  5. If lstart is not page-aligned (partial page at the start of the
     range): zero the tail of the partial page from lstart to the
     next page boundary. The page remains in the cache with its
     leading portion intact. Mark it dirty so the zeroed region is
     written back.

  6. If lend is not page-aligned and lend != u64::MAX (partial page
     at the end): zero the head of the partial page from the page
     start to lend. Mark it dirty. (This case arises only with
     FALLOC_FL_PUNCH_HOLE; ftruncate always has lend = u64::MAX.)

Conditional-invalidation path (invalidate_clean_pages_range(mapping, lstart, lend)):

Truncation and invalidation are two different contracts and therefore two different entry points. truncate_inode_pages_range is authoritative: it removes unconditionally, zeroes partial edge pages, and requires i_rwsem exclusive. Invalidation is advisory: it removes only what is safe to remove, never zeroes, and needs no exclusive lock. Fusing them behind a mode flag on one function would weld two preconditions and two error contracts together, so they stay separate.

/// Best-effort removal of cached pages overlapping `[lstart, lend]` (byte
/// offsets, inclusive) from `mapping`.
///
/// For each cached page in the range: if it is clean, not under writeback,
/// and unpinned (no `Mm::pin_pages` pin), unmap it from all page tables and
/// remove it from the cache; if it is dirty, under writeback, or pinned,
/// leave it in place and record the failure.
///
/// Unlike `truncate_inode_pages_range`, this NEVER zeroes partial edge pages
/// and does NOT require `i_rwsem` exclusive — it is safe under shared
/// `i_rwsem` or none at all. It participates in the SAME `InvalidateSeq`
/// coordination as `truncate_inode_pages_range` (bump before removal,
/// identical `invalidate_begin()` / `invalidate_end()` bracket), so a
/// lockless buffered reader racing the removal retries instead of
/// re-inserting a stale page.
///
/// The entire range is processed even after a failure — no short-circuit, so
/// one pinned page cannot leave the rest of the range cached.
///
/// # Errors
///
/// `EBUSY` — at least one page in the range could not be invalidated.
/// `Ok(())` means every cached page in the range was removed, or none was
/// cached. Callers own the failure policy (record in `wb_err`, warn, retry).
pub fn invalidate_clean_pages_range(
    mapping: &AddressSpace,
    lstart: u64,
    lend: u64,
) -> Result<(), Errno>;

Dirty page handling before eviction:

Dirty pages are NEVER silently discarded during eviction. The eviction sequence guarantees data integrity through the following invariants:

1. writeback_single_inode() (evict() step 3) is called with WritebackSyncMode::Wait, which means:
2. ALL dirty pages are submitted for writeback via AddressSpaceOps::writepage().
3. The caller blocks until every submitted bio has completed (waits on each page's PageFlags::WRITEBACK flag).

4. If the inode is already being written back by the periodic writeback thread, WritebackSyncMode::Wait waits for that in-progress writeback to finish, then re-scans for any pages dirtied in the interim.

5. If writeback I/O fails (disk error, transport error):

6. The error code is recorded in AddressSpace::wb_err (ErrSeq counter) so that any concurrent fsync() on another fd for this inode will observe the error.
7. AddressSpaceFlags::EIO or AddressSpaceFlags::ENOSPC is set in AddressSpace::flags.
8. The page is still freed in step 3 — there is no retry loop. The data is lost, but the error is recorded. This matches POSIX semantics: a subsequent fsync() returns -EIO exactly once per file descriptor.

9. An error-level kernel log message is emitted: "VFS: writeback error during eviction of inode {sb}:{ino}: {errno}".

10. truncate_inode_pages_range(&inode.i_mapping, 0, u64::MAX) (evict() step 4) asserts that no dirty pages remain (PageFlags::DIRTY must be clear after step 3). In debug builds, a dirty page at this point triggers a BUG (logic error in the writeback path). In release builds, it is logged as a warning and the page is freed without writeback.

Race prevention:

The eviction sequence must be safe against concurrent operations:

Race scenario	Prevention mechanism
inode_cache_lookup() during eviction	The inode is still hashed (unhashing is evict() step 6); the lookup finds it, sees I_FREEING under i_lock, and WAITS (wait_on_freeing_inode()), then retries. The retry misses (inode erased) and reads a fresh inode from POST-teardown disk state (or gets ENOENT if unlinked) — never a second live copy racing the dying inode's flush.
mark_inode_dirty() during eviction	I_FREEING flag checked — mark_inode_dirty() is a no-op when I_FREEING is set.
Page fault on evicting inode	Same as inode_cache_lookup() above: the fault-side lookup waits for teardown, then reads a fresh inode from disk. If the file is unlinked (i_nlink == 0), the on-disk inode is already marked free and the read fails with ESTALE.
Writeback thread picks inode during eviction	The writeback thread checks I_FREEING before acquiring the inode ref. If the flag is set, the inode is skipped. Writeback already in progress when I_FREEING is set is drained by evict() step 2 (wait for I_WRITEBACK clear + nrwriteback == 0) and step 3 (WritebackSyncMode::Wait).
Inode::get() racing with inode_put()'s final path	Both take i_lock: Inode::get()'s 0→1 revival happens only under i_lock with I_FREEING clear (INV-LC2); inode_put()'s 1→0 and I_FREEING set happen under i_lock with i_refcount == 0 observed (INV-LC1/LC3). Whichever acquires i_lock first wins; the loser observes a consistent state (either a live count > 0 — no eviction — or I_FREEING — wait and retry).
Inode::get() racing with the shrinker	The lock-free fast path is increment-if-positive and FAILS at i_refcount == 0, so it can never revive an inode the shrinker is isolating; the slow path serializes on i_lock against the shrinker's recheck (see inode_cache_evict() phase 2). If the lookup wins (0→1 under i_lock first), the shrinker's recheck sees i_refcount != 0 and skips the inode.
Truncate racing with readahead	truncate_inode_pages_range() holds i_rwsem exclusive; readahead in SUBMISSION context acquires i_rwsem shared, so the rwsem serializes those two directly. A readahead FILL that fails asynchronously delivers its error in I/O-completion context, which erases the slot from the page cache holding NO i_rwsem — that erasure is therefore not rwsem-serialized against truncate, but it is harmless: it only removes a page already off the truncate path, and truncate's post-lock residency re-check (the RESIDENCY RE-CHECK in the Truncate path above) sees the slot no longer maps that page and skips it.
Truncate racing with mmap read fault	The page fault handler does NOT acquire i_rwsem — truncation-fault coordination uses InvalidateSeq (Section 4.8), a lockless seqcount. Truncate increments the seq (odd = in progress) under i_rwsem exclusive before mutating the page cache; the fault detects the concurrent truncation via two atomic seq loads and retries (returns SIGBUS if the fault address is beyond the new EOF).

Cross-references: - Writeback thread organization and writeback_single_inode(): Section 4.6 - Buddy allocator (page frame release): Section 4.2 - fsync end-to-end flow and ErrSeq semantics: this section (fsync / fdatasync End-to-End Flow) - VFS ring buffer protocol (EvictInode dispatch): this section (VFS Ring Buffer Protocol) - Page cache XArray structure: Section 4.4 - LRU lists and page reclaim: Section 4.2 - DLM-aware page cache invalidation on lock release/downgrade: Section 15.15

14.1.2.4.1.9 Inode Cache (icache)

All in-memory inodes are registered in a per-superblock inode cache (icache). The cache serves two purposes: (1) deduplication — ensuring that only one Inode instance exists for any given (superblock, inode_number) pair via per-superblock XArray lookup, and (2) memory management — tracking unreferenced inodes on a global LRU list for eviction under memory pressure.

/// Lock level for `InodeCache::lru`. Sits between `XA_LOCK` (178) and
/// `PAGE_LOCK` (180): the two inbound chains are `i_lock`
/// (`INODE_LOCK`, 160) → `ICACHE_LRU_LOCK` (revival unlink / shrinker
/// rotation) and XArray internal lock (178) → `ICACHE_LRU_LOCK` — both
/// strictly ascending. Nothing is acquired under it. Master table:
/// [Section 3.4](03-concurrency.md#cumulative-performance-budget--lock-ordering).
pub const ICACHE_LRU_LOCK: u32 = 179;

/// Global inode cache LRU and shrinker state. Inode lookup is per-superblock
/// (via `SuperBlock.inode_cache: XArray<u64, Arc<Inode>>`), but the LRU list
/// for memory pressure eviction is global — the shrinker needs a single list
/// to scan across all filesystems.
///
/// **Design rationale**: Per-superblock XArray eliminates hash computation
/// (~15-25 cycles saved), provides O(1) guaranteed lookup (no collision
/// chains), and improves cache locality (per-filesystem working set stays
/// in its own radix tree). The caller always has the superblock from path
/// resolution (`dentry.d_sb`), so no extra lookup is needed.
///
/// **Singleton**: one global instance, initialized during VFS subsystem
/// init. Accessed via `inode_cache()` which returns `&'static InodeCache`.
pub struct InodeCache {
    /// LRU list of unreferenced inodes (i_refcount == 0, i_nlink > 0).
    ///
    /// Head = least recently used (oldest unreferenced inode).
    /// Tail = most recently unreferenced inode.
    ///
    /// An inode is added to the LRU tail when its `i_refcount` drops
    /// to 0 (via `inode_put()`) and `i_nlink > 0` (still has on-disk links).
    /// It is removed from the LRU when:
    ///   - A zero-count inode is revived (0→1 under `i_lock`,
    ///     `inode_ref_acquire()` slow path).
    ///   - The shrinker isolates a reclaim batch (`inode_cache_evict()`
    ///     phase 1).
    ///   - `inode_put()`'s final path evicts it (`i_nlink == 0` or sb
    ///     shutdown).
    ///
    /// Protected by `lru_lock` (level `ICACHE_LRU_LOCK`, 179). Lock
    /// ordering: `i_lock` (`INODE_LOCK`, 160) → `lru_lock` (revival
    /// unlink, shrinker rotation) and XArray internal lock (`XA_LOCK`,
    /// 178) → `lru_lock` are both ascending. The shrinker NEVER takes
    /// `i_lock` while holding `lru_lock` — it isolates a batch under
    /// `lru_lock` alone, drops it, then revalidates each inode under
    /// `i_lock` (see `inode_cache_evict()`); this avoids the inverted
    /// `lru_lock → i_lock` chain that forces Linux into
    /// Linux `inode_lru_isolate()` uses `spin_trylock(&inode->i_lock)`.
    pub lru: SpinLock<IntrusiveList<Inode>, ICACHE_LRU_LOCK>,

    /// Number of inodes currently on the LRU list. Updated atomically
    /// on LRU insert/remove. Used by the shrinker to estimate
    /// reclaimable memory without taking `lru_lock`.
    pub lru_count: AtomicU64,

    /// High watermark — when `lru_count` exceeds this value, the
    /// background reclaim kthread is woken to proactively evict cold
    /// inodes. Set during VFS init based on total system memory:
    ///   reclaim_watermark = max(1024, total_pages / 256)
    /// This keeps the LRU from growing unboundedly on large-memory
    /// systems while ensuring small systems still cache a useful
    /// number of inodes.
    pub reclaim_watermark: u64,
}

/// Return the boot-initialized global inode cache. The cache is a `&'static`
/// singleton established once during VFS init and never torn down, so callers
/// need no lifetime or refcount. Used by the inode shrinker and the cache
/// operation helpers below.
fn inode_cache() -> &'static InodeCache { /* boot-initialized singleton */ }

/// Unlink `inode` from the global LRU if it is currently linked (its
/// intrusive `IntrusiveListNode` linkage is testable under `lru_lock`);
/// decrements `lru_count` when an unlink happens. No-op when unlinked —
/// safe to call from the revival path even after the shrinker has
/// already isolated the inode (phase 1 of `inode_cache_evict()` unlinks
/// it first). Caller may hold `i_lock` (160 → 179, ascending).
fn inode_lru_remove_if_linked(inode: &Inode);

/// Insert `inode` at the LRU tail (most recently unreferenced) and
/// increment `lru_count`. Called by `inode_put()`'s final path for
/// `i_nlink > 0` inodes (CACHED_IDLE) and by the shrinker's rotation of
/// dirty/under-writeback candidates. Caller may hold `i_lock`.
fn inode_lru_add_tail(inode: &Inode);

Inode cache operations:

/// Look up an inode in the per-superblock XArray by inode number.
/// This is `Inode::get()` / `inode_cache_lookup()` — the names are used
/// interchangeably in this section.
///
/// The XArray load is RCU (lock-free); reference acquisition is
/// `inode_ref_acquire()` (lifecycle state machine above): one CAS on
/// the common `i_refcount > 0` path, per-inode `i_lock` only for
/// zero-count revival. (Linux parity: `igrab_from_hash()` lockless
/// Linux `find_inode()` uses `i_lock` on the slow path, fs/inode.c.)
///
/// Returns `None` ONLY when no inode with this ino is (or remains)
/// in the superblock's inode cache — the caller then reads a fresh
/// inode from disk. An inode found with `I_FREEING` set is NEVER
/// skipped-and-reread: the lookup WAITS for teardown to complete and
/// retries, so a fresh instance is only ever built from POST-teardown
/// disk state (a skip-and-reread here would create a second live
/// `(sb, ino)` instance while the dying one's `WritebackSyncMode::Wait` flush is
/// still landing — stale duplicate, divergent AddressSpaces; Linux
/// Linux waits identically in `__wait_on_freeing_inode()`).
///
/// **Hot path**: called on every `open()`, `stat()`, and path lookup
/// that misses the dentry cache. O(1) XArray lookup with no hash
/// computation and no collision chains; one refcount CAS in the
/// common case.
pub fn inode_cache_lookup(sb: &SuperBlock, ino: u64) -> Option<Arc<Inode>> {
    loop {
        // rcu_read_lock() (implicit in XArray::load)
        // let inode = sb.inode_cache.load(ino) else return None;
        // match inode_ref_acquire(inode):
        //   Acquired => {
        //       // NEW inodes: wait for I_NEW to clear
        //       // (inode_waitqueue; woken when initialization completes),
        //       // then return Some(inode).
        //       return Some(inode);
        //   }
        //   Freeing => {
        //       // Teardown in progress. inode_ref_acquire returned
        //       // with i_lock released; re-take it, re-check, and
        //       // sleep until evict() step 6c wakes us:
        //       // wait_on_freeing_inode(inode);
        //       // then RETRY the whole lookup (the old pointer is
        //       // dead; the retry misses or finds a new instance).
        //       continue;
        //   }
        // rcu_read_unlock() (implicit)
    }
}

/// Insert an inode into the global cache.
///
/// Called after a filesystem driver has allocated and filled a new inode
/// (from `InodeOps::lookup()`, `InodeOps::create()`, or `read_inode()`).
/// The inode must have `I_NEW` set in `i_state` — this flag is cleared
/// when the filesystem publishes completed initialization.
///
/// **Preconditions**:
///   - `inode.i_ino` and `inode.i_sb` are set (valid superblock + ino).
///   - `inode.i_refcount >= 1` (the caller holds a reference).
///   - `inode.i_seq == 0` (unassigned sentinel) — every freshly
///     instantiated inode enters the cache with no futex sequence; the
///     futex subsystem assigns one lazily on first futex use
///     ([Section 19.4](19-sysapi.md#futex-and-userspace-synchronization)).
///   - `inode.i_flags` initialized to
///     `fs_flags_to_inode_flags(attr.fs_flags).bits()` from the instantiating
///     attribute read, plus any kernel-managed bits the constructor owns
///     (for example, `InodeFlags::DAX` mirroring the mapping's DAX bit) — see
///     [Section 14.1](#virtual-filesystem-layer--inode-attribute-flags). `0` for
///     filesystems without persistent flags.
///   - No existing entry for `inode.i_ino` in `inode.i_sb.inode_cache`
///     — callers must check `inode_cache_lookup()` first.
///
/// **Panics** in debug builds if a duplicate entry exists (logic error
/// in the filesystem driver). In release builds, returns the existing
/// inode and drops the new one.
pub fn inode_cache_insert(inode: Arc<Inode>) {
    // inode.i_sb.inode_cache.store(inode.i_ino, inode.clone())
    // Set i_state |= I_HASHED (indicates presence in icache).
}

/// Number of LRU candidates isolated per batch by `inode_cache_evict()`.
/// Bounds the `lru_lock` hold time (phase 1) and the stack footprint of
/// the batch ArrayVec. Not a reclaim limit — the two-phase loop repeats
/// until `count` is reached or the LRU is exhausted.
pub const INODE_EVICT_BATCH: usize = 32;

/// Evict unreferenced inodes from the LRU list to reclaim memory.
///
/// Called by the memory shrinker when the slab allocator or page reclaim
/// needs to free memory. Reclaims up to `count` inodes from the LRU head
/// (least recently used first).
///
/// **LRU reclaim NEVER destroys on-disk state.** Inodes on the LRU have
/// `i_nlink > 0` (live files): reclaim runs `evict(inode,
/// EvictDisposition::Release)` — writeback + page cache teardown +
/// in-memory driver state release. The disposition is still computed
/// from `i_nlink` (`disposition_from_nlink()`) under the same `i_lock`
/// hold that sets `I_FREEING`, so even a theoretical nlink-reached-0
/// straggler is handled correctly rather than assumed away.
///
/// Two-phase loop (repeat until `count` reclaimed or LRU exhausted):
///
/// **Phase 1 — isolate** (under `lru_lock` ONLY; bounded hold time):
///   Pop up to `INODE_EVICT_BATCH` inodes from the LRU head into a
///   local `ArrayVec<Arc<Inode>, INODE_EVICT_BATCH>` (the `Arc` is
///   cloned from the still-hashed icache entry — an RCU read, no
///   `xa_lock`; the clone pins memory for phase 2). Skip entries not
///   matching `sb` when a single superblock is targeted.
///   `i_lock` is NEVER taken in this phase — see the `lru` field doc
///   (this is what lets UmkaOS avoid Linux's `spin_trylock` inversion
///   during inode LRU isolation).
///
/// **Phase 2 — revalidate and dispose** (per inode, NO `lru_lock`):
///   Take `i_lock` and re-check eligibility — the inode's state may
///   have changed between isolation and now:
///   - `i_refcount != 0`: a lookup revived it after phase 1 popped it.
///     Skip; do NOT reinsert — the holder's final `inode_put()` re-inserts
///     it (lazy LRU discipline, same as Linux "iput, sync, or the last
///     page cache deletion will requeue them").
///   - `I_FREEING` or `I_NEW` set: owned elsewhere; skip.
///   - `I_WRITEBACK` set, or any `I_DIRTY_*` flag set: not disposable
///     now. ROTATE: reinsert at the LRU tail (`inode_lru_add_tail()`,
///     `i_lock`(160) → `lru_lock`(179), ascending) so the periodic
///     writeback pass cleans it before it reaches the head again.
///   - Otherwise (clean, unreferenced): set `I_FREEING` under `i_lock`
///     (INV-LC1 — `i_refcount == 0` was just observed under this
///     lock), compute `disposition_from_nlink(inode)`, drop `i_lock`,
///     and run `evict(inode, disposition)` — writeback drain, page
///     cache teardown, `InodeOps::evict_inode(ino, disposition)`,
///     unhash + wake, RCU-deferred slab free (see the eviction
///     sequence above).
///
/// **Concurrency**: closed by the lifecycle invariants — a concurrent
/// `Inode::get()` either revives the inode under `i_lock` BEFORE phase 2's
/// recheck (recheck sees `i_refcount != 0`, shrinker skips) or finds
/// `I_FREEING` and waits for teardown (`wait_on_freeing_inode()`).
/// The lock-free increment-if-positive path cannot revive a zero-count
/// inode at all (INV-LC2), so "shrinker frees an inode with a live
/// reference" is unreachable.
///
/// **Scope**: `sb` constrains eviction to a single superblock. Pass
/// `None` to evict across all superblocks (global memory pressure).
///
/// May sleep (writeback waits, cross-domain `VfsOpcode::EvictInode`
/// ring round-trips) — process context only, no locks held on entry.
pub fn inode_cache_evict(sb: Option<&SuperBlock>, count: usize) -> usize {
    // Returns the number of inodes actually reclaimed (may be < count
    // if fewer reclaimable inodes exist or candidates were revived,
    // rotated, or skipped).
}

Shrinker integration:

A Shrinker registers a cache with the page-reclaim path (Section 4.2) so it can be shrunk under memory pressure. It is the UmkaOS analogue of Linux struct shrinker, reduced to the two callbacks and the cost hint the reclaim core actually consumes:

/// Type of a shrinker's object-count callback: returns the number of
/// currently-reclaimable objects in the cache (fast, lock-free where
/// possible). The reclaim core distributes scan pressure across all
/// registered shrinkers in proportion to this value.
pub type ShrinkerCountFn = fn() -> u64;

/// Type of a shrinker's scan callback: attempts to reclaim up to
/// `nr_to_scan` objects and returns the number actually freed.
pub type ShrinkerScanFn = fn(nr_to_scan: u64) -> u64;

/// A registered memory-reclaim shrinker. Static instances (one per cache)
/// are registered with the reclaim core during subsystem init. All fields
/// are plain data / fn pointers — a `Shrinker` is Copy-free `'static` state.
pub struct Shrinker {
    /// Reports reclaimable object count (drives proportional scan pressure).
    pub count_objects: ShrinkerCountFn,
    /// Reclaims up to `nr_to_scan` objects; returns the count freed.
    pub scan_objects: ShrinkerScanFn,
    /// Reclaim-cost hint (Linux `seeks`): higher = more expensive to
    /// rebuild, so less scan pressure. `DEFAULT_SEEKS` (2) for disk-backed
    /// caches (one seek to re-read).
    pub seeks: u32,
    /// Reserved shrinker behaviour flags (0 = none). Bit meanings are
    /// assigned by the reclaim core; VFS caches use 0.
    pub flags: u32,
}

/// Default `Shrinker::seeks` cost — matches Linux `DEFAULT_SEEKS`. Represents
/// the ~2 seeks needed to reconstruct a disk-backed cache object.
pub const DEFAULT_SEEKS: u32 = 2;

/// Upper bound on registered shrinkers (inode, dentry, and per-slab-cache
/// reclaimers). Exceeding it is a static misconfiguration caught at init.
pub const MAX_SHRINKERS: usize = 64;

/// Global registry of memory-reclaim shrinkers, populated at subsystem init
/// via `register_shrinker` and walked by the page-reclaim core under memory
/// pressure. Entries are `&'static` (registered once, never removed), so the
/// reclaim path snapshots the list under a short spinlock hold and runs the
/// (sleeping) scan callbacks with NO lock held — see
/// `shrink_registered_caches()`. Bounded — one entry per cache — so a fixed
/// `ArrayVec` holds every kernel shrinker with no heap allocation.
static SHRINKER_REGISTRY: SpinLock<ArrayVec<&'static Shrinker, MAX_SHRINKERS>> =
    SpinLock::new(ArrayVec::new_const());

/// Register a shrinker with the reclaim core. Called once per cache during
/// subsystem init (cold path). Fails with `Err(())` if `MAX_SHRINKERS` is
/// already reached.
pub fn register_shrinker(shrinker: &'static Shrinker) -> Result<(), ()> {
    SHRINKER_REGISTRY.lock().try_push(shrinker).map_err(|_| ())
}

/// Reclaim driver invoked by the page-reclaim core when free memory falls
/// below the low watermark ([Section 4.2](04-memory.md#physical-memory-allocator)). Distributes
/// `nr_to_scan` total scan pressure across all registered shrinkers in
/// proportion to each cache's `count_objects()`, and returns the total number
/// of objects freed. Warm/cold path (reclaim only), never per-syscall.
///
/// **Locking**: scan callbacks MAY SLEEP — `inode_cache_scan()` reaches
/// `writeback_single_inode()` (I/O waits) and cross-domain
/// `VfsOpcode::EvictInode` ring round-trips inside `evict()` — so the
/// registry spinlock is NEVER held across a callback. Entries are
/// `&'static` and registration is append-only, so a stack snapshot
/// taken under the lock stays valid forever; the walk runs unlocked.
/// Process context only.
pub fn shrink_registered_caches(nr_to_scan: u64) -> u64 {
    // Snapshot under the lock (bounded copy of `&'static` pointers into
    // a stack ArrayVec — no heap allocation), then drop the lock before
    // invoking any callback.
    let snapshot: ArrayVec<&'static Shrinker, MAX_SHRINKERS> =
        SHRINKER_REGISTRY.lock().clone();
    let total: u64 = snapshot.iter().map(|s| (s.count_objects)()).sum();
    if total == 0 {
        return 0;
    }
    let mut freed: u64 = 0;
    for s in snapshot.iter() {
        let share = nr_to_scan.saturating_mul((s.count_objects)()) / total;
        if share > 0 {
            freed += (s.scan_objects)(share);
        }
    }
    freed
}

/// Register the VFS inode and dentry shrinkers with the reclaim core. Runs
/// during VFS init, after the caches themselves are initialized.
#[module_init]
fn vfs_register_shrinkers() {
    register_shrinker(&INODE_CACHE_SHRINKER)
        .expect("shrinker registry has room for the inode cache");
    register_shrinker(&DENTRY_CACHE_SHRINKER)
        .expect("shrinker registry has room for the dentry cache");
}

The inode cache registers a memory shrinker callback during VFS init. The shrinker is invoked by the page reclaim path (Section 4.2) when free memory falls below the low watermark.

/// `count_objects` callback for `INODE_CACHE_SHRINKER`: the number of
/// reclaimable inodes, read as a plain atomic load of the global inode-cache
/// LRU length (no `lru_lock`, so it may be momentarily stale — acceptable for
/// proportional pressure distribution).
fn inode_cache_count() -> u64 {
    inode_cache().lru_count.load(Ordering::Relaxed)
}

/// `scan_objects` callback for `INODE_CACHE_SHRINKER`: reclaim up to
/// `nr_to_scan` inodes across all superblocks (`sb = None`) and report how
/// many were actually freed.
fn inode_cache_scan(nr_to_scan: u64) -> u64 {
    inode_cache_evict(None, nr_to_scan as usize) as u64
}

/// Inode cache shrinker. Registered as a global shrinker during VFS init.
///
/// `count_objects()`: returns `inode_cache().lru_count` (fast — no lock).
/// `scan_objects()`: calls `inode_cache_evict(None, nr_to_scan)`.
///
/// Priority: shrinker priority is 0.
/// The inode cache shrinker runs alongside the dentry cache shrinker and
/// slab shrinkers. The reclaim path distributes scan pressure across all
/// registered shrinkers proportionally to their `count_objects()` return
/// value — larger caches receive more scan pressure.
pub static INODE_CACHE_SHRINKER: Shrinker = Shrinker {
    count_objects: inode_cache_count,
    scan_objects: inode_cache_scan,
    seeks: DEFAULT_SEEKS,  // 2 — moderate cost to recreate (disk read)
    flags: 0,
};

Invariants: - Universal cache membership — every live inode is hashed and enumerable. EVERY Inode instantiation inserts into its SuperBlock.inode_cache via inode_cache_insert() on the I_NEW construction path at creation (setting I_HASHED) and is removed at destruction — with NO exception for pseudo-filesystem inodes. This covers regular filesystem inodes, the pipefs/sockfs synthetic inodes backing pipes and sockets (OpenFile.inode, "Open File Description" above), and InodeFlags::ANON_INODE anon-inode fds (Section 14.1). There is therefore NO live-but-unhashed inode class: an inode is reachable through exactly one SuperBlock.inode_cache for the whole of its lifetime — not merely while a filesystem elects to hash it — so live ⇔ I_HASHED ⇔ enumerable. The pseudo-filesystem superblocks (pipefs, sockfs, anon-inode) hosting these inodes are registered in SUPER_BLOCK_MAP at internal-mount setup (see SUPER_BLOCK_MAP above), so all_superblocks() enumerates them too. Cost: one per-superblock XArray insert per pipe(2)/socket(2)/anon-fd creation — a warm-path operation (bounded, off the per-syscall hot path), which is the price of making the Shadow-and-Migrate layout walk ("Layout migration of Inode and SuperBlock" above) complete by construction rather than by an unstated live ⇒ hashed assumption. - Every in-memory Inode with I_HASHED set is present in exactly one SuperBlock.inode_cache XArray entry. Removing from the XArray clears I_HASHED (evict() step 6b — the LAST visible lifecycle step before the RCU free). - If an inode is on the LRU list, then i_refcount == 0 AND i_nlink > 0 AND I_FREEING is clear. The converse is deliberately weak (lazy LRU): a zero-count idle inode may be transiently OFF the list while it sits in a shrinker isolation batch (inode_cache_evict() phase 1→2 window) or after a revival raced the shrinker (the holder's final inode_put() re-inserts it). - lru_count always equals the number of nodes in lru (maintained atomically — incremented on LRU insert, decremented on LRU remove). - Lock ordering (levels enforce it): XArray internal lock (XA_LOCK, 178) -> lru_lock (ICACHE_LRU_LOCK, 179), and i_lock (INODE_LOCK, 160) -> lru_lock. Never acquire an XArray lock or i_lock while holding lru_lock.

Cross-references: - Inode struct and lifecycle: this section (Inode above) - Eviction sequence: this section (Inode Lifecycle and Page Cache Teardown above) - Page reclaim and shrinker framework: Section 4.2 - Dentry cache (parallel deduplication cache for path components): this section (Dentry above) - Crash recovery inode iteration: this section (Dirty Page Handling on VFS Crash above)

14.1.2.4.1.10 SuperBlock

/// In-memory representation of a mounted filesystem.
///
/// Each mount creates one SuperBlock instance. The superblock holds
/// filesystem-level metadata (block size, feature flags, root inode)
/// and provides the interface between the VFS and the filesystem driver.
///
/// **Lifecycle**: Created by `FileSystemOps::mount()`. Destroyed by
/// `FileSystemOps::unmount()` after all references are released.
pub struct SuperBlock {
    /// Filesystem type identifier (e.g., "ext4", "xfs", "tmpfs").
    pub s_type: &'static str,

    /// Block size in bytes (typically 1024, 2048, or 4096).
    pub s_blocksize: u32,

    /// Log2 of block size (for bit-shift division).
    pub s_blocksize_bits: u8,

    /// Maximum file size supported by this filesystem.
    pub s_maxbytes: i64,

    /// Root dentry of the mounted filesystem.
    pub s_root: Arc<Dentry>,

    /// Filesystem operations (mount, unmount, statfs, sync).
    pub s_op: &'static dyn FileSystemOps,

    /// Mount flags (MS_RDONLY, MS_NOSUID, MS_NODEV, MS_DSM_COOPERATIVE, etc.).
    ///
    /// **`MS_DSM_COOPERATIVE` (bit 30)**: Set at mount time to indicate this
    /// filesystem participates in DSM cooperative caching. When set, the VFS
    /// page cache miss path (`filemap_get_pages`) uses the three-stage
    /// speculative protocol ([Section 6.11](06-dsm.md#dsm-distributed-page-cache)): (1) Bloom
    /// filter fast rejection (~15ns), (2) sequential access skip, (3)
    /// speculative parallel RDMA + NVMe for random misses. Zero overhead
    /// for ~95% of misses; ~5-10μs speedup for the rest. Enables
    /// cluster-wide page cache sharing for distributed filesystems.
    /// Only meaningful when DSM is enabled ([Section 6.11](06-dsm.md#dsm-distributed-page-cache)).
    /// Filesystems that do not set this flag use the standard local-only
    /// page cache lookup (no DSM overhead on the read path).
    /// AtomicU32: all defined flags fit in bits 0-30. If more than 31
    /// flags are needed, widen to AtomicU64.
    pub s_flags: AtomicU32,

    /// Filesystem-specific data. Opaque pointer used by the filesystem
    /// driver to attach its own per-superblock state (e.g., ext4_sb_info,
    /// xfs_mount).
    /// SAFETY: Set to a filesystem-specific type during mount(). The
    /// filesystem's `FileSystemOps::unmount()` must cast back and free. Valid
    /// for the lifetime of the superblock. Set once during mount,
    /// read-only thereafter.
    pub s_fs_info: *mut (),

    /// UUID of the filesystem (if supported). Used for persistent mount
    /// identification and `/proc/mounts` output.
    pub s_uuid: [u8; 16],

    /// Device identifier (major:minor encoded in the kernel-internal `DevId`
    /// layout). For block-backed filesystems this is the
    /// backing `BlockDevice`'s device number; for pseudo-filesystems (tmpfs,
    /// procfs) it is an anonymous device number allocated at mount. Set once
    /// during mount, read-only thereafter. Surfaced to userspace via
    /// `stat(2)`'s `st_dev` and used as the identity key in fault/recovery
    /// events (e.g. `FaultEvent::RecoveryQuiesceTimeout.sb_dev`).
    pub s_dev: u32,

    /// Per-superblock inode cache. Provides O(1) lookup by inode number
    /// via XArray (radix tree). RCU-protected reads on the hot path.
    ///
    /// Per-superblock XArray eliminates hash computation (~15-25 cycles
    /// saved vs. global RcuHashMap), provides O(1) guaranteed lookup
    /// (no collision chains), and improves cache locality (per-filesystem
    /// working set stays in its own radix tree). The caller always has the
    /// superblock from path resolution (`dentry.d_sb`), so no extra lookup
    /// is needed.
    /// On 32-bit architectures (ARMv7, PPC32), XArray stores u64 keys via
    /// a synthetic two-level index. Performance is O(1) for keys <= u32::MAX
    /// and O(log64(N)) for larger keys.
    pub inode_cache: XArray<u64, Arc<Inode>>,

    /// List of all inodes belonging to this superblock.
    /// Protected by `s_inode_list_lock`.
    pub s_inodes: IntrusiveList<Inode>,

    // Dirty inode tracking is handled exclusively by BdiWriteback.b_dirty
    // (see [Section 4.6](04-memory.md#writeback-subsystem--writeback-thread-organization)). No per-superblock dirty list.

    /// Per-superblock lock for inode list management.
    pub s_inode_list_lock: SpinLock<()>,

    /// Block device backing this filesystem (None for pseudo-filesystems
    /// like tmpfs, procfs, sysfs).
    pub s_bdev: Option<Arc<BlockDevice>>,

    /// Backing device info — controls writeback rate limiting, readahead
    /// window, and per-device dirty page accounting. Set during mount:
    ///
    /// - **Disk-backed filesystems** (ext4, XFS, Btrfs): points to the
    ///   `BackingDevInfo` owned by the underlying `BlockDevice`. The BDI
    ///   is shared if multiple mounts use the same block device (e.g.,
    ///   bind mounts). The writeback thread
    ///   ([Section 4.6](04-memory.md#writeback-subsystem--writeback-thread-organization)) uses `s_bdi` to
    ///   locate the inode dirty lists (`BdiWriteback.b_dirty`) and to
    ///   enforce per-device dirty page throttling via
    ///   `balance_dirty_pages()`.
    ///
    /// - **Network filesystems** (NFS, CIFS, 9P): allocate a dedicated
    ///   `BackingDevInfo` per superblock during mount. The BDI's `bdev`
    ///   field is `None`; writeback goes through the filesystem's own
    ///   network I/O path rather than the block layer.
    ///
    /// - **Pseudo-filesystems** (tmpfs, procfs, sysfs, devtmpfs): `None`.
    ///   These filesystems have no backing store and never produce dirty
    ///   pages that require writeback. The VFS skips all writeback and
    ///   dirty-throttling code paths when `s_bdi` is `None`.
    ///
    /// **Writeback chain**: inode → `i_sb` (`SuperBlock`) → `s_bdi`
    /// (`BackingDevInfo`) → `wb` (`BdiWriteback`). This chain is how the
    /// writeback subsystem discovers which device an inode's dirty pages
    /// should be flushed to. Breaking this chain (e.g., a disk-backed
    /// filesystem with `s_bdi = None`) would silently prevent writeback
    /// and leak dirty pages indefinitely.
    ///
    /// **Cross-reference**: `BackingDevInfo` struct definition and
    /// `BdiWriteback` internals are in [Section 4.6](04-memory.md#writeback-subsystem--writeback-thread-organization).
    pub s_bdi: Option<Arc<BackingDevInfo>>,

    /// Reference count. Held by Mount nodes and open file handles.
    pub s_refcount: AtomicU32,

    /// Per-superblock writer tracking for the freeze state machine.
    /// Implements the `SbFreezeLevel::Write -> SbFreezeLevel::PageFault ->
    /// SbFreezeLevel::Fs -> SbFreezeLevel::Complete` progression used by FIFREEZE/FITHAW ioctls and
    /// `do_remount()`.
    ///
    /// `s_writers.frozen` is the ONE freeze-state authority for this
    /// superblock (Linux parity: `sb->s_writers.frozen`). An earlier
    /// revision also carried a `s_freeze_count: AtomicU32` — a dead
    /// shadow that nothing incremented, decremented, or read (and whose
    /// count > 1 was unreachable anyway: `freeze_super` returns `EBUSY`
    /// on an already-frozen filesystem). It has been deleted; any future
    /// freeze-nesting support (Linux master's `freeze_kcount`/
    /// `freeze_ucount` kernel/userspace split) must be designed into the
    /// `SbWriters` state machine itself, not bolted on as a second
    /// representation.
    pub s_writers: SbWriters,

    /// Error handling behavior for this filesystem. Set at mount time from
    /// the `errors=` mount option (e.g., `errors=remount-ro`). Defaults to
    /// `FsErrorMode::Continue` unless the filesystem specifies otherwise.
    /// Consulted by the VFS error path when a filesystem reports an I/O or
    /// metadata corruption error.
    pub s_error_behavior: FsErrorMode,

    /// Per-mount VFS ring set for cross-domain filesystem driver
    /// communication. Contains N ring pairs (request + response), where
    /// N is negotiated at mount time. `None` for co-located deployments
    /// (the provider shares the Core domain — direct calls, no rings),
    /// which includes the pseudo-filesystems (tmpfs, procfs, sysfs).
    ///
    /// The ring_set is allocated at mount time and persists across driver
    /// crashes (rings are drained and reset, not recreated). The replacement
    /// driver re-binds to the existing ring_set during crash recovery
    /// Step U14 ([Section 14.3](#vfs-per-cpu-ring-extension--crash-recovery)).
    pub ring_set: Option<Box<VfsRingSet>>,

    /// Core-ONLY per-ring companion state, parallel to `ring_set.rings`
    /// (index = ring index): the in-flight request table + response
    /// drain lock (`VfsRingCoreSide`, [Section 14.2](#vfs-ring-buffer-protocol)).
    /// Never mapped into any driver domain — this is the state crash
    /// recovery trusts. `None` iff `ring_set` is `None`. Allocated at
    /// mount step 4a; freed at umount after the response worker exits
    /// and the recovery descriptor is unregistered.
    pub ring_core: Option<Box<[VfsRingCoreSide]>>,

    /// Exit signal for the per-mount Core response worker
    /// (`vfs_response_worker`, [Section 14.2](#vfs-ring-buffer-protocol)).
    /// 0 = run, 1 = exit at next wake. Set (Release) by umount, which
    /// then kicks `ring_set.completion_doorbell` and waits for the
    /// worker's task exit before freeing `ring_core`.
    pub ring_worker_exit: AtomicU8,

    /// This mount's crash-recovery descriptor, registered on the
    /// provider's domain at mount step 4a and unregistered at umount
    /// (`SubsystemRecoveryDescriptor` with `ctx = self`,
    /// [Section 11.9](11-drivers.md#crash-recovery-and-state-preservation--subsystem-recovery-descriptors)).
    /// The registration is what connects a domain fault to the unified
    /// VFS recovery sequence U1-U18 — there is no global
    /// DomainId → SuperBlock table.
    pub vfs_recovery_desc: SubsystemRecoveryDescriptor,

    /// Recovery-completion wait queue (Core-resident). VFS dispatchers that
    /// receive a boundary `ENXIO` from `select_ring()` while
    /// `ring_set.state != VFSRS_ACTIVE` sleep here interruptibly (crash
    /// recovery intro step b) and retry the dispatch after wake; a signal
    /// interrupts the sleep with `-ERESTARTSYS` (standard syscall-restart
    /// semantics). **Wake contract**: the recovery worker's `resume_fn`
    /// wakes ALL waiters immediately after Step U17's
    /// `ring_set.state.store(VFSRS_ACTIVE, Release)`
    /// ([Section 14.3](#vfs-per-cpu-ring-extension--crash-recovery)); umount wakes it
    /// after tearing down the ring set (waiters then observe the unmount
    /// and fail with `ENODEV`). Waiters re-check `ring_set.state` after
    /// every wake — the queue carries no payload.
    pub recovery_wait: WaitQueue,

    /// Driver generation counter. Incremented each time the filesystem
    /// driver is (re)loaded after a crash — at Step **U10a** of the
    /// unified VFS crash recovery sequence
    /// ([Section 14.3](#vfs-per-cpu-ring-extension--crash-recovery)), BEFORE the U11
    /// io_uring CQE drain and BEFORE the U14 reload. (An earlier
    /// revision said "Step U16"; U16 is a removed placeholder — bumping
    /// after U15 would make the new instance's writeback completions
    /// look stale and silently discard them.) Initialized to 0 at first
    /// mount.
    ///
    /// Used for:
    /// - **Stale response detection**: the Core response drain checks
    ///   `response.driver_generation == sb.driver_generation.load(Acquire)`
    ///   before processing any completion. Mismatched responses (from the
    ///   pre-crash driver instance) are discarded. The driver learns the
    ///   value from the read-mostly mirror `VfsRingSet.driver_generation`
    ///   (it cannot read this Core-only field); U10a bumps the sb field
    ///   first, then the mirror, both Release — the sb field is
    ///   authoritative.
    /// - **Stale file handle detection**: `OpenFile.open_generation` is
    ///   compared against this field; a mismatch routes the operation into
    ///   the lazy fd revalidation slow path
    ///   ([Section 14.1](#virtual-filesystem-layer--open-file-descriptor-recovery-generation-refresh)),
    ///   which refreshes the fd transparently — `ENOTCONN` is the
    ///   transport-level signal absorbed by that layer, not a userspace
    ///   outcome for recovered superblocks.
    ///
    /// One counter per superblock (not per ring) — all rings on a mount
    /// share the same generation. Persisted in Core (Tier 0) memory, so
    /// it survives driver crashes at any isolation tier.
    ///
    /// **Longevity**: u64 at crash-per-minute rate (extreme) wraps after
    /// ~35 trillion years. No wrap handling needed.
    pub driver_generation: AtomicU64,
}

/// Freeze level for the superblock writer tracking state machine.
///
/// The freeze process advances through levels in order:
/// `Unfrozen -> Write -> PageFault -> Fs -> Complete`.
/// Each level blocks a broader category of operations. Thaw reverses
/// the progression. Used by FIFREEZE/FITHAW ioctls and `do_remount()`.
#[repr(u8)]
pub enum SbFreezeLevel {
    /// Not frozen. All operations permitted.
    Unfrozen = 0,
    /// Block new writes (`write`, `truncate`, `fallocate`).
    Write = 1,
    /// Block page faults (prevent new page-cache population via mmap writes).
    PageFault = 2,
    /// Block filesystem operations (metadata updates, journal commits).
    Fs = 3,
    /// Fully frozen. No filesystem activity. Safe for snapshots.
    Complete = 4,
}

/// Per-superblock writer tracking for the freeze state machine.
///
/// The `frozen` field records the current freeze level. The `writers`
/// array tracks the number of active writers at each of the three
/// blockable levels (Write, PageFault, Fs). Each counter uses
/// `PerCpuCounter` for scalable per-CPU increment/decrement on the
/// write path, with a global `sum()` used only during freeze/thaw
/// transitions to wait for all writers to drain.
///
/// **Freeze protocol** (per level, see `freeze_super`):
/// 1. Set `frozen` to the target level (e.g., `SbFreezeLevel::Write`),
///    `Release` store.
/// 2. `rcu_synchronize()` — every `sb_start_write()` fast path runs its
///    increment + re-check inside one RCU read-side section, so after
///    the grace period every concurrent entry attempt has either
///    (a) observed `frozen` and blocked (net counter effect zero), or
///    (b) completed its increment, which the freezer's `sum()` below is
///    now guaranteed to observe. This closes the store-buffering race
///    between the freezer's `frozen` store and the writers' counter
///    increments — the same role `rcu_sync` plays inside Linux's
///    per-CPU write-side freeze path in Linux (`sb_wait_write`, fs/super.c).
/// 3. Wait for `writers[level - 1].sum() == 0`: sleep on `freeze_wait`,
///    re-sampling `sum()` after every wake (`SbWriteGuard::drop` wakes
///    the queue whenever a freeze is in progress).
/// 4. Advance to the next level. Repeat until `Complete`.
///
/// **Thaw protocol**: Set `frozen` back to `Unfrozen` and wake all
/// waiters blocked by the freeze.
///
/// **Shared queue note**: `freeze_wait` carries both blocked writers
/// (waiting for thaw) and the freezer (waiting for drain). Every waiter
/// re-checks its own condition after each wake; the spurious cross-wakes
/// are bounded by freeze/thaw frequency (cold path) and harmless.
pub struct SbWriters {
    /// Current freeze level — THE freeze-state authority for the
    /// superblock (see `SuperBlock::s_writers`). Read with `Acquire`,
    /// written with `Release`, always by `freeze_super`/`thaw_super`.
    pub frozen: AtomicU8,
    /// Per-level writer counts: `[0]` = Write level, `[1]` = PageFault
    /// level, `[2]` = Fs level. `PerCpuCounter` for scalable hot-path
    /// increment (no cross-CPU contention on the write path).
    pub writers: [PerCpuCounter; 3],
    /// WaitQueue for threads blocked by a freeze AND for the freezer
    /// awaiting writer drain. Woken on thaw, and by `SbWriteGuard::drop`
    /// while a freeze is in progress.
    pub freeze_wait: WaitQueue,
}

Writer entry/exit protocol (sb_start_write / SbWriteGuard::drop):

Every VFS operation that modifies the filesystem must bracket its work with the sb_start_write()/SbWriteGuard::drop protocol. This allows the freeze state machine to wait for all in-flight writers to drain before advancing to the next freeze level. The canonical Write-level entry points, each specified with its bracket in this chapter:

• Buffered write: page_cache_write_iter() Step 0 (guard held through the O_SYNC flush — Linux vfs_write() file_start_write() placement).
• Direct I/O write: the VFS O_DIRECT dispatch, entered before the exclusive i_rwsem (see "O_DIRECT (Direct I/O) Path").
• Truncate-on-open: handle_truncate() (O_TRUNC).
• Write-class ring operations (Create, Unlink, Rename, SetAttr, Truncate, Fallocate, …): the Core dispatch wrapper brackets the ring round-trip — see "Freeze/thaw interaction with VFS ring protocol" below.

SbFreezeLevel::PageFault is entered by the mmap write-fault path (page_mkwrite, Section 4.8) before dirtying a file-backed page, and SbFreezeLevel::Fs by filesystem-internal modification sources (journal commit threads) — both outside this file; the bracket obligation is normative here, the call sites live with their subsystems.

/// Attempt to enter the filesystem for a write-class operation.
/// Returns a guard that decrements the writer count on drop.
///
/// If the filesystem is frozen at or beyond the requested level,
/// the caller blocks on `sb.s_writers.freeze_wait` until thaw.
///
/// **Interruptibility**: If the calling task receives a fatal signal
/// while blocked, `sb_start_write` returns `Err(EINTR)`. The VFS
/// write path converts this to `-EINTR` for the syscall.
///
/// # Arguments
/// * `sb` — The superblock of the filesystem being written to.
/// * `level` — The freeze level this operation belongs to:
///   - `SbFreezeLevel::Write` (1): data writes (`write`, `truncate`, `fallocate`).
///   - `SbFreezeLevel::PageFault` (2): page fault writes (mmap dirty page).
///   - `SbFreezeLevel::Fs` (3): filesystem metadata updates (journal commit).
///
/// # Hot path
/// The common case (filesystem not frozen) is a single `Acquire` load
/// on `frozen` + one `PerCpuCounter::inc()` under an RCU read guard
/// (~5-10 cycles total, no contention, no shared-cacheline RMW). The
/// slow path (freeze in progress) blocks.
///
/// # Why the fast path is an RCU read-side section
/// The increment + re-check pair must be bounded by an RCU read-side
/// critical section so that `freeze_super`'s `rcu_synchronize()` (after
/// its `frozen` store) can conclude: every writer that slipped past the
/// `frozen` check has finished its increment and is visible to the
/// drain-side `sum()`. Without it, a writer's per-CPU increment could
/// still be in flight when the freezer samples `sum() == 0` — the
/// classic store-buffering race Linux closes the same way (`rcu_sync`
/// inside its per-CPU write-side freeze path).
pub fn sb_start_write(sb: &SuperBlock, level: SbFreezeLevel) -> Result<SbWriteGuard, Errno> {
    loop {
        let rcu = rcu_read_lock(); // bounds the inc + re-check (see above)
        let frozen = sb.s_writers.frozen.load(Ordering::Acquire);
        if frozen >= level as u8 {
            drop(rcu);
            // Filesystem is frozen at or beyond our level. Block.
            sb.s_writers.freeze_wait.wait_interruptible()?;
            continue;
        }
        sb.s_writers.writers[(level as u8 - 1) as usize].inc();
        // Re-check after increment (the freeze may have advanced between
        // our check and increment — same race as Linux's percpu_rwsem).
        let frozen_after = sb.s_writers.frozen.load(Ordering::Acquire);
        drop(rcu);
        if frozen_after >= level as u8 {
            sb.s_writers.writers[(level as u8 - 1) as usize].dec();
            // The freezer may already be draining: report our retraction.
            sb.s_writers.freeze_wait.wake_up_all();
            sb.s_writers.freeze_wait.wait_interruptible()?;
            continue;
        }
        return Ok(SbWriteGuard { sb, level });
    }
}

/// RAII guard that decrements the writer count on drop; callers spell exit as
/// `drop(guard)`. If a freeze is in progress, the drop also
/// wakes `freeze_wait` so the freezer re-samples the drain condition.
pub struct SbWriteGuard<'a> {
    sb: &'a SuperBlock,
    level: SbFreezeLevel,
}

impl Drop for SbWriteGuard<'_> {
    fn drop(&mut self) {
        self.sb.s_writers.writers[(self.level as u8 - 1) as usize].dec();
        // Drain wake: without this, `freeze_super`'s wait for
        // `sum() == 0` would never be woken — the freeze state machine
        // would drain a counter nobody signals. Two cases:
        // - our `frozen` load below sees the freezer's store (`Release`
        //   store / `Acquire` load on the same atomic): we wake it, it
        //   re-samples `sum()` and observes our decrement;
        // - our whole drop completed before the freezer's
        //   post-`rcu_synchronize()` FIRST `sum()` sample: that sample
        //   already observes the decrement, and no wake is needed.
        // Either way the freezer cannot sleep forever on a drained
        // counter. Cost on the non-frozen hot path: one `Acquire` load.
        if self.sb.s_writers.frozen.load(Ordering::Acquire)
            != SbFreezeLevel::Unfrozen as u8
        {
            self.sb.s_writers.freeze_wait.wake_up_all();
        }
    }
}

/// Freeze a filesystem to the specified level.
///
/// Called by `FIFREEZE` ioctl and `do_remount()` (remount read-only).
/// Advances through freeze levels sequentially:
/// `Write -> PageFault -> Fs -> Complete`.
///
/// At each level:
/// 1. Set `frozen` to the target level (Release store).
/// 2. `rcu_synchronize()` — after the grace period, every in-flight
///    `sb_start_write()` fast path has either blocked or made its
///    increment visible (see the `SbWriters` freeze protocol).
/// 3. Wait for `writers[level-1].sum() == 0`: sleep interruptibly on
///    `s_writers.freeze_wait`, re-sampling `sum()` after every wake
///    (`SbWriteGuard::drop` wakes the queue while a freeze is in
///    progress).
///
/// Between levels (Linux `freeze_super()`, `fs/super.c`): after the
/// Write-level drain, dirty data and metadata are flushed for `sb` so the
/// PageFault/Fs drains are not waiting on writeback that new writers
/// could otherwise keep regenerating. After the Fs-level drain, notify
/// the filesystem driver to quiesce its internal state
/// (`FileSystemOps::freeze`; cross-domain providers receive it as the
/// ring `Freeze` opcode) before storing `Complete`.
///
/// After reaching `Complete`, the filesystem is fully quiesced: no pending
/// writes, no page faults, no metadata updates. Safe for LVM snapshots,
/// device-mapper operations, and backup tools.
///
/// # Errors
/// Returns `Err(EBUSY)` if the filesystem is already frozen
/// (`s_writers.frozen != Unfrozen` — freezes do not nest; a second
/// FIFREEZE observes `EBUSY` exactly as on Linux).
/// Returns `Err(EINTR)` if the wait is interrupted by a fatal signal —
/// the partial freeze is rolled back before returning: `frozen` is
/// stored back to `Unfrozen` and `freeze_wait` is woken, releasing any
/// writers that blocked against the partial level.
pub fn freeze_super(sb: &SuperBlock) -> Result<(), Errno>;

/// Thaw a frozen filesystem.
///
/// Called by `FITHAW` ioctl. Notifies the driver to resume
/// (`FileSystemOps::thaw` / ring `Thaw` opcode), sets `frozen` back to
/// `Unfrozen` (Release), and wakes all threads blocked on `freeze_wait`.
/// Returns `Err(EINVAL)` if the filesystem is not frozen.
pub fn thaw_super(sb: &SuperBlock) -> Result<(), Errno>;

Freeze/thaw interaction with VFS ring protocol: freeze protection for cross-domain filesystems is enforced Core-side, before the ring. The Core dispatch wrapper brackets every write-class ring operation (Write, Truncate, Fallocate, Create, Mkdir, Symlink, Link, Unlink, Rmdir, Rename, Mknod, SetAttr, SetXattr, RemoveXattr) with sb_start_write(sb, SbFreezeLevel::Write) … drop(guard) around the ring round-trip, so during a freeze these operations block until thaw (interruptibly, -EINTR on a fatal signal) — never reaching the ring. Blocking is the Linux-exact ABI: write(2) to a frozen filesystem sleeps in sb_start_write until FITHAW; it does not fail. (An earlier revision returned -EROFS for write-class operations during freeze — wrong twice over: userspace-visible errno divergence, and it left the Tier-0/co-located write paths entirely outside the freeze machinery.) Read-class operations (Read, Lookup, Getattr, Readdir, ReadPage, Readahead) continue to function during freeze. The Freeze and Thaw opcodes in the ring protocol (Section 14.2) are used by freeze_super/thaw_super to notify the filesystem driver to quiesce/resume its own internal state (journal, allocator).

/// Filesystem error handling behavior (set via mount option `errors=`).
///
/// When a filesystem encounters an I/O error or metadata corruption,
/// the VFS consults `SuperBlock.s_error_behavior` to determine the
/// system-level response. This is separate from the error returned to
/// the calling application (which always gets an appropriate errno).
///
/// **Linux compatibility**: The `errors=` mount-option keywords
/// (`continue` / `remount-ro` / `panic`) and their runtime semantics match
/// Linux exactly. The numeric discriminants below are UmkaOS-internal
/// (0-based) and are deliberately NOT the ext4 on-disk encoding: ext4's
/// `s_errors` superblock field is `Le16` with `EXT4_ERRORS_CONTINUE = 1`,
/// `EXT4_ERRORS_RO = 2`, `EXT4_ERRORS_PANIC = 3` (`fs/ext4/ext4.h`). The ext4
/// driver MUST remap between the on-disk `s_errors` value and `FsErrorMode`
/// at mount and writeback — it must never cast one directly to the other.
#[repr(u8)]
pub enum FsErrorMode {
    /// Continue operation after error (default for ext4).
    /// The error is reported to the application via errno, but the
    /// filesystem remains mounted read-write. Suitable for non-critical
    /// filesystems where availability is preferred over safety.
    Continue = 0,

    /// Remount filesystem read-only on error.
    /// This is the safest non-destructive option: it prevents further
    /// data corruption while keeping existing data readable.
    ///
    /// **Remount-ro procedure**:
    /// 1. Set `SuperBlock.s_flags |= MS_RDONLY` (atomic OR).
    /// 2. Flush all dirty pages via `sync_fs(sb, wait=true)`. Pages that
    ///    fail to flush are marked with `PageFlags::ERROR` and left in the cache
    ///    (they cannot be written back to a read-only filesystem).
    /// 3. Reject all future write operations (`write`, `truncate`,
    ///    `fallocate`, `rename`, `unlink`, `mkdir`, etc.) with `EROFS`.
    /// 4. Log the error and the remount event to the kernel log and
    ///    the fault management subsystem ([Section 20.1](20-observability.md#fault-management-architecture)).
    /// 5. Existing read-only file descriptors continue to work.
    ///    Existing read-write file descriptors remain open but all
    ///    write operations return `EROFS`.
    RemountRo = 1,

    /// Kernel panic on filesystem error.
    /// Used for critical root filesystems where continuing with a
    /// corrupted filesystem is worse than rebooting. This should only
    /// be set on the root filesystem in environments with automatic
    /// reboot and fsck (e.g., servers with watchdog timers).
    Panic = 2,
}

Writeback error integration: The writeback subsystem calls check_fs_error_mode(sb) when a page writeback I/O completes with an error. This function inspects sb.s_error_behavior and takes the configured action (log, remount-ro, or panic). Without this hook, errors=remount-ro would be meaningless for asynchronous writeback errors — the writeback subsystem would mark pages PageFlags::ERROR but never trigger the VFS-level error policy. See Section 4.6 for the writeback I/O completion path that invokes this check.

See Section 14.2 for the VFS ring buffer cross-domain dispatch protocol (request/response ring pairs, opcodes, marshaling, timeout, cancellation, crash recovery).

See Section 14.4 for the fsync/fdatasync end-to-end flow and Copy-on-Write / Redirect-on-Write infrastructure (WriteMode, ExtentSharingOps, shared-extent page cache, reflink ioctls, CoW-aware writeback, free space accounting).

14.1.2.5 End-to-End Write Path: Userspace to Hardware

This walkthrough traces a single buffered write(2) from a userspace application through every kernel layer to stable media. It serves as a cross-reference map connecting the VFS, page cache, writeback, block layer, and device driver specifications.

1. USERSPACE: write(fd, buf, len)
   → Syscall entry ([Section 19.1](19-sysapi.md#syscall-interface))
   → umka-sysapi resolves fd to OpenFile

2. VFS DISPATCH: vfs_write(file, buf, len, &pos)
   → fdget(): lockless RCU fd lookup (zero refcount traffic for an
     unshared fd table — see "File Descriptor Lookup" above)
   → fdget_pos(): f_pos serialization (lock-free for private
     descriptors, f_pos_lock otherwise)
   → File operations dispatch via file.f_ops.write
   → Calls page_cache_write_iter() for regular files

3. PAGE CACHE WRITE: page_cache_write_iter()
   → sb_start_write(sb, Write): freeze protection entered before
     i_rwsem; blocks here until FITHAW if the filesystem is frozen
   → mapping.ops.write_begin(mapping, pos, len, 0)
     → Page cache lookup via XArray ([Section 4.4](04-memory.md#page-cache))
     → On miss: allocate page, insert into XArray
   → copy_from_user(page_addr + offset, buf, len)
     → Data copied from userspace buffer to page cache page
   → set_page_dirty(page) → marks PageFlags::DIRTY
   → vfs_dirty_extent_reserve() ([Section 14.4](#vfs-fsync-and-cow))
     → Reserves writeback intent in the dirty extent tracker

4. FILESYSTEM NOTIFICATION: .write() callback
   → For ext4: mapping.ops.write_begin() / mapping.ops.write_end()
     → Journal reservation (JBD2) for metadata
     → Delayed allocation: logical blocks reserved, physical not yet assigned
   → For XFS: xfs_file_write_iter() → iomap framework
   → For Btrfs: CoW reservation via extent tree
   **Tier boundary**: For Tier 1 filesystems, write_begin/write_end are
   dispatched via KABI ring. The Tier 1 filesystem's write_end() response
   includes `dirty: bool`. The Tier 0 VFS ring consumer calls
   set_page_dirty() on behalf of the filesystem -- the filesystem never
   calls set_page_dirty() directly across the domain boundary.
   ([Section 12.8](12-kabi.md#kabi-domain-runtime))

5. WRITEBACK (ASYNCHRONOUS): triggered by dirty ratio threshold,
   periodic writeback timer (default 5s), or explicit fsync()
   → Writeback thread ([Section 4.6](04-memory.md#writeback-subsystem--writeback-thread-organization))
     picks dirty inode from per-bdi writeback list
   → writeback_single_inode() → .writepages() or .writepage()
   → Filesystem allocates physical blocks (delayed allocation commit):
     - ext4: ext4_writepages() → ext4_map_blocks() assigns physical extents
     - XFS: xfs_vm_writepages() → xfs_bmapi_write()
     - Btrfs: extent_writepages() → CoW extent allocation
   → vfs_dirty_extent_commit() binds physical block address to intent

6. BIO CONSTRUCTION: filesystem builds Bio from dirty pages
   → Bio { op: Write, start_lba, segments: [page, ...] }
   → Sets BioFlags: FUA for journal commits, PERSISTENT for critical I/O
   → ([Section 15.2](15-storage.md#block-io-and-volume-management--bio-crash-recovery))

7. BLOCK LAYER: bio_submit_raw()
   → Cgroup I/O throttling check ([Section 15.2](15-storage.md#block-io-and-volume-management--cgroup-io-throttling))
   → I/O scheduler path (if attached):
     bio_to_io_request() → scheduler merges, reorders
     ([Section 15.18](15-storage.md#io-priority-and-scheduling))
   → Direct dispatch path (NVMe multi-queue): bypass scheduler

8. DEVICE DRIVER: BlockDeviceOps::submit()
   → Tier 0: direct function call in kernel context
   → Tier 1: KABI ring dispatch through DomainRingBuffer
     ([Section 12.3](12-kabi.md#kabi-bilateral-capability-exchange))
   → Tier 2: IPC message to userspace driver process

9. HARDWARE DMA: driver programs NVMe SQ / AHCI command slot / virtio desc
   → DMA from page cache page to device
   → DmaDevice::dma_map_sgl() creates IOMMU mapping
     ([Section 4.14](04-memory.md#dma-subsystem))
   → Device writes data to stable media

10. COMPLETION: device signals IRQ → driver processes CQ entry
    → bio_complete() invokes bio.end_io callback (interrupt context)
    → Deferred to blk-io workqueue for page cache updates:
      - Clear PageFlags::WRITEBACK on the page
      - Wake fsync() waiters if applicable
      - Update AddressSpace.wb_err on error

Design note — write() and async writeback visibility: write() returns success as soon as data is in the page cache (step 3 above). Asynchronous writeback failures (step 10) are NOT visible to write() — they are visible only to fsync() via the ErrSeq mechanism (Section 15.1). This is intentional and Linux-compatible: write() is a buffer-fill operation, not a durability guarantee. Applications that need durability must call fsync() or use O_SYNC/O_DSYNC. The ErrSeq mechanism ensures each open file descriptor sees each writeback error exactly once on the next fsync() call — the fd snapshots AddressSpace::wb_err at open() time (file.f_wb_err), and fsync() compares the snapshot to the current wb_err generation to detect new errors. If the application never calls fsync(), writeback errors are silently absorbed (the data is lost, but the application was not requesting durability guarantees). This matches POSIX semantics and Linux 4.13+ behavior (errseq_t).

14.1.2.6 O_SYNC / O_DSYNC Write Path

When a file is opened with O_SYNC or O_DSYNC, write() must not return until the data (and possibly metadata) is on stable storage. This guarantee is enforced after step 3 (page cache write) completes, before returning to userspace.

The synchronous write path reuses the normal page-cache write path above — data is still copied to a page cache page and the page is marked dirty. The difference is that the caller blocks on writeback before returning, instead of deferring to the asynchronous writeback thread. This design keeps the page cache as the single source of truth for dirty tracking and avoids duplicating writeback logic.

O_SYNC/O_DSYNC branch (inserted between steps 3 and 4 above):

3a. SYNC CHECK: after set_page_dirty() and vfs_dirty_extent_reserve():
    if file.f_flags & (O_SYNC | O_DSYNC) != 0 or inode.is_sync():
        // inode.is_sync() = per-inode InodeFlags::SYNC (chattr +S) or MS_SYNCHRONOUS
        // mount — forces full O_SYNC semantics (Linux IS_SYNC(); see the
        // enforcement map in the Inode Attribute Flags subsection).
        // Flush the dirty range we just wrote to stable storage.
        err = filemap_write_and_wait_range(
            mapping,
            offset,           // start of the write
            offset + len - 1, // end of the write (inclusive)
        )
        // filemap_write_and_wait_range():
        //   1. Calls writeback_range(mapping, start, end) which triggers
        //      AddressSpaceOps::writepages() for the dirty pages in [start, end].
        //   2. Waits for PageFlags::WRITEBACK to clear on all pages in the range
        //      (blocks until device DMA + completion for those pages).
        //   3. Returns the first error from wb_err in the range, if any.
        if err != 0:
            // Writeback failed — propagate error to write() caller.
            // The page remains in the page cache (still dirty or errored).
            // AddressSpace.wb_err records the error for subsequent fsync().
            return Err(err)

        // O_SYNC: data + ALL metadata must be stable.
        // O_DSYNC: data must be stable; metadata only if file size changed.
        // (O_SYNC is the composite __O_SYNC|O_DSYNC — test the __O_SYNC
        // bit to distinguish; InodeFlags::SYNC counts as O_SYNC.)
        if file.f_flags & (O_SYNC & ~O_DSYNC) != 0 or inode.is_sync():
            // Full sync: flush inode metadata (timestamps, size, blocks).
            err = vfs_fsync_metadata(inode)
            if err != 0:
                return Err(err)
        else:
            // O_DSYNC: flush metadata only if i_size changed (data integrity).
            // File size changes affect data recoverability — a crash after
            // extending the file but before updating i_size on disk would lose
            // the new data (it would be beyond the on-disk EOF). Timestamp
            // updates (mtime, ctime) are NOT required for data integrity.
            if offset + len > old_i_size:
                err = vfs_fsync_metadata(inode)
                if err != 0:
                    return Err(err)

vfs_fsync_metadata(inode) calls InodeOps::write_inode(InodeId(inode.i_ino), WriteSyncMode::Sync) with Linux WB_SYNC_ALL semantics to flush the inode's on-disk metadata. For journaling filesystems (ext4, XFS), this commits the journal transaction containing the inode update. For non-journaling filesystems, this writes the inode block and issues a cache flush.

Performance: O_SYNC adds the device write latency to every write() call (~10-15 us on NVMe, ~3-8 ms on SATA). This is inherent — the user requested durability. The page cache write (step 3) remains ~1-5 us; the additional cost is entirely device I/O.

Interaction with writeback: The synchronous flush in step 3a writes back the same dirty pages that the asynchronous writeback thread (step 5) would eventually process. After filemap_write_and_wait_range() completes, the pages are clean (PageFlags::DIRTY cleared), so the writeback thread skips them. No double-write occurs. Dirty page accounting (AddressSpace.page_cache.nr_dirty, BDI dirty counters) is correctly decremented by the writeback completion path, regardless of whether writeback was triggered synchronously or asynchronously.

Error semantics: If the device reports a write error, the error is: 1. Stored in AddressSpace.wb_err (for subsequent fsync() error reporting). 2. Returned from write() to the caller (the write "failed" from the durability perspective, even though the data is in the page cache). 3. The page may remain dirty in the cache (for retry on next writeback attempt).

14.1.2.7 O_DIRECT (Direct I/O) Path

O_DIRECT bypasses the page cache entirely: data is transferred via DMA directly between the user buffer and the block device. This eliminates double-copying (user buffer to page cache to device) and avoids polluting the page cache with streaming I/O data that will never be re-read.

Alignment requirements: O_DIRECT requires sector-aligned file offset and transfer length. The required alignment is filesystem-dependent and reported via statx() (dio_offset_align, dio_mem_align fields in UmkaStatx). Typical values: - ext4/XFS on NVMe: 512 bytes (sector size) - ext4 with bigalloc: filesystem block size (e.g., 4096) - Btrfs: sector size (4096)

Unaligned offset or length returns EINVAL from write() / read().

The user buffer must also be aligned to dio_mem_align (typically 512 bytes). This ensures the DMA controller can transfer directly to/from the buffer without bounce buffering.

/// Direct I/O operations. Returned by `AddressSpaceOps::direct_io()` for
/// filesystems that support O_DIRECT. The VFS calls these methods instead
/// of the page-cache path when `FMODE_DIRECT` is set on the file.
///
/// **Locking**: the VFS DIO dispatch — not the implementation — acquires
/// `i_rwsem` (EXCLUSIVE around `direct_write`, SHARED around
/// `direct_read`; see the coherence protocol below) and performs the
/// flush/invalidate brackets (coherence rules 2-4) around these calls.
/// Implementations MUST NOT take `i_rwsem` themselves.
///
/// **Freeze protection**: the same dispatch brackets `direct_write` with
/// `sb_start_write(sb, SbFreezeLevel::Write)` … `drop(guard)`, entered
/// BEFORE the exclusive `i_rwsem` and released after the post-write
/// invalidation (coherence rule 4) — a DIO write to a frozen filesystem
/// blocks until thaw, exactly like the buffered path
/// (`page_cache_write_iter()` Step 0). `direct_read` takes no freeze
/// protection (reads are never freeze-blocked).
pub trait DirectIoOps: Send + Sync {
    /// Perform a direct read from the block device into the user buffer.
    ///
    /// `file`: the open file (provides inode, block mapping).
    /// `buf`: user-space destination buffer (must be dio_mem_align-aligned).
    /// `offset`: file offset (must be dio_offset_align-aligned).
    /// `len`: number of bytes to read.
    ///
    /// Returns the number of bytes actually read (may be less than `len`
    /// on EOF or partial DMA completion).
    ///
    /// The implementation must:
    /// 1. Map the file offset range to block device LBAs via the filesystem's
    ///    extent/block map.
    /// 2. Pin the user buffer pages via `Mm::pin_pages(buf, len, WRITE)`.
    /// 3. Build a Bio with the pinned user pages as DMA targets.
    /// 4. Submit the Bio and wait for completion.
    /// 5. Unpin the user pages on completion.
    fn direct_read(
        &self,
        file: &OpenFile,
        buf: UserSliceMut,
        offset: u64,
        len: u64,
    ) -> Result<u64, IoError>;

    /// Perform a direct write from the user buffer to the block device.
    ///
    /// `file`: the open file.
    /// `buf`: user-space source buffer (must be dio_mem_align-aligned).
    /// `offset`: file offset (must be dio_offset_align-aligned).
    /// `len`: number of bytes to write.
    ///
    /// Returns the number of bytes actually written (short write on error).
    ///
    /// The implementation must:
    /// 1. Allocate blocks if writing beyond current extents (fallocate or
    ///    delayed allocation commit).
    /// 2. Pin the user buffer pages via `Mm::pin_pages(buf, len, READ)`.
    /// 3. Build a Bio with the pinned user pages as DMA sources.
    /// 4. Submit the Bio and wait for completion.
    /// 5. Update i_size if the write extended the file.
    /// 6. Unpin the user pages on completion.
    fn direct_write(
        &self,
        file: &OpenFile,
        buf: UserSlice,
        offset: u64,
        len: u64,
    ) -> Result<u64, IoError>;
}

Cache coherence protocol: O_DIRECT and buffered I/O on the same file must never produce LOST WRITES, and stale buffered reads must be bounded to the documented best-effort window (rule 5 below). The protocol is built on the REAL lock model of the I/O paths specified in this section (see page_cache_write_iter() — i_rwsem exclusive; page_cache_read_iter() — no i_rwsem at all, page cache reads are lockless):

Path	i_rwsem (level 80)
Buffered write (page_cache_write_iter)	EXCLUSIVE
Buffered read (page_cache_read_iter)	NONE (lockless page cache)
Direct I/O write (DirectIoOps::direct_write)	EXCLUSIVE
Direct I/O read (DirectIoOps::direct_read)	SHARED

Direct I/O cache coherence:

1. WRITE-SIDE EXCLUSION via i_rwsem:
   - DIO write vs buffered write vs truncate: all take i_rwsem
     EXCLUSIVE — fully serialized. A DIO write can never race a
     buffered write or a concurrent dirty-page producer on the range.
   - DIO read (SHARED) vs any writer (EXCLUSIVE): excluded for the
     whole duration of the device I/O — a DIO read never observes a
     half-applied buffered write, and concurrent DIO reads proceed in
     parallel.
   (A filesystem MAY narrow DIO write to SHARED for pure
   non-extending, non-allocating overwrites — the ext4/XFS overwrite
   optimization; the generic contract is EXCLUSIVE.)
   Buffered READS take no i_rwsem and are NOT excluded — their
   coherence is invalidation-based (rules 3-5).

2. BEFORE DIO READ — flush only, no invalidation:
   filemap_write_and_wait_range(mapping, offset, offset + len - 1)
   → Dirty cached pages in the range reach disk before the device
     read, so the DIO read returns current data. Clean cached pages
     MAY remain cached — they match disk by definition.
   (Linux parity: generic_file_read_iter() IOCB_DIRECT branch →
   Linux `kiocb_write_and_wait()`, mm/filemap.c.)

3. BEFORE DIO WRITE — flush + invalidate:
   filemap_write_and_wait_range(mapping, offset, offset + len - 1)
   → Writes back dirty pages in the range (their writeback would
     otherwise race the DIO DMA and overwrite it with stale cache
     contents).
   invalidate_clean_pages_range(mapping, offset, offset + len - 1)
   → Removes the now-clean pages so post-write buffered reads miss
     and re-fetch the new data from disk.
   → Err(EBUSY) here does NOT abort the DIO write: a page that is
     pinned or re-dirtied at this instant cannot be waited out
     without an unbounded flush-invalidate retry loop. Proceed with
     the write; the step-4 invalidation plus its error path bound
     the staleness, exactly as rule 5 describes for the general
     lockless-reader case.
   (Linux parity: filemap_invalidate_pages(), mm/filemap.c.)

4. AFTER DIO WRITE COMPLETION — second invalidation:
   invalidate_clean_pages_range(mapping, offset, offset + len - 1)
   → Catches pages RE-instantiated during the DIO write by lockless
     buffered readers, readahead, or Mm::pin_pages() faulting an
     mmap of the written region (including the self-referential case
     where the DIO source buffer is an mmap of the same file). Runs
     after the filesystem's completion work (unwritten-extent
     conversion), for the same reason as Linux: invalidating before
     conversion would let a racing buffered read cache zeros.
   → Err(EBUSY) — at least one page in the range could not be
     invalidated (dirtied or pinned meanwhile): record -EIO in
     mapping.wb_err (per-fd fsync visibility) and emit a
     rate-limited "stale pagecache" warning. The DIO write itself
     still succeeded.
   (Linux parity: kiocb_invalidate_post_direct_write() +
   dio_warn_stale_pagecache(), mm/filemap.c; ordering-after-end_io
   per fs/iomap/direct-io.c.)

5. RESIDUAL WINDOW (documented, matches Linux): buffered reads are
   lockless, so a buffered read racing an in-flight DIO write may
   return pre-write data and may momentarily re-cache it; rule 4
   bounds the staleness. POSIX makes no ordering promise for this
   race, and open(2) tells applications not to mix O_DIRECT and
   buffered I/O on overlapping ranges without their own
   serialization. What IS guaranteed: no lost writes (rule 1), and
   read-after-completed-write coherence — once the DIO write has
   returned, no pre-write page remains cached (rules 3-4).

VMA integration: O_DIRECT does NOT allocate page cache pages. The user buffer pages are pinned in physical memory via Mm::pin_pages() for the duration of the DMA transfer and unpinned on completion. This is fundamentally different from buffered I/O, where data passes through kernel-owned page cache pages.

No-DIO filesystems: If AddressSpaceOps::direct_io() returns None, the filesystem cannot service O_DIRECT. Opening such a file with O_DIRECT set returns EINVAL at open time — there is no silent buffered downgrade. The authoritative gate is open_and_install step 3h (Section 14.1), which rejects the open with Err(EINVAL) when direct_io() is None and never sets FMODE_DIRECT. This matches Linux fs/open.c do_dentry_open(), which sets Linux FMODE_CAN_ODIRECT is set only when a_ops->direct_IO exists and returns -EINVAL for O_DIRECT without it.

Error handling: If DMA fails mid-transfer (device error, IOMMU fault), the Bio completion callback reports the error. The DIO path returns the number of bytes successfully transferred (short read/write). If zero bytes were transferred, the error code from the Bio is returned directly (e.g., EIO).

Key latency contributors (approximate, NVMe on x86-64): - Steps 1-4 (VFS + page cache): ~1-5 us (CPU-bound, no I/O) - Step 3a (O_SYNC/O_DSYNC): +10-15 us NVMe, +3-8 ms SATA (device write latency) - Step 5 (writeback trigger): 0-5s delay (async) or 0 (fsync path) - Steps 6-7 (bio construction + block layer): ~1-3 us - Steps 8-9 (driver + DMA): ~1-2 us (Tier 0/1) or ~5-10 us (Tier 2) - Step 10 (hardware): 10-100 us (NVMe) / 1-10 ms (SATA) - O_DIRECT path (bypasses steps 3-5): ~15-120 us total (DMA + device)

Cross-references for each step: - Syscall dispatch: Section 19.1 - Page cache: Section 4.4 - Dirty extent tracking: Section 14.4 - Writeback subsystem: Section 4.6 - Bio and block device trait: Section 15.2 - I/O scheduling: Section 15.18 - KABI ring dispatch: Section 12.3 - DMA subsystem: Section 4.14 - Tier 1 crash recovery for in-flight writes: Section 11.9

The following sections (Pipe Subsystem, Inode Cache, Dentry Cache, Path Resolution, Mount Namespace) remain in this file.

14.1.3 Pipe Subsystem

For pipe implementation, see Section 14.17.

14.1.4 Inode Cache (icache)

The inode cache uses per-superblock XArray lookup (SuperBlock.inode_cache) with a global LRU list (InodeCache) for eviction. See the Core VFS Data Structures section above for struct definitions. It provides:

• inode_cache_lookup(sb, ino) — O(1) per-superblock XArray lookup under RCU (hot path).
• inode_cache_insert(inode) — inserts into the inode's superblock XArray.
• inode_cache_evict(sb, count) — LRU eviction for memory pressure.
• INODE_CACHE_SHRINKER — registered shrinker for integration with the page reclaim subsystem.

Each superblock holds an XArray<u64, Arc<Inode>> keyed by inode number for O(1) lookup with no hash computation. Unreferenced inodes (i_refcount == 0, i_nlink > 0) are placed on a global LRU list for eviction under memory pressure.

Memory pressure integration: The inode cache registers a shrinker with Core's memory reclaim subsystem (Section 4.2). When the page allocator signals pressure, inode_cache_evict() reclaims up to nr_to_scan inodes from the LRU with EvictDisposition::Release — each reclaim frees the inode struct, its page cache pages (via truncate_inode_pages_range(&inode.i_mapping, 0, u64::MAX)), its driver-private state, and its LSM blob; on-disk state is never touched (the files are live, i_nlink > 0). The dentry cache shrinker runs first (evicting dentries drops inode refcounts, making more inodes eligible for LRU eviction).

14.1.5 Dentry Cache

The dentry (directory entry) cache is the performance-critical data structure of the VFS. It maps (parent_inode, name) pairs to child inodes, eliminating repeated disk lookups for path resolution.

Data structure: RCU-protected hash table. Read-side lookups are lock-free — no atomic operations on the read path, only a memory barrier on RCU read lock entry/exit. This matches Linux's dentry cache design, which is similarly RCU-protected for the same performance reasons.

Negative dentries: When a lookup() returns ENOENT, the VFS caches a negative dentry for that (parent, name) pair. Subsequent lookups for the same nonexistent path component return ENOENT immediately without calling into the filesystem driver. This is critical for workloads like $PATH searches where the shell looks for an executable in 5-10 directories, finding it only in one. Without negative dentries, every command invocation would perform 4-9 unnecessary disk lookups.

Eviction: LRU eviction under memory pressure. The dentry cache integrates with Core's memory reclaim (Section 4.12 — Memory Compression Tier, in 04-memory.md) — when the page allocator signals memory pressure, the dentry cache shrinker evicts least-recently-used entries. Negative dentries are evicted preferentially (they are cheaper to re-create than positive dentries). Every eviction returns the dentry's storage slot to the Nucleus tracked allocator via free_tracked::<Dentry>() (RCU-deferred — see the free path in "Dentry Allocation — Nucleus Tracked Storage" below), so reclaim pressure directly replenishes the dentry instance budget.

14.1.5.1 Dentry Allocation — Nucleus Tracked Storage

Dentry is a migration-tracked type (Section 13.18): dentries are long-lived, and cache-line repacking of the hot lookup fields (d_name hash prefix, d_flags, cached_perm) is exactly the class of layout evolution the tracked allocator exists for. Every live dentry must be enumerable by the Nucleus evolution orchestrator, so dentries allocate from Nucleus tracked storage — never from an untracked slab cache. The conversion follows the Task template (Section 13.18, Section 8.1 step 6).

Hot-path cost (negative-overhead analysis): dentry allocation runs only on a dcache MISS — the >99%-hit-rate RCU-walk fast path never allocates. On a miss, alloc_tracked's fast path is a per-CPU magazine pop with no lock, using the same magazine implementation as the slab allocator (Section 4.3) — cycle-equivalent to a slab_alloc fast path (~20-40 cycles). The conversion is therefore cost-neutral on the allocation path, and the miss path is dominated by the filesystem driver lookup (microseconds when I/O is involved) regardless. The free path is a magazine push; the RCU deferral it sits behind was already required by the dcache design (d_rcu) and is not an added cost. Slow path: depot refill briefly holds TRACKED_REGISTRY_LOCK(135) — legal from dentry_alloc()'s call sites because callers hold NO locks at all when calling dentry_alloc() (see the locking contract on dentry_alloc() below: the parent's d_lock is taken by dentry_alloc() itself, internally, only for the insertion step AFTER allocation), and amortized over the magazine size.

Type registration (from the VFS module's #[module_init] constructor, which runs during boot before the first path lookup):

// umka-vfs/src/dcache/dcache_init.rs — Evolvable

impl TrackedType for Dentry {
    fn type_id() -> TypeId {
        DENTRY_TYPE_ID.get().copied().expect("Dentry not yet registered")
    }
}

static DENTRY_TYPE_ID: BootOnceCell<TypeId> = BootOnceCell::new();

#[module_init]
fn dcache_register_tracked_type() {
    let slot = round_up(core::mem::size_of::<Dentry>(),
                        core::mem::align_of::<Dentry>()) as u64;
    let template = TypeDescriptorTemplate {
        size: core::mem::size_of::<Dentry>() as u32,
        alignment: core::mem::align_of::<Dentry>() as u32,
        // Runtime-derived f(memory): the dcache instance budget defaults to
        // min(physical_memory / 128, 2 GiB) of tracked storage — exactly
        // half of the tracked region's own default of
        // min(physical_memory / 64, 4 GiB)
        // ([Section 13.18](13-device-classes.md#live-kernel-evolution--generic-tracked-allocator)), so the
        // default registration always fits and boot cannot fail on it.
        // The dcache is the highest-instance-count tracked type in the
        // kernel; it deliberately receives the largest share. Deployments
        // with tens of millions of dentries (file servers) raise
        // `umka.tracked_storage_size` and `umka.vfs.dcache_slots` together.
        // This budget is not a workload limit in the Linux sense — the
        // dcache was never allowed to grow without bound in Linux either;
        // there the bound is implicit (reclaim pressure), here it is
        // explicit and the shrinker enforces it (see below).
        max_instances: boot_param("umka.vfs.dcache_slots",
            default = (core::cmp::min(
                total_ram_pages() as u64 * PAGE_SIZE as u64 / 128,
                2u64 << 30 /* 2 GiB */) / slot) as u32),
        migration_fn: Some(dentry_migrate),
        checker_id: None,
    };
    DENTRY_TYPE_ID.set(register_tracked_type(template)
        .expect("Dentry descriptor registration failed"))
        .expect("Dentry type_id already set");
}

dentry_migrate follows the task_migrate field-copy template (Section 13.18): copy preserved fields, initialize added fields from schema defaults, return Err(MigrationError::Incompatible) on an irreconcilable retype. migration_fn = None (Extension-Array-only) is not acceptable: hot-field repacking is an anticipated Shadow-and-Migrate change. The initially registered function is an identity field-copy; each evolution payload ships its own replacement.

Allocation path (dentry_alloc):

/// Allocate and initialize a dentry for `name` under `parent`.
/// Process context; may sleep (the budget-exhaustion slow path waits for
/// an RCU grace period). Called on every dcache miss, by both positive
/// lookups and negative-dentry creation.
///
/// **Locking contract (Linux convention — `d_alloc()` in fs/dcache.c
/// takes `parent->d_lock` itself)**: callers hold NO dentry locks, in
/// particular NOT the parent's `d_lock`, and no spinlock of any level.
/// `dentry_alloc()` acquires `parent.d_lock` and the hash bucket lock
/// INTERNALLY, only around the final insertion step — strictly AFTER the
/// allocation and field initialization below. This ordering is what makes
/// the two sleep points legal: the `OutOfInstances` arm's
/// `dcache_lru_scan()` + `rcu_synchronize()` and `alloc_tracked`'s depot
/// refill (`TRACKED_REGISTRY_LOCK(135)`) both run with no lock held.
/// A caller that held `parent.d_lock` across `dentry_alloc()` would deadlock
/// on the internal acquisition — the contract is structural, not advisory.
fn dentry_alloc(parent: &Arc<Dentry>, name: &[u8]) -> Result<Arc<Dentry>, Errno> {
    if name.len() > DENTRY_MAX_NAME_LEN {
        return Err(Errno::ENAMETOOLONG);
    }
    let ptr: TrackedPtr<Dentry> = match alloc_tracked::<Dentry>() {
        Ok(p) => p,
        Err(AllocError::OutOfInstances) => {
            // Instance budget exhausted: the dcache has grown to its
            // configured share of tracked storage. Run a synchronous
            // targeted shrink (negative dentries first, then LRU tail),
            // then wait one grace period — freed slots only return to
            // the per-type free list after the RCU-deferred final Arc
            // drop (see the free path below) — then retry ONCE.
            // Cold path by construction: it fires only at budget
            // exhaustion, and each firing frees a batch.
            dcache_lru_scan(DCACHE_EMERGENCY_SCAN_BATCH); // default 1024
            rcu_synchronize();
            alloc_tracked::<Dentry>().map_err(|_| Errno::ENOMEM)?
        }
        Err(AllocError::OutOfStorage) => return Err(Errno::ENOMEM),
        // The buddy/slab-only variants are unreachable from the tracked
        // allocator but are listed explicitly (no wildcard) so a future
        // AllocError variant forces a compile error rather than a silent
        // ENOMEM.
        Err(AllocError::OutOfMemory)
        | Err(AllocError::CgroupLimit)
        | Err(AllocError::CacheDraining)
        | Err(AllocError::TooLarge)
        | Err(AllocError::WouldSleep) => return Err(Errno::ENOMEM),
    };
    // Initialize every field (tracked slots are not zeroed): d_name from
    // `name` (inline if <= DENTRY_INLINE_NAME_LEN), d_inode = None (all
    // dentries start negative; attaching the inode makes them positive), d_parent =
    // Arc::clone(parent), d_flags = 0, d_mount_refcount/d_refcount/
    // cached_perm/d_mount_seq = 0, d_sb = Arc::clone(&parent.d_sb),
    // d_ops = None (filesystem sets it during lookup), and self-linked
    // list nodes for d_hash/d_children/d_sibling/d_lru.
    //
    // SAFETY: ptr is a valid, exclusive TrackedPtr freshly returned by
    // alloc_tracked; ownership moves into the Arc. The last strong-ref
    // drop invokes free_tracked::<Dentry>().
    let dentry: Arc<Dentry> = unsafe { Arc::from_tracked(ptr) };
    // ... field initialization as above, then hash insertion under the
    // parent's d_lock + hash bucket lock ...
    Ok(dentry)
}

ENOMEM from dentry_alloc() propagates to the syscall exactly as a failed Linux d_alloc() (NULL return) does. The FMA framework (Section 20.1) records the OutOfInstances cause distinctly so operators can distinguish "raise umka.vfs.dcache_slots" from general memory pressure.

Free path and shrinker interaction: RCU-walk resolves paths through reference-less dentry pointers, so a dentry's tracked slot must never be reused while such a reader can still hold it. The kill path enforces this:

1. dentry_kill(dentry) (final Dentry::put() of an unhashed dentry, LRU eviction by the shrinker, or explicit cache invalidation): under the dentry's d_lock and the hash bucket lock, unhash from the dcache hash table (new RCU walkers can no longer find it), unlink from the LRU and from the parent's d_children.
2. Call d_ops.d_release() if set; drop the d_inode reference (for positive dentries — this is what makes inodes eligible for the inode LRU, which is why the dentry shrinker runs before the inode shrinker).
3. rcu_call(&dentry.d_rcu, dentry_free_rcu) — defer past a grace period.
4. dentry_free_rcu drops the dcache's owning Arc<Dentry> (the reference created by Arc::from_tracked in dentry_alloc()). If it is the last strong reference, the embedded TrackedPtr<Dentry> drops, invoking free_tracked::<Dentry>() — returning the slot to the per-type free list AND removing the instance from the Nucleus live registry. If other holders remain (an open File, a MountDentry, a child's d_parent back-reference), the slot is freed at their final drop — safe, because they hold strong references. There is no separate slab-free path.

Invariant: free_tracked::<Dentry>() never runs earlier than one full RCU grace period after the dentry was unhashed. Reference-less readers can only discover dentries through the hash table and parent/child links, all severed in step 1, so no such reader can hold the pointer when the slot is reused.

Invalidating one cached name: the "explicit cache invalidation" entry into step 1 above is this helper. It is the only sanctioned way to drop a cached name: the d_lock / bucket-lock ordering stays encapsulated in the dcache, and no caller ever names those locks.

/// Remove the cached child entry (positive or negative) of `parent` named
/// `name` from the dentry cache, if present.
///
/// Looks up the (parent dentry, name) key in the dcache hash — the way the
/// dcache is actually indexed — and, on a hit, unhashes and retires the
/// dentry through the `dentry_kill()` sequence (steps 1-4 of "Free path and
/// shrinker interaction" above), preserving the RCU grace-period reuse
/// invariant.
///
/// A child that is currently a mountpoint (`DCACHE_MOUNTED`) is left in
/// place: detaching mounts is a umount concern, not a dcache concern.
///
/// Idempotent — a no-op when nothing is cached under that name.
///
/// Callers are stacking-filesystem events that change what a name resolves
/// to: whiteout removal, and upper-layer creation over a cached negative
/// entry ([Section 14.8](#overlayfs-union-filesystem-for-containers)). Copy-up is NOT
/// one of them — it updates the overlay inode in place, so the cached
/// dentry stays correct.
///
/// **Caller discipline** (normative): the callers are the VFS-side flows
/// that already hold the parent dentry from path resolution — the create
/// dispatch for creation-over-negative, the unlink/rename dispatch for
/// whiteout removal. Filesystem `InodeId`-world code never resolves a
/// dentry to call this: dentries do not cross a domain boundary ("Tier 1
/// drivers never receive raw `Dentry` pointers"), so a filesystem-internal
/// call site would be a hidden same-domain assumption.
///
/// After the call, the next lookup of that name runs the filesystem's
/// `lookup()`.
pub fn dentry_invalidate_child(parent: &Arc<Dentry>, name: &OsStr);

The shrinker itself is registered alongside the inode cache shrinker (see the Inode section above) and its scan path is dentry_kill in a loop:

/// Emergency dcache scan batch: how many dentries `dentry_alloc()` reclaims in one
/// synchronous shrink when the dcache instance budget is exhausted. Default
/// 1024 — large enough to amortise the `rcu_synchronize()` that follows,
/// small enough to bound the stall. Evolvable tunable (policy, not Nucleus).
pub const DCACHE_EMERGENCY_SCAN_BATCH: u64 = 1024;

/// Reclaim up to `nr_to_scan` unused dentries from the dcache LRU; returns
/// the number actually freed (the `Shrinker::scan_objects` callback for
/// `DENTRY_CACHE_SHRINKER`, and the emergency shrink invoked by `dentry_alloc()`).
///
/// Walks the dcache LRU tail under `lru_lock`, PREFERRING negative dentries
/// (they pin no inode, so freeing them is pure gain), and routes every
/// eviction through `dentry_kill()` — which unhashes, unlinks from the LRU,
/// drops the inode reference, and RCU-defers the final `Arc<Dentry>` drop so
/// the tracked slot returns to storage only after a grace period. Only
/// dentries with `d_refcount == 0` are eligible; in-use dentries are skipped.
///
/// The ordering heuristic and batch shape are Evolvable policy; the
/// tracked-storage lifecycle it drives is the Nucleus-owned mechanism.
/// Warm/cold path (reclaim / budget exhaustion), never per-syscall.
fn dcache_lru_scan(nr_to_scan: u64) -> u64;

/// `count_objects` callback for `DENTRY_CACHE_SHRINKER`: the number of
/// reclaimable (unused, `d_refcount == 0`) dentries currently on the dcache
/// LRU — exactly the set `dcache_lru_scan` walks. Read from the dcache LRU's
/// atomic length counter without taking `lru_lock` (fast; may be momentarily
/// stale, which only perturbs proportional pressure distribution).
fn dcache_lru_count() -> u64;

/// Dentry cache shrinker. Runs BEFORE the inode cache shrinker (evicting
/// dentries drops inode refcounts, making inodes LRU-eligible).
/// `scan_objects` walks the dcache LRU, preferring negative dentries,
/// and routes every eviction through dentry_kill — i.e., every reclaimed
/// dentry returns its slot to tracked storage via free_tracked after the
/// grace period. The shrinker is thus both the memory-pressure valve AND
/// the enforcement mechanism for the dcache instance budget.
pub static DENTRY_CACHE_SHRINKER: Shrinker = Shrinker {
    count_objects: dcache_lru_count,   // LRU length (fast — no lock)
    scan_objects: dcache_lru_scan,     // dentry_kill() over up to nr_to_scan LRU entries
    seeks: DEFAULT_SEEKS,
    flags: 0,
};

The eviction policy (LRU ordering, negative-dentry preference, batch size, DCACHE_EMERGENCY_SCAN_BATCH) is Evolvable, per the classification table in this section; the tracked-storage lifecycle above (alloc, unhash, RCU deferral, free_tracked) is the Nucleus-owned mechanism and is identical for positive and negative dentries.

14.1.5.2 Dentry Slot Reclamation on VFS Crash

The free path above assumes the dcache management structures are alive: an eviction unhashes, then RCU-defers the drop of the dcache's owning Arc<Dentry>. A crash of the domain hosting the umka-vfs instance breaks that assumption for every dentry at once — the hash table and LRU linkage die with the domain, and the crashed instance's owning references are never dropped. Without an explicit reclaim, every one of those slots would stay "live" in the Nucleus tracked registry forever: a few crashes on a loaded machine would permanently exhaust the dcache instance budget (umka.vfs.dcache_slots), after which dentry_alloc() fails with ENOMEM on every cache miss. That is a 50-year-uptime resource leak; this section is the reclaim that prevents it.

Two crash shapes:

1. A filesystem provider crashes but the domain hosting umka-vfs survives (VFS and the FS driver bound to different domains). The dcache is structurally intact. No registry walk is needed or permitted: the surviving VFS prunes the recovering superblock's dentries through the NORMAL kill path (dentry subtree walk from sb.s_root — every eviction routes through dentry_kill() and returns its slot via free_tracked after the grace period). This runs as part of the superblock's recovery, before Step U14's reload, so post-replay lookups repopulate from disk rather than serving pre-crash cached state.

2. The crashed domain hosted umka-vfs itself (the default shared VFS+FS domain, or any grouping that includes the VFS module). The dcache structure is gone; recovery reclaims slots through the Nucleus registry:

/// Nucleus tracked-allocator crash-reclaim primitive.
///
/// CANONICAL HOME: [Section 13.18](13-device-classes.md#live-kernel-evolution--generic-tracked-allocator) —
/// declared here because the VFS crash-recovery protocol is its first
/// consumer; the generic tracked-allocator section owns the definitive
/// specification (generic, type-parametric, zero type-specific knowledge —
/// the same registry walk the migration engine uses).
///
/// For every instance of `T` in the live registry, drop ONE strong
/// reference from the slot's side header — reconstructing the owning
/// reference that a crashed manager (e.g. the dcache) held and can no
/// longer drop. Slots whose strong count reaches zero are freed exactly as
/// a normal final `Arc` drop: the payload's `Drop` runs (cascading
/// `Arc` fields — for `Dentry`: `d_parent`, `d_inode`, `d_sb`), then
/// `free_tracked::<T>()` returns the slot and removes it from the live
/// registry. Slots still referenced by other holders are NOT freed — they
/// free at those holders' final drop.
///
/// Returns the number of slots freed (including cascade frees).
///
/// # Safety
/// The caller must guarantee that (1) the manager that held the owning
/// references is dead and quiesced — no thread can concurrently run the
/// normal free path for these instances — and (2) every free callback the
/// dead manager queued before crashing (`rcu_call`) has already executed,
/// so no queued callback will drop the same owning reference again.
pub unsafe fn tracked_reclaim_owner_refs<T: TrackedType>() -> u64;

The reclaim step runs once per VFS-module crash, in the recovery worker's process context between Step U13 (driver unload) and Step U14 (reload) of the unified sequence (Section 14.3):

1. Quiesce preconditions (already established by the sequence): U5/U5a guarantee no old producer or consumer thread survives; U13 has freed the old domain's memory and completed the module-unload RCU-callback drain (rcu_barrier() — all dentry_free_rcu callbacks queued before the crash have executed, so their owning references are already dropped and their slots already left the live registry; the walk cannot double-drop them). One additional rcu_synchronize() covers any Core-side reference-less reader — none can exist, since RCU-walk path resolution ran only inside the dead domain, but the grace period keeps the slot reuse invariant unconditional.
2. Walk and drop: unsafe { tracked_reclaim_owner_refs::<Dentry>() }. The unpinned bulk of the dcache (typically >99% of instances) reaches strong count zero and is freed; parent chains cascade through payload Drop (a freed child drops its d_parent reference, freeing ancestors bottom-up in the same pass).
3. Pinned survivors: instances still held by Core-side references — SuperBlock.s_root, Mount mountpoint/root dentries, and OpenFile.dentry pins — are not freed by the pass. They are orphans (unreachable by lookup — the new instance's dcache never learns about them) but remain refcount-correct and free at their holders' final drop (close(2), umount). Bounded by open fds + mounts, not by dcache size.
4. FMA accounting: the pass emits the freed count, and the post-recovery budget headroom, as an FMA event — a recovery that reclaims suspiciously few slots (< the pre-crash live count minus pins) indicates registry inconsistency and flags the recovery degraded.

Aliasing note: an OpenFile.dentry orphan may coexist with a fresh post-recovery dentry for the same (parent, name) — the same benign aliasing Linux exhibits for unhashed/disconnected dentries: fd-based I/O uses the inode, not the dentry; rendering the orphan's path keeps reporting the pre-crash path (matching Linux behavior for deleted/renamed files).

Trust note: slot ENUMERATION and refcounts come from Nucleus-owned metadata (allocation registry + side headers), which the driver domain cannot write — the walk itself trusts no in-domain data. Payload Drop of freed dentries does read payload fields (d_parent/d_inode/d_sb Arcs) that were domain-writable; this is the same accepted co-tenancy exposure as the rest of crash recovery (see "Domain grouping limitation" in the intro) — Rust memory safety plus the U9 integrity check bound it, and a corrupted payload pointer surfaces as a fault in the recovery worker, which aborts the recovery per Reload Failure Handling rather than corrupting Core silently.

14.1.6 Path Resolution

Path resolution walks the dentry cache component by component. For example, /usr/lib/libfoo.so resolves as: root dentry -> lookup("usr") -> lookup("lib") -> lookup("libfoo.so").

RCU path walk (fast path): The entire resolution is attempted under an RCU read-side critical section. No dentry reference counts are taken, no locks are acquired. If every component is in the dentry cache and no concurrent renames or unmounts are in progress, the entire path resolves with zero atomic operations.

Ref-walk fallback (slow path): If any component is not cached, or if a concurrent mount/rename is detected (via sequence counters), the RCU walk aborts and restarts in ref-walk mode. Ref-walk takes dentry reference counts and inode locks as needed. This two-phase approach is identical to Linux's rcu-walk (LOOKUP_RCU set) -> ref-walk (LOOKUP_RCU cleared via Linux fs/namei.c try_to_unlazy()) fallback; Linux has no LOOKUP_LOCKED flag.

Mount point traversal: When a dentry is flagged as a mount point, resolution crosses into the mounted filesystem's root dentry. The mount table is consulted via RCU lookup (no lock) in the fast path.

Symlink resolution: The VFS follows up to 40 nested symlinks before returning ELOOP. This matches the Linux limit and prevents infinite symlink loops. The budget is a hard count (MAX_SYMLINK_FOLLOWS), not a visited-set — a symlink cycle terminates by exhausting the count. The follow step is the operation-time enforcement site of the vfs_path_walk_safety invariant checker (Section 14.1): before each follow it calls op_check on the walk state's symlink_follows counter and maps a budget-exceeded result to ELOOP (walk-core excerpt below).

Symlink namespace semantics: Symlink targets are always resolved relative to the current task's mount namespace, not the symlink inode's namespace. Absolute symlink targets (/foo/bar) start from task.fs.root (the task's chroot/pivot_root). Relative symlink targets are resolved from the symlink's parent directory. The AT_SYMLINK_NOFOLLOW flag prevents resolution entirely (returns the symlink inode). This matches Linux behavior and ensures that symlinks do not become cross-namespace escape vectors — a symlink created in one mount namespace cannot force resolution through a different namespace's mount tree.

14.1.6.1 Walk-core structure and provider parameterization

The component walk is one algorithm shared by both modes (RCU-walk and ref-walk); it is parameterized over a component-lookup provider so the same code backs both the live dentry cache and the evolution harness's synthetic fixture. The provider is a compile-time generic — walk_components::<P> is monomorphized at each call site (static dispatch, zero-cost) — NOT a runtime replaceable-trait object: the hot path never dispatches through a vtable, per the hot-path rule (CLAUDE.md §Collection and Allocation Policy). The live syscall path binds DcacheWalkProvider (defined below): path_lookup runs both the RCU-walk and ref-walk phases as walk_components::<DcacheWalkProvider> (the instantiation is in Section 14.1). The vfs_path_walk_safety checker binds FixtureWalkProvider (Section 14.1) over the SAME monomorphized walk_components core, so the checker exercises the exact walk the syscall path runs, not a fixture-only copy.

/// Per-lookup mutable walk bookkeeping, provider-independent. The current node
/// is threaded through `walk_components` as `P::Node`; this carries only the
/// counters the safety invariants constrain. Kernel-internal, not KABI.
pub struct WalkState {
    /// Symlinks followed so far in this resolution. Checked against
    /// `MAX_SYMLINK_FOLLOWS` at every follow (INV-1); the `op_check` subject.
    pub symlink_follows: u32,
    /// Bytes consumed across the path and every followed symlink target,
    /// bounded by `PATH_MAX` (`ENAMETOOLONG`).
    pub bytes_walked: u32,
}

impl WalkState {
    pub fn new() -> Self { WalkState { symlink_follows: 0, bytes_walked: 0 } }
}

/// Kind of a resolved component, reported by the provider so the walk core can
/// apply symlink / mount / terminal handling without knowing the backing store.
pub enum WalkNodeKind { Dir, File, Symlink, MountPoint }

/// One resolved component step: the child node, its kind, and whether the RCU
/// seqcount stayed valid between lookup and validation. `seq_valid == false`
/// forces the RCU-walk to retry or drop to ref-walk (INV-3); it is always true
/// on the locked ref-walk path. Kernel-internal — never crosses a KABI boundary.
pub struct WalkStep<N> {
    pub node: N,
    pub kind: WalkNodeKind,
    pub seq_valid: bool,
}

/// Component-lookup seam the walk core is generic over. The live path binds a
/// dcache/mount-hash implementation; the evolution harness binds
/// `FixtureWalkProvider`
/// ([Section 14.1](#virtual-filesystem-layer--nucleus-invariant-checker-path-walk-safety)).
pub trait WalkProvider {
    /// Provider-native node handle (a live dentry reference on the syscall
    /// path; a `FixtureNodeId` in the harness).
    type Node: Copy;

    /// The provider's root node — binds absolute paths and `IN_ROOT`.
    fn root(&self) -> Self::Node;

    /// Resolve one component `name` within directory `dir` to its child node
    /// and kind, revalidating the RCU seqcount. `Err(Errno::ENOENT)` if absent.
    fn lookup(&self, dir: Self::Node, name: &[u8])
        -> Result<WalkStep<Self::Node>, Errno>;

    /// Read a symlink node's target bytes into `out`, returning the length.
    /// `Err(Errno::EINVAL)` if `node` is not a symlink.
    fn read_link(&self, node: Self::Node, out: &mut [u8]) -> Result<usize, Errno>;
}

/// The shared walk core. Monomorphized over `P`: the live path specializes it
/// over the dcache provider, the harness over `FixtureWalkProvider`. Splits
/// `path` into components, resolves each via `provider.lookup`, follows
/// intermediate symlinks (and the terminal one when `FOLLOW` is set) under the
/// symlink-follow budget, crosses mount points, and enforces
/// `BENEATH`/`NO_XDEV`/`IN_ROOT`/`NO_SYMLINKS`. Returns the resolved node.
pub fn walk_components<P: WalkProvider>(
    provider: &P,
    walk: &mut WalkState,
    start: P::Node,
    path: &[u8],
    flags: LookupFlags,
) -> Result<P::Node, Errno>

At the shared symlink-follow step — reached by both RCU-walk and ref-walk whenever a resolved component is a symlink to be followed — the walk enforces the budget through the checker's op_check predicate before consuming another follow:

// Inside walk_components::<P>(), before following a symlink component:
if !op_check(
    PATH_WALK_CHECKER_ID.get().copied().expect("walk checker not registered"),
    &walk.symlink_follows as *const u32 as *const u8,
    core::ptr::null(),
) {
    return Err(Errno::ELOOP);
}
walk.symlink_follows += 1;

rcu_walk_step is the single-component entry the RCU-walk uses (and the checker drives for INV-3): it performs one provider.lookup plus its seqcount revalidation and returns Err(Errno::EAGAIN) when the component's seqcount was bumped between lookup and validation (WalkStep::seq_valid == false), signalling the caller to retry or drop to ref-walk — it never returns Ok with a stale node.

/// One RCU-walk component step: lookup + seqcount revalidation.
/// `Err(Errno::EAGAIN)` means the seqcount moved (retry / ref-walk); `Ok`
/// carries a node validated under a stable seqcount.
pub fn rcu_walk_step<P: WalkProvider>(
    provider: &P,
    at: P::Node,
    name: &[u8],
) -> Result<P::Node, Errno>

14.1.6.1.1 Live binding — the syscall path's DcacheWalkProvider

DcacheWalkProvider is the syscall path's WalkProvider — the concrete companion to the harness's FixtureWalkProvider. It is a thin adapter: lookup composes the existing dentry-cache hash probe with the existing mount-hash crossing, and read_link delegates to the filesystem symlink op — the provider introduces no lookup of its own. path_lookup instantiates walk_components::<DcacheWalkProvider> for both walk phases (Section 14.1), monomorphized identically to the harness's walk_components::<FixtureWalkProvider>, so the vfs_path_walk_safety checker validates the very core the syscall path runs. Static dispatch throughout — no dyn, no vtable on the walk hot path.

/// The live walk's node currency: a `Copy` (mount, dentry) raw-pointer pair.
/// It threads through `walk_components` exactly as `FixtureNodeId` does on the
/// harness side — the reason the `WalkProvider::Node` associated type exists.
/// The node owns NO refcount (hence `Copy`): its targets are kept live by the
/// provider's mode discipline — the RCU read-side section that brackets the
/// entire RCU-walk phase, or the reference ref-walk pins on the current
/// component — never by the node itself. Kernel-internal, not KABI.
#[derive(Copy, Clone)]
pub struct LiveWalkNode {
    /// The mount whose filesystem `dentry` belongs to. Carried for
    /// mount-crossing and `NO_XDEV`/`BENEATH` confinement.
    pub mnt: NonNull<Mount>,
    /// The dentry reached at this step.
    pub dentry: NonNull<Dentry>,
}

/// RCU-walk vs ref-walk. Selects the per-component revalidation discipline the
/// provider applies: `Rcu` is seqcount-validated and may report
/// `seq_valid == false` to force retry / ref-walk; `Ref` is reference-pinned
/// and every step is `seq_valid`.
#[derive(Copy, Clone, PartialEq, Eq)]
pub enum WalkMode {
    Rcu,
    Ref,
}

/// Live `WalkProvider`: the dentry-cache / mount-hash binding of the shared
/// walk core. Holds no owned state beyond the resolution scope — the caller
/// brackets the RCU-walk phase in one `rcu_read_lock()` (RCU mode) or pins the
/// current component (ref mode); `mode` tells each `lookup` which discipline to
/// apply. Kernel-internal, not KABI.
pub struct DcacheWalkProvider<'w> {
    /// Mount namespace scoping mount-crossing lookups
    /// (`mnt_ns.hash_table.lookup`, [Section 14.6](#mount-tree-data-structures-and-operations)).
    mnt_ns: &'w MountNamespace,
    /// Chroot / `IN_ROOT` boundary node (`..` clamping, `BENEATH`). Returned by
    /// `WalkProvider::root`; the core enforces confinement against it.
    root: LiveWalkNode,
    /// RCU-walk or ref-walk (see `WalkMode`).
    mode: WalkMode,
}

impl<'w> DcacheWalkProvider<'w> {
    /// RCU-walk provider. Caller MUST hold an `rcu_read_lock()` spanning the
    /// whole `walk_components` call — the node pointers borrow dentries that
    /// are valid only inside that section.
    pub fn rcu_walk(mnt_ns: &'w MountNamespace, root: LiveWalkNode) -> Self {
        Self { mnt_ns, root, mode: WalkMode::Rcu }
    }

    /// Ref-walk provider. Nodes are reference-pinned as the walk advances (the
    /// Linux `nd->path` discipline); every step is `seq_valid`.
    pub fn ref_walk(mnt_ns: &'w MountNamespace, root: LiveWalkNode) -> Self {
        Self { mnt_ns, root, mode: WalkMode::Ref }
    }
}

impl<'w> WalkProvider for DcacheWalkProvider<'w> {
    type Node = LiveWalkNode;

    fn root(&self) -> LiveWalkNode {
        self.root
    }

    fn lookup(&self, dir: LiveWalkNode, name: &[u8])
        -> Result<WalkStep<LiveWalkNode>, Errno>
    {
        // Inode type bits (Linux `include/uapi/linux/stat.h`) — the same set
        // `open_and_install` uses
        // ([Section 14.1](#virtual-filesystem-layer--path-lookup-entry-point)).
        const S_IFMT: u32 = 0o170000;
        const S_IFDIR: u32 = 0o040000;
        const S_IFLNK: u32 = 0o120000;

        // SAFETY: `dir` was produced by this provider's `root()`/`lookup()`;
        // its dentry is kept live by the active walk discipline (the RCU
        // section in `Rcu` mode, the pinned component in `Ref` mode) — see
        // `WalkMode`.
        let parent = unsafe { dir.dentry.as_ref() };

        // Sample the mount-crossing seqcount BEFORE the probe (the
        // `d_mount_seq` reader protocol on `Dentry`: Acquire sample -> lookup
        // -> Acquire resample -> retry on change).
        let seq0 = parent.d_mount_seq.load(Ordering::Acquire);

        // The EXISTING dentry-cache probe + mount-hash crossing, exposed
        // through the provider surface — no new lookup is defined here.
        let resolved = dcache_walk_lookup(self.mnt_ns, dir, name, self.mode)?;

        // (INV-3) In RCU mode a bumped `d_mount_seq` means a mount changed
        // under this step: report `seq_valid == false` so `rcu_walk_step` / the
        // core retries or drops to ref-walk. A racing rename is caught
        // separately — the dcache probe's own RCU revalidation surfaces it as
        // `EAGAIN` from `dcache_walk_lookup`. Ref-walk holds references and is
        // always stable.
        let seq_valid = match self.mode {
            WalkMode::Ref => true,
            WalkMode::Rcu => {
                parent.d_mount_seq.load(Ordering::Acquire) == seq0
            }
        };

        // Classify the resolved child. A negative dentry (no inode) is a cached
        // "does not exist" — `ENOENT`, matching the fixture provider.
        // SAFETY: as for `parent` — `resolved` is provider-produced.
        let child = unsafe { resolved.dentry.as_ref() };
        let guard = rcu_read_lock();
        let i_fmt = match child.d_inode.read(&guard) {
            // `RcuCell::read` yields `&Option<Arc<Inode>>`: a single `Option`
            // (positive dentry -> `Some(inode)`, negative -> `None`). Match
            // ergonomics binds `inode: &Arc<Inode>` — no `Arc` is moved out.
            Some(inode) => inode.i_mode.load(Ordering::Relaxed) & S_IFMT,
            None => {
                drop(guard);
                return Err(Errno::ENOENT);
            }
        };
        drop(guard);
        let mounted = (child.d_flags.load(Ordering::Acquire)
            & DcacheFlags::DCACHE_MOUNTED.bits()) != 0;
        let kind = if mounted {
            WalkNodeKind::MountPoint
        } else {
            match i_fmt {
                S_IFDIR => WalkNodeKind::Dir,
                S_IFLNK => WalkNodeKind::Symlink,
                _ => WalkNodeKind::File,
            }
        };

        Ok(WalkStep { node: resolved, kind, seq_valid })
    }

    fn read_link(&self, node: LiveWalkNode, out: &mut [u8]) -> Result<usize, Errno> {
        const S_IFMT: u32 = 0o170000;
        const S_IFLNK: u32 = 0o120000;

        // SAFETY: `node` is provider-produced and kept live by the active walk
        // discipline (see `WalkMode`).
        let dentry = unsafe { node.dentry.as_ref() };
        let guard = rcu_read_lock();
        let inode = match dentry.d_inode.read(&guard) {
            // `RcuCell::read` yields `&Option<Arc<Inode>>` (single `Option`);
            // `i: &Arc<Inode>`, cloned to own a reference past the RCU section.
            Some(i) => Arc::clone(i),
            None => {
                drop(guard);
                return Err(Errno::EINVAL); // negative dentry: not a symlink
            }
        };
        drop(guard);
        if (inode.i_mode.load(Ordering::Relaxed) & S_IFMT) != S_IFLNK {
            return Err(Errno::EINVAL); // trait contract: EINVAL if not a symlink
        }
        // Read the target through the filesystem symlink op. A target-read
        // failure aborts the walk as an I/O error (the walk operates in `Errno`
        // currency; `InodeOps` reports `KernelError`, mapped here via the
        // codebase's `map_err(|_| Errno::…)` boundary idiom).
        inode
            .i_op
            .readlink(InodeId(inode.i_ino), out)
            .map_err(|_| Errno::EIO)
    }
}

lookup builds on one seam — the dentry-cache probe, exposed under a stable name. It is NOT a new lookup: it composes the two lookups the spec already defines.

/// The dentry-cache component probe the live `WalkProvider` adapts — NOT a new
/// lookup. It composes the two lookups the spec already defines: the dcache
/// hash probe keyed by `(parent, name-hash)`
/// ([Section 14.1](#virtual-filesystem-layer--dentry-cache)) and the mount-hash crossing
/// `mnt_ns.hash_table.lookup(...)`
/// ([Section 14.6](#mount-tree-data-structures-and-operations)). Given `parent` (already
/// at a directory) it (1) follows any mount stacked at `parent` to the
/// mounted-filesystem root, then (2) resolves `name` in that directory via the
/// dcache hash — populating the child from the filesystem `InodeOps::lookup`
/// slow path on a ref-walk miss. `mode` picks the discipline: `Rcu` borrows
/// dentries under the caller's RCU section and returns `Err(Errno::EAGAIN)` on
/// a dcache miss (drop to ref-walk); `Ref` pins references and populates
/// misses. The resolved child's own `DCACHE_MOUNTED` is left for the caller to
/// report as `WalkNodeKind::MountPoint` — the core crosses it on the next
/// step, exactly as the harness fixture models a mount.
fn dcache_walk_lookup(
    mnt_ns: &MountNamespace,
    parent: LiveWalkNode,
    name: &[u8],
    mode: WalkMode,
) -> Result<LiveWalkNode, Errno>

Capability checks: Traverse permission is checked at each path component, but not via an inter-domain ring call on every component. Instead, the dentry cache stores a cached_perm: AtomicU64 field: the traverse grants validated by the last authoritative check, tagged with the SUBJECT'S permission identity. During RCU-walk, the VFS reads cached_perm from the dentry (one atomic load, no ring call) and compares its tag against the caller's tag and its grant bits against the requested access. On a hit (common case — the same subject re-walking the same path), no domain crossing occurs.

Permission cache encoding (cached_perm: AtomicU64; value 0 = empty/invalidated): - Bits [63:12]: 52-bit subject tag (see below). - Bits [11:4]: Reserved (zero). - Bit [3]: VALID (always 1 in a populated entry — guarantees a populated entry is never all-zero, so 0 is unambiguously "empty"). - Bits [2:0]: Union of rwx access modes the authoritative slow path has VALIDATED for this subject on this component. Positive-only: a bit is set only after the slow path granted that access.

Subject tag: tag52 = (cred.perm_tag ^ (policy_gen * PERM_TAG_SPREAD)) & ((1 << 52) - 1) where: - cred.perm_tag: u64 is the credential's permission-identity hash, computed ONCE at TaskCredential construction (Section 9.9): SipHasher13 keyed with the boot-random 128-bit HASH_SEED over the permission-relevant identity — (fsuid, fsgid, supplementary_groups contents, the DAC-relevant effective capabilities CAP_DAC_OVERRIDE and CAP_DAC_READ_SEARCH, user_ns.ns_id). Credentials are immutable (Section 9.9), so the tag never changes for a live credential object; reading it is one dependent load off the already-resolved cred pointer. - policy_gen is LSM_REGISTRY.policy_generation (Section 9.8) — the same counter that invalidates cached CapValidationToken LSM decisions. PERM_TAG_SPREAD is a fixed odd 64-bit constant (splitmix-style multiplier) diffusing the low-entropy counter across the tag bits: one multiply + XOR per component check (~3-5 cycles).

Hit rule: entry != 0 && entry[63:12] == tag52 && requested ⊆ entry[2:0]. Anything else — empty entry, tag mismatch, or a requested mode not yet validated — is a miss: the VFS performs the full authoritative check via the inter-domain ring and, on ALLOW, stores (tag52 << 12) | VALID | granted_modes (single Release store; concurrent updaters are last-writer-wins). Denials are never cached — a deny verdict always comes from the authoritative path, so the cache can only accelerate allows, never manufacture or serve a denial.

Why the tag closes the subject-side staleness hole: credentials are immutable copy-on-write — setuid(), setgid(), setgroups(), execve() credential transforms, setns() into another user namespace, and every other subject-side transition install a NEW TaskCredential via install_credentials(). A permission-relevant change (different fsuid/fsgid/groups/DAC-caps/userns) produces a different perm_tag, so every entry cached for the OLD identity misses for the new one — no invalidation broadcast, correct by construction. (The previous design tagged entries with a 16-bit UID hash only: a task that dropped a supplementary group, or transitioned via setgid-execve, kept hitting its stale ALLOW entries because its UID — hence the tag — was unchanged. That was a real privilege-retention hole, not a theoretical one.) Conversely, a credential re-commit that does NOT change the permission identity (a plain execve) hashes to the SAME tag, so the cache stays hot across exec-heavy workloads. Two unrelated credential objects with identical permission identity (two logins of the same user) also share entries — the tag is content-derived, not object-derived. LSM policy reloads fold in via policy_gen: a reload changes every tag, orphaning all cached decisions at once (repopulated on next access), mirroring the CapValidationToken policy_gen invalidation rule.

Object-side invalidation (unchanged): chmod(), chown(), ACL changes, and capability revocation store 0 to cached_perm on the affected dentries. These are the operations that change what a given subject may do to the OBJECT; subject-side changes need no invalidation per the tag argument above.

Mount-dependent id mappings — cache bypassed: when the traversal mount translates file ownership through the mount's user namespace with a non-identity mapping (Section 17.1), the permission result depends on the (mount, subject) pair, and the per-dentry tag carries only the subject. Dentries are shared across mounts of one superblock, so a grant computed under one mount's mapping must not be served under another. On such mounts the cache is neither consulted nor populated — every component check takes the authoritative path. Identity-mapped mounts (the overwhelmingly common case) are unaffected.

Collision honesty — what the encoding can and cannot carry: a tag collision (two different permission identities producing equal tag52) is UNDETECTABLE at check time by construction — a colliding subject would be served the cached subject's validated grants. The design therefore makes collisions unproducible rather than pretending to detect them: - Accidental: 2⁻⁵² per pair of distinct identities per dentry — negligible. - Adversarial: the hash is keyed with the boot-random secret HASH_SEED, so collisions cannot be computed offline; an online search must MINT credentials (unprivileged user namespaces allow this) and probe a populated entry, expecting ~2⁵¹ attempts — and each missing probe runs the slow path and OVERWRITES the entry with the prober's own tag, destroying the target. This is infeasible. (The former 16-bit UID-hash tag failed exactly this test: ~2¹⁵ mintable identities to a collision, minutes of work — and the old text's claim that "a collision always causes a cache miss" was self-contradictory, since an undetectable collision is by definition a hit.) - The 52-bit field cannot carry: mount-dependent mappings (hence the bypass above), negative verdicts (denials always authoritative), or per-LSM state finer than the policy generation (a policy reload invalidates everything).

Multi-subject ping-pong: On shared directory trees traversed by multiple distinct permission identities concurrently, the single-entry cache ping-pongs (alternating misses as subjects overwrite each other's entry). This is acceptable — identical-identity access patterns dominate, value-equal credentials share entries by content hash, and each miss costs one domain crossing, identical to the no-cache case. A multi-entry cache was considered and rejected: each additional entry adds 8 bytes to every dentry, multiplied by millions of cached dentries.

This design is correct because: 1. A hit requires the subject tag to match, and the tag identifies the permission-relevant credential content through an unforgeable keyed hash (collision analysis above) crossed with the LSM policy generation. 2. Grants are positive-only and object-side invalidation clears entries on every operation that changes the object's permission state, so a served grant is one the authoritative path issued for this subject-identity on this object under the current LSM policy. 3. Only the slow-path inter-domain ring call is authoritative. umka-vfs cannot grant access that Core's capability tables do not authorize.

Only on a cache miss (first access, different subject identity, mode not yet validated, or invalidated entry) does the VFS call Core via the inter-domain ring to perform a full capability check and update the dentry's cached entry. This amortized design preserves the security guarantee (umka-vfs cannot bypass capability checks — it has no access to capability tables, per Section 11.2 and Section 11.3) while keeping the hot-path overhead to a single atomic load plus a register compare per component, comparable to Linux's inode->i_mode check.

14.1.6.2 MountDentry — VFS Location Pair

A MountDentry is the fundamental VFS location type: a (mount, dentry) pair that uniquely identifies a point in the mount tree. It is the result of every path resolution and the primary reference type passed between VFS operations.

/// A reference to a location in the mount tree: the specific mount and the
/// dentry within that mount's filesystem. Two dentries with the same inode
/// in different mounts are different `MountDentry` values.
///
/// `MountDentry` holds Arc references to both the `Mount` and the `Dentry`,
/// keeping both alive for the duration of the reference. Dropping a
/// `MountDentry` decrements both refcounts.
pub struct MountDentry {
    /// The mount containing this dentry.
    pub mnt: Arc<Mount>,
    /// The dentry within the mount's filesystem.
    pub dentry: Arc<Dentry>,
}

/// Resolve an open file descriptor to its `MountDentry`.
///
/// Used by `open_by_handle_at()`, `fstatat(AT_EMPTY_PATH)`, and io_uring
/// `AT_FDCWD` resolution. The returned `MountDentry` identifies the mount
/// and dentry that the file descriptor was opened on.
///
/// # Errors
/// - `EBADF`: `fd` is not a valid open file descriptor.
/// - `ENOENT`: The file descriptor's dentry has been unlinked (deleted)
///   and `FMODE_PATH` is not set.
pub fn fd_to_mount_dentry(fd: i32) -> Result<MountDentry, Errno> {
    let file = fget(fd)?;
    // `OpenFile.mount` is an RcuCell (rebound once after a VFS-module
    // crash — see the field doc); load it under an RCU read-side guard.
    let guard = rcu_read_lock();
    let mnt = Arc::clone(&*file.mount.read(&guard));
    drop(guard);
    Ok(MountDentry {
        mnt,
        dentry: Arc::clone(&file.dentry),
    })
}

14.1.6.3 File Descriptor Lookup — Lockless Fast Paths

fd → OpenFile resolution runs on EVERY read/write/fstat/ioctl — it is one of the hottest paths in the kernel, and it takes no lock. The FdTable's fds XArray (Section 8.1) is read under RCU exactly like the page cache and the PID table; the fd-table lock (FdTable.inner) is a writer-side lock only (fd allocation, close, dup2, the exec O_CLOEXEC sweep, cloexec bitmap updates). Two invariants make the lockless readers sound:

• INV-FD1 (slot-clear locality): an fd-table slot is cleared only (a) from a task that has the table installed as its files (close(2), dup2(2), exec, exit all run on an owning task), or (b) at table teardown, after every owning task has exited. Kernel-held descriptors comply because no kernel-side closer exists: the exit-cleanup action list (Section 8.1) holds objects (Arc<EventFd>), not descriptors, so it never clears a slot. Any future kernel-side closer — including any consumer of OwnedFd — MUST either run on the owning task or hold a strong Arc<FdTable> for its whole lifetime (which makes the table observably shared, forcing every fdget() onto the owned path).
• INV-FD2 (deferred release): every path that clears a slot releases the table's owning Arc<OpenFile> through rcu_call — never a direct drop. The strong count therefore cannot reach zero before every RCU read-side section that could have loaded the slot has ended. This is the same pattern that protects the dcache's owning dentry references ("Dentry Allocation — Nucleus Tracked Storage" above). FileOps::release() consequently runs at most one grace period after the last close(2) — asynchronous release is already the Linux contract (final file-reference release defers to task work).

/// Look up `fd` in the calling task's fd table and return the backing open
/// file description with its refcount incremented (`Arc::clone`).
///
/// The returned `Arc<OpenFile>` keeps the description alive independently of
/// the fd table, so a concurrent `close(fd)` cannot free it under the caller.
/// The caller drops the `Arc` when done. Position-aware read/write paths use
/// `fdget()` + `fdget_pos()` instead; `fget()` is the plain (offset-supplied
/// / metadata / long-lived reference) lookup. `RawFd` is the Linux `int` fd
/// type.
///
/// **Lockless**: the lookup runs entirely inside one RCU read-side
/// section — no fd-table spinlock, no per-file lock:
///
/// 1. `rcu_read_lock()`.
/// 2. RCU-load the slot: `files.fds.load(fd)` (XArray reads are RCU-safe;
///    slot writers hold `FdTable.inner`).
/// 3. `Arc::clone` the entry.
/// 4. Drop the RCU guard and return.
///
/// Step 3 cannot race a concurrent `close(2)` into a use-after-free:
/// INV-FD2 (above) keeps the `OpenFile` — including its refcount word —
/// alive for the whole read-side section. A caller that wins the race
/// holds a fully valid reference to a description whose last fd was
/// concurrently closed, identical to having completed `fget()` just
/// before the `close()`. No slot re-check loop is needed, unlike Linux
/// Linux `__fget_files_rcu()` (`fs/file.c`), whose `SLAB_TYPESAFE_BY_RCU` file
/// slab can hand the reader a RECYCLED `struct file` and therefore needs
/// Linux uses the `file_ref_get()` + slot re-check dance; INV-FD2's deferral
/// guarantees the loaded pointer still names the same object.
///
/// **Cost** (hot path): RCU guard (per-CPU nesting bookkeeping only) +
/// one XArray walk (≤ 2 node hops for fd < 4096) + one uncontended
/// atomic increment — the same order as Linux's lockless
/// Linux `__fget_files_rcu()` performs additional re-check loads. (An earlier revision
/// took the process-wide fd-table spinlock here, serializing every fd
/// lookup of a `CLONE_FILES` group on one cache line — slower than
/// Linux on the hottest VFS path, and contradicting the `fds` XArray's
/// own RCU-read rationale.)
///
/// # Errors
/// - `EBADF`: `fd` is negative or not an open descriptor in this task.
pub fn fget(fd: RawFd) -> Result<Arc<OpenFile>, Errno> {
    if fd < 0 {
        return Err(Errno::EBADF);
    }
    // SAFETY: `current_task()` returns the running task's pointer, which is
    // valid for this synchronous syscall — the task cannot exit while it is
    // executing here.
    let task = unsafe { &*current_task() };
    let files = task.files.load(); // ArcSwap<FdTable> snapshot
    let guard = rcu_read_lock();
    let result = match files.fds.load(fd) {
        Some(file) => Ok(Arc::clone(file)),
        None => Err(Errno::EBADF),
    };
    drop(guard);
    result
}

/// Result of `fdget()`: a possibly-borrowed open file description — the
/// UmkaOS analogue of Linux's `struct fd` (`fs/file.c`). The hot
/// read/write path avoids ALL refcount traffic when the fd table is
/// private to the calling task. Kernel-internal; never crosses a KABI or
/// userspace boundary; syscall-scoped (a `FdGuard` must not be stored
/// beyond the current syscall — long-lived holders use `fget()`).
pub enum FdGuard<'t> {
    /// The calling task's `FdTable` is unshared: INV-FD1 guarantees the
    /// slot cannot be cleared concurrently, so the description is
    /// borrowed with ZERO refcount traffic (Linux `__fget_light()`
    /// borrowed-fd parity).
    Borrowed {
        /// The table's open file description, borrowed.
        file: &'t OpenFile,
        /// True iff the `OpenFile`'s owning reference count was exactly 1
        /// (the fd table's own) at lookup time — input to the
        /// `fdget_pos()` fast-path decision. Stable for the guard's
        /// lifetime: with the table unshared, only the current task
        /// could `dup`/`fork`/`SCM_RIGHTS`-send this description, and it
        /// is executing here.
        pos_exclusive: bool,
    },
    /// The fd table is shared (`CLONE_FILES`, or a kernel-side strong
    /// table reference exists): an owned reference was taken via the
    /// `fget()` lockless protocol.
    Owned(Arc<OpenFile>),
}

impl core::ops::Deref for FdGuard<'_> {
    type Target = OpenFile;
    fn deref(&self) -> &OpenFile {
        match self {
            FdGuard::Borrowed { file, .. } => file,
            FdGuard::Owned(arc) => arc,
        }
    }
}

/// Syscall-duration fd lookup — Linux `fdget()`/`__fget_light()` parity.
///
/// `task` MUST be the calling task (`current_task()`); the parameter
/// exists to name the borrow lifetime. Discriminator: the `FdTable`'s
/// `Arc` strong count, observed through the `ArcSwap` load guard (which
/// pins without bumping). Each task sharing the table holds exactly one
/// strong reference in its `files` cell, so count == 1 ⇔ no
/// `CLONE_FILES` sibling and no kernel-side strong holder. Transient
/// kernel snapshot holders (`load_full()` users) can only INFLATE the
/// count — an inflated reading errs toward the safe `Owned` path.
///
/// Fast path cost for an unshared table: RCU guard + XArray walk +
/// **zero** atomic RMW — matching Linux, where single-threaded
/// processes pay no refcount traffic on `read(2)`/`write(2)`.
///
/// # Errors
/// - `EBADF`: `fd` is negative or not an open descriptor in this task.
pub fn fdget(task: &Task, fd: RawFd) -> Result<FdGuard<'_>, Errno> {
    if fd < 0 {
        return Err(Errno::EBADF);
    }
    let files = task.files.load(); // ArcSwap pin — does not bump the count
    if Arc::strong_count(&files) == 1 {
        let guard = rcu_read_lock();
        let entry = match files.fds.load(fd) {
            Some(entry) => entry,
            None => return Err(Errno::EBADF),
        };
        let pos_exclusive = Arc::strong_count(entry) == 1;
        // SAFETY: extending the borrow past the RCU section is sound by
        // INV-FD1: the table is reachable only by `task`, which is
        // executing this syscall and therefore can neither clear the slot
        // nor swap its `files` cell (`unshare(CLONE_FILES)` runs on the
        // task itself) concurrently. The slot's owning Arc pins the
        // `OpenFile` for the guard's lifetime.
        let file: &OpenFile = unsafe { &*Arc::as_ptr(entry) };
        drop(guard);
        Ok(FdGuard::Borrowed { file, pos_exclusive })
    } else {
        Ok(FdGuard::Owned(fget(fd)?))
    }
}

/// Obtain the f_pos serialization right for a looked-up file — see the
/// `fdget_pos()` protocol on `OpenFile::f_pos` for the full decision
/// table and its Linux parity argument. Fast path (private descriptor,
/// non-directory): zero atomics beyond the position load. Slow path:
/// `f_pos_lock` (sleeping mutex, uncontended in the absence of actual
/// position races).
pub fn fdget_pos<'f>(fd_guard: &'f FdGuard<'_>) -> FdPosGuard<'f> {
    let file: &'f OpenFile = fd_guard;
    let needs_lock = file.f_mode.contains(FileMode::FMODE_ATOMIC_POS) && {
        let exclusive = matches!(
            fd_guard,
            FdGuard::Borrowed { pos_exclusive: true, .. }
        );
        // S_IFDIR (Linux include/uapi/linux/stat.h): directories always
        // take the locked path — see the protocol's Directories arm.
        let is_dir = (file.inode.i_mode.load(Ordering::Relaxed) & 0o170000) == 0o040000;
        !exclusive || is_dir
    };
    // Lock BEFORE loading the position — loading first would read a
    // stale cursor that a racing holder is about to write back.
    let lock = if needs_lock {
        Some(file.f_pos_lock.lock())
    } else {
        None
    };
    FdPosGuard {
        file,
        pos: file.f_pos.load(Ordering::Acquire),
        _lock: lock,
    }
}

14.1.6.4 Syscall-Boundary VFS Types

These VFS types are referenced by the syscall translation layer (Section 19.1) — the kernel-owned path buffer produced by copy_path_from_user, the access-intent mask consumed by the LSM check_file_permission hook, and the vfs_open entry point behind sys_open/sys_openat/sys_openat2.

// Linux open(2)/mount(2) flag constants used by the open path in this
// section — userspace ABI values, verified against `torvalds/linux`
// master `include/uapi/asm-generic/fcntl.h`. The kernel-INTERNAL
// encoding is the asm-generic (canonical) one on all eight targets. Two
// arch groups permute O_DIRECTORY/O_NOFOLLOW/O_LARGEFILE/O_DIRECT in their
// userspace ABI: the ARM group — ARMv7 AND AArch64 (`arch/arm64` overrides
// asm-generic even for native LP64) — and the PPC group — PPC32 and PPC64LE
// (`arch/powerpc/include/uapi/asm/fcntl.h`). The SysAPI translation layer
// ([Section 19.1](19-sysapi.md#syscall-interface), which holds the two normalization tables)
// normalizes those bits to the canonical encoding at syscall entry (and
// denormalizes for `fcntl(F_GETFL)`), so VFS-internal checks compare against
// exactly one encoding.

/// open(2) `O_TRUNC` — truncate a regular file to length 0 at open.
/// Same value in every architecture's userspace ABI.
pub const O_TRUNC: u32 = 1 << 9; // 0o1000

/// open(2) `O_DIRECT` — bypass the page cache. Canonical (asm-generic)
/// encoding; PowerPC userspace uses `1 << 17` and the ARM group (ARMv7,
/// AArch64) uses `1 << 16`, both normalized at the SysAPI boundary (see the
/// block comment above).
pub const O_DIRECT: u32 = 1 << 14; // 0o40000

/// open(2) `O_NOATIME` — suppress atime updates (file owner or
/// CAP_FOWNER only). Same value in every architecture's userspace ABI.
pub const O_NOATIME: u32 = 1 << 18; // 0o1000000

/// open(2) `O_CLOEXEC` — close-on-exec disposition for the new fd.
/// Same value in every architecture's userspace ABI.
pub const O_CLOEXEC: u32 = 1 << 19; // 0o2000000

/// open(2) `O_EMPTYPATH` — with an empty `pathname`, resolve to the `dirfd`
/// itself (the `AT_EMPTY_PATH`-as-open-flag mechanism; sets `LookupFlags::EMPTY_PATH`).
/// Member of Linux `O_PATH_FLAGS`, so it is O_PATH-compatible and survives an
/// `O_PATH` open. Canonical asm-generic value on all eight targets — no
/// architecture permutes bit 26.
pub const O_EMPTYPATH: u32 = 1 << 26; // 0o400000000

/// open(2) `O_APPEND` — every write appends at EOF. Same value in every
/// architecture's userspace ABI (`include/uapi/asm-generic/fcntl.h`).
pub const O_APPEND: u32 = 1 << 10; // 0o2000

/// open(2) `O_DSYNC` — data-integrity synchronous writes. Same value on
/// all eight targets.
pub const O_DSYNC: u32 = 1 << 12; // 0o10000

/// open(2) `O_SYNC` — file-integrity synchronous writes. Linux-exact
/// composite: `__O_SYNC (1 << 20) | O_DSYNC`, so testing `O_DSYNC` alone
/// matches both (the asm-generic compatibility encoding).
pub const O_SYNC: u32 = (1 << 20) | O_DSYNC; // 0o4010000

/// Superblock-flag constants. The VALUES are the `mount(2)` userspace ABI
/// (Linux `include/uapi/linux/mount.h`); the Rust TYPE is `u32` because
/// their sole in-kernel consumer is `SuperBlock::s_flags: AtomicU32`
/// (all defined superblock flags fit in bits 0-30 — see the `s_flags`
/// field doc). The `mount(2)`/`mount_setattr(2)` compat shims do NOT
/// compare against these: they translate the raw userspace flag word to
/// `MountFlags` (a `u64` bitflags type) at syscall entry
/// ([Section 14.6](#mount-tree-data-structures-and-operations)), so no `MS_*`
/// constant ever meets a `KernelULong` mount-flags argument. Widen these
/// together with `s_flags` if a superblock flag is ever assigned bit 31+.

/// mount(2)/superblock `MS_RDONLY` (Linux `SB_RDONLY`) — read-only
/// filesystem. Bit 0 of `SuperBlock::s_flags` on every architecture.
pub const MS_RDONLY: u32 = 1;

/// mount(2)/superblock `MS_SYNCHRONOUS` (Linux `SB_SYNCHRONOUS`) —
/// all writes synced at once (`mount -o sync`). Feeds
/// `Inode::is_sync()`/`is_dirsync()`. Linux `include/uapi/linux/mount.h`.
pub const MS_SYNCHRONOUS: u32 = 16;

/// mount(2)/superblock `MS_DIRSYNC` (Linux `SB_DIRSYNC`) — directory
/// modifications synchronous (`mount -o dirsync`). Feeds
/// `Inode::is_dirsync()`.
pub const MS_DIRSYNC: u32 = 128;

/// mount(2)/superblock `MS_NOATIME` (Linux `SB_NOATIME`) — suppress all
/// atime updates on this mount/superblock. Consulted by the atime-update
/// decision after the per-inode `InodeFlags::NOATIME` check.
pub const MS_NOATIME: u32 = 1024;

/// Kernel-owned, NUL-free path buffer produced by `copy_path_from_user`
/// ([Section 19.1](19-sysapi.md#syscall-interface)). A `#![no_std]`-compatible replacement for
/// `std::path::PathBuf`: it owns a heap `Vec<u8>` of raw path bytes with no
/// interior or trailing NUL (the copy helper scans for the first NUL and
/// truncates there). Path resolution borrows it as a byte slice via
/// `as_bytes()`; a `KernelPath` is not itself interned in the dcache.
pub struct KernelPath {
    /// Raw path bytes: no interior NUL, no trailing NUL. Length bounded by the
    /// caller's `max_len` (typically `PATH_MAX` = 4096).
    bytes: Vec<u8>,
}

impl KernelPath {
    /// Wrap already-validated path bytes. The caller (`copy_path_from_user`)
    /// has copied at most `max_len` bytes, scanned for the first NUL, and
    /// `set_len`-truncated the `Vec` there — so `bytes` is the exact path with
    /// no NUL. Warm path: the allocation is the caller's `Vec`; no copy here.
    pub fn from_bytes(bytes: Vec<u8>) -> KernelPath {
        KernelPath { bytes }
    }

    /// Borrow the path as raw bytes for component-by-component resolution.
    pub fn as_bytes(&self) -> &[u8] {
        &self.bytes
    }
}

bitflags! {
    /// VFS access-intent mask passed to the LSM `check_file_permission` hook
    /// ([Section 19.1](19-sysapi.md#syscall-interface)) and internal permission checks. This is a
    /// kernel-internal mask, NOT a userspace ABI.
    ///
    /// The low three bits are laid out to mirror the `rwx` triad of the mode
    /// word — `MAY_READ = 4`, `MAY_WRITE = 2`, `MAY_EXEC = 1`, i.e. the same
    /// `r=4 / w=2 / x=1` weighting as `S_IRUSR>>6 / S_IWUSR>>6 / S_IXUSR>>6`.
    /// This lets a requested-access mask be derived from the relevant
    /// owner/group/other mode-bit triad with a shift and a 3-bit mask, with no
    /// per-bit remap table on the permission hot path. (This low-bit layout
    /// coincides with Linux's `MAY_*` in `include/linux/fs.h`; the shared value
    /// is a consequence of the `rwx` convention, not an imported dependency.)
    pub struct FilePermission: u32 {
        /// Execute a file / search (traverse) a directory.
        const MAY_EXEC      = 0x0000_0001;
        /// Write / modify.
        const MAY_WRITE     = 0x0000_0002;
        /// Read.
        const MAY_READ      = 0x0000_0004;
        /// Append-only write (`O_APPEND`).
        const MAY_APPEND    = 0x0000_0008;
        /// `access(2)`/`faccessat(2)` probe (no open performed).
        const MAY_ACCESS    = 0x0000_0010;
        /// Open (checked at open time, in addition to the read/write intent).
        const MAY_OPEN      = 0x0000_0020;
        /// `chdir(2)`/`fchdir(2)` into a directory.
        const MAY_CHDIR     = 0x0000_0040;
        /// Called from RCU-walk (non-blocking) resolution: the hook must not
        /// sleep and may return `-ECHILD` to force a ref-walk retry.
        const MAY_NOT_BLOCK = 0x0000_0080;
    }
}

/// Open a resolved mount-tree location and return an installed file descriptor.
/// This is the VFS entry point behind `sys_open`/`sys_openat`/`sys_openat2`
/// ([Section 19.1](19-sysapi.md#syscall-interface)), called after `path_lookup` produced the target
/// `MountDentry` and the LSM `check_open` hook authorized the open.
///
/// `mode` supplies the creation mode only when the inode was just created
/// upstream under `O_CREAT` (path resolution / `InodeOps::create` performs the actual
/// creation); it is ignored for an existing inode.
///
/// # Errors
/// - `ENOENT` — the dentry is negative (a lost `O_CREAT` creation race).
/// - Everything `open_and_install` can return (`ELOOP`, `ENOTDIR`,
///   `EISDIR`, `EROFS`, `EPERM`, `EINVAL`, `EMFILE`, driver open and
///   `O_TRUNC` truncation errors — see its doc for the conditions).
pub fn vfs_open(md: MountDentry, flags: u32, mode: u32) -> Result<Fd, Errno> {
    // `path_lookup` returns a positive dentry; fetch its inode under RCU.
    let guard = rcu_read_lock();
    let inode = match md.dentry.d_inode.read(&guard) {
        Some(Some(i)) => Arc::clone(i),
        _ => return Err(Errno::ENOENT), // negative dentry (lost O_CREAT race)
    };
    drop(guard);
    // Build the open file description and install a descriptor.
    OpenFile::open_and_install(md, inode, flags, mode)
}

impl OpenFile {
    /// Build an open file description for `inode` (opened via `md`) and install
    /// it at a fresh descriptor in the calling task's `FdTable`, returning the
    /// new `Fd`. This is the shared open→install path behind `vfs_open`; the
    /// pseudo-filesystem openers (`pipe(2)`, `socket(2)`, `eventfd(2)`, …) build
    /// their `OpenFile` the same way over a synthetic inode.
    ///
    /// Steps (Linux parity: `fs/namei.c may_open()`/`do_open()` +
    /// Linux `fs/open.c do_dentry_open()`, verified against `torvalds/linux`
    /// master):
    /// 1. `O_PATH` short-circuit — build a pure location reference
    ///    (`FMODE_PATH`, no driver open, no access modes) and install it.
    /// 2. Derive `f_mode` from the access-mode bits.
    /// 3. `do_open()`/`may_open()`-parity flag/type validation, in Linux
    ///    order: `EFTYPE` for `OPENAT2_REGULAR` on a non-regular target
    ///    (Linux `do_open()` gate, ahead of `may_open()`),
    ///    then `ELOOP` for symlinks, `ENOTDIR`/`EISDIR` type mismatches,
    ///    `O_TRUNC` stripped for
    ///    device/FIFO/socket inodes, `EROFS` for write-class opens on
    ///    read-only superblocks/mounts, `EPERM` for write-class opens of
    ///    `InodeFlags::IMMUTABLE` inodes and for `InodeFlags::APPEND` violations (write
    ///    access without `O_APPEND`, or `O_TRUNC` — the attribute-flag
    ///    substrate, [Section 14.1](#virtual-filesystem-layer--inode-attribute-flags)),
    ///    `EPERM` for non-owner `O_NOATIME`,
    ///    `EINVAL` for `O_DIRECT` without filesystem support.
    /// 4. Positional-capability mode bits: `FMODE_LSEEK`/`FMODE_PREAD`/
    ///    `FMODE_PWRITE` (seekable inode types), `FMODE_ATOMIC_POS`
    ///    (regular files and directories), `FMODE_DIRECT`.
    /// 5. Run `FileOps::open` for the `OpenOutcome` — the driver's private
    ///    token plus the stacking-filesystem data-inode binding.
    /// 6. `O_TRUNC` — truncate regular files to length 0
    ///    (`handle_truncate()`, after the driver open exactly as Linux
    ///    Linux `do_open()` orders it), rolling back the driver open on failure.
    /// 7. Assemble the `OpenFile` per its field-init contract, then reserve
    ///    and fill the lowest free descriptor (RLIMIT_NOFILE-checked)
    ///    honoring `O_CLOEXEC` — rolling back the driver open if descriptor
    ///    allocation fails.
    ///
    /// `mode` is consumed upstream by `InodeOps::create` when `O_CREAT` actually
    /// created the inode; the inode already exists here, so it is not re-applied.
    ///
    /// The DAC/LSM authorization for the open ran upstream (see `vfs_open`);
    /// the access mask it checked includes `MAY_WRITE` whenever `O_TRUNC`
    /// is set (Linux `build_open_flags()`: "O_TRUNC implies we need access
    /// checks for write permissions"), so `O_RDONLY | O_TRUNC` was
    /// write-authorized before the truncation in step 6 runs.
    ///
    /// # Errors
    /// - `EFTYPE` — `OPENAT2_REGULAR` was set and the resolved inode is not
    ///   a regular file. Takes precedence
    ///   over `ELOOP` when both apply (a symlink target under
    ///   `O_NOFOLLOW | OPENAT2_REGULAR`): Linux tests `__O_REGULAR` in
    ///   Linux `do_open()` precedes `may_open()`'s `ELOOP` check.
    /// - `ELOOP` — the resolved inode is a symlink (`O_NOFOLLOW` without
    ///   `O_PATH` landed on a symbolic link).
    /// - `ENOTDIR` — `O_DIRECTORY` requested but the inode is not a directory.
    /// - `EISDIR` — a directory was opened for write (or with `O_TRUNC`).
    /// - `EROFS` — write-class open of a regular file on a read-only
    ///   superblock or read-only mount.
    /// - `EPERM` — write-class open of an `InodeFlags::IMMUTABLE` inode; write
    ///   access to an `InodeFlags::APPEND` inode without `O_APPEND`, or `O_TRUNC`
    ///   on it; or `O_NOATIME` by a caller that is neither the file
    ///   owner nor `CAP_FOWNER`-capable.
    /// - `EINVAL` — the access-mode bits are not one of
    ///   `O_RDONLY`/`O_WRONLY`/`O_RDWR`, or `O_DIRECT` was requested and the
    ///   filesystem has no direct-I/O implementation.
    /// - `EMFILE` — the per-process `RLIMIT_NOFILE` limit is reached (`alloc_fd`).
    /// - Any `Errno` propagated by `FileOps::open` or the `O_TRUNC` truncation.
    pub fn open_and_install(
        md: MountDentry,
        inode: Arc<Inode>,
        flags: u32,
        _mode: u32,
    ) -> Result<Fd, Errno> {
        // Linux `S_IF*` constants (`include/uapi/linux/stat.h`) — inode type
        // is the top bits of `i_mode`.
        const S_IFMT: u32 = 0o170000;
        const S_IFSOCK: u32 = 0o140000;
        const S_IFLNK: u32 = 0o120000;
        const S_IFREG: u32 = 0o100000;
        const S_IFBLK: u32 = 0o060000;
        const S_IFDIR: u32 = 0o040000;
        const S_IFCHR: u32 = 0o020000;
        const S_IFIFO: u32 = 0o010000;
        let i_fmt = inode.i_mode.load(Ordering::Relaxed) & S_IFMT;
        let is_dir = i_fmt == S_IFDIR;
        let mut flags = flags;

        let task = current_task();
        // Capture the opener's credentials (used for the O_NOATIME owner
        // check below, and by writeback / async completion that runs after
        // the syscall returns).
        let rcu = rcu_read_lock();
        let f_cred = Arc::clone(&task.cred.read(&rcu));
        drop(rcu);

        // Mount pin cell (`RcuCell::new` allocates its Box — fallible) and
        // the crash-epoch snapshot for the rebind keys. Built BEFORE the
        // driver open in step 5, so this failure path needs no `release()`
        // rollback (see the step-7 rollback note).
        let mount_cell =
            RcuCell::new(Arc::clone(&md.mnt)).map_err(|_| Errno::ENOMEM)?;
        let mount_epoch_now =
            SHADOW_MOUNT_REGISTRY.crash_epoch.load(Ordering::Acquire);

        // Step 1: O_PATH short-circuit (Linux `do_dentry_open()`:
        // Linux: `f_mode = FMODE_PATH | FMODE_OPENED; f_op = &empty_fops;
        // return 0;` — nothing else runs). sys_open/sys_openat have
        // already stripped O_PATH-incompatible flags
        // (`normalize_open_flags()`, "O_PATH beats everything else");
        // sys_openat2 rejected them with EINVAL. An O_PATH description is
        // a pure location reference: no `FileOps::open`, no permission
        // check on the final component, no access modes, no freeze/EROFS
        // interaction, and symlinks are legal targets (the
        // `O_PATH | O_NOFOLLOW` symlink-fd idiom). Its permitted uses
        // (dirfd anchor, fstat, fchdir, close, dup, fcntl,
        // /proc/self/fd re-open, linkat/execveat with `AT_EMPTY_PATH`)
        // never dispatch through `f_ops`; every data operation is
        // rejected by the FMODE gates with `EBADF` — neither `FMODE_READ`
        // nor `FMODE_WRITE` is set — and `epoll_ctl(2)` rejects it with
        // `EPERM` (Linux: files without poll support).
        if flags & O_PATH != 0 {
            if (flags & O_DIRECTORY) != 0 && !is_dir {
                return Err(Errno::ENOTDIR);
            }
            let of = Arc::new(OpenFile {
                inode: Arc::clone(&inode),
                // `O_PATH` performs no data I/O at all, so the resolved data
                // inode is trivially the opened inode.
                data_inode: Arc::clone(&inode),
                dentry: Arc::clone(&md.dentry),
                mount: mount_cell,
                // Rebind keys: Core-written ONCE here, never re-read from
                // `Mount` payload memory afterwards (see the field docs).
                mount_id: md.mnt.mount_id,
                mount_ns_id: task.namespace_set.load().mount_ns.ns_id,
                mount_epoch: AtomicU64::new(mount_epoch_now),
                f_ops: &PATH_FILE_OPS,
                f_pos: AtomicI64::new(0),
                f_pos_lock: Mutex::new(()),
                f_flags: AtomicU32::new(flags),
                f_mode: FileMode::FMODE_PATH,
                f_cred,
                f_wb_err: WbErrSnapshot::new(inode.i_mapping.wb_err.sample()),
                ra_state: Mutex::new(FileRaState::default()),
                private_data: AtomicPtr::new(core::ptr::null_mut()),
                open_generation: AtomicU64::new(
                    inode.i_sb.driver_generation.load(Ordering::Acquire)),
                revalidate_lock: Mutex::new(()),
                reopen_errno: AtomicI32::new(0),
            });
            return Self::install_fd(task, of, flags);
        }

        // Step 2: Derive the internal file mode from the access-mode bits
        // (the low two bits; `O_ACCMODE == 0o3`).
        let mut f_mode = FileMode::empty();
        match flags & 0o3 {
            0 => f_mode |= FileMode::FMODE_READ,                         // O_RDONLY
            1 => f_mode |= FileMode::FMODE_WRITE,                        // O_WRONLY
            2 => f_mode |= FileMode::FMODE_READ | FileMode::FMODE_WRITE, // O_RDWR
            _ => return Err(Errno::EINVAL),                             // invalid (0o3)
        }

        // Step 3: may_open()-parity validation (Linux fs/namei.c, order
        // preserved: type switch, then write-access, then O_NOATIME).
        //
        // 3a'. `OPENAT2_REGULAR` — carried internally as `OPEN_REGULAR_ONLY`
        //      (`sys_openat2` translated UAPI bit 32 down to the
        //      kernel-internal carrier before VFS entry; see the
        //      openat2-exclusive upper-flag space in [Section 19.1](19-sysapi.md#syscall-interface)).
        //      Fail unless the resolved object is a regular file. Checked
        //      FIRST — ahead of the symlink `ELOOP` arm (3a) — because
        //      Linux `fs/namei.c do_open()` tests `__O_REGULAR` before
        //      Linux `may_open()`, where a trailing symlink
        //      yields `-ELOOP` (`case S_IFLNK: return -ELOOP;`); the
        //      Linux condition is `if ((open_flag & __O_REGULAR) && !d_is_reg(nd->path.dentry))`
        //      Linux returns `-EFTYPE` before the `LOOKUP_DIRECTORY`/`ENOTDIR`
        //      test. So `openat2(O_NOFOLLOW | OPENAT2_REGULAR)` on a symlink
        //      returns `EFTYPE`, not `ELOOP`. `O_PATH` opens never reach here
        //      (`O_PATH | OPENAT2_REGULAR` is `EINVAL` at `sys_openat2`).
        if (flags & OPEN_REGULAR_ONLY) != 0 && i_fmt != S_IFREG {
            return Err(Errno::EFTYPE);
        }
        // 3a. A symlink can only be opened via O_PATH (handled above):
        //     a non-O_PATH open that resolved to a symlink means
        //     O_NOFOLLOW stopped resolution at the link itself. This is
        //     Linux `may_open()`'s `case S_IFLNK: return -ELOOP;` arm, which
        //     runs AFTER the regular-only `EFTYPE` gate above.
        if i_fmt == S_IFLNK {
            return Err(Errno::ELOOP);
        }
        // 3b. Flag/type compatibility.
        if (flags & O_DIRECTORY) != 0 && !is_dir {
            return Err(Errno::ENOTDIR);
        }
        if is_dir && (f_mode.contains(FileMode::FMODE_WRITE) || (flags & O_TRUNC) != 0) {
            return Err(Errno::EISDIR);
        }
        // 3c. O_TRUNC is meaningless for objects without file storage:
        //     silently stripped for device nodes, FIFOs, and sockets
        //     (Linux `may_open()`: `flag &= ~O_TRUNC` in the S_IFBLK/S_IFCHR/
        //     S_IFIFO/S_IFSOCK arms).
        if matches!(i_fmt, S_IFBLK | S_IFCHR | S_IFIFO | S_IFSOCK) {
            flags &= !O_TRUNC;
        }
        // 3d. "Nobody gets write access to a read-only fs" (Linux
        //     Linux `sb_permission()` + `mnt_get_write_access()`): a write-class
        //     open — write access mode or O_TRUNC — fails with EROFS when
        //     the superblock or the mount is read-only. Only regular files
        //     remain to check here: directory and symlink write-opens were
        //     rejected above (EISDIR/ELOOP), and device/FIFO/socket writes
        //     do not modify the filesystem (exempt in Linux too).
        let write_class =
            f_mode.contains(FileMode::FMODE_WRITE) || (flags & O_TRUNC) != 0;
        if write_class && i_fmt == S_IFREG {
            if inode.i_sb.s_flags.load(Ordering::Relaxed) & MS_RDONLY != 0
                || md.mnt.flags.load(Ordering::Relaxed) & MNT_READONLY != 0
            {
                return Err(Errno::EROFS);
            }
        }
        // 3e. Immutable file: "Nobody gets write access to an immutable
        //     file" (Linux fs/namei.c inode_permission(), reached from
        //     Linux `may_open()`'s `inode_permission` call — EROFS-before-EPERM
        //     order preserved). One Relaxed i_flags load
        //     ([Section 14.1](#virtual-filesystem-layer--inode-attribute-flags));
        //     deliberately NOT served from the dentry `cached_perm`
        //     grants, so a cached write grant can never override the
        //     flag. Applies to every inode type (a write-open of an
        //     immutable device node is EPERM too, matching Linux).
        if write_class && inode.is_immutable() {
            return Err(Errno::EPERM);
        }
        // 3f. Append-only file: must be opened in append mode for
        //     writing, and can never be opened with O_TRUNC:
        //     Linux `may_open(): if (IS_APPEND(inode))` block — the EPERM
        //     rules this gate implements are rows of the normative
        //     enforcement map in
        //     [Section 14.1](#virtual-filesystem-layer--inode-attribute-flags).
        //     O_TRUNC on device/FIFO/socket inodes was stripped in 3c,
        //     so this cannot mis-fire on them.
        if inode.is_append() {
            if f_mode.contains(FileMode::FMODE_WRITE) && flags & O_APPEND == 0 {
                return Err(Errno::EPERM);
            }
            if flags & O_TRUNC != 0 {
                return Err(Errno::EPERM);
            }
        }
        // 3g. O_NOATIME can only be set by the file owner or a
        //     CAP_FOWNER-capable caller (Linux `may_open()`:
        //     `inode_owner_or_capable()` → EPERM).
        if flags & O_NOATIME != 0
            && f_cred.fsuid != inode.i_uid.load(Ordering::Relaxed)
            && !has_cap(task, SystemCaps::CAP_FOWNER)
        {
            return Err(Errno::EPERM);
        }
        // 3h. O_DIRECT requires a direct-I/O implementation; absence
        //     yields EINVAL.
        if flags & O_DIRECT != 0 {
            if inode.i_mapping.ops.direct_io().is_none() {
                return Err(Errno::EINVAL);
            }
            f_mode |= FileMode::FMODE_DIRECT;
        }

        // Step 4: Positional-capability bits (Linux do_dentry_open():
        // `f_mode |= FMODE_LSEEK | FMODE_PREAD | FMODE_PWRITE`, cleared
        // for stream-like files). Regular files, directories, and device
        // nodes have a byte offset; FIFOs and sockets are streams — their
        // lseek/pread/pwrite fail with ESPIPE at the FMODE gate. Character
        // devices that cannot actually seek (ttys) keep the bits and
        // return ESPIPE from `FileOps::llseek` — the mode bit only says
        // the operation may be DISPATCHED.
        if !matches!(i_fmt, S_IFIFO | S_IFSOCK) {
            f_mode |= FileMode::FMODE_LSEEK
                | FileMode::FMODE_PREAD
                | FileMode::FMODE_PWRITE;
        }
        // POSIX-atomic f_pos (Linux: `S_ISREG || S_ISDIR` — see the
        // `fdget_pos()` protocol on `OpenFile::f_pos`).
        if matches!(i_fmt, S_IFREG | S_IFDIR) {
            f_mode |= FileMode::FMODE_ATOMIC_POS;
        }

        // Step 5: Run the filesystem/driver open hook. It returns an
        // `OpenOutcome`: the private token stashed in `private_data` and
        // handed back to every later `FileOps` call, plus the data-inode
        // binding a stacking filesystem resolved for this open (`None` for
        // every non-stacking filesystem, which is every filesystem that
        // reaches this generic path directly).
        // Device nodes (S_IFCHR/S_IFBLK) do NOT open through the generic
        // inode `i_fop`: they route through `device_node_open()`
        // ([Section 14.5](#device-node-framework)), which runs the cgroup
        // `BPF_CGROUP_DEVICE` access check, resolves the CHRDEV/BLKDEV region by
        // `i_rdev`, and binds the driver's region `FileOps` — bypassing it would
        // defeat container device-cgroup isolation and leave registered drivers
        // unreachable. `open_fops` is the surface installed as `f_ops` and used
        // for the release rollback below.
        let (open_fops, outcome) = if matches!(i_fmt, S_IFCHR | S_IFBLK) {
            let dopen = device_node_open(&inode, flags)?;
            // A device-node region open never rebinds data I/O to another
            // inode — a character/block device's data path IS the driver's
            // `FileOps`, not a page cache on some other inode.
            (dopen.fops, OpenOutcome { private: dopen.private, data_inode: None })
        } else {
            let o = inode
                .i_fop
                .open(InodeId(inode.i_ino), OpenFlags::from_bits_truncate(flags))?;
            (inode.i_fop, o)
        };

        // Step 6: O_TRUNC — truncate AFTER the driver open, matching Linux
        // driver-open-before-truncate ordering (the filesystem may need
        // its per-open state to truncate; truncating a freshly
        // O_CREAT-created file is a no-op on an already-empty file). Only
        // regular files reach here (3b/3c stripped or rejected the rest).
        if flags & O_TRUNC != 0 && i_fmt == S_IFREG {
            if let Err(e) = handle_truncate(&inode, outcome.private) {
                // Roll back the driver open — see the step-7 rollback
                // comment; the same leak argument applies. `open_fops` is the
                // surface `open()` ran through (region ops for device nodes).
                let _ = open_fops.release(InodeId(inode.i_ino), outcome.private);
                return Err(e);
            }
        }

        // Assemble the description per each field's documented init contract.
        // `f_pos` starts at 0 even for `O_APPEND` — the write path re-seeks to
        // EOF before every write regardless of the stored position.
        let of = Arc::new(OpenFile {
            inode: Arc::clone(&inode),
            // Resolved data inode, exactly as step 5's hook returned it. A
            // stacking filesystem resolved the real backing inode during its
            // own open (after any copy-up it performed) and handed it back in
            // `OpenOutcome::data_inode`; every non-stacking filesystem
            // returned `None`, meaning the data lives in the opened inode
            // itself, so the binding is the same `Arc`. It is written here,
            // while the description is still exclusively owned, and never
            // mutated after publication.
            data_inode: outcome.data_inode.unwrap_or_else(|| Arc::clone(&inode)),
            dentry: Arc::clone(&md.dentry),
            mount: mount_cell,
            // Rebind keys: Core-written ONCE here, never re-read from
            // `Mount` payload memory afterwards (see the field docs).
            mount_id: md.mnt.mount_id,
            mount_ns_id: task.namespace_set.load().mount_ns.ns_id,
            mount_epoch: AtomicU64::new(mount_epoch_now),
            // For device nodes this is the resolved region `FileOps`, not the
            // inode's generic `i_fop` (see step 5).
            f_ops: open_fops,
            f_pos: AtomicI64::new(0),
            f_pos_lock: Mutex::new(()),
            // `OPEN_REGULAR_ONLY` is never user-visible: stripped before the
            // description is published so `F_GETFL` cannot report it
            // (Linux `do_dentry_open()` strips its carrier likewise).
            f_flags: AtomicU32::new(flags & !OPEN_REGULAR_ONLY),
            f_mode,
            f_cred,
            f_wb_err: WbErrSnapshot::new(inode.i_mapping.wb_err.sample()),
            ra_state: Mutex::new(FileRaState::default()),
            private_data: AtomicPtr::new(outcome.private as *mut ()),
            open_generation: AtomicU64::new(
                inode.i_sb.driver_generation.load(Ordering::Acquire)),
            revalidate_lock: Mutex::new(()),
            reopen_errno: AtomicI32::new(0),
        });

        // Step 7: install, rolling back the driver open on failure.
        match Self::install_fd(task, Arc::clone(&of), flags) {
            Ok(fd) => Ok(fd),
            Err(e) => {
                // `FileOps::open` succeeded in step 5 and its per-open state
                // (journal handle, delegation, device context) is owned by
                // `private` — dropping the `OpenFile` does NOT release it
                // (no Drop hook crosses the KABI boundary; `release()` is
                // invoked explicitly by the close path). Without this call,
                // every EMFILE-losing open would leak driver per-open state
                // — a 50-year-uptime violation. Linux avoids the window by
                // reserving the descriptor before the driver open; UmkaOS rolls back
                // instead — the same
                // no-leak postcondition. A `release()` error is swallowed:
                // the open itself already failed with `e`.
                let _ = open_fops.release(InodeId(inode.i_ino), outcome.private);
                Err(e)
            }
        }
    }

    /// Reserve the lowest free descriptor (RLIMIT_NOFILE-checked via
    /// `alloc_fd`, [Section 8.8](08-process.md#resource-limits-and-accounting)), then fill the
    /// reserved slot and record its close-on-exec disposition. Both steps
    /// run against the same `FdTable` snapshot; the reservation keeps a
    /// concurrent `alloc_fd` from handing out the same number before the
    /// store completes. The slot store runs under the fd-table writer lock
    /// (`FdTable.inner`), never bare — see "File Descriptor Lookup —
    /// Lockless Fast Paths" for the reader side.
    ///
    /// On `Err` the caller still owns its `Arc<OpenFile>` (and the driver
    /// open-state rollback, if any).
    fn install_fd(task: &Task, of: Arc<OpenFile>, flags: u32) -> Result<Fd, Errno> {
        let fd = alloc_fd(task)?;
        let files = task.files.load(); // ArcSwap<FdTable> snapshot
        let mut inner = files.inner.lock(); // fd-table writer lock
        files.fds.store(fd, of);
        // O_CLOEXEC == 0o2000000.
        inner.cloexec.set(fd as usize, (flags & 0o2000000) != 0);
        Ok(fd)
    }
}

/// Truncate-on-open: the `O_TRUNC` backend — Linux
/// `fs/namei.c handle_truncate()` parity, called from
/// `OpenFile::open_and_install` step 6 after `FileOps::open` succeeded.
///
/// The write authorization for the truncation happened at the open's
/// DAC/LSM check: the access mask carried `MAY_WRITE` whenever `O_TRUNC`
/// was set (see `open_and_install`), so a
/// dedicated truncate-time permission hook is not re-run here. Linux
/// master no longer refuses truncation of a running executable either —
/// exec stopped holding Linux's `i_writecount` ("fs: don't block i_writecount
/// during exec", v6.11) — so there is no `ETXTBSY` arm.
///
/// Ordering: freeze protection (`sb_start_write`, Write level) is entered
/// BEFORE `i_rwsem`, mirroring every other write-class path; the in-memory
/// size is zeroed before `FileOps::truncate` per that method's contract
/// ("the VFS calls truncate after updating the in-memory inode size").
fn handle_truncate(inode: &Inode, private: u64) -> Result<(), Errno> {
    // Freeze protection: truncation is a Write-level filesystem
    // modification. Blocks here (not EROFS) if the filesystem is frozen.
    let _write = sb_start_write(&inode.i_sb, SbFreezeLevel::Write)?;
    // Exclusive inode lock (I_RWSEM, level 80): excludes concurrent
    // writers and readers-of-size for the whole size change.
    let _guard = inode.i_rwsem.write();
    // In-memory size first (FileOps::truncate contract), then the page
    // cache — nothing beyond offset 0 may survive.
    inode.i_size.store(0, Ordering::Release);
    truncate_inode_pages_range(&inode.i_mapping, 0, u64::MAX);
    // Filesystem frees blocks/extents and updates its on-disk metadata.
    inode.i_fop.truncate(InodeId(inode.i_ino), private, 0)?;
    // ctime/mtime (Linux do_truncate(): ATTR_MTIME | ATTR_CTIME), both
    // under one INODE_LOCK hold so readers never observe a torn pair.
    let now = Timespec::from_ns(current_time_ns());
    {
        let mut times = inode.i_times.lock();
        times.i_mtime = now;
        times.i_ctime = now;
    }
    Ok(())
}

/// The `FileOps` vtable installed on `O_PATH` open file descriptions
/// (Linux `empty_fops` parity — a `file_operations` with every pointer
/// NULL). Never legitimately invoked: the VFS dispatch layer rejects
/// every data and metadata operation on an `FMODE_PATH` file before
/// reaching `f_ops` (`EBADF` from the FMODE gates — neither `FMODE_READ`
/// nor `FMODE_WRITE` nor any positional-capability bit is set; `EPERM`
/// for `epoll_ctl`). Every method returns `EBADF` defensively.
pub struct PathFileOps;

/// The single static instance referenced by `open_and_install`'s O_PATH arm.
pub static PATH_FILE_OPS: PathFileOps = PathFileOps;

impl FileOps for PathFileOps {
    fn open(&self, _inode: InodeId, _flags: OpenFlags) -> Result<OpenOutcome> {
        Err(Errno::EBADF)
    }
    fn release(&self, _inode: InodeId, _private: u64) -> Result<()> {
        Err(Errno::EBADF)
    }
    fn read(
        &self,
        _file: &OpenFile,
        _buf: &mut UserSliceMut,
        _offset: &mut i64,
    ) -> Result<usize, IoError> {
        Err(IoError::EBADF)
    }
    fn write(
        &self,
        _file: &OpenFile,
        _buf: &UserSlice,
        _offset: &mut i64,
    ) -> Result<usize, IoError> {
        Err(IoError::EBADF)
    }
    fn truncate(&self, _inode: InodeId, _private: u64, _new_size: u64) -> Result<()> {
        Err(Errno::EBADF)
    }
    fn fsync(
        &self,
        _inode: InodeId,
        _private: u64,
        _start: u64,
        _end: u64,
        _datasync: u8,
    ) -> Result<()> {
        Err(Errno::EBADF)
    }
    fn fallocate(
        &self,
        _inode: InodeId,
        _private: u64,
        _offset: u64,
        _len: u64,
        _mode: FallocateMode,
    ) -> Result<()> {
        Err(Errno::EBADF)
    }
    fn readdir(
        &self,
        _inode: InodeId,
        _private: u64,
        _offset: u64,
        _emit: &mut dyn FnMut(InodeId, u64, FileType, &OsStr) -> bool,
    ) -> Result<()> {
        Err(Errno::EBADF)
    }
    fn llseek(
        &self,
        _inode: InodeId,
        _private: u64,
        _offset: i64,
        _whence: SeekWhence,
    ) -> Result<u64> {
        Err(Errno::EBADF)
    }
    fn mmap(
        &self,
        _inode: InodeId,
        _private: u64,
        _offset: u64,
        _len: usize,
        _vm_flags: u64,
    ) -> Result<MmapResult> {
        Err(Errno::EBADF)
    }
    fn ioctl(&self, _inode: InodeId, _private: u64, _cmd: u32, _arg: u64) -> Result<i64> {
        Err(Errno::EBADF)
    }
    fn splice_read(
        &self,
        _inode: InodeId,
        _private: u64,
        _offset: u64,
        _pipe: PipeId,
        _len: usize,
    ) -> Result<usize> {
        Err(Errno::EBADF)
    }
    fn splice_write(
        &self,
        _pipe: PipeId,
        _inode: InodeId,
        _private: u64,
        _offset: u64,
        _len: usize,
    ) -> Result<usize> {
        Err(Errno::EBADF)
    }
    fn poll(
        &self,
        _inode: InodeId,
        _private: u64,
        _events: PollEvents,
        _pt: Option<&mut PollTable>,
    ) -> Result<PollEvents> {
        Err(Errno::EBADF)
    }
}

14.1.6.5 Path Lookup Entry Point

The path_lookup function is the primary entry point for all VFS path resolution. Every syscall that accepts a pathname (open, stat, access, mkdir, unlink, mount, execve, etc.) calls path_lookup to translate the user-provided path string into a MountDentry pair identifying the target location in the mount tree.

bitflags! {
    /// Flags controlling path resolution behavior. Passed to `path_lookup()`
    /// by syscall handlers to customize resolution semantics.
    ///
    /// These flags correspond to Linux's internal `LOOKUP_*` flags (not
    /// directly visible to userspace, but indirectly controlled by syscall
    /// flags like `O_NOFOLLOW`, `O_DIRECTORY`, `O_CREAT`, `AT_SYMLINK_NOFOLLOW`,
    /// `AT_EMPTY_PATH`, `RESOLVE_BENEATH`, `RESOLVE_NO_XDEV`, etc.).
    ///
    /// A flag-set is a plain `u32` value type: derive `Copy` (with `Clone`)
    /// so it passes by value without moving — `path_lookup` hands the same
    /// `flags` to both the RCU-walk and ref-walk `walk_components` calls, and
    /// the syscall bridges pass it by value throughout. `bitflags` 2.x does
    /// not auto-derive these, so they are stated explicitly (matching the
    /// `MsgFlags` convention in [Section 16.3](16-networking.md#socket-abstraction)).
    #[derive(Copy, Clone, Debug, PartialEq, Eq)]
    pub struct LookupFlags: u32 {
        /// Follow the terminal symlink. If the final path component is a
        /// symlink and this flag is set, resolution follows it to the target.
        /// If not set, resolution returns the symlink inode itself.
        ///
        /// Default for most syscalls (`open`, `stat`). Cleared by `O_NOFOLLOW`
        /// and `AT_SYMLINK_NOFOLLOW`. `lstat()` clears this flag.
        ///
        /// Note: intermediate symlinks (non-terminal components) are ALWAYS
        /// followed regardless of this flag — only the final component is
        /// affected. This matches POSIX behavior.
        const FOLLOW        = 0x0001;

        /// The final component must be a directory. If it resolves to a
        /// non-directory inode, return `ENOTDIR`.
        ///
        /// Set by `O_DIRECTORY` (openat2), `mkdir` (parent lookup), and
        /// `rmdir` (target validation). Also implicitly set when the path
        /// ends with a trailing `/` (POSIX: trailing slash implies directory).
        const DIRECTORY     = 0x0002;

        /// Resolve the parent directory of the final component, not the
        /// final component itself. The final component name is returned
        /// separately (not resolved to an inode). Used by syscalls that
        /// create or remove entries: `mkdir`, `mknod`, `unlink`, `rmdir`,
        /// `rename`, `link`, `symlink`.
        ///
        /// When set, `path_lookup` returns the `MountDentry` of the parent
        /// directory, and the final component name is stored in a separate
        /// output parameter (not shown in this simplified signature).
        const PARENT        = 0x0004;

        /// The syscall is creating a new entry (`O_CREAT`). This flag is
        /// informational — it does not change resolution behavior, but it
        /// is checked by audit/LSM hooks to distinguish "create" from
        /// "open existing".
        const CREATE        = 0x0008;

        /// The syscall requires exclusive creation (`O_EXCL`). Combined
        /// with `CREATE`. If the final component already exists, return
        /// `EEXIST`. The VFS checks this after resolution completes.
        const EXCL          = 0x0010;

        /// The resolution is part of an `open()` operation. This flag
        /// enables open-intent optimizations: the VFS can pass an open
        /// intent to the filesystem's `lookup()` so that NFS can perform
        /// an atomic lookup-and-open in a single RPC, avoiding a TOCTOU
        /// race between lookup and open.
        const OPEN          = 0x0020;

        /// Resolution must not cross the `root` boundary upward. If the
        /// path contains `..` components that would ascend above `root`,
        /// return `EXDEV` instead of silently clamping to `root`.
        ///
        /// Maps to `RESOLVE_BENEATH` (openat2). Provides a stronger
        /// security guarantee than chroot: even a privileged process
        /// cannot escape the `root` boundary via `..` traversal.
        const BENEATH       = 0x0040;

        /// Resolution must not cross mount point boundaries. If the path
        /// traverses a mount point (in either direction — into a mounted
        /// filesystem or back out via `..`), return `EXDEV`.
        ///
        /// Maps to `RESOLVE_NO_XDEV` (openat2). Used by sandboxed
        /// processes and container runtimes that want to confine path
        /// resolution to a single filesystem.
        const NO_XDEV       = 0x0080;

        /// Do not trigger automounts during resolution. If a path
        /// component is an autofs trigger point, return `ENOENT` instead
        /// of mounting the remote filesystem.
        ///
        /// Maps to `AT_NO_AUTOMOUNT`. This is separate from
        /// `RESOLVE_NO_MAGICLINKS` — automount suppression and magic-link
        /// suppression are independent concepts.
        const NO_AUTOMOUNT  = 0x0100;

        /// Fail if any path component (including the terminal) is a
        /// symlink. Stricter than clearing `FOLLOW` (which only affects
        /// the terminal component).
        ///
        /// Maps to `RESOLVE_NO_SYMLINKS` (openat2, 0x04).
        const NO_SYMLINKS   = 0x0200;

        /// Fail on `/proc/[pid]/fd/*` style magic symlinks (procfs magic
        /// links that jump to arbitrary filesystem locations). Regular
        /// symlinks are still followed unless `NO_SYMLINKS` is also set.
        ///
        /// Maps to `RESOLVE_NO_MAGICLINKS` (openat2, 0x02).
        const NO_MAGICLINKS = 0x0400;

        /// Treat `dirfd` as the filesystem root. `..` at `dirfd` stays
        /// at `dirfd` (like chroot but per-syscall). Combined with
        /// `BENEATH`, provides a complete sandboxed lookup.
        ///
        /// Maps to `RESOLVE_IN_ROOT` (openat2, 0x10).
        const IN_ROOT       = 0x0800;

        /// Non-blocking lookup. If the resolution would block (uncached
        /// dentry, lazy NFS lookup, autofs trigger), return `EAGAIN`
        /// instead of blocking. Used by io_uring for async path ops.
        ///
        /// Maps to `RESOLVE_CACHED` (openat2, 0x20).
        const CACHED        = 0x1000;

        /// Empty path resolution. When set with an empty path string,
        /// the resolution returns the `MountDentry` of the `dirfd`
        /// itself (or `pwd` if `dirfd` is `AT_FDCWD`). Used by
        /// `AT_EMPTY_PATH` (fstatat, linkat, etc.) and `fexecve`.
        const EMPTY_PATH    = 0x2000;
    }
}

/// VFS path resolution entry point. Called from syscall handlers to resolve
/// a user-provided path string to a `MountDentry` pair
/// ([Section 8.1](08-process.md#process-and-task-management)).
///
/// This function implements the two-phase resolution protocol described above:
/// first attempts RCU-walk (lockless, zero atomic operations on hit), then
/// falls back to ref-walk on miss or concurrent modification.
///
/// # Arguments
///
/// - `mnt_ns`: Mount namespace for mount point traversal. Determines which
///   mounts are visible during resolution. Obtained from
///   `task.namespace_set.mount_ns` ([Section 17.1](17-containers.md#namespace-architecture)).
/// - `root`: Chroot root boundary from `task.fs.load().read().root`. Path
///   resolution never ascends above this point via `..`. If `LookupFlags::BENEATH`
///   is set, attempting to ascend above `root` returns `EXDEV` instead of
///   clamping.
/// - `pwd`: Current working directory from `task.fs.load().read().pwd`. Used as the
///   starting point for relative path resolution. Ignored for absolute paths
///   (paths starting with `/`).
/// - `path`: Path string (absolute or relative). Kernel-space byte slice —
///   the syscall layer has already copied this from userspace via
///   `copy_from_user`. Must be null-terminated or bounded by `PATH_MAX`
///   (4096 bytes). An empty `path` is valid only if `LOOKUP_EMPTY_PATH` is
///   set in `flags`.
/// - `flags`: `LookupFlags` bitflags controlling resolution behavior (see
///   the `LookupFlags` definition above).
///
/// # Returns
///
/// On success, returns a `MountDentry` identifying the resolved location
/// in the mount tree. The returned `MountDentry` holds references to both
/// the mount and the dentry (refcounts incremented). The caller is
/// responsible for releasing these references when done.
///
/// # Errors
///
/// | Error | Condition |
/// |-------|-----------|
/// | `ENOENT` | A path component does not exist (and `LookupFlags::CREATE` is not set) |
/// | `EACCES` | Traverse (execute) permission denied on a directory component |
/// | `ENOTDIR` | A non-terminal component is not a directory, or `LookupFlags::DIRECTORY` is set and the final component is not a directory |
/// | `ELOOP` | More than 40 nested symlinks encountered during resolution |
/// | `ENAMETOOLONG` | A path component exceeds `NAME_MAX` (255 bytes) or the total path exceeds `PATH_MAX` (4096 bytes) |
/// | `EXDEV` | `LookupFlags::BENEATH`: path escapes `root` via `..`. `LookupFlags::NO_XDEV`: path crosses a mount boundary |
/// | `EINVAL` | Empty path without `LOOKUP_EMPTY_PATH` |
///
/// # Concurrency
///
/// Thread-safe. Multiple threads may call `path_lookup` concurrently. The
/// RCU-walk phase is fully lockless. The ref-walk fallback acquires per-dentry
/// spinlocks and inode `i_rwsem` (shared) as needed.
///
/// # Performance
///
/// Hot path (all components cached, no concurrent mutations): O(n) where n is
/// the number of path components. Each component costs one dentry hash lookup
/// (~5-10ns) plus one `cached_perm` check (~1-3ns). No domain crossings, no
/// locks, no atomic RMW operations.
pub fn path_lookup(
    mnt_ns: &MountNamespace,
    root: &MountDentry,
    pwd: &MountDentry,
    path: &[u8],
    flags: LookupFlags,
) -> Result<MountDentry, Errno> {
    // Relative paths start at `pwd`, absolute paths at the `root` boundary
    // (chroot / `IN_ROOT`). Empty-path resolution (`EMPTY_PATH`) and the
    // dirfd-relative base are settled by the caller (the syscall bridge below);
    // `path_lookup` receives the resolved `pwd`/`root` pair.
    let start = if path.first() == Some(&b'/') { root } else { pwd };
    let root_node = LiveWalkNode {
        mnt: NonNull::from(&*root.mnt),
        dentry: NonNull::from(&*root.dentry),
    };
    let start_node = LiveWalkNode {
        mnt: NonNull::from(&*start.mnt),
        dentry: NonNull::from(&*start.dentry),
    };

    // Phase 1 — RCU-walk (zero refcount traffic). One RCU section brackets the
    // whole core call; the provider borrows dentries validated inside it. The
    // walk IS `walk_components::<DcacheWalkProvider>` — the same monomorphized
    // core the `vfs_path_walk_safety` harness drives over `FixtureWalkProvider`.
    {
        let guard = rcu_read_lock();
        let provider = DcacheWalkProvider::rcu_walk(mnt_ns, root_node);
        let mut walk = WalkState::new();
        match walk_components::<DcacheWalkProvider>(
            &provider, &mut walk, start_node, path, flags,
        ) {
            // Terminal node reached: legitimize it (take Arc refs + revalidate
            // the seqcount, Linux `try_to_unlazy`). On success it is a stable
            // owning `MountDentry`.
            Ok(node) => {
                if let Some(md) = legitimize_walk_node(node, &guard) {
                    return Ok(md);
                }
                // Lost the legitimization race — restart under ref-walk.
            }
            // Seqcount moved mid-walk (INV-3): drop to ref-walk.
            Err(Errno::EAGAIN) => {}
            // Definite resolution failure (ENOENT/ELOOP/EXDEV/…): final.
            Err(e) => return Err(e),
        }
    }

    // Phase 2 — ref-walk (reference-pinned, always `seq_valid`). SAME core,
    // SAME instantiation — only the provider's mode differs.
    let provider = DcacheWalkProvider::ref_walk(mnt_ns, root_node);
    let mut walk = WalkState::new();
    let node = walk_components::<DcacheWalkProvider>(
        &provider, &mut walk, start_node, path, flags,
    )?;
    mount_dentry_from_node(node)
}

Both phases feed the shared core through two terminal-node materializers — the ref-walk one packages held references; the RCU-walk one legitimizes (refs + a final seqcount revalidation) and can fail the race, forcing the ref-walk fallback:

/// Legitimize an RCU-walk terminal node into an owning `MountDentry`: take Arc
/// references on the mount and dentry and revalidate the seqcount under the
/// still-held RCU `guard` (the Linux `try_to_unlazy` / `complete_walk`
/// transition). `None` if a concurrent mutation invalidated the node between
/// resolution and legitimization — the caller restarts under ref-walk. Never
/// returns a node whose references were taken under a stale seqcount.
fn legitimize_walk_node(node: LiveWalkNode, guard: &RcuReadGuard) -> Option<MountDentry>

/// Materialize a ref-walk terminal node into an owning `MountDentry`. Ref-walk
/// already pinned the terminal component, so this only packages the held mount
/// and dentry references into the result pair — no revalidation needed.
fn mount_dentry_from_node(node: LiveWalkNode) -> Result<MountDentry, Errno>

Credential resolution — transport-dependent, tier-agnostic: the VFS, like every module, can be deployed at any tier (Section 11.3), so path_lookup() obtains the caller's credentials per the transport selected at bind time, never by assuming a deployment:

• Same-domain deployment (VFS bound in the caller's domain — kabi_call! resolved to a direct call): the dispatch path reads current_task().cred directly (RCU-protected read). No domain crossing, no snapshot.
• Cross-domain deployment (kabi_call! resolved to a ring): the CALLER side — the syscall layer, running in Core — resolves current_task().cred BEFORE submission and delivers the credential context WITH the request, per the KABI caller-credential model (Section 12.3: validation takes the caller's credentials as an argument, and tokens pin cred_generation). Concretely, the request carries the caller's credential reference (kept live for the request's duration by the task's blocked syscall) plus its perm_tag and cred_generation snapshot; the serving domain performs its per-component checks against THAT delivered identity. The serving domain never dereferences Core per-CPU state — current_task lives on the CpuLocalBlock, which is Core-image-only under the isolation key model (Section 11.2) and is not readable from any non-Core domain image.

Either way, permission checks at each path component evaluate the SAME credential the caller entered the syscall with; a concurrent install_credentials() does not retroactively alter an in-flight lookup (credentials are immutable objects — the lookup holds the entry-time object).

RESOLVE_IN_ROOT capture timing: The root boundary is captured at the start of path_lookup() from task.fs.root. If the task's namespace changes between syscall entry and path_lookup(), the root captured at path_lookup entry is authoritative. This is consistent with Linux openat2() behavior.

dirfd validity across unshare(CLONE_NEWNS): After unshare(CLONE_NEWNS), existing file descriptors (including dirfd values) remain valid. The dentry referenced by a dirfd is in the mount tree — after unshare, the new mount namespace is a copy of the old, and existing dentries are shared (copy-on-write mount points). Operations using AT_FDCWD or an explicit dirfd resolve in the calling task's current mount namespace. The dirfd does not become invalid.

pwd after unshare(CLONE_NEWNS): If the current working directory is unreachable from the new namespace's root (e.g., the mount point was not copied), the pwd becomes a "floating" dentry. File operations relative to pwd succeed (the dentry is still valid). getcwd() returns ENOENT. This matches Linux behavior.

Syscall-to-VFS bridge: Syscall handlers construct the path_lookup call from the current task's state:

// Example: openat(dirfd, pathname, flags, mode) syscall handler sketch.
// Shows how SyscallContext fields feed into path_lookup.
fn sys_openat(ctx: &mut SyscallContext) -> i64 {
    let dirfd = ctx.args[0] as i32;
    let pathname = copy_path_from_user(ctx.args[1] as *const u8, PATH_MAX)?;
    // "O_PATH beats everything else" — strip O_PATH-incompatible flags
    // (Linux build_open_how(); openat2 EINVALs instead, in its handler).
    let flags = normalize_open_flags(ctx.args[2] as u32);
    let mode = ctx.args[3] as u32;

    // `fs` is `ArcSwap<RwLock<FsStruct>>`: load() pins the current Arc, then
    // read() takes the RwLock. Bind the load guard so the read guard does not
    // borrow a dropped temporary.
    let fs_guard = ctx.task.fs.load();
    let fs = fs_guard.read();
    let namespace_set = ctx.task.namespace_set.load();
    let mnt_ns = &namespace_set.mount_ns;

    // Determine the base directory for relative paths.
    let base = if dirfd == AT_FDCWD {
        &fs.pwd
    } else {
        &fd_to_mount_dentry(ctx.task.files.load().get(dirfd)?)?
    };

    let lookup_flags = open_flags_to_lookup_flags(flags);
    let target = path_lookup(mnt_ns, &fs.root, base, &pathname, lookup_flags)?;

    // ... proceed with open using the resolved MountDentry ...
}

Flag translation functions:

/// Normalize `open(2)` / `openat(2)` O_* flags BEFORE lookup translation.
/// Linux `fs/open.c build_open_how()` parity (verified against
/// `torvalds/linux` master): "O_PATH beats everything else" — when
/// `O_PATH` is set, every flag outside the O_PATH-compatible set
/// (`O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC | O_EMPTYPATH`) is SILENTLY
/// stripped. In particular `O_CREAT` is stripped, so an `O_PATH` open can
/// never create a file, and the access-mode bits are stripped, so the
/// resulting description carries neither read nor write access.
/// `O_EMPTYPATH` is preserved because it is a member of Linux `O_PATH_FLAGS`
/// (`= O_DIRECTORY | O_NOFOLLOW | O_PATH | O_CLOEXEC | O_EMPTYPATH` on
/// master) — an `O_PATH | O_EMPTYPATH` open of a directory fd is valid.
///
/// `openat2(2)` is stricter — it REJECTS `O_PATH` combined with any flag
/// outside that set with `EINVAL` instead of stripping
/// (Linux `build_open_flags()`: `if (flags & ~O_PATH_FLAGS) return -EINVAL`);
/// `sys_openat2` performs that check in its own handler before calling
/// the translation functions below.
///
/// Called by sys_open / sys_openat before `open_flags_to_lookup_flags()`;
/// the normalized value is also what reaches `open_and_install()` and is
/// stored in `OpenFile::f_flags`.
fn normalize_open_flags(o_flags: u32) -> u32 {
    if o_flags & O_PATH != 0 {
        o_flags & (O_DIRECTORY | O_NOFOLLOW | O_PATH | O_CLOEXEC | O_EMPTYPATH)
    } else {
        o_flags
    }
}

/// Translate `open(2)` / `openat(2)` O_* flags to internal LookupFlags.
/// Used by sys_open, sys_openat, and legacy open paths. The input must
/// already be normalized by `normalize_open_flags()`.
///
/// `O_PATH` adds NO lookup flag of its own: it changes what `open`
/// builds (an `FMODE_PATH` location reference — see `open_and_install()`),
/// not how the path resolves. The symlink-fd idiom `O_PATH | O_NOFOLLOW`
/// works through the `O_NOFOLLOW` arm below. (An earlier revision mapped
/// `O_PATH` to `LookupFlags::EMPTY_PATH`, which is wrong — `EMPTY_PATH`
/// means "an empty pathname resolves to dirfd itself" and belongs to
/// `AT_EMPTY_PATH`. The `O_EMPTYPATH` open flag is the open-family spelling
/// of that same knob, and IS mapped to `EMPTY_PATH` below — distinct from
/// `O_PATH`.)
fn open_flags_to_lookup_flags(o_flags: u32) -> LookupFlags {
    let mut lf = LookupFlags::FOLLOW; // default: follow terminal symlinks
    if o_flags & O_NOFOLLOW != 0 {
        lf.remove(LookupFlags::FOLLOW);
    }
    if o_flags & O_DIRECTORY != 0 {
        lf |= LookupFlags::DIRECTORY;
    }
    if o_flags & O_CREAT != 0 {
        lf |= LookupFlags::CREATE;
    }
    if o_flags & O_EMPTYPATH != 0 {
        // Empty pathname resolves to dirfd itself (Linux `build_open_flags()`:
        // the Linux condition maps `O_EMPTYPATH` to the empty-path lookup bit).
        lf |= LookupFlags::EMPTY_PATH;
    }
    lf
}

/// Translate `openat2(2)` resolve flags (from `struct open_how.resolve`) to
/// internal LookupFlags. Called by sys_openat2 AFTER `open_flags_to_lookup_flags()`
/// to layer the RESOLVE_* restrictions on top of the O_* translations.
///
/// Linux `openat2(2)` resolve flag values (from `include/uapi/linux/openat2.h`):
///   RESOLVE_NO_XDEV       = 0x01
///   RESOLVE_NO_MAGICLINKS = 0x02
///   RESOLVE_NO_SYMLINKS   = 0x04
///   RESOLVE_BENEATH       = 0x08
///   RESOLVE_IN_ROOT       = 0x10
///   RESOLVE_CACHED        = 0x20
///
/// Returns `Err(EINVAL)` if mutually exclusive resolve flags are combined:
/// `RESOLVE_BENEATH | RESOLVE_IN_ROOT` is rejected;
/// Linux `fs/open.c build_open_flags()` checks `if ((how->resolve & RESOLVE_BENEATH) &&
/// (how->resolve & RESOLVE_IN_ROOT)) return -EINVAL;`. Both flags shipped
/// with `openat2()` in Linux 5.6 (only `RESOLVE_CACHED` is 5.12+), so the
/// combination has been rejected since 5.6.
fn resolve_flags_to_lookup_flags(
    base: LookupFlags,
    resolve: u64,
) -> Result<LookupFlags, Errno> {
    // RESOLVE_BENEATH and RESOLVE_IN_ROOT are mutually exclusive
    // (Linux build_open_flags(): both set -> -EINVAL).
    if resolve & RESOLVE_BENEATH != 0 && resolve & RESOLVE_IN_ROOT != 0 {
        return Err(Errno::EINVAL);
    }
    let mut lf = base;
    if resolve & RESOLVE_NO_XDEV != 0 {
        lf |= LookupFlags::NO_XDEV;
    }
    if resolve & RESOLVE_NO_MAGICLINKS != 0 {
        lf |= LookupFlags::NO_MAGICLINKS;
    }
    if resolve & RESOLVE_NO_SYMLINKS != 0 {
        lf |= LookupFlags::NO_SYMLINKS;
        lf.remove(LookupFlags::FOLLOW); // NO_SYMLINKS implies no terminal follow
    }
    if resolve & RESOLVE_BENEATH != 0 {
        lf |= LookupFlags::BENEATH;
    }
    if resolve & RESOLVE_IN_ROOT != 0 {
        lf |= LookupFlags::IN_ROOT;
    }
    if resolve & RESOLVE_CACHED != 0 {
        lf |= LookupFlags::CACHED;
    }
    // Reject unknown bits (forward compatibility).
    let known_bits = RESOLVE_NO_XDEV | RESOLVE_NO_MAGICLINKS | RESOLVE_NO_SYMLINKS
        | RESOLVE_BENEATH | RESOLVE_IN_ROOT | RESOLVE_CACHED;
    if resolve & !known_bits != 0 {
        return Err(Errno::EINVAL);
    }
    Ok(lf)
}

14.1.7 Mount Namespace and Capability-Gated Mounting

Each process belongs to a mount namespace containing its own mount tree.

Mount operations are capability-gated:

Operation	Required Capability	Scope
mount	CAP_MOUNT	Mount namespace
bind mount	CAP_MOUNT + read access to source	Mount namespace + source
remount	CAP_MOUNT	Mount namespace
umount	CAP_MOUNT	Mount namespace
pivot_root	CAP_SYS_ADMIN	Mount namespace

CAP_MOUNT is scoped to the calling process's mount namespace — it does not grant mount authority in other namespaces. A container with its own mount namespace can mount filesystems within that namespace without affecting the host.

Mount propagation: Shared, private, slave, and unbindable propagation types, with the same semantics as Linux (MS_SHARED, MS_PRIVATE, MS_SLAVE, MS_UNBINDABLE). This is essential for container runtimes that rely on mount propagation for volume mounts.

Filesystem type registration: Only Core can register new filesystem types with the VFS. Filesystem drivers request registration via the inter-domain ring, and Core verifies the driver's identity and KABI certification before granting registration.

14.1.7.1 Mount Lifecycle

The mount(2) syscall drives a multi-step flow that creates or reuses a SuperBlock, allocates a Mount node, and inserts it into the calling process's mount tree. Each step has a defined rollback on failure, ensuring no resource leaks.

Mount flow (mount_filesystem) (SUMMARY — the canonical mount_filesystem algorithm, including the authoritative superblock resolve-or-create step, error-arm ordering, and the shadow-registry reporting bracket, is Section 14.6; where this summary and the canonical algorithm differ, the canonical algorithm governs):

1. Lookup filesystem type. Search the global FS_TYPE_TABLE (XArray keyed by filesystem name hash) for the requested fs_type string (e.g., "ext4", "tmpfs"). If not found, return ENODEV.

2. Cgroup device controller check. For block-backed filesystems (source resolves to a block device), check the calling task's cgroup device controller allowlist (Section 17.2). If the device's (major, minor) is not in devices.allow for the task's cgroup, return EPERM. This prevents containers from mounting arbitrary block devices. Pseudo-filesystems (tmpfs, procfs, sysfs) skip this check.

3. Resolve or reuse the SuperBlock. For device-backed (FS_REQUIRES_DEV) filesystems, search the filesystem type's superblock table (FsType.sb_table, keyed by DevId — canonical definition in Section 14.6, step 6b):

4. Existing superblock found: Increment s_refcount and REUSE the superblock — step 4 is skipped entirely, the filesystem instance is already live. First verify mount flags are compatible (e.g., cannot mount the same device MS_RDONLY and read-write simultaneously; Linux compat anchor: fs/super.c get_tree_bdev_flags() returns EBUSY on an RO/RW state mismatch, torvalds/linux master). If incompatible, return EBUSY.

5. No existing superblock: proceed to step 4 — the DRIVER creates the superblock; the VFS layer never pre-allocates one.

6. Call FileSystemOps::mount(source, flags, data) (signature per the FileSystemOps vtable — the driver CREATES and returns the SuperBlock; there is no VFS-allocated sb argument). The filesystem driver reads the on-disk superblock (for block-backed filesystems), fills s_type, s_blocksize, s_maxbytes, s_root, s_fs_info, sets s_dev to the backing BlockDevice's device number (or an anonymous device number for diskless filesystems), performs journal replay if needed (ext4, XFS), and returns the superblock with s_refcount = 1. The VFS inserts it into FsType.sb_table. For pseudo-filesystems (tmpfs, procfs), this step populates the root inode and dentry without any block I/O.

7. On error: return the filesystem's error code (creation failed inside the driver — there is no superblock to release).

8. Create Mount node. Allocate a Mount (Section 14.6) linking:

9. parent: the dentry where this mount is attached (e.g., /mnt/data).
10. source: the device path or source string (e.g., /dev/sda1).
11. sb: the SuperBlock from step 3/4.
12. mount_id: a per-namespace monotonic u64 mount identifier, never reused within the namespace (exposed to userspace via STATX_MNT_ID; allocated from MountNamespace.id_allocator).
13. flags: mount flags (MS_RDONLY, MS_NOSUID, MS_NODEV, etc.).
14. propagation: propagation type (MS_SHARED, MS_PRIVATE, etc.), defaulting to MS_PRIVATE.

15. On error: decrement s_refcount; only if it reaches zero, call FileSystemOps::unmount(sb) and free the SuperBlock (the canonical superblock release rule — Section 14.6, step 6b). In the reused-superblock arm of step 3 the superblock is still serving other live mounts — an unconditional unmount() here would tear down a filesystem out from under them.

16. Bind BackingDevInfo. For block-backed filesystems, associate the BackingDevInfo (BDI) from the block device with the superblock. The BDI controls writeback rate limiting, readahead window defaults, and dirty page accounting per backing device (Section 4.6).

17. For pseudo-filesystems (tmpfs, procfs), a default BDI with no writeback is used.

18. Insert into mount tree. Acquire the mount namespace write lock. Attach the Mount as a child of the parent dentry in the namespace's mount tree. Set d_mount_seq on the parent dentry (incremented to invalidate any in-flight RCU-walk lookups that cached the old state). Apply mount propagation rules: if the parent mount is MS_SHARED, replicate the new mount into all peer mount namespaces. Release the mount namespace lock.

19. On error: deallocate the Mount, then decrement s_refcount; only if it reaches zero, call FileSystemOps::unmount(sb) and free the SuperBlock — never unconditionally (reused superblocks, step 3, are still serving other live mounts; mirror of the step-5 error arm).

20. Shadow commit. Report the committed transition (new mounts including propagation clones, and any new superblock record) to the Shadow Mount Registry — the stage/commit bracket detailed in the canonical algorithm (Section 14.6).

21. Return success. The filesystem is now accessible at the mount point.

Unmount flow (do_umount):

1. Check reference count. If the mount has active open files, child mounts, or CWD references, return EBUSY (unless MNT_FORCE or MNT_DETACH is specified).

2. Detach from mount tree. Acquire the mount namespace write lock. Remove the Mount from its parent's child list. Increment d_mount_seq on the parent dentry. For MNT_DETACH (lazy umount), the mount is detached from the namespace tree immediately but the SuperBlock is kept alive until all references are released. Report the committed removal to the Shadow Mount Registry (the registry drops the mount's owning Arc<Mount> after an RCU grace period — Section 14.6).

3. Sync dirty data. Call FileSystemOps::sync_fs(sb, wait=true) to flush all dirty pages and metadata. This invokes the writeback thread (Section 4.6) for the superblock's BDI. For MNT_FORCE, skip the sync and proceed with best-effort teardown (in-flight I/O is drained with -EIO).

4. Tear down SuperBlock. Decrement s_refcount. If the refcount reaches zero (no other mounts share this superblock): a. Evict all inodes: walk s_inodes, call the eviction sequence (evict(): writeback dirty pages, InodeOps::evict_inode, remove from sb.inode_cache XArray). Disposition per inode via disposition_from_nlink(): still-linked inodes get EvictDisposition::Release (unmount must NOT free live files' disk blocks); unlinked-but-open stragglers finally reaped here get EvictDisposition::Delete. b. Call FileSystemOps::unmount(sb) — the filesystem flushes its journal, writes the clean-unmount marker, and releases s_fs_info. b2. Ring/recovery teardown (ring-backed mounts — sb.ring_set is Some; runs AFTER step b because the unmount call itself may ride the ring): signal sb.ring_worker_exit (Release) + kick ring_set.completion_doorbell and WAIT for the response worker's task exit; call unregister_recovery_descriptor(&sb.vfs_recovery_desc) — it returns only after any in-flight recovery walk has drained (Section 11.9), so no later domain-fault walk can invoke a descriptor whose ctx points at freed storage; wake sb.recovery_wait (parked dispatchers observe the unmount and fail with ENODEV — the umount arm of that field's wake contract); then free sb.ring_core and sb.ring_set. The normative step list is do_umount step 16 in Section 14.6. c. Release s_bdev reference (if block-backed). d. Free the SuperBlock slab object.

5. Release the Mount node. The shadow registry's owning Arc<Mount> drop (step 2's report, after the grace period) plus the release of any remaining fd references frees the Mount back to Nucleus tracked storage (free_tracked::<Mount>() at final Arc drop — there is no direct slab free).

Force unmount (umount2 with MNT_FORCE): Calls FileSystemOps::force_umount(sb), which aborts in-flight I/O with -EIO and skips journal commit. Used when a network filesystem server is unreachable or a device has been physically removed. Data loss may occur for unflushed dirty pages.

Remount (mount -o remount): Does not create a new Mount node. Instead, calls FileSystemOps::remount(sb, new_flags, data) to update mount options on the existing superblock. The VFS validates flag transitions (e.g., MS_RDONLY → read-write requires CAP_MOUNT and a journal replay check).

See Section 14.5 for the character/block device node framework (chrdev/blkdev registration, major number table, devtmpfs automatic /dev node lifecycle).

14.1.7.2 ML Policy Integration for VFS

The VFS subsystem emits observations and exposes tunable parameters through the ML policy framework (Section 23.1). This enables closed-loop optimization of readahead, writeback scheduling, and dirty page throttling.

Observation hooks: The following observe_kernel! call sites are placed in VFS hot/warm paths. Each call is zero-cost (NOP) when no policy service consumer is attached (static key patching; see Section 23.1).

Call site	Subsystem	Observation type	Path class	Data emitted
filemap_get_pages() cache miss	VfsLayer	VfsObs::PageCacheMiss	Hot	(ino, file_offset, ra_window_size, sequential: bool)
filemap_get_pages() cache hit	VfsLayer	VfsObs::PageCacheHit	Hot	(ino, file_offset) — sampled at 1/64 rate to bound overhead
page_cache_write_iter()	VfsLayer	VfsObs::BufferedWrite	Hot	(ino, bytes_written, dirty_pages_after) — sampled at 1/16
writeback_single_inode() completion	VfsLayer	VfsObs::WritebackComplete	Warm	(ino, pages_written, elapsed_us, sequential_ratio)
balance_dirty_pages() throttle	VfsLayer	VfsObs::DirtyThrottle	Warm	(bdi_id, dirty_pages, dirty_limit, throttle_ms)
page_cache_readahead() trigger	VfsLayer	VfsObs::ReadaheadTrigger	Warm	(ino, start_offset, window_pages, sequential: bool)
Dentry cache miss in path_lookup()	VfsLayer	VfsObs::DentryCacheMiss	Hot	(parent_ino, name_hash) — sampled at 1/32
VFS ring request enqueue	VfsLayer	VfsObs::RingEnqueue	Hot	(mount_id, opcode, ring_index) — sampled at 1/128
path_lookup() completion	VfsLayer	VfsObs::PathLookupLatency	Hot	(path_components, elapsed_ns, rcu_walk_success: bool) — sampled at 1/64. Measures end-to-end path resolution latency including mount crossings and symlink follows. RCU-walk success rate is a key metric: low success rate indicates contention forcing ref-walk fallbacks.
select_ring() → response dequeue	VfsLayer	VfsObs::RingUtilization	Warm	(mount_id, ring_index, ring_depth, pending_slots, response_latency_ns) — emitted on every response dequeue. Measures ring fill level and round-trip latency. High pending_slots/ring_depth ratio signals the ring is saturated and ring count should be increased (or ring depth enlarged).
Readahead completion audit	VfsLayer	VfsObs::ReadaheadHitRate	Warm	(bdi_id, window_pages, pages_used_before_eviction, hit_ratio_pct) — emitted when a readahead window is fully consumed or evicted. Tracks how many prefetched pages were actually accessed before eviction. Low hit rate means the readahead window is oversized (wasting memory and I/O bandwidth).

/// VFS-specific observation types for the ML policy framework.
/// Used as the `obs_type` field in `observe_kernel!` calls.
#[repr(u16)]
pub enum VfsObs {
    /// Page cache miss — readahead evaluation opportunity.
    PageCacheMiss       = 0,
    /// Page cache hit — confirms readahead effectiveness.
    PageCacheHit        = 1,
    /// Buffered write completion — dirty page accumulation signal.
    BufferedWrite       = 2,
    /// Writeback completion for a single inode.
    WritebackComplete   = 3,
    /// Dirty page throttling engaged — backpressure signal.
    DirtyThrottle       = 4,
    /// Readahead triggered — window sizing feedback.
    ReadaheadTrigger    = 5,
    /// Dentry cache miss — path resolution pressure signal.
    DentryCacheMiss     = 6,
    /// VFS ring enqueue — cross-domain I/O pressure signal.
    RingEnqueue         = 7,
    /// Path resolution end-to-end latency — RCU-walk success rate signal.
    PathLookupLatency   = 8,
    /// Ring utilization — fill level and response latency signal.
    RingUtilization     = 9,
    /// Readahead hit rate — window sizing effectiveness feedback.
    ReadaheadHitRate    = 10,
}

Tunable parameters: The following VFS parameters are registered in the Kernel Tunable Parameter Store (Section 23.1). ParamId values are allocated in the I/O Scheduler range (0x0300-0x03FF) since VFS readahead and writeback are I/O-adjacent. Each parameter has a default, bounds, and a cooldown period to prevent oscillation.

ParamId	Name	Default	Min	Max	Cooldown	Description
IoReadaheadPages (0x0300)	readahead_pages	32	1	512	30s	Per-BDI max readahead window in pages
0x0303	vfs_dirty_ratio_pct	20	5	80	60s	vm.dirty_ratio equivalent — percentage of total memory that can be dirty before synchronous writeback
0x0304	vfs_dirty_bg_ratio_pct	10	1	50	60s	vm.dirty_background_ratio — background writeback trigger threshold
0x0305	vfs_writeback_interval_cs	500	100	6000	30s	Writeback timer interval in centiseconds (default 5s = 500cs)
0x0306	vfs_ra_sequential_threshold	4	1	32	30s	Number of sequential page accesses before readahead window doubles
0x0307	vfs_ring_coalesce_batch	8	1	64	10s	Default VFS ring doorbell coalescing batch size for regular I/O
0x0308	vfs_ring_coalesce_timeout_us	20	1	200	10s	Default VFS ring doorbell coalescing timeout in microseconds
0x0309	vfs_completion_coalesce_batch	8	1	32	10s	Response-direction completion coalescing batch size (Section 14.3). Number of completions batched before waking the VFS consumer.
0x030A	vfs_completion_coalesce_timeout_us	10	1	100	10s	Response-direction completion coalescing timeout in microseconds. Bounds worst-case latency for sparse completion streams.

Closed-loop example — readahead window auto-tuning:

1. Policy service observes VfsObs::PageCacheMiss and VfsObs::ReadaheadTrigger on a per-BDI basis. High miss rate after readahead suggests the window is too small.
2. Policy service computes the optimal readahead_pages using the PID controller (Section 23.1). Target metric: page cache hit rate > 95% for sequential workloads.
3. Policy service sends a ParamAdjust { param_id: IoReadaheadPages, value: N } message.
4. The VFS readahead engine reads the updated value via KERNEL_PARAM_STORE.get(IoReadaheadPages) on the next readahead evaluation (warm path, no hot-path overhead).

Phase assignment: VFS observation hooks are Phase 3 (functional without ML; ML provides optimization). Parameter registration is Phase 2 (parameters are readable by sysctl even without a policy service).

14.2 VFS Ring Buffer Protocol (Cross-Domain Dispatch)

The tier model (Section 11.3) requires ALL cross-domain communication to use ring buffer IPC. However, the FileSystemOps, InodeOps, and FileOps traits defined in Section 14.1 use direct Rust function call signatures. This section specifies how trait method calls are marshaled across the isolation domain boundary between umka-nucleus (VFS layer) and Tier 1 filesystem drivers.

Architecture: Each mounted filesystem has a dedicated request/response ring pair:

/// Maximum inline I/O data size in bytes. Reads/writes at or below this
/// threshold carry data inline in the ring entry, avoiding DMA buffer
/// allocation and IOMMU mapping. Covers >90% of procfs/sysfs reads.
///
/// 192 bytes fits within the ring entry without bloating large-I/O variants
/// (the `VfsRequestArgs` union is already dominated by `SetXattr` at ~280 bytes).
/// Saves ~150-300ns per small I/O by eliminating DMA alloc/free + IOMMU map/unmap.
pub const INLINE_IO_MAX: usize = 192;

/// Sentinel value for `DmaBufferHandle` indicating no DMA buffer is
/// allocated. Used by the inline small I/O path: when `buf == ZERO`,
/// data is carried inline in the ring entry (request's `inline_data` for
/// writes, response's `inline_data` for reads). The driver checks
/// `buf == DmaBufferHandle::ZERO` to select the inline path.
impl DmaBufferHandle {
    /// All-zero sentinel indicating no DMA buffer is allocated.
    /// Pool ID 0 is reserved as invalid and never allocated by the DMA
    /// buffer pool allocator ([Section 4.14](04-memory.md#dma-subsystem)). Combined with
    /// `iova_base = 0` (page 0 is never mapped by any IOMMU implementation),
    /// `ZERO` is guaranteed to never match any valid handle.
    pub const ZERO: Self = DmaBufferHandle { pool_id: 0, generation: 0, offset: 0, iova_base: 0 };
}

/// VFS-specific ring buffer. Extends `DomainRingBuffer`
/// ([Section 11.8](11-drivers.md#ipc-architecture-and-message-passing)) with per-slot state tracking
/// for the split reservation protocol (`EMPTY -> RESERVED -> FILLED ->
/// CONSUMED -> EMPTY`) and a per-ring in-flight operation counter for
/// crash recovery quiescence.
///
/// The `DomainRingBuffer` header occupies 128 bytes (2 cache lines).
/// The `slot_states` array is allocated contiguously after the ring data
/// region. The `inflight_ops` counter is used by crash recovery
/// (`drain_all_vfs_rings()`) and live evolution quiescence to wait for
/// producers to complete their current operations before draining.
///
/// **Relationship to DomainRingBuffer**: `RingBuffer<T>` composes (not
/// inherits) `DomainRingBuffer`. All ring pointer fields (`head`, `tail`,
/// `published`, `state`) are accessed through `inner`. The generic
/// parameter `T` is the entry type (`VfsRequest` or `VfsResponseWire`)
/// for type-safe entry access via `read_entry()`.
// kernel-internal, not KABI
pub struct RingBuffer<T> {
    /// Base ring buffer header (128 bytes) + data region.
    /// Contains `head`, `tail`, `published`, `state`, `size`, `entry_size`.
    pub inner: DomainRingBuffer,

    /// Per-slot state for the split reservation protocol.
    /// Length: `inner.size` entries. Allocated contiguously after the
    /// ring data region. Each entry is an `AtomicU8` holding one of
    /// the `RingSlotState` values (`Empty`, `Reserved`, `Filled`,
    /// `Consumed`).
    ///
    /// SAFETY: Pointer is valid for `inner.size` elements. Allocated
    /// from kernel slab at mount time, valid for the lifetime of the
    /// mount. Freed during umount after all rings are drained.
    pub slot_states: *const AtomicU8,

    /// Number of in-flight producer operations on this ring. Incremented
    /// in `reserve_slot()` after successful slot reservation (after CAS
    /// on `inner.head`). Decremented in `complete_slot()` after marking
    /// the slot `FILLED` and advancing `inner.published`.
    ///
    /// Used by crash recovery (`drain_all_vfs_rings()`) and live evolution
    /// quiescence to wait for all producers to complete before draining.
    /// The counter reaching zero guarantees no producer is between
    /// `reserve_slot()` and `complete_slot()` (i.e., no producer is
    /// mid-`copy_from_user` with a RESERVED slot).
    ///
    /// `AtomicU32`: maximum concurrent producers per ring is bounded by
    /// CPU count (≤256 for MAX_VFS_RINGS). `u32` is sufficient.
    pub inflight_ops: AtomicU32,

    /// Phantom type for entry type safety.
    _marker: core::marker::PhantomData<T>,
}

impl<T> RingBuffer<T> {
    /// Read a typed entry at the given slot index.
    ///
    /// SAFETY: `idx` must be < `self.inner.size`. The caller must ensure
    /// the slot contains valid data (slot_state == FILLED or the ring is
    /// being drained during crash recovery with all producers quiesced).
    pub unsafe fn read_entry(&self, idx: usize) -> &T {
        debug_assert!(idx < self.inner.size as usize);
        let data_base = (&self.inner as *const DomainRingBuffer as *const u8)
            .add(size_of::<DomainRingBuffer>());
        &*(data_base.add(idx * self.inner.entry_size as usize) as *const T)
    }

    /// Raw pointer to the entry at `idx` WITHOUT dereferencing it.
    /// Used by the driver-side prefetch path
    /// ([Section 14.3](#vfs-per-cpu-ring-extension--driver-side-ring-entry-prefetch)),
    /// which must not require a valid reference (the slot may be
    /// unpublished — prefetching stale bytes is harmless, forming a
    /// reference to them is not).
    ///
    /// `idx` must be < `self.inner.size` (debug-asserted). The returned
    /// pointer is always within the ring's allocated data region.
    #[inline(always)]
    pub fn entry_ptr(&self, idx: usize) -> *const T {
        debug_assert!(idx < self.inner.size as usize);
        // Same base + idx * entry_size arithmetic as read_entry(), minus
        // the dereference.
        let data_base = (&self.inner as *const DomainRingBuffer as *const u8)
            .wrapping_add(size_of::<DomainRingBuffer>());
        data_base.wrapping_add(idx * self.inner.entry_size as usize) as *const T
    }

    /// Borrow the slot-state atomic for slot `idx`.
    ///
    /// `slot_states` is `*const AtomicU8` — a raw pointer does not support
    /// `[]` indexing, so ALL slot-state accesses (producer reserve/complete,
    /// consumer drain, recovery reset) go through this accessor.
    ///
    /// SAFETY (encapsulated): `slot_states` points to an array of
    /// `inner.size` `AtomicU8` elements allocated at mount time and valid
    /// for the lifetime of the mount; `idx < inner.size` is the caller
    /// invariant, debug-asserted here.
    #[inline(always)]
    pub fn slot_state(&self, idx: usize) -> &AtomicU8 {
        debug_assert!(idx < self.inner.size as usize);
        // SAFETY: see accessor doc — bounded index into a mount-lifetime
        // allocation.
        unsafe { &*self.slot_states.add(idx) }
    }
}

/// Per-mount ring buffer pair for VFS <-> filesystem driver communication.
///
/// The VFS (in umka-evolvable) enqueues requests on `request_ring`; the filesystem
/// driver dequeues, processes, and enqueues responses on `response_ring`.
/// Both rings are in shared memory (PKEY 1 on x86-64 — read-only for both
/// domains; actual data in PKEY 14 shared DMA pool).
pub struct VfsRingPair {
    /// Request ring: VFS -> filesystem driver. Ring size: 256 entries
    /// (configurable per-mount via mount options).
    ///
    /// **Producer model**: Under PerCpu granularity, each ring has exactly
    /// one producer (pure SPSC). Under PerNuma/PerLlc/Fixed granularity,
    /// multiple CPUs may share a ring; the producer side claims `head`
    /// via the guarded position-claim discipline (`reserve_head_claim()`
    /// called by `reserve_slot()`, [Section 14.3](#vfs-per-cpu-ring-extension) —
    /// fullness validated inside the claim window; commit conditional on
    /// an unbroken claim). The consumer side is always single-
    /// threaded per ring (driver consumer thread).
    pub request_ring: RingBuffer<VfsRequest>,

    /// Whether this ring is shared by multiple CPUs. Set at mount time
    /// based on the ring granularity and CPU-to-ring mapping. When `true`,
    /// `reserve_slot()` claims `head` via the guarded position claim
    /// (`reserve_head_claim()`, [Section 14.3](#vfs-per-cpu-ring-extension)). When
    /// `false` (PerCpu mode), `reserve_slot()` uses a
    /// simple load/store on `head` (no contention possible).
    ///
    /// Invariant: `shared_ring == false` implies exactly one CPU maps to
    /// this ring in the `cpu_to_ring` table. This is verified at mount time.
    pub shared_ring: bool, // Kernel-internal, not KABI.

    /// Response ring: filesystem driver -> VFS. SPSC (driver produces, VFS
    /// consumes). Same size as request ring.
    pub response_ring: RingBuffer<VfsResponseWire>,

    /// Doorbell: filesystem driver writes to signal request availability.
    /// Uses the doorbell coalescing mechanism (Section 11.5.1.1) to batch
    /// notifications when multiple requests are enqueued.
    pub doorbell: DoorbellRegister,

    /// Completion WaitQueue: VFS threads wait here when a synchronous
    /// operation needs a response. Multiple threads may be blocked on
    /// the same WaitQueue simultaneously (one per in-flight request).
    ///
    /// **Response matching protocol** (request_id -> waiting thread) —
    /// built on the Core-resident in-flight table (`VfsInflightEntry`,
    /// below). The table is the single source of truth for every request
    /// between slot reservation and response retrieval; recovery trusts
    /// ONLY the table, never driver-writable ring memory.
    ///
    /// 1. **Enroll + submit**: The VFS caller allocates a `request_id`
    ///    from the mount-global `VfsRingSet::next_request_id`
    ///    (AtomicU64, `fetch_add(1, Relaxed)`), reserves a slot
    ///    (`reserve_slot()`, [Section 14.3](#vfs-per-cpu-ring-extension)), inserts a
    ///    `VfsInflightEntry` into the ring's Core-side table
    ///    (`core_side.inflight.store(request_id, entry)`) BEFORE
    ///    `complete_slot()` publishes the slot — so the entry exists
    ///    before any response can arrive — then parks on `completion`
    ///    via `wait_event!` (synchronous operations only; async
    ///    ReadPage/Readahead/WritePage submitters do not park here —
    ///    their completion is executed by the response drain, step 3).
    ///
    /// 2. **Wait condition** (`Fn() -> bool` per the canonical
    ///    `WaitQueueHead::wait_event` contract,
    ///    [Section 3.6](03-concurrency.md#lock-free-data-structures--waitqueuehead-blocking-wait-queue); the condition MUST
    ///    NOT `return` out of the enclosing function — an early return
    ///    would leave the on-stack wait entry enqueued, dangling into a
    ///    dead frame):
    ///    ```rust
    ///    wait_event!(ring.completion,
    ///        ring_set.state.load(Ordering::Acquire) != VFSRS_ACTIVE
    ///            || vfs_inflight_is_completed(core_side, our_request_id));
    ///    // AFTER the wait returns (wait entry dequeued), classify:
    ///    match core_side.inflight.remove(our_request_id) {   // claim-by-remove
    ///        Some(entry) if entry.state.load(Ordering::Acquire)
    ///                == VfsInflightState::Completed as u8 => {
    ///            // Sole owner of the entry: consume entry.response,
    ///            // then free entry (call_rcu) and any DMA handle per
    ///            // the normal completion path.
    ///        }
    ///        Some(entry) => {
    ///            // Recovery/quiescence woke us before a response
    ///            // arrived. We claimed the entry: free
    ///            // entry.dma_handle (if != ZERO) and the entry
    ///            // (call_rcu), return Err(EIO).
    ///        }
    ///        None => {
    ///            // Another owner (crash recovery sweep) already
    ///            // claimed and cleaned the entry. Return Err(EIO).
    ///        }
    ///    }
    ///    ```
    ///    The state check uses `!= VFSRS_ACTIVE` (not `== RECOVERING`)
    ///    so QUIESCING (live evolution) waiters also resolve — matching
    ///    `select_ring()`'s gate. `vfs_inflight_is_completed()` is an
    ///    RCU lookup + `state` Acquire load returning `bool`; the waiter
    ///    holds NO reference to the entry across the sleep (the entry
    ///    may be claimed and RCU-freed by recovery while it sleeps).
    ///
    /// 3. **Response drain** (cooperative + worker; replaces the former
    ///    free-floating "VFS response consumer thread"): the driver
    ///    enqueues `VfsResponseWire` entries and, when the per-ring
    ///    `CompletionCoalescer` fires, calls
    ///    `ring.completion.wake_up_all()` and
    ///    `ring_set.completion_doorbell.notify(ring_index, true)`.
    ///    The response ring's SPSC consumer role is held by whoever
    ///    holds the ring's `core_side.response_drain_lock`:
    ///    - a woken synchronous waiter opportunistically `try_lock`s it
    ///      and drains (lowest latency for the sync case), and
    ///    - the per-mount Core drain worker (`vfs_response_worker`,
    ///      defined below) waits on `completion_doorbell` and drains
    ///      every pending ring — this is what completes ASYNC requests
    ///      that have no parked waiter.
    ///    Draining, per response: verify
    ///    `response.driver_generation == ring_set.driver_generation`
    ///    (stale-instance filter, see Step 3a below); RCU-look up
    ///    `inflight[request_id]`; then:
    ///    - entry absent → stale/duplicate response, drop it;
    ///    - `Abandoned` → the waiter already returned (timeout/cancel):
    ///      claim-by-remove, free `dma_handle`, free entry, drop the
    ///      response;
    ///    - async entry (no parked waiter — `entry.sync_waiter == 0`) →
    ///      execute the Core-side completion for the opcode
    ///      (`vfs_async_complete()`, below), then claim-by-remove + free;
    ///    - sync entry → copy the response into `entry.response`, then
    ///      `entry.state.store(Completed, Release)`. After the batch,
    ///      drop the lock, then `ring.completion.wake_up_all()`
    ///      (two-phase: never wake while holding the drain lock).
    ///
    /// 4. **Retrieval**: exactly one owner removes each entry from the
    ///    table (XArray `remove` is atomic — one winner): the sync
    ///    waiter (step 2), the drain (async/Abandoned, step 3), or the
    ///    crash-recovery table sweep
    ///    ([Section 14.3](#vfs-per-cpu-ring-extension--crash-recovery) Step U6).
    ///    The owner that removes the entry is responsible for freeing
    ///    `entry.dma_handle` (unless the normal completion path already
    ///    consumed and freed the buffer) and for `call_rcu`-freeing the
    ///    entry itself. This single-owner rule is what makes crash
    ///    recovery double-free-proof: an entry present in the table ⇔
    ///    its DMA handle has not been freed.
    pub completion: WaitQueue,

    /// Per-ring completion coalescing state (driver side batches
    /// response-direction wakeups). Defined in
    /// [Section 14.3](#vfs-per-cpu-ring-extension--completion-coalescing-response-direction).
    pub completion_coalescer: CompletionCoalescer,
}
// NOTE: the former per-ring `response_table: XArray<VfsResponseWire>` and
// per-ring `next_request_id` fields are GONE. Request IDs are mount-global
// (`VfsRingSet::next_request_id`, [Section 14.3](#vfs-per-cpu-ring-extension)), and
// response matching moved to the Core-resident in-flight table below —
// VfsRingPair pages are mapped read-write into the driver's domain, and a
// matching table living in driver-writable memory could be corrupted by
// the very crash it must recover from.

/// State of one in-flight cross-domain VFS request.
/// Values stored in `VfsInflightEntry.state` (AtomicU8).
#[repr(u8)]
pub enum VfsInflightState {
    /// Submitted; response not yet drained. A synchronous submitter is
    /// parked on `ring.completion`; an async submitter is not parked.
    InFlight = 0,
    /// Response copied into `entry.response` by the response drain
    /// (synchronous entries only). The waiter claims the entry by
    /// removing it from the table.
    Completed = 1,
    /// The synchronous waiter gave up (per-request timeout, or caller
    /// cancellation via signal/thread-exit) and already returned to
    /// userspace. The entry MUST stay in the table: the driver may still
    /// write into the request's DMA buffer and will eventually respond
    /// (`-ECANCELED` or a normal result — cancellation protocol step f).
    /// The response drain (on the late response) or the crash-recovery
    /// sweep claims the entry and frees `dma_handle`.
    Abandoned = 2,
}

/// Cause of a ring-level VFS request failure, distinguishing a genuine
/// provider-returned errno from a failure INDUCED by crash recovery /
/// evolution quiescence — the request's ring was drained while the request
/// was in flight, or a waiter observed a non-ACTIVE ring set and its entry
/// was claimed by the U6/U7 sweep ([Section 14.3](#vfs-per-cpu-ring-extension)).
///
/// The distinction is INTERNAL. Syscall adapters flatten ALL THREE to a
/// userspace errno — `Provider(e)` -> `e`, `LostByRecovery` and
/// `GenerationBoundary` -> `EIO` (the `-EIO` in-flight drain contract above) —
/// so the userspace contract is unchanged. Internal consumers that can retry —
/// notably open-file revalidation ([Section 14.1](#virtual-filesystem-layer)) — use the
/// distinction to avoid latching an fd dead over a recovery-induced loss:
/// `LostByRecovery` and `GenerationBoundary` are both propagated WITHOUT
/// latching (sleep on `sb.recovery_wait`, retry the next-generation instance),
/// whereas `Provider(errno)` is a real terminal provider error that ALWAYS
/// latches — no per-errno exceptions.
pub enum VfsFailureCause {
    /// The filesystem provider returned this errno from the (reloaded)
    /// instance. ALWAYS latches — a real terminal provider verdict, with NO
    /// per-errno carve-outs.
    Provider(Errno),
    /// The in-flight request was lost to crash recovery / quiescence: its
    /// ring was drained (non-ACTIVE ring set, entry claimed by the U6/U7
    /// sweep) AFTER submission, before a provider response could be matched.
    /// Never latched by internal consumers; retryable against the next
    /// generation.
    LostByRecovery,
    /// The transport rejected the request at the ring boundary BEFORE it
    /// reached a provider: `select_ring()` returned its transport-level
    /// `ENXIO` because the ring set was non-ACTIVE (recovery/quiescence
    /// re-entered ahead of the generation bump) — an instance-generation
    /// boundary, not a provider result. Never latched; the dispatch wrapper
    /// sleeps on `sb.recovery_wait` and retries against the next generation.
    GenerationBoundary,
}

/// Core-resident record of one in-flight cross-domain VFS request,
/// tracked from slot reservation to response retrieval. Mirrors the
/// block layer's per-device in-flight bio table
/// (`BlkInflightTable`, [Section 15.2](15-storage.md#block-io-and-volume-management--per-device-in-flight-bio-table)):
/// every value that recovery, cancellation, or completion needs is
/// captured HERE, in Core memory, at submit time — recovery never reads
/// (and never trusts) driver-writable ring memory.
///
/// Allocation: `VFS_INFLIGHT_SLAB` (slab objects are hot-path-legal per
/// the collection policy). Freed via `call_rcu` by whichever owner
/// claims the entry (the response drain holds only RCU-lifetime
/// references while writing `response`/`state`).
// kernel-internal, not KABI — lives in Core-only memory; no driver
// domain has a mapping for it. Native field types are fine.
pub struct VfsInflightEntry {
    /// Ring the request was submitted on. Routes `vfs_cancel()`'s
    /// `CancelToken` to the correct ring's cancellation side-channel.
    pub ring_index: u16,
    /// 1 = a synchronous waiter is (or was) parked on `ring.completion`;
    /// 0 = async submission (ReadPage/Readahead/WritePage) — the
    /// response drain executes the completion itself.
    pub sync_waiter: u8,
    /// Entry state: `VfsInflightState` value. Written with Release,
    /// read with Acquire (the Release store of `Completed` publishes
    /// the `response` payload written just before it).
    pub state: AtomicU8,
    /// Opcode — recovery classification (ReadPage/Readahead entries get
    /// inline terminal page state — PageFlags::ERROR + unlock_page — as
    /// they are claimed in Step U6; see [Section 14.3](#vfs-per-cpu-ring-extension)).
    pub opcode: VfsOpcode,
    /// Core-captured copy of the request's DMA buffer handle
    /// (`DmaBufferHandle::ZERO` if the request carries none). The
    /// claiming owner frees it exactly once (see retrieval rule above).
    pub dma_handle: DmaBufferHandle,
    /// Pin on the target inode — an `Arc<Inode>` captured by the Core
    /// dispatch path at submit time. Pins the inode (and its embedded
    /// `i_mapping` AddressSpace) until the entry reaches a terminal
    /// state; this is the recovery path's ONLY route to the page cache
    /// (`entry.inode.i_mapping.page_cache`), and it REPLACES the deleted
    /// raw `page_cache_id` (a cast-to-u64 AddressSpace pointer). A pin,
    /// not an address: a corrupted ring cannot redirect it, and there is
    /// no lifetime gap for concurrent eviction to free the AddressSpace
    /// under a raw deref. Dropping the entry drops the pin.
    pub inode: Arc<Inode>,
    /// First page index (ReadPage/Readahead; 0 otherwise).
    pub page_index: u64,
    /// Page count (Readahead batch size; 1 for ReadPage; 0 otherwise).
    pub nr_pages: u32,
    /// Typed single-page fill obligation, moved off the faulter's `FillLease`
    /// at submit (`lease.into_inflight()` → `FillCompletion`, [Section 4.4](04-memory.md#page-cache))
    /// by the Core dispatch that enrolls this entry. `Some` for demand
    /// `ReadPage` fills ONLY; `None` for `Readahead` batch entries (their pages
    /// carry no per-entry obligation — the completion publishes their flags
    /// directly) and for every non-fill opcode. `Cell` because the completion /
    /// crash-claim path consumes it (`fill.take()`) through a shared
    /// `&VfsInflightEntry`; discharge is EXACTLY ONCE via `FillCompletion`'s
    /// `complete_ok`/`complete_err` terminal bodies — the single place content
    /// state (`UPTODATE`/`ERROR`) is published on the physical `Page.flags`.
    pub fill: Cell<Option<FillCompletion>>,
    /// Explicit pad to align `response` (`#[repr(C, align(256))]`).
    pub _pad: [u8; 4],
    /// Response payload, written by the response drain for synchronous
    /// entries before the `Completed` Release store.
    ///
    /// `UnsafeCell` because the payload is written through a shared
    /// `&VfsInflightEntry` (the drain holds only an RCU-lifetime reference,
    /// never `&mut`). SAFETY contract: the SOLE writer is the drainer holding
    /// the ring's `response_drain_lock` and observing `state == InFlight`;
    /// readers touch it ONLY after the subsequent `state.store(Completed,
    /// Release)` they pair with via Acquire. This is sound interior mutability;
    /// a const→mut cast would be UB.
    pub response: UnsafeCell<VfsResponseWire>,
}

// SAFETY: `VfsInflightEntry` is shared across CPUs (RCU lookups in the
// per-ring `inflight` XArray), so it must be `Sync`; the
// `UnsafeCell<VfsResponseWire>` and `Cell<Option<FillCompletion>>` fields
// otherwise make it `!Sync`. Both are accessed under strict single-consumer
// discipline: `response` by the sole writer holding `response_drain_lock` (then
// read only after the `Completed`-Release/Acquire), and `fill` consumed exactly
// once — by the async completion (drain-lock held, entry `InFlight`) or by the
// crash-claim path that has already removed the entry from the table. No two
// actors touch either field concurrently.
unsafe impl Sync for VfsInflightEntry {}

/// Core-only per-ring companion state, allocated at mount time as an
/// array parallel to `VfsRingSet.rings` (index = ring index) and owned
/// by the superblock (`SuperBlock.ring_core`,
/// [Section 14.1](#virtual-filesystem-layer)). NOT mapped into any driver domain —
/// this is the state crash recovery trusts.
// kernel-internal, not KABI.
pub struct VfsRingCoreSide {
    /// In-flight request table for this ring, keyed by request_id (u64).
    /// XArray per the integer-key rule; per-ring sharding bounds
    /// contention (PerCpu mode: single inserter per ring).
    ///
    /// **Hot-path cost + compensation**: one XArray `store` at submit
    /// and one `remove` at retrieval per cross-domain request — the SAME
    /// two locked table operations the deleted per-ring `response_table`
    /// performed (its store happened at response time instead); the only
    /// addition is one lockless RCU lookup per response (~10-20 cycles)
    /// on a path that already costs a domain crossing. Net ≈ 0, and the
    /// table replaces three whole mechanisms: the in-slot CANCEL flag
    /// machinery, the trusted-ring-walk recovery extraction, and the
    /// `is_core_memory_range()` pointer-validation defense.
    pub inflight: XArray<VfsInflightEntry>,
    /// Serializes the SPSC consumer role on this ring's RESPONSE ring
    /// between opportunistic sync-waiter drains, the per-mount response
    /// worker, and the crash-recovery reset (U7(a)).
    ///
    /// Subsystem-internal leaf SpinLock (not in the global level table,
    /// same discipline as `DomainDescriptor.entry_lock`): nothing is
    /// acquired under it — drain does ring pops (atomics), RCU lookups,
    /// and entry-payload stores only; entry insert/remove (XA_LOCK) and
    /// all wakeups happen OUTSIDE the lock (two-phase discipline).
    /// Holders MUST check `ring_set.state == VFSRS_ACTIVE` under the
    /// lock before touching the response ring and back off otherwise —
    /// this is how recovery (which holds the lock across the U7 ring
    /// reset) excludes every other consumer without thread quiescence.
    /// Recovery also uses an acquire+immediate-release handoff of this
    /// lock as the U5b drainer barrier before the U6 sweep: because every
    /// drainer holds it for its whole pass and re-checks `ring_set.state`
    /// under it, the handoff proves no drain pass is in progress or can
    /// start, so the U6 sweep is the sole terminal-side-effect actor.
    pub response_drain_lock: SpinLock<()>,

    /// Armed-waiter count for producer quiescence on THIS ring. Non-zero
    /// only while a recovery/evolution quiescence wait is parked on
    /// `producer_quiesce_wq` (armed by `wait_until_quiesced`, disarmed when
    /// it returns). `complete_slot()` consults it (Acquire) after the
    /// `inflight_ops` decrement that reaches zero and, if armed, wakes the
    /// queue below — a plain `AtomicU8`, not `AtomicBool`, per the
    /// repr(C)-adjacent-flag convention. Core-only (never mapped to a
    /// driver); the repr(C) `VfsRingSet` and its `const_assert` are
    /// deliberately NOT touched — a `WaitQueue` cannot live in that
    /// pointer-width-independent ABI struct.
    pub quiesce_waiters: AtomicU8,

    /// Sleep target for producer quiescence on THIS ring. `wait_until_quiesced`
    /// sleeps here (canonical `wait_event_timeout`) until `inflight_ops == 0`
    /// for this ring or the window elapses; `complete_slot()` wakes it on the
    /// `inflight_ops` 1→0 edge WHEN `quiesce_waiters != 0`. Lost-wake-proof:
    /// the arm store (Release) precedes the wait, the decrement's wake reads
    /// `quiesce_waiters` (Acquire), and `wait_event_timeout` re-checks the
    /// predicate after being queued, so a decrement that races the arm is
    /// caught by the re-check. Replaces the former `core::hint::spin_loop()`
    /// busy-wait (a SCHED_FIFO-90 spinner that starved normal-priority
    /// producers on a single CPU). Core-only, non-KABI.
    pub producer_quiesce_wq: WaitQueue,
}

/// Per-mount Core response drain worker (`umkad-vfsresp-<mount>`), the
/// defined successor of the formerly-unspecified "VFS response consumer
/// thread". Created at mount (after ring negotiation), exits at umount.
/// Runs entirely in Core; NEVER enters the driver domain; is NOT
/// registered in `DomainDescriptor.consumer_threads` (it is not a
/// driver-side consumer) and does not need quiescing during recovery —
/// the `response_drain_lock` + state gate excludes it.
fn vfs_response_worker(sb: &SuperBlock) {
    let ring_set = sb.ring_set.as_ref().expect("worker only for ring mounts");
    loop {
        ring_set.completion_doorbell.doorbell.wait();
        // Umount signal: set by the umount path (which then kicks
        // completion_doorbell) before the ring_core array is freed.
        if sb.ring_worker_exit.load(Ordering::Acquire) != 0 {
            kthread_exit();
        }
        let pending = ring_set.completion_doorbell.take_pending();
        for word_idx in 0..4 {
            let mut bits = pending[word_idx];
            while bits != 0 {
                let ring_idx = (word_idx * 64) + bits.trailing_zeros() as usize;
                bits &= bits - 1;
                vfs_drain_response_ring(sb, ring_idx as u16);
            }
        }
    }
}

/// Drain one ring's response ring into the in-flight table. Called by
/// the worker above and opportunistically (try_lock) by woken
/// synchronous waiters. Batch-copies under the lock, wakes after.
fn vfs_drain_response_ring(sb: &SuperBlock, ring_idx: u16) {
    let ring_set = sb.ring_set.as_ref().unwrap();
    // SAFETY: ring_idx < ring_count (pending bits are set only for
    // valid rings).
    let ring = unsafe { &*ring_set.rings.add(ring_idx as usize) };
    // ring_core is Some whenever ring_set is Some (mount step 4a).
    let core_side = &sb.ring_core.as_ref().unwrap()[ring_idx as usize];

    let mut wake_needed = false;
    {
        let _g = core_side.response_drain_lock.lock();
        // Recovery gate: during RECOVERING/QUIESCING the recovery worker
        // (or evolution) owns the rings; back off. Checked under the
        // lock so the U7(a) holder observes no concurrent consumer.
        if ring_set.state.load(Ordering::Acquire) != VFSRS_ACTIVE {
            return;
        }
        let mut tail = ring.response_ring.inner.tail.load(Ordering::Acquire);
        let published = ring.response_ring.inner.published.load(Ordering::Acquire);
        while tail != published {
            let idx = (tail & (ring.response_ring.inner.size as u64 - 1)) as usize;
            // SAFETY: idx within ring bounds; we are the sole consumer
            // (drain lock held).
            let resp: &VfsResponseWire = unsafe { ring.response_ring.read_entry(idx) };
            // Stale-instance filter (Step 3a): responses stamped with a
            // pre-crash generation are discarded.
            if resp.driver_generation
                == ring_set.driver_generation.load(Ordering::Acquire)
            {
                let _rcu = rcu_read_lock();
                match core_side.inflight.load(resp.request_id) {
                    None => { /* stale/duplicate response — drop */ }
                    Some(entry) => {
                        let st = entry.state.load(Ordering::Acquire);
                        if st == VfsInflightState::Abandoned as u8 {
                            // Late response for a timed-out/cancelled
                            // request: claim outside RCU (below).
                            vfs_inflight_reap(core_side, resp.request_id);
                        } else if entry.sync_waiter == 0 {
                            // Async completion: execute it, then claim.
                            vfs_async_complete(entry, resp);
                            vfs_inflight_reap(core_side, resp.request_id);
                        } else {
                            // Sync: publish payload, then state. The
                            // Release store pairs with the waiter's
                            // Acquire load in step 2.
                            // SAFETY: entry payload writes are exclusive
                            // to the drain-lock holder while state ==
                            // InFlight.
                            unsafe { vfs_inflight_write_response(entry, resp); }
                            entry.state.store(
                                VfsInflightState::Completed as u8,
                                Ordering::Release,
                            );
                            wake_needed = true;
                        }
                    }
                }
            }
            tail = tail.wrapping_add(1);
        }
        ring.response_ring.inner.tail.store(tail, Ordering::Release);
    } // drain lock dropped
    if wake_needed {
        ring.completion.wake_up_all(); // two-phase: wake after unlock
    }
}

/// Claim-by-remove + cleanup for an entry the DRAIN owns (Abandoned or
/// async). `remove()` is atomic — if the crash-recovery sweep won the
/// race, this is a no-op.
fn vfs_inflight_reap(core_side: &VfsRingCoreSide, request_id: u64) {
    if let Some(entry) = core_side.inflight.remove(request_id) {
        // The CLAIMER owns the DMA free — always, exactly once. The
        // completion helpers (`vfs_complete_read_pages`,
        // `vfs_complete_writepage`) copy data OUT of the buffer but MUST
        // NOT free it; the sync-waiter path and the crash-recovery sweep
        // follow the same rule for their claims.
        if entry.dma_handle != DmaBufferHandle::ZERO {
            dma_pool_free(entry.dma_handle);
        }
        rcu_free_inflight_entry(entry); // call_rcu — drainers may still
                                        // hold RCU references
    }
}

/// Wait-condition helper for synchronous waiters: RCU lookup + state
/// check, returning bool (never a reference — the entry may be claimed
/// and RCU-freed by recovery while the waiter sleeps).
fn vfs_inflight_is_completed(core_side: &VfsRingCoreSide, id: u64) -> bool {
    let _rcu = rcu_read_lock();
    match core_side.inflight.load(id) {
        Some(entry) => entry.state.load(Ordering::Acquire)
            == VfsInflightState::Completed as u8,
        None => false, // claimed by recovery — waiter's post-wait match
                       // handles the None arm (EIO)
    }
}

/// Copy a drained response into a sync entry's payload slot.
///
/// # Safety
/// Caller holds the ring's `response_drain_lock` and observed
/// `entry.state == InFlight` — the drain-lock holder is the only writer
/// of `response` while InFlight, and the waiter reads it only after the
/// subsequent `Completed` Release store (Acquire pairing).
unsafe fn vfs_inflight_write_response(
    entry: &VfsInflightEntry,
    resp: &VfsResponseWire,
) {
    // `UnsafeCell::get()` yields the `*mut` soundly — no const→mut cast. The
    // drain-lock + `InFlight` precondition (see the `response` SAFETY contract)
    // make this the sole writer; the caller's `Completed` Release publishes it.
    core::ptr::write(entry.response.get(), core::ptr::read(resp));
}

/// Core-side completion execution for async (no-waiter) requests.
/// Runs in the drain context (process context, sleepable).
fn vfs_async_complete(entry: &VfsInflightEntry, resp: &VfsResponseWire) {
    match entry.opcode {
        // Page-populating reads: on success mark the page(s) uptodate
        // and unlock; on error set PageFlags::ERROR and unlock. Waiters
        // on wait_on_page_locked() wake and observe the outcome
        // ([Section 4.4](04-memory.md#page-cache)).
        VfsOpcode::ReadPage | VfsOpcode::Readahead => {
            vfs_complete_read_pages(entry, resp.status);
        }
        // Reclaim single-page writeback: funnel through the tier-agnostic
        // epilogue ([Section 15.2](15-storage.md#block-io-and-volume-management--writeback-io-completion-callback)).
        VfsOpcode::WritePage => {
            let errno = if resp.status < 0 { resp.status as i32 } else { 0 };
            vfs_complete_writepage(entry, errno);
        }
        // All other opcodes are synchronous (sync_waiter == 1) and never
        // reach this function; a driver protocol violation lands here.
        _ => {
            log_fma_warning!(
                "vfs_async_complete: unexpected async completion opcode {:?}",
                entry.opcode
            );
        }
    }
}

/// Async completion for page-populating reads (`ReadPage`/`Readahead`).
///
/// Resolves the target page(s) through the entry's inode pin
/// (`entry.inode.i_mapping`) — NEVER a raw pointer read back from the ring, so a
/// corrupted driver cannot redirect the wake, and the pin keeps the embedded
/// AddressSpace live across the deref — using `page_index`/`nr_pages`, and
/// publishes the outcome to page-cache waiters: on success (`status >= 0`) set
/// `UPTODATE` and unlock; on error set `ERROR` (leaving `UPTODATE` clear so
/// subsequent access returns `-EIO`) and unlock. `unlock_page()` wakes any task
/// parked in `wait_on_page_locked()`. This copies the completion status OUT of
/// the response but MUST NOT free the DMA buffer — the in-flight claimer
/// (`vfs_inflight_reap`) owns that, exactly once.
///
/// Runs in the response-drain context (process context, sleepable). Mirrors the
/// recovery-time inline page resolution in Step U6
/// ([Section 14.3](#vfs-per-cpu-ring-extension)).
fn vfs_complete_read_pages(entry: &VfsInflightEntry, status: i64) {
    let ok = status >= 0;
    // Demand `ReadPage` fills carry a linear `FillCompletion`: discharge it
    // through its terminal body — the SINGLE place content state (UPTODATE /
    // ERROR) is published on the physical `Page.flags` ([Section 4.4](04-memory.md#page-cache)) — then
    // return. The completion owns its page's unlock+wake, so there is NO direct
    // flag write here for a fill-carrying entry (it would be a double publish).
    if let Some(fc) = entry.fill.take() {
        // Revoke the FS domain's DMA grant for this page BEFORE publishing it
        // readable — the FS domain must lose DMA access before the page becomes
        // readable (grant lifetime = enrollment lifetime,
        // [Section 14.2](#vfs-ring-buffer-protocol--ringmount-addressspaceops-provider)).
        // Orthogonal to the in-flight reap's buffer reclaim below: revoke drops
        // the FS's DMA ACCESS (generation bump); the reap returns the slot.
        // Same-domain providers have no grant table (`dma_grant_table()` is
        // `None`) and skip this.
        if let Some(grant) = entry.inode.i_mapping.ops.dma_grant_table() {
            grant.revoke(entry.dma_handle);
        }
        if ok { fc.complete_ok(); } else { fc.complete_err(Errno::EIO); }
        return;
    }
    // Readahead batch entries carry NO per-entry `FillCompletion` — publish
    // content state directly on each physical `Page.flags` (still exactly one
    // place per page — these pages have no fill obligation). The inode pin keeps
    // the embedded i_mapping AddressSpace live across this deref — no raw
    // pointer, no cast, no lifetime gap.
    let address_space = &entry.inode.i_mapping;
    // Page-populating reads only target regular files (page_cache == Some).
    let Some(page_cache) = address_space.page_cache.as_ref() else { return };
    // RCU: the page may be concurrently evicted between lookup and flag update.
    let _rcu = rcu_read_lock();
    for i in 0..entry.nr_pages as u64 {
        let Some(page_entry) = page_cache.pages.load(entry.page_index + i) else {
            continue; // evicted or never inserted — no waiter can exist
        };
        // Content state on the PHYSICAL frame's `Page.flags` — the single
        // authority; the dirty-scan index is the XArray's own per-node
        // `XA_TAG_DIRTY` tag bitmap, not a slot field.
        let page = page_entry.page.page();
        if ok {
            page.flags_fetch_or(PageFlags::UPTODATE, Ordering::Release);
        } else {
            page.flags_fetch_or(PageFlags::ERROR, Ordering::Release);
        }
        unlock_page(page); // clears LOCKED, wakes waiters
    }
}

/// Async completion for reclaim single-page writeback (`WritePage`).
///
/// Resolves the single target page through the entry's inode pin
/// (`entry.inode.i_mapping`) + `page_index` and funnels it through the
/// tier-agnostic writeback epilogue
/// ([Section 15.2](15-storage.md#block-io-and-volume-management--writeback-io-completion-callback)),
/// which clears `WRITEBACK` (completion owns only the WRITEBACK→done teardown;
/// `DIRTY` and `nr_dirty` were cleared at submission on the DIRTY 1→0 edge,
/// [Section 4.6](04-memory.md#writeback-subsystem) D1-A), preserves a `DIRTY` bit set by a concurrent
/// redirty during the writeback round-trip, may re-dirty the page on a
/// retryable error, records any error in the mapping's `wb_err`, and wakes
/// writeback waiters. `errno` is 0 on success or a negative errno on failure.
/// Does NOT free the DMA buffer (the in-flight claimer does).
///
/// Runs in the response-drain context (process context, sleepable).
fn vfs_complete_writepage(entry: &VfsInflightEntry, errno: i32) {
    // The inode pin keeps the embedded i_mapping AddressSpace live across this
    // deref — no raw pointer, no cast (see `vfs_complete_read_pages`).
    let address_space = &entry.inode.i_mapping;
    let Some(page_cache) = address_space.page_cache.as_ref() else { return };
    let _rcu = rcu_read_lock();
    if let Some(page_entry) = page_cache.pages.load(entry.page_index) {
        writeback_page_epilogue(page_entry.page.page(), errno);
    }
    // Page not found: already evicted after writeback was queued — nothing
    // to wake (no thread can wait on a page that left the cache).
}

/// VFS request message. Serialized representation of a trait method call.
/// Fixed-size header + variable-length payload.
///
/// **Layout**: The header fields (`request_id`, `opcode`, `ino`, `fh`) are
/// followed by the tagged-union `args` payload. The `opcode` field is `u32`
/// (from `VfsOpcode`); `_pad_opcode` provides explicit padding to maintain
/// natural alignment of the subsequent `u64` fields. This prevents
/// information disclosure via implicit compiler-inserted padding bytes.
///
/// **Size**: Header is 32 bytes + `VfsRequestArgs` (largest variant is
/// `SetXattr` at ~280 bytes). Total entry size ~320 bytes (see per-CPU ring
/// extension memory analysis). `const_assert!` below verifies the header.
#[repr(C)]
pub struct VfsRequest {
    /// Unique request ID for matching responses. Globally unique per mount
    /// (allocated from `VfsRingSet::next_request_id`). IDs are unique but
    /// not necessarily monotonic within a single ring — two CPUs sharing a
    /// ring may allocate IDs before ring slot assignment, so a lower ID can
    /// appear in a later slot. The protocol uses IDs for response matching
    /// only, not ordering. u64 counter: at 10M ops/sec, wraps after
    /// ~58,000 years (well beyond the 50-year uptime target). No wrap
    /// handling needed.
    pub request_id: u64,

    /// Operation code identifying the trait method.
    pub opcode: VfsOpcode,

    /// Explicit padding after the u32 `opcode` to align `ino` to 8 bytes.
    /// Must be zero. Prevents information disclosure from implicit padding.
    pub _pad_opcode: u32,

    /// Inode number (for InodeOps/FileOps calls). 0 for FileSystemOps calls.
    pub ino: u64,

    /// File handle (for FileOps calls). u64::MAX for non-file operations.
    pub fh: u64,

    /// Operation-specific arguments. The variant must match `opcode`.
    /// Variable-length data (filenames, xattr values, write data) is
    /// passed via shared DMA buffer references embedded in the variant,
    /// not stored inline in the ring entry.
    ///
    /// The VFS dispatcher validates that the `args` variant matches
    /// `opcode` before dispatching; a mismatch is a kernel bug and
    /// triggers a panic in debug builds, a silent no-op error response
    /// in release builds.
    pub args: VfsRequestArgs,
}
// Verify header layout: request_id(8) + opcode(4) + pad(4) + ino(8) + fh(8) = 32.
const_assert!(core::mem::offset_of!(VfsRequest, args) == 32);
// Verify total size: header(32) + VfsRequestArgs(288, largest variant SetXattr) = 320.
// VfsRequestArgs = 4 (discriminant) + 256 (KernelString) + 4 (padding) + 16 (DmaBufferHandle)
//                + 4 (value_len) + 4 (flags) = 288 bytes, aligned to 8.
//
// Memory tradeoff: 320 bytes per entry × 256 entries × N rings = ~80 KiB per ring.
// At 64 CPUs × 256 entries = ~5 MiB per mount. The inline_data optimization
// (192 bytes embedded in the Write variant) eliminates DMA alloc + IOMMU mapping
// for small I/O (saving ~150-300 ns per operation). Ring memory is pinned DMA
// pages pre-allocated from the shared DMA pool — these pages are committed
// regardless of entry size and cannot be used for other purposes.
const_assert!(core::mem::size_of::<VfsRequest>() == 320);

/// Per-opcode argument payload for a `VfsRequest`.
///
/// `#[repr(C, u32)]` tagged union: the discriminant is a `u32` matching
/// `VfsOpcode`, and each variant is an independent `#[repr(C)]` struct.
/// This ensures a stable ABI across the Tier 0 / Tier 1 KABI boundary
/// (zero-copy ring, matching the io_uring SQE pattern). The `VfsRequest.opcode`
/// field in the header serves as the authoritative discriminant; the
/// in-union discriminant is redundant but guarantees Rust's safety
/// invariant (no invalid discriminant UB).
///
/// Every `VfsOpcode` variant has a corresponding `VfsRequestArgs` variant
/// with the exact parameters that the trait method requires. Variants
/// that carry no extra data beyond what is already in the `VfsRequest`
/// header (opcode, ino, fh) use an empty body `{}`.
///
/// **Inline string limits**: `KernelString` holds up to 255 bytes. Names
/// longer than 255 bytes (possible on some exotic filesystems) must be
/// passed via a `DmaBufferHandle` placed in the `buf` field of the
/// relevant variant; the VFS sets the string `len` to 0 as a sentinel in
/// that case.
///
/// **Caller contract**: The caller fills `VfsRequest { opcode, args, .. }`
/// and enqueues it on `request_ring`. The VFS dispatcher validates that
/// the `args` variant matches `opcode` before dispatching to the
/// filesystem driver.
#[repr(C, u32)]
pub enum VfsRequestArgs {
    // ---------------------------------------------------------------
    // FileSystemOps
    // ---------------------------------------------------------------

    /// `FileSystemOps::mount`. No extra args; mount options are passed
    /// via a separate `DmaBufferHandle` in the ring header.
    Mount {},
    /// `FileSystemOps::unmount`. Graceful unmount; all dirty data must
    /// be flushed before the response is sent.
    Unmount {},
    /// `FileSystemOps::force_unmount`. Best-effort: abandon in-flight
    /// I/O and free resources.
    ForceUnmount {},
    /// `FileSystemOps::statfs`. No per-call arguments.
    Statfs {},
    /// `FileSystemOps::sync_fs`. `wait` controls whether the driver
    /// must block until all I/O is complete (`true`) or may return once
    /// I/O is queued (`false`).
    SyncFs { wait: u8 }, // 0 = no-wait, 1 = wait. u8 for cross-domain safety.
    /// `FileSystemOps::remount`. New flags; updated option string is in
    /// a `DmaBufferHandle` in the ring header.
    Remount { flags: u32 },
    /// `FileSystemOps::freeze`. Quiesce all writes for snapshotting.
    Freeze {},
    /// `FileSystemOps::thaw`. Resume writes after a freeze.
    Thaw {},

    // ---------------------------------------------------------------
    // InodeOps
    // ---------------------------------------------------------------

    /// `InodeOps::lookup`. Look up `name` in the directory identified
    /// by `VfsRequest::ino`.
    Lookup { name: KernelString },
    /// `InodeOps::create`. Create a regular file named by the dentry
    /// already allocated by the VFS. `mode` is the combined file-type
    /// and permission bits.
    Create { mode: FileMode },
    /// `InodeOps::link`. Create a hard link whose new name is
    /// `new_name` inside the directory inode of the request.
    Link { src_ino: u64, new_name: KernelString },
    /// `InodeOps::unlink`. Remove a directory entry. The inode is freed
    /// when its link count reaches zero and all file descriptors are
    /// closed.
    Unlink { name: KernelString },
    /// `InodeOps::mkdir`. Create a directory with the given permission
    /// bits.
    Mkdir { mode: FileMode },
    /// `InodeOps::rmdir`. Remove an empty directory.
    Rmdir { name: KernelString },
    /// `InodeOps::rename`. Move or rename a directory entry.
    /// `new_dir_ino` is the inode number of the destination directory.
    /// `new_name` is the destination name. `flags` carries `RENAME_*`
    /// constants (e.g., `RENAME_NOREPLACE`, `RENAME_EXCHANGE`).
    Rename { new_dir_ino: u64, new_name: KernelString, flags: u32 },
    /// `InodeOps::symlink`. Create a symbolic link whose target path is
    /// `target`. The created inode is named by the dentry pre-allocated
    /// by the VFS.
    Symlink { target: KernelString },
    /// `InodeOps::readlink`. Resolve the symlink target into
    /// `buf`. The driver writes the target string into the DMA buffer
    /// identified by `buf`.
    Readlink { buf: DmaBufferHandle },
    /// `InodeOps::mknod`. Create a special file (block device, character
    /// device, FIFO, or socket). `dev` carries the (major, minor) pair
    /// using the `DevId` type with Linux MKDEV encoding: `(major << 20) | minor`.
    /// See [Section 14.5](#device-node-framework) for encoding details.
    Mknod { mode: FileMode, dev: DevId },
    /// `InodeOps::getattr`. Retrieve inode attributes into an
    /// `InodeAttr`. `request_mask` is a bitmask of `STATX_*` fields the
    /// caller wants. `flags` is `AT_*` flags from `statx(2)`.
    GetAttr { request_mask: u32, flags: u32 },
    /// `InodeOps::setattr`. Modify inode attributes. `valid` is a
    /// bitmask of `ATTR_*` flags indicating which fields in `attr` the
    /// driver must update.
    SetAttr { attr: InodeAttr, valid: u32 },
    /// `InodeOps::truncate`. Set the file size to `size` bytes,
    /// releasing or zero-extending as needed.
    Truncate { size: u64 },
    /// `InodeOps::getxattr`. Retrieve the extended attribute `name` into
    /// `buf`. On return, the response `status` field (>= 0) carries the
    /// attribute value length.
    GetXattr { name: KernelString, buf: DmaBufferHandle },
    /// `InodeOps::setxattr`. Set extended attribute `name` to `value`.
    /// `flags` is `XATTR_CREATE`, `XATTR_REPLACE`, or 0.
    SetXattr { name: KernelString, value: DmaBufferHandle, value_len: u32, flags: u32 },
    /// `InodeOps::listxattr`. Enumerate all extended attribute names into
    /// `buf` as a sequence of NUL-terminated strings. On return, the
    /// response `status` field (>= 0) carries the total length written.
    ListXattr { buf: DmaBufferHandle },
    /// `InodeOps::removexattr`. Delete the extended attribute `name`.
    RemoveXattr { name: KernelString },
    /// `InodeOps::fileattr_get`. Read the persistent `FS_*_FL` attribute
    /// word of the inode named by `VfsRequest::ino`. The response
    /// carries the `FileAttr` record; drivers without persistent flags
    /// respond `ENOTTY` ([Section 14.1](#virtual-filesystem-layer--inode-attribute-flags)).
    FileattrGet {},
    /// `InodeOps::fileattr_set`. Persist a new `FS_*_FL` attribute word.
    /// Core performed every permission check (owner/CAP_FOWNER,
    /// CAP_LINUX_IMMUTABLE, EROFS, freeze bracket) in
    /// `vfs_fileattr_set()` before enqueuing; the driver validates only
    /// filesystem support (unsupported bits → `EOPNOTSUPP`, nonzero
    /// reserved words → `EINVAL`).
    FileattrSet { attr: FileAttr },
    /// `FileSystemOps::show_options`. Write the filesystem-specific
    /// mount options (as they would appear in `/proc/mounts`) into
    /// `buf`.
    ShowOptions { buf: DmaBufferHandle },

    // ---------------------------------------------------------------
    // AddressSpaceOps (page cache → filesystem driver)
    // ---------------------------------------------------------------

    /// `AddressSpaceOps::read_page`. Populate one page from backing
    /// store on a page cache miss. `page_index` is the page-aligned
    /// file offset divided by `PAGE_SIZE`. The driver reads data from
    /// the backing block device and writes it into the DMA buffer
    /// identified by `buf` (exactly `PAGE_SIZE` bytes). The page has
    /// already been allocated and inserted into the page cache by the
    /// caller; the driver only needs to fill it.
    ///
    /// `buf` PROVENANCE: for a cross-domain ring mount, `buf` is minted by
    /// `FsDmaGrantTable::grant_page` — the already-cache-inserted page frame
    /// granted to the FS domain for the lifetime of this fill and revoked at
    /// the response drain (see
    /// [Section 14.2](#vfs-ring-buffer-protocol--ringmount-addressspaceops-provider)). No
    /// other provenance is legal for a ring mount.
    ///
    /// `page_cache_id`: Core-resident handle identifying the
    /// `AddressSpace`/`PageCache` that owns this page. Set by the VFS
    /// dispatch path (in Core, Tier 0) before enqueuing. Used by crash
    /// recovery to resolve orphaned pages WITHOUT traversing VFS-domain
    /// state (the inode cache is in VFS/Tier 1 and may be corrupted).
    /// The handle is the `AddressSpace` pointer cast to `u64` — valid
    /// because `AddressSpace` is in Core memory and pinned for the
    /// lifetime of the superblock.
    ReadPage { page_index: u64, page_cache_id: u64, buf: DmaBufferHandle },
    /// `AddressSpaceOps::readahead`. Batch read for the readahead
    /// engine ([Section 4.4](04-memory.md#page-cache--readahead-engine)). `start_index` is
    /// the first page index; `nr_pages` is the count. The driver
    /// should submit I/O for the entire range in a single Bio batch.
    /// `buf` is a DMA buffer large enough for `nr_pages * PAGE_SIZE`
    /// bytes. Pages have been pre-allocated and cache-inserted by the
    /// readahead engine; the driver fills them sequentially.
    /// Filesystems that do not implement batched readahead return
    /// `EOPNOTSUPP`; the VFS falls back to per-page `ReadPage` calls.
    ///
    /// `page_cache_id`: Same semantics as `ReadPage::page_cache_id`.
    /// All pages in the readahead batch belong to the same
    /// `AddressSpace`.
    Readahead { start_index: u64, nr_pages: u32, page_cache_id: u64, buf: DmaBufferHandle },
    /// `AddressSpaceOps::writepage`. Write a single dirty page to
    /// the backing store. Used by the page reclaimer when it needs to
    /// evict a dirty page. For normal writeback, the `WritebackRequest`
    /// ring ([Section 4.6](04-memory.md#writeback-subsystem--writeback-cross-domain-dispatch))
    /// is used instead (batched, higher throughput). `writepage` is
    /// the single-page fallback for reclaim pressure.
    WritePage { page_index: u64, buf: DmaBufferHandle, sync_mode: u8 },
    /// `AddressSpaceOps::dirty_extent`. Notify the filesystem that a
    /// page range is about to be dirtied. The filesystem records the
    /// affected extent for crash-recovery journaling. `offset` and
    /// `len` are byte offsets within the file.
    DirtyExtent { offset: u64, len: u64 },
    /// `AddressSpaceOps::releasepage`. Ask the filesystem whether a
    /// clean page may be evicted from the cache. The driver responds
    /// with `ok = true` (permit eviction) or `ok = false` (page is
    /// pinned for journaling or other reasons). Must not block.
    ReleasePage { page_index: u64 },

    // ---------------------------------------------------------------
    // FileOps
    // ---------------------------------------------------------------

    /// `FileOps::open`. Open the file. `flags` are the `O_*` open
    /// flags from `open(2)`/`openat(2)`. `mode` is relevant only when
    /// `O_CREAT` is set.
    Open { flags: u32, mode: FileMode },
    /// `FileOps::release`. The last reference to this open file
    /// descriptor has been closed. The driver must flush any cached
    /// state for `fh`.
    Release {},
    /// `FileOps::read`. Read up to `count` bytes starting at `offset`
    /// from the file into `buf`. The driver writes data into the DMA
    /// buffer identified by `buf`. On return, `VfsResponseWire::status`
    /// (>= 0) carries the number of bytes actually read.
    ///
    /// **Inline small I/O path**: If `count <= INLINE_IO_MAX` (192 bytes),
    /// the VFS sets `buf` to `DmaBufferHandle::ZERO` (sentinel: no DMA
    /// buffer allocated). The driver writes read data into the response's
    /// `inline_data` field instead of a DMA buffer. This eliminates DMA
    /// alloc/free + IOMMU map/unmap for small reads (procfs, sysfs, small
    /// config files). Saves ~150-300ns per small read. Covers >90% of
    /// procfs/sysfs reads. See `VfsResponseWire::inline_data`.
    Read { buf: DmaBufferHandle, offset: u64, count: u32 },
    /// `FileOps::write`. Write `count` bytes from `buf` into the file
    /// starting at `offset`. `buf` points to a DMA buffer the VFS has
    /// already filled with the data to be written.
    ///
    /// **Inline small I/O path**: If `count <= INLINE_IO_MAX` (192 bytes),
    /// the VFS places write data inline in `inline_data` and sets `buf` to
    /// `DmaBufferHandle::ZERO` (sentinel). The driver reads from
    /// `inline_data` instead of a DMA buffer. The `copy_from_user()` that
    /// fills `inline_data` happens after slot reservation but before
    /// `complete_slot()` — enabled by the split reservation/completion
    /// protocol ([Section 14.3](#vfs-per-cpu-ring-extension)).
    Write { buf: DmaBufferHandle, offset: u64, count: u32,
            inline_data: [u8; INLINE_IO_MAX] },
    /// `FileOps::fsync`. Flush dirty data and metadata to stable
    /// storage. If `datasync` is `true`, only data blocks need to be
    /// flushed (equivalent to `fdatasync(2)`). `start`..`end` is the
    /// byte range to sync; `end == u64::MAX` means "to end of file".
    Fsync { datasync: u8, start: u64, end: u64 }, // 0 = fsync, 1 = fdatasync. u8 for cross-domain safety.
    /// `FileOps::readdir`. Enumerate directory entries into `buf`
    /// starting after the position identified by `cookie`. A `cookie` of
    /// 0 means start from the beginning. The driver fills `buf` with
    /// `linux_dirent64` records. `VfsResponseWire::status` (>= 0) carries
    /// the number of bytes written.
    ReadDir { buf: DmaBufferHandle, cookie: u64 },
    /// `FileOps::ioctl`. Pass a device-specific command to the
    /// filesystem driver. `cmd` is the ioctl number; `arg` is the raw
    /// usize argument (may be a user pointer, a small integer, or a
    /// `DmaBufferHandle` depending on the command).
    Ioctl { cmd: u32, arg: usize },
    /// `FileOps::mmap`. Establish a memory mapping — the RING-TRANSPORT form of
    /// the decomposed `FileOps::mmap(inode, private, offset, len, vm_flags) ->
    /// MmapResult` ([Section 14.1](#virtual-filesystem-layer)). This variant carries the
    /// decomposed arguments verbatim: the inode is the enclosing
    /// `VfsRequest::ino`, so it is not repeated here; `private` is the
    /// filesystem-private context from `open()`; `offset`/`len` are the mapping
    /// range; `vm_flags` is the kernel-internal `VmFlags` bits (mapping type +
    /// VM_MAY* mask + PROT-derived bits) — the SAME encoding `MmapResult.vm_flags`
    /// returns. The driver's `MmapResult` (adjusted `vm_flags` + `vm_ops_handle`)
    /// travels back in the response envelope — see `VfsResponseWire`'s Mmap
    /// packing. This op ONLY runs the filesystem's `mmap` callback; PTE
    /// installation on subsequent faults is a SEPARATE mechanism (the per-VMA
    /// custom-fault / `FaultHandlerRequest` ring, [Section 4.8](04-memory.md#virtual-memory-manager)),
    /// NOT an opaque vma_token callback from this op. Tier-agnostic:
    /// `kabi_call!` selects direct-vtable (`mmap_direct`) or this ring transport
    /// at bind time, so one protocol serves VFS at Tier 0/1/2.
    Mmap { private: u64, offset: u64, len: u64, vm_flags: u64 },
    /// `FileOps::fallocate`. Pre-allocate or manipulate storage for the
    /// given byte range. `mode` carries `FALLOC_FL_*` flags.
    Fallocate { mode: u32, offset: u64, len: u64 },
    /// `FileOps::seek_data`. Find the next byte range containing data
    /// at or after `offset` (implements `SEEK_DATA` from `lseek(2)`).
    SeekData { offset: u64 },
    /// `FileOps::seek_hole`. Find the next hole (unallocated range) at
    /// or after `offset` (implements `SEEK_HOLE` from `lseek(2)`).
    SeekHole { offset: u64 },
    /// `FileOps::poll`. Query which I/O events are ready. `events` is
    /// a bitmask of `POLLIN`, `POLLOUT`, `POLLERR`, etc. The driver
    /// responds immediately with the currently ready events; the VFS
    /// handles `epoll`/`select` wait registration separately.
    Poll { events: u32 },
    /// `FileOps::splice_read`. Transfer up to `len` bytes from the file
    /// at `offset` into an in-kernel pipe identified by `pipe_ino`,
    /// without copying through userspace. `flags` carries `SPLICE_F_*`
    /// flags.
    SpliceRead { pipe_ino: u64, offset: u64, len: u32, flags: u32 },
    /// `FileOps::splice_write`. Transfer up to `len` bytes from the
    /// in-kernel pipe identified by `pipe_ino` into the file at
    /// `offset`. `flags` carries `SPLICE_F_*` flags.
    SpliceWrite { pipe_ino: u64, offset: u64, len: u32, flags: u32 },

    // ---------------------------------------------------------------
    // Inode lifecycle operations
    // ---------------------------------------------------------------

    /// `InodeOps::evict_inode`. Inode teardown — disposition-split. Sent after
    /// the VFS has completed page cache teardown (`evict()` step 4). The inode
    /// number is in `VfsRequest::ino`; `disposition` is the wire form of
    /// `EvictDisposition` (0 = Release, 1 = Delete — any other value is a
    /// protocol error, request rejected with EINVAL). The canonical trait and
    /// enum live in [Section 14.1](#virtual-filesystem-layer--inode-lifecycle-and-page-cache-teardown).
    /// - `disposition == 0` (Release, `i_nlink > 0` — LRU reclaim, umount of a
    ///   still-linked file): free ONLY the driver's in-memory per-inode state.
    ///   The driver MUST NOT touch on-disk allocation.
    /// - `disposition == 1` (Delete, `i_nlink == 0`): additionally release all
    ///   on-disk resources (extent tree entries, block allocations, journal
    ///   reservations, inode-table entry) associated with the inode.
    /// The response is `VfsResponse::Ok(0)` on success or
    /// `VfsResponse::Err(-errno)` on failure (which is logged but does not
    /// prevent inode freeing — the VFS continues eviction regardless).
    EvictInode { disposition: u8 },

    /// `InodeOps::truncate_range`. Deallocate blocks within
    /// `[offset, offset+len)` without changing file size.
    /// Used by `FALLOC_FL_PUNCH_HOLE`, `FALLOC_FL_ZERO_RANGE`, and
    /// `FALLOC_FL_COLLAPSE_RANGE`. Separate from `Truncate` (which sets
    /// `i_size` via `setattr`). The VFS evicts page cache pages in the
    /// affected range before sending this request.
    TruncateRange { offset: u64, len: u64 },

    /// `InodeOps::write_inode`. Flush inode metadata to stable storage.
    /// Called by `vfs_fsync_metadata()` for O_SYNC/O_DSYNC writes when the
    /// inode's on-disk metadata must be updated (timestamps, size, block map).
    /// `sync_mode`: 0 = `WriteSyncMode::Async` (schedule I/O, don't wait),
    /// 1 = `WriteSyncMode::Sync` (wait for I/O completion). u8 for cross-domain safety.
    WriteInode { sync_mode: u8 },

    // ---------------------------------------------------------------
    // Batched metadata operations (io_uring coalescing)
    // ---------------------------------------------------------------

    /// Batched `statx()` request. Generated by the io_uring dispatch path
    /// when consecutive `IORING_OP_STATX` SQEs are detected. The VFS
    /// resolves all paths in a single domain stay. Never sent by
    /// filesystem drivers.
    /// See [Section 14.1](#virtual-filesystem-layer--mechanism-2-iouring-statx-coalescing).
    StatxBatch { count: u8, entries_buf: DmaBufferHandle },

    // ---------------------------------------------------------------
    // Protocol-internal
    // ---------------------------------------------------------------

    /// Aborted-slot sentinel. Written by `abort_slot()`
    /// ([Section 14.3](#vfs-per-cpu-ring-extension)) when a producer must release a
    /// RESERVED slot it can no longer fill (`fill_slot_data()` failed
    /// with EFAULT, a fatal signal interrupted the fill, or the
    /// post-reservation state re-check observed RECOVERING/QUIESCING).
    /// A RESERVED slot cannot revert to EMPTY (later slots may already
    /// be FILLED past it, and EMPTY inside `[tail..published)` would
    /// stall the consumer's tail advance), so abort = fill-with-Nop:
    /// the slot completes normally and all ring invariants hold.
    ///
    /// Driver contract: respond `Ok(0)` immediately, with NO side
    /// effects and NO I/O. The producer never enrolls a Nop in the
    /// in-flight table, so the response drain finds no entry and drops
    /// the response — by design.
    Nop {},
}

/// Bounded kernel-internal string. Avoids heap allocation for the common
/// case of short names (directory entries, xattr names, symlink targets
/// ≤ 255 bytes).
///
/// For strings longer than 255 bytes the caller must use a
/// `DmaBufferHandle` instead and set `len = 0` as a sentinel.
#[repr(C)]
pub struct KernelString {
    /// Byte length of the string, not including any NUL terminator.
    /// Range: 0 (sentinel for "use DMA buffer") to 255.
    pub len: u8,
    /// Inline storage. Valid bytes are `data[..len]`. The remainder
    /// is zero-padded. Not NUL-terminated; callers must use `len`.
    pub data: [u8; 255],
}
// Layout: 1 + 255 = 256 bytes.
const_assert!(size_of::<KernelString>() == 256);

/// VFS operation codes. One-to-one mapping to trait methods.
#[repr(u32)]
pub enum VfsOpcode {
    // FileSystemOps
    Mount = 1,
    Unmount = 2,
    ForceUnmount = 3,
    Statfs = 4,
    SyncFs = 5,
    Remount = 6,
    Freeze = 7,
    Thaw = 8,
    ShowOptions = 37,    // → FileSystemOps::show_options; called by /proc/mounts, mount(8)

    // InodeOps
    Lookup = 20,
    Create = 21,
    Link = 22,
    Unlink = 23,
    Mkdir = 24,
    Rmdir = 25,
    Rename = 26,
    Symlink = 27,
    Readlink = 28,
    Getattr = 29,
    Setattr = 30,
    Truncate = 35,
    Getxattr = 31,
    Setxattr = 32,
    Listxattr = 33,
    Removexattr = 34,
    Mknod = 36,          // → InodeOps::mknod; called by mknod(2) for device nodes
    EvictInode = 38,     // → InodeOps::evict_inode; inode teardown (Release = in-memory only, Delete = also on-disk)
    TruncateRange = 39,  // → InodeOps::truncate_range; FALLOC_FL_PUNCH_HOLE/ZERO_RANGE
                         //   Separate from Truncate (35) which sets i_size via setattr.
                         //   TruncateRange deallocates blocks within [offset, offset+len)
                         //   without changing file size.
    WriteInode = 55,     // → InodeOps::write_inode; flush inode metadata to stable storage
                         //   Called by vfs_fsync_metadata() for O_SYNC/O_DSYNC writes.
    FileattrGet = 56,    // → InodeOps::fileattr_get; FS_IOC_GETFLAGS/lsattr backend
                         //   ([Section 14.1](#virtual-filesystem-layer--inode-attribute-flags)).
    FileattrSet = 57,    // → InodeOps::fileattr_set; FS_IOC_SETFLAGS/chattr backend.
                         //   All permission checks ran Core-side in vfs_fileattr_set()
                         //   before this request was enqueued.

    // AddressSpaceOps (page cache ↔ filesystem)
    ReadPage = 60,       // → AddressSpaceOps::read_page; page cache miss
    Readahead = 61,      // → AddressSpaceOps::readahead; batched readahead
    WritePage = 62,      // → AddressSpaceOps::writepage; reclaim single-page writeback
    DirtyExtent = 63,    // → AddressSpaceOps::dirty_extent; journal pre-registration
    ReleasePage = 64,    // → AddressSpaceOps::releasepage; reclaim eviction check

    // FileOps
    Open = 40,
    Release = 41,
    Read = 42,
    Write = 43,
    Fsync = 44,
    Readdir = 45,
    Ioctl = 46,
    Mmap = 47,
    Fallocate = 48,
    SeekData = 49,
    SeekHole = 50,
    Poll = 51,
    SpliceRead = 52,     // → FileOps::splice_read; called by splice(2), sendfile(2)
    SpliceWrite = 53,    // → FileOps::splice_write; called by splice(2) write side

    // Batched metadata operations (io_uring coalescing)
    // These opcodes are generated only by the io_uring statx coalescing path
    // ([Section 14.1](#virtual-filesystem-layer--mechanism-2-iouring-statx-coalescing)).
    // They are never exposed to filesystem drivers directly — the VFS
    // dispatches individual Getattr calls internally for each batch entry.
    StatxBatch = 70,       // → Batched statx; args in DmaBufferHandle as StatxBatchEntry[]
    StatxBatchResult = 71, // → Response-only opcode (no VfsRequestArgs variant). Carries
                         //   per-entry StatxBuf or error as a batched response payload.
                         //   Used only by VFS internal response routing; never sent on
                         //   the request ring.

    // Protocol-internal
    Nop = 72,            // → Aborted-slot sentinel (see VfsRequestArgs::Nop).
                         //   Driver responds Ok(0), no side effects; the
                         //   response matches no in-flight entry and is dropped.
}

/// VFS response message — wire-level representation on the response ring.
///
/// Every request placed on the `request_ring` eventually produces exactly one
/// `VfsResponseWire` on the paired `response_ring`. The `request_id` field
/// matches the request it completes, enabling out-of-order completion.
///
/// **Status encoding**: `status` is a signed 64-bit value.
/// - `status >= 0`: Success. For data-transfer operations (`Read`, `Write`,
///   `Readdir`, `ReadPage`, `Readahead`, `SpliceRead`, `SpliceWrite`), the
///   value is the byte count transferred. For operations that return a new
///   inode (`Lookup`, `Create`, `Mkdir`, `Symlink`, `Mknod`), the value
///   is the new inode number. For all other operations, the value is 0.
/// - `status == -4095..-1`: Error. The negated Linux errno (e.g., `-2` for
///   `ENOENT`). Matches the kernel's standard error encoding.
/// - `status == i64::MIN` (`0x8000_0000_0000_0000`): Pending — the driver
///   has acknowledged the request but not yet completed it. The VFS must
///   continue waiting for the final response. At most one `Pending` response
///   per request is permitted.
///
/// **Size**: Header is 40 bytes (8 + 8 + 8 + 8 + 4 + 4). For responses
/// carrying inline read data (small I/O path), `inline_data` adds up to
/// `INLINE_IO_MAX` (192) bytes. Total response entry: 256 bytes
/// (40 header + 192 inline_data + 24 padding, aligned to 256 for cache
/// efficiency on response ring). For responses without inline data
/// (large I/O, non-read operations), `inline_data_len` is 0 and the
/// consumer can skip the inline data region.
#[repr(C, align(256))]
pub struct VfsResponseWire {
    /// Request ID this response completes. Matches `VfsRequest::request_id`.
    pub request_id: u64,

    /// Driver generation counter at the time this response was produced.
    /// The response drain discards responses whose generation does not
    /// match the current mount generation (stale responses from a
    /// pre-crash driver instance). See Step 3a below.
    ///
    /// **Where the driver gets the value**: the isolated driver cannot
    /// read `SuperBlock.driver_generation` (Core-only memory). Core
    /// maintains a read-mostly mirror in the driver-mapped ring set —
    /// `VfsRingSet.driver_generation`
    /// ([Section 14.3](#vfs-per-cpu-ring-extension--ring-topology)) — written at
    /// mount init and bumped together with the SuperBlock counter in
    /// recovery Step U10a. The driver stamps
    /// `ring_set.driver_generation.load(Acquire)` into every response
    /// at construction time (contract stated in the `vfs_init()` doc).
    pub driver_generation: u64,

    /// Status code: >= 0 for success (byte count or inode number),
    /// negative for error (negated errno), `i64::MIN` for Pending.
    pub status: i64,

    /// Operation-specific supplementary data. Currently used by:
    /// - `Lookup`/`Create`/`Mkdir`/`Symlink`/`Mknod`: inode generation
    ///   counter in `aux[0]` (u32, for NFS file handle staleness detection).
    /// - `Open`: the `OpenOutcome` ([Section 14.1](#virtual-filesystem-layer)) split
    ///   across three fields — `private` in `status` (u64, stored by VFS in
    ///   `OpenFile::private_data`); the resolved data inode's NUMBER in
    ///   `aux` (`aux[0]` = low 32 bits, `aux[1]` = high 32 bits, the same
    ///   split `Mmap` uses below); and the layer that inode lives on in
    ///   `aux2[0]`. `aux2[0] == 0` means `data_inode: None` — no rebinding,
    ///   and `aux` is ignored.
    /// - `GetAttr`: `STATX_*` result mask in `aux[0]`.
    /// - `ReleasePage`: `aux[0]` = 1 if eviction is permitted, 0 if denied.
    /// - `Mmap`: the filesystem's `MmapResult` — adjusted `vm_flags` in `status`
    ///   (u64, `>= 0` on success; `VmFlags` bits never exceed `i64::MAX`), and
    ///   `vm_ops_handle` split across `aux` (`aux[0]` = low 32 bits, `aux[1]` =
    ///   high 32 bits). The caller-side ring stub reconstructs
    ///   `MmapResult { vm_flags, vm_ops_handle }` and applies it to the VMA. Same
    ///   status-carries-a-u64 packing `Open` uses; a negative `status` is the
    ///   error errno as for every op.
    /// - All other operations: `aux` is zero.
    pub aux: [u32; 2],

    /// Number of valid bytes in `inline_data`. 0 for non-inline responses.
    /// Range: 0..=INLINE_IO_MAX (192). When > 0, the VFS copies
    /// `inline_data[..inline_data_len]` directly to userspace, bypassing
    /// the DMA buffer entirely.
    pub inline_data_len: u32,

    /// Padding after inline_data_len to maintain 8-byte alignment.
    pub _pad_len: u32,

    /// Inline read data for small I/O responses. Used when the original
    /// `Read` request had `count <= INLINE_IO_MAX` and `buf == ZERO`.
    /// The filesystem driver writes read data here instead of into a DMA
    /// buffer. Eliminates DMA alloc/free + IOMMU map/unmap for small reads.
    /// For non-inline responses, this region is unused (content undefined).
    pub inline_data: [u8; INLINE_IO_MAX],

    /// Second supplementary word pair, carved out of the response tail (the
    /// padding it replaces was never carrying anything). Offset 232, natural
    /// `u32` alignment. Currently used by:
    /// - `Open`: the stacking-filesystem data-inode binding.
    ///   `aux2[0]` = layer designator + 1, so 0 means NO rebinding
    ///   (`OpenOutcome::data_inode` is `None`, which is every non-stacking
    ///   filesystem). The designator indexes the mount's Core-side ORDERED
    ///   LAYER TABLE — a mount-scoped Core structure, sibling of
    ///   `Mount.ring_aspace_ops`
    ///   ([Section 14.6](#mount-tree-data-structures-and-operations)), recorded when
    ///   Core resolved the stacking mount's layer paths at mount time. The
    ///   filesystem domain can therefore name only the layers its own mount
    ///   declared; an out-of-range designator is a protocol violation and
    ///   fails the open. `aux2[1]` is reserved and MUST be zero.
    /// - All other operations: `aux2` is zero.
    pub aux2: [u32; 2],

    /// Padding to fill the 256-byte struct size mandated by `#[repr(C, align(256))]`.
    /// 8 + 8 + 8 + 8 + 4 + 4 + 192 + 8 = 240 bytes of fields. 256 - 240 = 16 bytes pad.
    /// The `align(256)` attribute ensures each response entry is cache-line-aligned
    /// and power-of-two sized for efficient ring indexing (index × 256 = byte offset).
    pub _pad: [u8; 16],
}
const_assert!(core::mem::size_of::<VfsResponseWire>() == 256);

Open response reconstruction (Core side). The ring stub that completes an Open rebuilds the very OpenOutcome the same-domain path returns: private from status; then, when aux2[0] != 0, it resolves layer aux2[0] - 1 in the mount's ordered layer table, looks the inode number carried in aux up in THAT layer superblock's inode_cache (instantiating through the layer filesystem's inode-get path on a miss), and hands the result back as data_inode. aux2[0] == 0 yields data_inode: None. open_and_install (Section 14.1) then applies the outcome exactly as it does for a same-domain filesystem — the transport is invisible to the binding. The layer table is Core-owned and built at MOUNT time, when Core resolves the stacking mount's layer paths: the isolated filesystem never supplies a superblock pointer, only an index into what its own mount declared, so a compromised filesystem domain cannot name a layer outside its mount.

14.2.1 RingMount AddressSpaceOps Provider

The cross-domain arm of the page-cache demand-fill seam (dispatch_read_page, Section 14.1) is realized by ONE generic provider, RingMountAddressSpaceOps — mount-scoped (one instance per ring mount, OWNED by the mount in Mount.ring_aspace_ops, Section 14.6) and generic over EVERY ring-backed filesystem. Per-filesystem AddressSpaceOps impls for ring mounts are FORBIDDEN: a single audited seam, no 50-year drift. The ring op set is UNCHANGED — ReadPage already carries everything; what this section legislates is buf's PROVENANCE (a mount-scoped DMA grant) and the transport binding.

Bind-grant DMA window. Each ring mount owns an FsDmaGrantTable — a mount-scoped bind-grant window established as part of the mount's KABI bind-grant (Section 12.8) — through which Core grants the filesystem domain per-fill DMA access to exactly the cache pages it must populate, and no others. The FS domain can DMA only pages Core has granted, each for the lifetime of one fill, generation-checked against reuse.

/// Mount-scoped bind-grant DMA window. Inserts already-cache-inserted page
/// frames into the mount's grant table for the FS domain to DMA into, and
/// revokes them when the fill completes. Bounds exposure: the FS domain never
/// sees a page Core did not grant; a revoked grant's `DmaBufferHandle` fails
/// the generation check ([Section 16.5](16-networking.md#netbuf-packet-buffer)).
pub struct FsDmaGrantTable {
    /// Base of the mount's bind-grant DMA region (device-visible), minted by
    /// the KABI bind-grant at mount bind ([Section 12.8](12-kabi.md#kabi-domain-runtime)).
    region_iova: u64,
    /// Pool id identifying this window among the mount's DMA pools — the
    /// `DmaBufferHandle::pool_id` that grants carry.
    pool_id: u16,
    /// Per-slot generation counters — the ABA guard bumped by `revoke` so a
    /// stale handle fails validation ([Section 16.5](16-networking.md#netbuf-packet-buffer)). Length =
    /// window slot count, fixed at mount bind.
    generations: Box<[AtomicU16]>,
}

impl FsDmaGrantTable {
    /// Insert `page`'s frame into this mount's grant table and mint the
    /// `DmaBufferHandle` naming it (the `generation` bits are the opaque-handle
    /// ABA guard). The FS domain can DMA into this page until the matching
    /// `revoke`. `Err` (window full / frame unmappable) enrolls nothing.
    pub fn grant_page(&self, page: &Page) -> Result<DmaBufferHandle, IoError>;

    /// Revoke the grant named by `buf`, bumping the slot generation so the FS
    /// domain loses DMA access to the page. Idempotent for an already-revoked
    /// handle. Called from the response drain (or the crash/abort arms) BEFORE
    /// the page is published readable.
    pub fn revoke(&self, buf: DmaBufferHandle);
}

/// The ONE generic cross-domain `AddressSpaceOps` provider. Constructed at
/// mount bind (`Mount::resolve_aspace_ops`, [Section 14.1](#virtual-filesystem-layer))
/// with the mount's ring transport handle and its bind-grant DMA window.
pub struct RingMountAddressSpaceOps {
    /// The mount's bind-time transport handle (its VFS ring).
    ring: KabiServiceHandle,
    /// The mount's bind-grant DMA window.
    grant: FsDmaGrantTable,
}

impl AddressSpaceOps for RingMountAddressSpaceOps {
    /// Cross-domain demand fill: grant the page, enroll the obligation, submit
    /// `ReadPage`. Completion is ASYNCHRONOUS — the response drain
    /// (`vfs_complete_read_pages`) discharges `fill` and revokes the grant. The
    /// provider NEVER completes `fill` on the success path.
    fn read_page_async(
        &self,
        mapping: &AddressSpace,
        index: u64,
        page: &Page,
        fill: FillCompletion,
    ) -> Result<(), (FillCompletion, IoError)> {
        // (a) Grant the FS domain DMA access to this already-cache-inserted
        //     page and mint the ReadPage `buf`. On failure NOTHING is enrolled;
        //     the obligation returns to the caller unchanged.
        let buf = match self.grant.grant_page(page) {
            Ok(buf) => buf,
            Err(e) => return Err((fill, e)),
        };
        // (b)+(c) Enroll a `VfsInflightEntry`, MOVE `fill` into `entry.fill`
        //     (the :396 contract — the obligation leaves the faulter's lease at
        //     submit), and submit `ReadPage { page_index: index, page_cache_id,
        //     buf }` on `self.ring` via the protocol's existing submission path
        //     (the mount-dispatch flow below). `fill` is consumed on `Ok`.
        //     On enrollment/submit failure BEFORE the entry is live, `fill` is
        //     returned undischarged — undo the grant and hand it back.
        vfs_ring_submit_readpage(&self.ring, mapping, index, buf, fill)
            .map_err(|(fill, e)| {
                self.grant.revoke(buf);
                (fill, e)
            })
    }

    /// Expose the mount-scoped grant table so the ring response drain
    /// (`vfs_complete_read_pages`) can revoke a page's grant before publishing
    /// it readable. This is the ONE override of the trait's `None` default.
    fn dma_grant_table(&self) -> Option<&FsDmaGrantTable> {
        Some(&self.grant)
    }
}

/// Enroll a `VfsInflightEntry` for a cross-domain `ReadPage`, MOVE `fill` into
/// `entry.fill` (the :396 contract), and submit
/// `ReadPage { page_index: index, page_cache_id, buf }` on `ring` via the
/// protocol's existing submission path (the mount-dispatch flow below).
/// `page_cache_id` is `mapping`'s Core-resident handle (`&AddressSpace as u64`,
/// per the `ReadPage::page_cache_id` doc). On `Ok(())` the enrolled entry owns
/// the obligation and completion is asynchronous. On enrollment/submit failure
/// BEFORE the entry is live, `fill` is returned undischarged in the `Err`
/// payload (the caller undoes the grant). Exactly one entry is enrolled per
/// call; this is the single named entry to the existing submission path.
fn vfs_ring_submit_readpage(
    ring: &KabiServiceHandle,
    mapping: &AddressSpace,
    index: u64,
    buf: DmaBufferHandle,
    fill: FillCompletion,
) -> Result<(), (FillCompletion, IoError)>;

Grant lifetime = enrollment lifetime. The grant is minted at submit and revoked exactly once — at the response drain (success or error) or at an enrollment/submit abort — never held by any other owner. Discharge of the fill is owned exclusively by vfs_complete_read_pages, which revokes the grant BEFORE it publishes the page readable (the FS domain must lose DMA access before the page becomes readable). The crash/teardown arm (the in-flight crash-drain) likewise revokes the grant and completes the fill with EIO.

Dispatch flow (read syscall example):

1. Userspace calls read(fd, buf, len).
2. Syscall entry point resolves fd to a ValidatedCap (Section 9.1).
3. VFS checks the page cache (Section 4.4). On cache HIT: data is served from core memory with zero domain crossings. On cache MISS: continue.
4. VFS constructs a VfsRequest:
5. Large I/O (count > INLINE_IO_MAX): { opcode: Read, buf: DmaBufferHandle, offset, count }. The buf is a DmaBufferHandle pointing to a shared-memory region where the driver will write the read data (zero-copy).
6. Small I/O (count <= INLINE_IO_MAX): { opcode: Read, buf: DmaBufferHandle::ZERO, offset, count }. No DMA buffer is allocated. The driver writes data into VfsResponseWire::inline_data.
7. VFS reserves a slot, enrolls a VfsInflightEntry in the ring's Core-side in-flight table (capturing the DMA handle and the target inode as an Arc<Inode> pin — the recovery/completion route to the page cache via inode.i_mapping), fills the slot, completes it, and rings the doorbell. Enrollment precedes complete_slot() so the entry exists before any response can arrive.
8. The filesystem driver (in its Tier 1 domain) dequeues the request. It checks buf == DmaBufferHandle::ZERO to select the path:
9. DMA path: reads via BlockDevice, writes data to the shared DMA buffer.
10. Inline path: reads from page cache or block device, writes data into VfsResponseWire::inline_data[..inline_data_len] and sets inline_data_len.
11. Driver enqueues a VfsResponseWire { request_id, status, inline_data_len, ... } on response_ring.
12. VFS dequeues the response:
13. DMA path: populates the page cache, copies data from DMA buffer to userspace.
14. Inline path: copies inline_data[..inline_data_len] directly to userspace. No DMA buffer to free, no IOMMU unmap. Saves ~150-300ns per small read.

Key design properties:

• Page cache absorbs most I/O: Only cache misses cross the domain boundary. On a warm cache (common for frequently accessed files), read() has zero domain crossings — data is served directly from core memory. This is why the page cache lives in umka-nucleus, not in the filesystem driver.
• Zero-copy data path: Read/write data is transferred via shared DMA buffer handles, not copied into the ring buffer. The ring carries only the metadata (opcode, offsets, lengths, buffer handles). Data pages are in the shared DMA pool (PKEY 14 / domain 2).
• Batching: The doorbell coalescing mechanism (Section 11.5.1.1) batches multiple requests into a single domain switch. readahead() enqueues multiple read requests before ringing the doorbell once.
• Trait interface as specification: The FileSystemOps, InodeOps, FileOps, and AddressSpaceOps traits defined in Section 14.1 serve as the SPECIFICATION of the ring protocol. Each trait method maps to exactly one VfsOpcode. The trait signatures define the arguments; the ring protocol serializes them into VfsRequestArgs. Filesystem driver developers implement the traits; the KABI code generator (Section 12.1) produces the serialization/deserialization stubs.

VFS Ring Error Handling and Cancellation:

Every cross-domain VFS request is subject to timeout, cancellation, and driver crash handling. This section specifies the complete lifecycle of a request that does not complete normally.

1. Timeout: Every VFS request has a per-operation timeout based on the expected latency class of the operation:

Timeout class	Operations	Default timeout
Regular	Read, Write, Stat, Lookup, Create, Open, Release, Getattr, Setattr, Readdir, Readlink, Link, Unlink, Mkdir, Rmdir, Rename, Symlink, Getxattr, Setxattr, Listxattr, Removexattr, FileattrGet, FileattrSet, Mmap, SeekData, SeekHole, Poll, Ioctl	30 seconds
Slow	Fsync, Truncate, Fallocate	120 seconds
Mount	Mount, Unmount, ForceUnmount, Remount, Statfs, SyncFs, Freeze, Thaw	300 seconds

The kernel VFS layer starts a per-request timer when the request is enqueued on the request_ring. If the timer fires before a VfsResponse::Ok or VfsResponse::Err arrives on the response_ring, the kernel performs the following steps:

a. Marks the request's Core-resident in-flight entry Abandoned (entry.state.store(VfsInflightState::Abandoned as u8, Release) via an RCU lookup in core_side.inflight). The entry is NOT removed: the driver may still write into the request's DMA buffer and will eventually respond — the response drain reaps the Abandoned entry (and frees its DMA handle) when that late response arrives, or the crash-recovery sweep reaps it if the driver dies first. Nothing is written to shared ring metadata — the in-slot state byte carries RingSlotState only. b. Returns ETIMEDOUT to the waiting syscall (waking the blocked thread via the VfsRingPair::completion wait queue). c. Enqueues a CancelToken { request_id, reason: CancelReason::Timeout } on the cancellation side-channel of the ring recorded in entry.ring_index, so the filesystem driver can detect the cancellation and avoid processing a stale request. The driver is expected to check the cancellation channel before beginning I/O for each dequeued request.

Timeouts are per-mount configurable via mount options (vfs_timeout_regular=<secs>, vfs_timeout_slow=<secs>, vfs_timeout_mount=<secs>). The values above are defaults.

2. Crash handling (filesystem driver crashes): When a Tier 1 filesystem driver crashes (detected by the isolation recovery mechanism described in Section 11.6), the kernel VFS layer performs the following recovery sequence:

Step content superseded: the four steps below are the mount-level SUMMARY; their content is normative only as refined by the unified sequence Steps U3-U18 in Section 14.3. In particular, the rings are NEVER unmapped on crash and the remount is AUTOMATIC — an earlier revision of this section specified "unmap the shared ring pages, mark the VfsRingPair defunct, manual remount required," which contradicted the ring-reuse design (U7 reset + U14(c') key re-arm) and was removed.

a. New operations are rejected first (ring_set.state = RECOVERING at Step U3 → select_ring() returns ENXIO), then all pending requests are failed with EIO: every thread blocked on VfsRingPair::completion for that mount is woken (Step U7(b)), observes the non-ACTIVE state in its wait condition, claims-or-loses its in-flight entry, and returns EIO. b. The rings are RETAINED, drained, and reset in place (Steps U6-U7): the ring memory allocation is kernel-owned and survives the crash; only the crashed domain's ACCESS to it was revoked (Step U2), and Step U14(c') re-arms the retained key for the reloaded instance. Tier 2 redeployments re-map the SAME physical ring pages into the replacement process. c. The remount is AUTOMATIC (Step U14: reload driver → Hello → vfs_init(ring_set, ...) → mount RO → fsck_fast() → remount RW). Stale open files survive TRANSPARENTLY: an OpenFile whose open_generation predates the crash fails vfs_check_open_generation() against the bumped sb.driver_generation (Step U10a / Step 3a below), but ENOTCONN is an INTERNAL transport signal, not a userspace error — the dispatch wrapper routes it into vfs_revalidate_open_file(), which re-runs FileOps::open against the reloaded driver and refreshes the fd's private_data and open_generation in place. ENOTCONN does NOT reach userspace and no close/reopen is required; only a PERMANENT revalidation failure (backing inode gone after journal replay, or the driver refusing the re-open) is latched and surfaced as EIO (Section 14.1). The mount itself, dentry cache, and inode cache remain valid; files opened AFTER recovery completes work normally with no administrator action. d. Pending request state is lost — applications whose requests were failed with EIO/ETIMEDOUT must retry, exactly as after any other I/O error.

Crash Recovery Algorithm — Complete Specification:

VFS crash recovery runs when a Tier 1 VFS driver (e.g., ext4, XFS) crashes and is reloaded (Section 11.9).

Synchronization during recovery (no lock-based ordering — uses atomics):

The unified VFS crash recovery sequence (Section 14.3) uses VfsRingSet.state atomics (VFSRS_RECOVERING) to block new operations, NOT explicit locks. The previous lock-based model (vfs_global_lock, sb.recovery_lock) was replaced by the atomic state machine approach: - ring_set.state.store(VFSRS_RECOVERING, Release) blocks all new select_ring() calls. - Per-ring inflight_ops counters provide the quiescence barrier. - Per-inode inode.lock (level 185) is acquired only if individual inodes need repair (e.g., truncate-on-recovery for partially-written files).

This eliminates lock ordering complexity and avoids adding a global lock to the recovery path. See the unified U1-U18 sequence in Section 14.3 for the authoritative step ordering.

Step 1: Quiesce in-flight operations - Set ring_set.state = VFSRS_RECOVERING (atomic store, Release ordering). Note: the recovery state is tracked on the VfsRingSet, not on the SuperBlock. See Section 14.3 for the unified U1-U18 recovery sequence. - All new VFS operations on this superblock return ENXIO immediately (no-op check at syscall entry). - Wait for all per-ring inflight_ops counters to reach zero: sum(ring.request_ring.inflight_ops for all rings in ring_set) == 0 (a SLEEPING wait — wait_until_quiesced, on each ring's Core-only producer_quiesce_wq, woken by complete_slot() on the 1→0 edge; NOT a busy spin — with a 5s timeout; if not drained after 5s, send SIGKILL to processes with operations stuck in the crashed driver's domain, then RE-ENTER the wait for a second 5s window — the killed producers must unwind through abort_slot(), which balances inflight_ops, before recovery may touch the rings; if the count is STILL nonzero after the second window, ABORT the recovery sequence with Err(QuiesceTimeout) per Step U5's abort arm — NEVER proceed to the drain with a live producer). SIGKILL is the escalation path of last resort: it is used only when a process cannot be unblocked by returning EIO on its stuck syscall (i.e., the process is in an uninterruptible task state waiting on a ring response that will never arrive). SIGTERM cannot wake an uninterruptible process. Only processes with operations stuck in the crashed driver's domain are affected. - The inflight_ops counter (defined in RingBuffer<T>) is incremented in reserve_slot() after successful slot reservation and decremented in complete_slot() after marking the slot FILLED. Per-ring counters avoid false sharing between CPUs on different rings. See Section 14.3 for the per-CPU ring extension that defines these counters. - Why per-ring, not per-sb: A single per-sb AtomicU32 would be a cache line contention point for N concurrent producers. Per-ring counters eliminate cross-ring false sharing. The crash recovery path sums all N counters (cold path, O(N) where N <= 256).

Step 2: Sweep the in-flight table (inline terminal page state), then drain and reset rings

This step has two phases. The sweep phase claims every in-flight table entry and, for page-populating reads, publishes its terminal page state INLINE as the entry is claimed; the drain phase resets ring pointers. There is NO separate orphaned-page collection and NO separate wake pass — the fixed-cap OrphanedPageEntry buffer is gone, and pages can never remain locked pending "manual intervention". The unified recovery sequence in Section 14.3 specifies the exact interleaving (Steps U5b/U6/U7) with the general crash recovery steps from Section 11.9.

Phase 2a: SWEEP + INLINE WAKE (from the Core-resident in-flight table — NEVER from ring memory) - Sweep each ring's core_side.inflight table (claim-by-remove: XArray remove per key — atomic, so a concurrently woken waiter claiming its own entry cannot double-process). For each claimed VfsInflightEntry: - If entry.opcode is ReadPage or Readahead: for each page index in entry.page_index .. + entry.nr_pages, look up the page in entry.inode.i_mapping.page_cache under RCU and, if present, set PageFlags::ERROR and unlock_page() — INLINE, as the entry is claimed (waking any wait_on_page_locked() waiter, which returns EIO). The entry.inode Arc<Inode> pin keeps the embedded i_mapping AddressSpace live across the deref; there is no raw AddressSpace pointer to forge or to dangle. - If entry.dma_handle != DmaBufferHandle::ZERO: free it (dma_pool_free) — the claim rule guarantees no other path freed or will free it (a completed request's entry was already removed by its claiming owner and is not in the table). - Ring memory is untrusted: the request/response rings are mapped read-write into the crashed domain — a corrupted driver could have scribbled arbitrary bytes into them (including replayed DMA handles). Every value recovery acts on was captured into the Core-only table by the DISPATCH path at submit time. This also covers the dispatched-but-uncompleted class that a ring walk cannot see: the consumer sets a slot EMPTY immediately after dispatching it to a work queue, so at crash time [tail..published) legally contains EMPTY slots whose requests are still in flight inside the (dead) driver — their table entries are still present and are cleaned here, while already-completed requests (entry removed at completion) are never double-processed. - Inode-pin independence: the page cache lives in Core (Tier 0); the table entry carries an Arc<Inode> pin, so orphaned pages are resolved WITHOUT traversing VFS-domain state and WITHOUT a raw pointer. No inode lookup, no ring read. - Exactly-once terminal state: claim-by-remove yields one winner per entry; Step U5b's response_drain_lock barrier excluded every response drainer, so none completes an entry concurrently; a completed ReadPage's entry was already removed (and its page unlocked) by the pre-crash response drain; waiters unlock only via their own sync completions and never touch these pages; and a LOCKED page cannot be evicted, so the RCU-lookup miss arm is pure defense-in-depth.

Phase 2b: DRAIN (reset rings) - Acquire each ring's core_side.response_drain_lock (now uncontended defense-in-depth — U5b already excluded every response drainer) and hold it across the ring reset. - Discard the response ring contents wholesale (pointer reset below) — pre-crash completions are suspect and are NOT delivered; their waiters observe the RECOVERING state and return EIO. Table entries for them were reaped in Phase 2a. - After the reset and the explicit drain-lock DROP, wake blocked submitter threads (completion.wake_up_all()) — UNCONDITIONAL, outside the lock (two-phase discipline: response_drain_lock is a leaf); woken threads observe state != ACTIVE in their wait condition and return EIO (claim-or-lose per the response matching protocol). - Reset all slot_states to EMPTY and ring pointers to 0. See drain_all_vfs_rings() in Section 14.3. - After Phase 2a's sweep, each ring's core_side.inflight table is empty — no stale pre-crash entries survive recovery (50-year no-leak rule: request IDs are u64 and never reused, so an unreaped entry would otherwise leak permanently).

Step 2.5: Page cache integrity verification - Before inspecting page cache state, verify page cache metadata integrity. Page cache metadata (XArray nodes, xa_node.count fields) is owned by Core — VFS drivers never hold xa_lock directly. All page cache mutations from VFS driver domains go through Core services via kabi_call! (see XA_LOCK entry in Section 3.4). A crashed VFS driver cannot have left the page cache's xa_lock held, and cannot have partially modified XArray tree nodes directly. However, in-flight kabi_call! completions that Core was executing on behalf of the driver at crash time may have been interrupted. The integrity walk detects any such torn state. - For each inode with a non-empty page cache: walk the XArray tree and verify: (a) all slot pointers are within valid slab regions, (b) the xa_node.count field matches the actual non-null slot count, (c) no cycles exist (bounded walk depth = XArray max height = 6 for 64-bit).

Synchronization protocol for the XArray walk: - Read-only validation phase: Acquire rcu_read_lock(). The XArray walk reads node pointers and slot entries under RCU protection. reclaimd may run concurrently (it removes pages from the XArray via xa_erase()), but RCU protects node lifetimes — freed XArray nodes are not reused until the grace period ends. - Mutation phase (corruption repair): If corruption is detected, drop the RCU lock, acquire the per-AddressSpace page-cache writer lock via Core's xa_lock service (see Section 3.4 XA_LOCK = 181, the page-cache XArray level), then call truncate_inode_pages_range(mapping, 0, u64::MAX) to drop the entire page cache for that inode. Because VFS drivers never hold xa_lock directly, this acquisition cannot deadlock on a crashed VFS domain. - The walk is bounded (max XArray height = 6 levels, each level is 64-way fanout = 64^6 = ~68 billion pages). For a 1TB file with 4KB pages, the tree has ~256M entries across ~4M nodes, requiring ~100ms to walk. This is acceptable for a cold crash recovery path.

• If corruption is detected: mark the inode as I_PAGE_CACHE_CORRUPT, drop the entire page cache for that inode (truncate_inode_pages_range(mapping, 0, u64::MAX)), and log an FMA event. The data will be re-read from disk after remount.

Step 3: Dirty page detection (Core-only rule — defer everything) - Walk each inode's page cache (skipping DAX inodes where page_cache is None and inodes marked I_PAGE_CACHE_CORRUPT) for all dirty pages: pages with PageFlags::Dirty set. - ALL dirty pages are kept in memory (pinned) across the outage and re-written through the NEW driver instance in Step 5 (writeback_deferred_dirty()). Core does NOT consult journal transaction state: the page→Transaction.tid association and the committed-tid watermark are FILESYSTEM DRIVER state, living inside the crashed (revoked, possibly corrupted) domain — recovery must resolve page disposition WITHOUT traversing VFS-domain state (the same independence principle as Phase 2a). An earlier revision instructed Core to "mark clean" pages whose transaction had committed; that was only ever an optimization, and it was unimplementable from Core. - Correctness comes from machinery that is already required: Step 4's journal replay (U14(e)) restores committed metadata exactly as after a power failure, and Step 5 re-writes every deferred dirty page — re-writing a page whose transaction had committed is a redundant but idempotent data write. - The committed-extent FAST FLUSH does exist, but on the Core-resident substrate: the writeback crash-recovery bypass flushes DirtyIntentEntry records whose block_addr is committed — those carry block_addr/sb_dev/block_dev in Core memory (Section 4.6, Writeback Cross-Domain Dispatch) — and runs as unified-sequence Step U10b (Section 14.3), ordered before the U14(e) journal replay.

Step 4: Reload driver and remount - Load the new driver image (Section 11.3 reload protocol). - Call driver.mount(sb.device, sb.flags) with MS_RDONLY first (safe mode). - Run the filesystem's built-in consistency check (ext4 replay journal; XFS log recovery; Btrfs tree walk) via driver.fsck_fast(). - If fsck_fast() returns Ok(()): remount read-write; resume normal operations. - If fsck_fast() returns Err: emit FMA fault event, keep read-only, require manual intervention.

Step 5: Flush deferred dirty pages - After successful RW remount, call writeback_deferred_dirty(sb) to flush the dirty pages held since Step 3.

Step 3a: Generation counter bump (before CQE drain and driver reload) - The SuperBlock has a driver_generation: AtomicU64 counter that is incremented each time the driver is (re)loaded. The VFS sets sb.driver_generation = old_generation + 1 BEFORE the io_uring CQE drain (unified sequence Step U11), BEFORE the driver is reloaded (Step 4) and BEFORE dirty page writeback (Step 5). This ordering is critical twice over: the replacement driver's responses (including writeback completions) must carry the NEW generation so the VFS consumer does not discard them, and CQE-triggered io_uring resubmissions must observe the new generation so they cannot be admitted at the stale one. (Previously numbered Step 5.5 and placed after Step 5 — moved to avoid silent data loss on the writeback flush path, then moved before the CQE drain to close the weak-memory resubmission window. See Section 14.3 Step U10a for the unified sequence rationale.) This ensures: - Stale responses from the pre-crash ring (if any were in-flight) are detected and discarded: the VFS checks response.driver_generation == sb.driver_generation.load(Acquire) before processing any completion. - Open file handles acquired before the crash carry the old generation (stored in OpenFile.open_generation, set at open time). Any VFS operation using an old-generation file handle hits the generation check:

if file.open_generation.load(Acquire) != file.inode.i_sb.driver_generation.load(Acquire) {
    return Err(ENOTCONN);
}

ENOTCONN here is the INTERNAL transport signal, not a userspace error: the dispatch wrapper absorbs it via vfs_revalidate_open_file() (re-runs FileOps::open, refreshes private_data/open_generation in place), so the fd survives transparently; only a permanent revalidation failure is latched and surfaced as EIO (Section 14.1). - The generation counter is persisted in the SuperBlock struct (not in driver-owned memory), so it survives driver crashes. One counter per mount (not per ring) — all rings on a mount share the same generation.

Recovery latency target: ≤500ms for ≤1 million in-flight operations and ≤10 million dirty pages.

3. Cancellation protocol: A caller (or the kernel on behalf of a caller) can cancel a pending request through the following protocol:

a. The caller invokes vfs_cancel(request_id) (internal kernel API, not exposed as a syscall — cancellation is triggered by signal delivery, thread exit, or timeout). b. The kernel resolves the request via the Core-resident in-flight table: RCU-look up core_side.inflight[request_id] (the per-ring tables of the mount are indexed by the ring recorded at submit; the lookup starts at the entry's ring_index when the caller knows it — the timeout path does — and falls back to probing the mount's per-ring tables, a cold path bounded by ring_count). If no entry exists, the request already completed — cancellation is a no-op. Otherwise mark the entry Abandoned (Release) after waking/returning per the timeout steps above. There is NO in-slot cancel flag: the slot-state byte carries RingSlotState only. (An earlier revision packed a CANCEL bit into bit 7 of the state byte; every state-machine operation on the byte — swap(RESERVED), store(FILLED), exact-value matches in the consumer — clobbered or misread it, and a cancelled slot wedged the consumer scan. The side-channel below, which was already mandatory, is the sole cancellation signal.) c. The kernel enqueues a CancelToken on the cancellation side-channel of the ring identified by entry.ring_index. d. The filesystem driver drains the cancellation side-channel at the top of every drain_ring() pass into a small cancelled-ID set (bounded by ring depth — driver-private memory, e.g. a fixed-capacity open-addressed set; entries are dropped when the matching request is answered). Before starting I/O for each dequeued request, the driver checks the set. On a hit, it writes VfsResponse::Err(-ECANCELED) to the response ring and moves on. The driver MUST NOT begin any side-effecting I/O (block reads, metadata updates) for a cancelled request. e. If the driver has already started processing the request (e.g., issued a block I/O read before the token arrived), the driver completes the operation normally and writes the result to the response ring. The response drain finds the entry Abandoned, reaps it, and discards the response — the request is already resolved from the caller's perspective. f. The driver always eventually produces exactly one final response — either VfsResponse::Err(-ECANCELED) (token seen before I/O) or the normal VfsResponse::Ok/VfsResponse::Err (I/O already started). The per-request timeout still applies; if neither response arrives within the timeout, the request enters the crash-recovery path (step 2 above).

/// Token placed on the cancellation side-channel of a VfsRingPair to notify
/// the filesystem driver that a previously enqueued request should be skipped.
#[repr(C)]
pub struct CancelToken {
    /// The `request_id` of the cancelled request. Matches `VfsRequest::request_id`.
    pub request_id: u64,
    /// Why the request was cancelled.
    pub reason: CancelReason,
    /// Explicit trailing padding (struct alignment = 8 from u64 field).
    _pad: [u8; 4],
}
const_assert!(size_of::<CancelToken>() == 16);

/// Reason for request cancellation.
#[repr(u32)]
pub enum CancelReason {
    /// The per-operation timeout expired before the driver responded.
    Timeout = 1,
    /// The calling thread was interrupted (signal delivery or thread exit).
    CallerCancelled = 2,
    /// The filesystem driver crashed; all pending requests are being flushed.
    DriverCrash = 3,
}

Cancellation state machine — the lifecycle of a cancelled request from the driver's perspective:

Driver drain pass:
  0. Drain the cancellation side-channel into the driver-private
     cancelled-ID set (one pass, bounded by side-channel depth).
Driver dequeues request from ring:
  1. Check cancelled_ids.contains(request.request_id).
  2. If cancelled:
       → Write VfsResponse::Err(-ECANCELED) to response ring.
       → Remove the ID from the set. Done — no I/O.
  3. If not cancelled:
       → Begin I/O processing.
       → Re-drain the side-channel and re-check periodically for long
         operations (optional optimization — not required for
         correctness).
       → Complete I/O. Write VfsResponse::Ok/Err to response ring.
       → If a CancelToken arrived between steps 3 and completion:
         the response drain finds the in-flight entry Abandoned and
         discards the response (request already resolved).

This protocol guarantees: (1) the caller always receives exactly one error or the original lower-layer file remains untouched, (2) no I/O is wasted on requests that were cancelled before the driver began processing, (3) I/O that has already started is never aborted mid-flight (which would risk filesystem inconsistency).

4. VfsResponse::Pending semantics: A VfsResponse::Pending response from the filesystem driver means the request has been accepted and acknowledged but not yet completed (for example, the driver has issued a block I/O request and is waiting for device completion). The contract is:

• The caller must poll the response_ring or sleep on VfsRingPair::completion for the final VfsResponse::Ok or VfsResponse::Err.
• Pending does NOT reset the per-request timeout timer. The maximum time in Pending state is bounded by the operation timeout defined above. If the final response does not arrive within the timeout, the request is cancelled using the standard cancellation protocol (step 3).
• A driver may send at most one Pending response per request. Sending multiple Pending responses for the same request_id is a protocol violation; the kernel logs a warning and ignores duplicate Pending responses.
• Pending is optional: a driver may respond directly with Ok or Err without ever sending Pending. It exists to allow the VFS layer to distinguish "driver has seen the request" from "request is still sitting in the ring unprocessed" for diagnostic and health-monitoring purposes (Section 20.1).

5. Filesystem Driver Concurrency Requirements:

Driver concurrency model: The SPSC ring serializes delivery of requests to the filesystem driver, not processing. Filesystem driver implementations MUST process dequeued requests concurrently using internal work queues. A driver that processes requests sequentially will exhibit head-of-line blocking (e.g., a stat() waiting behind a fsync()).

The request_id-based response matching already supports out-of-order completion. A compliant driver implementation:

1. Dequeues requests in batches (up to ring depth).
2. Dispatches each request to an appropriate internal work queue (e.g., metadata operations to a fast-path thread, journal commits to a dedicated journal thread).
3. Responds via VfsResponse with the matching request_id as each operation completes -- responses may arrive in any order.

The VfsResponse::Pending mechanism (see above) provides acknowledgment for long-running operations, allowing the VFS layer to distinguish "driver is processing" from "driver is stuck."

Rationale: Unlike Linux, where multiple threads enter the filesystem driver concurrently on different CPUs (with filesystem-internal locking providing serialization at a finer granularity), UmkaOS's SPSC ring serializes delivery. The page cache absorbs >95% of read/stat operations with zero domain crossings (see "Key design properties" above), so the ring is crossed primarily for cache-miss reads, writes, and metadata mutations. For these operations, concurrent driver-internal processing is essential for matching Linux's multi-threaded I/O throughput on a single filesystem.

Scaling beyond single-ring: For workloads with high write concurrency on a single mount (e.g., PostgreSQL checkpoint with 64 backends issuing concurrent fsyncs), the single SPSC ring becomes a producer-side contention point. The per-CPU VFS ring extension (Section 14.3) replaces the single ring with N SPSC rings (one per CPU or per CPU group), eliminating cross-CPU cache line bouncing while preserving all SPSC lock-free invariants per ring.

14.3 Per-CPU VFS Ring Extension

14.3.1 Motivation

The baseline VFS ring protocol (Section 14.2) allocates a single SPSC VfsRingPair per mounted filesystem. The VFS (umka-nucleus) is the sole producer on the request ring; the filesystem driver is the sole consumer. This design is correct and efficient for single-threaded workloads, but creates a producer-side serialization bottleneck when many CPUs issue concurrent filesystem operations on the same mount.

PostgreSQL checkpoint scenario (the motivating workload): PostgreSQL's checkpoint process (and since PG 15, the checkpointer plus background writer) issues fsync() on hundreds to thousands of relation files concurrently. With 64 backends each doing fsync() on different files, all fsync requests serialize through the single request ring. The ring depth is 256 entries by default; under contention, producers spin waiting for the ring to drain. The filesystem driver dequeues and dispatches to internal work queues, but the single-ring bottleneck means:

1. Producer serialization: The SPSC ring protocol requires a single producer. Since the VFS runs on any CPU, slot reservation must be serialized. Under 64-way contention with a single ring, this serialization becomes the bottleneck. Each CPU contends for slot reservation, waiting for the lock, and the lock holder's cache line bounces between CPUs via the coherence protocol. At ~50-70 cycles per bounce on x86-64, the lock acquisition alone costs ~3.2-4.5 us under 64-way contention.

2. Head-of-line blocking: A slow fsync that fills the ring blocks all subsequent producers from enqueueing any request (reads, stats, lookups) on the same mount. The single ring's depth (256 entries by default) is shared across all CPUs.

3. Doorbell storm: Each CPU that reserves a slot and enqueues a request rings the doorbell independently. With 64 concurrent producers, the driver receives up to 64 doorbells for requests that could have been coalesced.

This extension replaces the single SPSC ring per mount with N rings per mount, one per CPU or per CPU group. Under PerCpu granularity (the primary mode), each ring is pure SPSC — no CAS, no contention between producers. Under shared-ring modes (PerNuma/PerLlc/Fixed), multiple CPUs share a ring with atomic (CAS) head allocation. The CPU that issues a VFS operation uses its assigned ring, eliminating or greatly reducing cross-CPU cache line bouncing.

14.3.2 Design Principles

1. PerCpu rings are pure SPSC — the primary mode. Under PerCpu granularity (the default for >= 65 CPUs), each ring has exactly one producer. All SPSC invariants, memory ordering guarantees, and backpressure semantics are preserved per ring. No CAS on the ring head — just relaxed load/store.

2. Shared-ring modes claim head with the guarded position claim — safe but slower. Under PerNuma, PerLlc, and Fixed granularity, multiple CPUs share a ring. The reserve_slot() function claims head via reserve_head_claim() — the guarded position-claim discipline (Section 3.1): a CAS-loop lowering on 64-bit-atomic legs, the ll/sc reservation lowering on the 32-bit-atomic leg, fullness validated inside the claim window on both. This is lock-free (no spinlock) but has O(N) expected claim retries under N-way contention, bounded by ring size.

3. The driver side multiplexes N rings. The filesystem driver's consumer thread(s) poll all N rings. This is the only new complexity — the driver must handle multiple input rings instead of one.

4. Backward compatible. Drivers that only support single-ring mode (ring_count_max = 1 in their KABI manifest) continue to work unchanged. The VFS falls back to single-ring mode for such drivers.

5. No new lock types. Ring selection is determined by cpu_id at VFS entry — no lock, no CAS, no arbitration. The mapping is a static array lookup. The shared-ring CAS operates on the existing head atomic.

---

14.3.3 Ring Topology

14.3.3.1 VfsRingSet: Per-Mount Ring Collection

The single VfsRingPair is replaced by a VfsRingSet that contains 1..N ring pairs, where N is negotiated at mount time.

/// VfsRingSet state constants. These are the mount-level recovery states,
/// distinct from the per-ring DomainRingBuffer.state values (Active=0,
/// Disconnected=1).
pub const VFSRS_ACTIVE: u8 = 0;
pub const VFSRS_RECOVERING: u8 = 1;
pub const VFSRS_QUIESCING: u8 = 2;

/// Per-mount collection of VFS ring pairs. Replaces the single `VfsRingPair`
/// from [Section 14.2](#vfs-ring-buffer-protocol).
///
/// Each ring pair is a full SPSC channel (request + response + doorbell +
/// completion wait queue). Rings are indexed by CPU group — each CPU is
/// statically assigned to exactly one ring at mount time.
///
/// Placement: Tier 0 (Core). The VfsRingSet is owned by the superblock
/// and lives in umka-nucleus memory. The ring data regions are in shared
/// memory (PKEY 14 shared DMA pool on x86-64).
// Kernel-allocated, kernel-owned. Layout is depended upon by Tier 1
// drivers via `&VfsRingSet` reference passed at vfs_init() time.
// `#[repr(C)]` ensures stable field offsets across compilation units.
#[repr(C)]
pub struct VfsRingSet {
    /// Array of ring pairs. Length is `ring_count`. Index 0 is always valid.
    /// Allocated from the kernel slab at mount time (warm path, bounded N).
    /// Maximum N = MAX_VFS_RINGS (256, sufficient for 256-core systems
    /// with 1:1 CPU-to-ring mapping).
    ///
    /// SAFETY: Allocated from kernel slab at mount time. Valid for the
    /// lifetime of the VfsRingSet (which is the lifetime of the mount).
    /// `ring_count` is the element count. Freed in umount after all rings
    /// are drained. During crash recovery, `drain_all_vfs_rings()` accesses
    /// rings under `VfsRingSet.state == RECOVERING`, which prevents
    /// concurrent `select_ring()` access. Raw pointer required because
    /// VfsRingSet is `#[repr(C)]` for KABI transport.
    /// `debug_assert!(ring_count <= MAX_VFS_RINGS)` at mount-time init.
    ///
    /// VfsRingSet implements Send + Sync because `rings` points to
    /// slab-allocated memory that outlives all users; ring_count is
    /// immutable after mount.
    ///
    /// `*const` because VfsRingPair's mutable fields (via `RingBuffer<T>`:
    /// `inner.head`, `inner.tail`, `inner.published`, `slot_states`,
    /// `inflight_ops`) are all atomics — shared access is safe via atomic
    /// operations. The pointer itself is immutable after mount (the array
    /// base never changes); interior mutability is provided by the atomic
    /// fields within each VfsRingPair's `RingBuffer<T>` members.
    ///
    /// Raw pointer required because `#[repr(C)]` layout is depended upon
    /// by Tier 1 VFS drivers that receive `&VfsRingSet` via `vfs_init()`.
    /// Although VfsRingSet is kernel-allocated and kernel-owned, its field
    /// offsets are ABI-visible to the driver through the reference.
    pub rings: *const VfsRingPair,

    /// Number of active ring pairs. Range: 1..=MAX_VFS_RINGS.
    /// Set at mount time after negotiation with the driver.
    /// Invariant: ring_count >= 1 (single-ring mode is the minimum).
    pub ring_count: u16,

    /// CPU-to-ring mapping table. Index: CPU ID (0..num_possible_cpus()).
    /// Value: ring index (0..ring_count). Populated at mount time.
    /// Updated atomically on CPU hotplug events.
    ///
    /// This is a read-only lookup table on the hot path — no lock, no CAS.
    /// The table is allocated once at mount time with capacity for
    /// `num_possible_cpus()` entries (runtime-discovered, not hardcoded).
    ///
    /// For CPU IDs beyond the table size (should not happen — table is
    /// sized to num_possible_cpus() at mount time), the fallback is ring index 0.
    ///
    /// SAFETY: Same lifetime as `rings` — allocated at mount, freed at
    /// umount. `cpu_to_ring_len` is the element count. Raw pointer for
    /// `#[repr(C)]` KABI transport compatibility. `*mut` because
    /// `cpu_to_ring` entries are updated by CPU hotplug
    /// (`vfs_rings_cpu_online`) via `AtomicU16::store()`.
    pub cpu_to_ring: *mut AtomicU16,

    /// Number of entries in the cpu_to_ring table (== num_possible_cpus() at mount time).
    /// u32: supports systems with >65535 CPUs (large HPC / datacenter nodes).
    pub cpu_to_ring_len: u32,

    /// Ring allocation granularity used at mount time.
    pub granularity: RingGranularity,

    /// Driver-readable MIRROR of `SuperBlock.driver_generation`.
    ///
    /// The isolated filesystem driver cannot read the SuperBlock (Core-
    /// only memory), yet every `VfsResponseWire.driver_generation` must
    /// carry the CURRENT generation for the stale-response filter to
    /// work. This mirror is the delivery channel: the VfsRingSet is
    /// mapped read-write into the driver's domain, so the driver stamps
    /// `ring_set.driver_generation.load(Acquire)` into each response at
    /// construction time (contract in the `vfs_init()` doc below).
    ///
    /// Single writer: Core — the mount path writes the initial value
    /// (equal to `sb.driver_generation`), and recovery Step U10a bumps
    /// BOTH (`sb` field first, then this mirror, both Release; the sb
    /// field is authoritative, this is a pure copy). A crashed old
    /// instance cannot forge the NEW value: it is dead (domain revoked,
    /// U2) before U10a bumps, and its in-ring responses are discarded by
    /// the U7 reset anyway — the generation check is the second,
    /// independent line of defense exactly as U10a argues. A LIVE driver
    /// could scribble this mirror (the page is driver-writable), but
    /// that only corrupts the value the driver itself stamps — i.e., it
    /// can only invalidate its own responses, never forge acceptance of
    /// a stale instance's (the filter compares against the Core-side
    /// `sb` copy, not this mirror).
    pub driver_generation: AtomicU64,

    /// Response-direction doorbell: the driver notifies Core that one or
    /// more rings have new RESPONSE entries; the per-mount Core response
    /// worker (`vfs_response_worker`, [Section 14.2](#vfs-ring-buffer-protocol))
    /// waits on it and drains the pending rings. Same `CoalescedDoorbell`
    /// machinery as the request direction below, pointed the other way.
    /// Reset in U7 alongside `coalesced_doorbell`.
    pub completion_doorbell: CoalescedDoorbell,

    /// Per-ring doorbell coalescing state for cross-ring coalesced doorbells.
    /// See "Doorbell Coalescing" section below.
    pub coalesced_doorbell: CoalescedDoorbell,

    /// Global request_id generator for this mount. All rings draw from this
    /// single atomic counter to ensure mount-wide unique request IDs.
    /// See "Request ID Generation" below.
    ///
    /// **Cache line isolation**: This field is hot-path (atomically incremented
    /// on every VFS operation). It must NOT share a cache line with other
    /// frequently-written fields. The `state` field below is cold-path (written
    /// only during crash recovery), so false sharing with `state` is acceptable.
    /// The `CacheLinePadded` wrapper ([Section 3.6](03-concurrency.md#lock-free-data-structures)) gives
    /// `next_request_id` its own fixed 64-byte cache line (exactly 64 bytes on
    /// every arch, load-bearing for the `VfsRingSet` size `const_assert!`).
    pub next_request_id: CacheLinePadded<AtomicU64>,

    /// Mount-level state for crash recovery coordination.
    /// Set to RECOVERING during crash recovery to block all rings.
    /// Cold path only — written during crash recovery, read (Relaxed) on
    /// ring selection hot path for early-exit check.
    ///
    /// Constants:
    /// - `VFSRS_ACTIVE = 0u8`: Normal operation. select_ring() proceeds.
    /// - `VFSRS_RECOVERING = 1u8`: Crash recovery in progress. select_ring()
    ///   returns ENXIO. Set at Step U3, cleared at Step U17.
    /// - `VFSRS_QUIESCING = 2u8`: Live evolution quiescence in progress.
    ///   select_ring() returns ENXIO. Set at evolution initiation, cleared
    ///   when the replacement driver's consumer loops are ready.
    ///
    /// These are DISTINCT from the per-ring `DomainRingBuffer.state` values
    /// (0 = Active, 1 = Disconnected). The VfsRingSet state gates ALL rings
    /// in the mount; the per-ring state controls individual ring operation.
    pub state: AtomicU8,

    /// Padding to fill remaining bytes in the struct layout after state,
    /// preventing adjacent slab allocations from sharing this cache line.
    /// (Not cache-line alignment of `state` itself — for that, add
    /// `#[repr(C, align(64))]` to the struct.)
    _pad: [u8; 5],
}
// VfsRingSet is kernel-allocated but its layout is depended upon by Tier 1
// drivers that receive &VfsRingSet via vfs_init(). Struct alignment is 64
// (inherited from the align(64) CoalescedDoorbell and CacheLinePadded members),
// so every big member is placed on a 64-byte boundary. The leading scalar
// prefix — rings *const (ptr) + ring_count u16 + cpu_to_ring *mut (ptr) +
// cpu_to_ring_len u32 + granularity u8 + driver_generation AtomicU64 — fits
// within offsets 0..64 on BOTH 32- and 64-bit (the pointer-width difference is
// absorbed by padding before the first 64-aligned member at offset 64), then
// completion_doorbell CoalescedDoorbell (192) + coalesced_doorbell
// CoalescedDoorbell (192) + next_request_id CacheLinePadded<AtomicU64> (64) +
// state AtomicU8 + _pad[5], rounded up to the struct's 64-byte alignment:
// 64 + 192 + 192 + 64 = 512, then the state/pad tail rounds to 576. The size is
// pointer-width-INDEPENDENT because the align(64) members absorb the delta.
const_assert!(core::mem::size_of::<VfsRingSet>() == 576);

// SAFETY: `rings` and `cpu_to_ring` point to slab-allocated memory that
// outlives all users of VfsRingSet (freed at umount after all rings are
// drained). All mutable state within VfsRingPair uses atomics. ring_count
// and cpu_to_ring_len are immutable after mount-time initialization.
unsafe impl Send for VfsRingSet {}
unsafe impl Sync for VfsRingSet {}

/// Maximum number of VFS rings per mount. Sized for 256-core systems
/// with 1:1 CPU-to-ring mapping. Systems with >256 CPUs use CPU-group
/// mapping (multiple CPUs share one ring). The value is a compile-time
/// upper bound for array sizing; the actual ring count is negotiated
/// at mount time and is typically much smaller.
pub const MAX_VFS_RINGS: usize = 256;

/// Ring allocation granularity — how CPUs are mapped to rings.
#[repr(u8)]
pub enum RingGranularity {
    /// One ring per CPU. Maximum parallelism, maximum memory usage.
    /// Best for high-IOPS workloads (databases, storage servers).
    PerCpu = 0,

    /// One ring per NUMA node. CPUs on the same NUMA node share one ring.
    /// Good balance of parallelism and memory. Reduces cross-NUMA cache
    /// bouncing while keeping ring count manageable.
    PerNuma = 1,

    /// One ring per LLC (Last-Level Cache) group. CPUs sharing an L3 cache
    /// share one ring. Finer than PerNuma on multi-CCX/chiplet designs
    /// (AMD EPYC, Intel Sapphire Rapids). Within an LLC group, cache line
    /// bouncing for the ring head is L3-local (~10-15 cycles) rather than
    /// cross-socket (~50-70 cycles).
    PerLlc = 2,

    /// Fixed number of rings (specified via mount option). CPUs are
    /// distributed round-robin across rings. Used when the operator
    /// wants explicit control (e.g., `vfs_ring_count=4`).
    Fixed = 3,

    /// Single ring (legacy mode). Equivalent to the baseline protocol.
    /// Used when the driver reports `ring_count_max = 1`.
    Single = 4,
}

14.3.3.2 CPU-to-Ring Assignment

At mount time, the VFS builds the cpu_to_ring mapping table based on the negotiated ring_count and granularity:

/// Build the CPU-to-ring mapping table at mount time.
///
/// # Arguments
/// * `ring_count` — Negotiated number of rings (1..=MAX_VFS_RINGS).
/// * `granularity` — How CPUs are grouped into rings.
///
/// # Returns
/// Slab-allocated mapping table of length `num_possible_cpus()`.
///
/// Hot path access: `cpu_to_ring[current_cpu().index()].load(Relaxed)` — one
/// atomic load (~1 cycle on x86-64 TSO, ~1-3 cycles on ARM/RISC-V).
fn build_cpu_to_ring_map(
    ring_count: u16,
    granularity: RingGranularity,
) -> &'static [AtomicU16] {
    let nr_cpus = num_possible_cpus();
    let table = slab_alloc_zeroed::<AtomicU16>(nr_cpus);

    match granularity {
        RingGranularity::PerCpu => {
            // 1:1 mapping: CPU i → ring min(i, ring_count - 1).
            // If nr_cpus > ring_count, wrap with modulo.
            for cpu in 0..nr_cpus {
                table[cpu].store((cpu % ring_count as usize) as u16, Relaxed);
            }
        }
        RingGranularity::PerNuma => {
            // One ring per NUMA node (up to ring_count nodes).
            // NUMA node IDs are discovered at boot via ACPI SRAT / device tree.
            for cpu in 0..nr_cpus {
                let node = arch::current::cpu::cpu_to_node(cpu);
                table[cpu].store((node % ring_count as usize) as u16, Relaxed);
            }
        }
        RingGranularity::PerLlc => {
            // One ring per LLC group. LLC group IDs are discovered at boot
            // via CPUID (x86), CLIDR_EL1 (AArch64), or device tree.
            for cpu in 0..nr_cpus {
                let llc_id = arch::current::cpu::cpu_to_llc_group(cpu);
                table[cpu].store((llc_id % ring_count as usize) as u16, Relaxed);
            }
        }
        RingGranularity::Fixed => {
            // Round-robin distribution.
            for cpu in 0..nr_cpus {
                table[cpu].store((cpu % ring_count as usize) as u16, Relaxed);
            }
        }
        RingGranularity::Single => {
            // All CPUs map to ring 0.
            // Table is already zero-initialized.
        }
    }

    table
}

compute_ring_for_cpu() is the single-CPU form of the same mapping, used by the CPU-hotplug path to (re)assign one CPU without rebuilding the whole table:

/// Compute the ring index for a single CPU under the mount's granularity.
///
/// Returns the same value `build_cpu_to_ring_map()` would store for `cpu`,
/// derived from `ring_set.granularity` and `ring_set.ring_count`. The result
/// is always in `0..ring_set.ring_count` (never out of bounds).
///
/// Warm path (CPU online/offline), not hot path: one topology query plus a
/// modulo. No allocation.
fn compute_ring_for_cpu(cpu: CpuId, ring_set: &VfsRingSet) -> u16 {
    let cpu = cpu.index();
    let count = ring_set.ring_count as usize; // invariant: >= 1
    let idx = match ring_set.granularity {
        RingGranularity::PerCpu | RingGranularity::Fixed => cpu % count,
        RingGranularity::PerNuma => {
            arch::current::cpu::cpu_to_node(cpu) % count
        }
        RingGranularity::PerLlc => {
            arch::current::cpu::cpu_to_llc_group(cpu) % count
        }
        RingGranularity::Single => 0,
    };
    idx as u16
}

Hot path ring selection — the VFS dispatch path (step 4 in the dispatch flow from Section 14.2) selects the ring as follows:

/// Ring slot states for the split reservation/completion protocol.
///
/// This protocol separates slot reservation (which requires producer ordering)
/// from data fill (which may fault on `copy_from_user`). The key insight:
/// `preempt_disable` is needed only around `select_ring()` + `reserve_slot()`
/// (~3 instructions, ~nanoseconds), NOT around the entire ring operation.
/// After reservation, the slot is owned by the reserving task regardless of
/// which CPU it runs on. This enables:
/// - Inline small I/O: `copy_from_user` during ring fill
/// - FUSE passthrough: userspace access during ring submission
/// - Any path needing page faults during data fill
///
/// ```
/// EMPTY → RESERVED → FILLED → EMPTY
///   ↑                            |
///   +----------------------------+
/// ```
///
/// - `EMPTY`: Slot is available for reservation (producer may claim it),
///   OR already processed by the consumer while `tail` is pinned behind
///   an earlier RESERVED blocker. EMPTY inside `[tail..published)` is
///   therefore a LEGAL state the consumer scan skips — not an invariant
///   violation.
/// - `RESERVED`: Slot is claimed by a producer. The producer owns the slot
///   exclusively. Consumer stops at RESERVED slots (head-of-line
///   blocking — the slot is not yet ready). The consumer does NOT skip past
///   RESERVED slots for tail-advance purposes; it processes later FILLED
///   slots but `tail` stays pinned. A producer that cannot fill a
///   RESERVED slot (fill fault, fatal signal, state re-check failure)
///   releases it via `abort_slot()` (fill-with-Nop) — RESERVED never
///   reverts directly to EMPTY.
/// - `FILLED`: Producer has written data and marked the slot complete.
///   The consumer processes it and stores EMPTY directly (single owner;
///   no intermediate state is ever visible).
/// - `Consumed = 3` is RESERVED for diagnostics/ABI stability of the u8
///   encoding: no current code path stores it. (An earlier revision
///   specified a two-step FILLED→CONSUMED→EMPTY transition; the consumer
///   was always specified to store EMPTY directly, so the doc now matches
///   the code.)
///
/// **State ownership**: Only the producer writes EMPTY→RESERVED and
/// RESERVED→FILLED (including the Nop abort fill). Only the consumer
/// writes FILLED→EMPTY and advances `tail`. The `head` is producer-owned.
/// This two-party protocol has no ABA risk because each slot has exactly
/// one owner at a time.
///
/// The state byte holds a `RingSlotState` value and NOTHING ELSE — no
/// flag bits are packed into it. (An earlier revision packed a CANCEL
/// flag into bit 7; every whole-byte state operation clobbered or
/// misread it. Cancellation is signaled exclusively via the side-channel
/// `CancelToken` + the Core-resident in-flight table —
/// [Section 14.2](#vfs-ring-buffer-protocol).)
#[repr(u8)]
pub enum RingSlotState {
    Empty    = 0,
    Reserved = 1,
    Filled   = 2,
    Consumed = 3,
}

/// Check open_generation before dispatching any VFS operation.
/// This is the VFS dispatch entry point referenced in the OpenFile
/// struct doc comment. Must be called before `select_ring()`.
///
/// `file` is the canonical open-file description (`OpenFile`,
/// [Section 14.1](#virtual-filesystem-layer--openfile-open-file-description)) —
/// there is no separate `File` type in UmkaOS.
///
/// Returns `Err(ENOTCONN)` if the file was opened with a different driver
/// generation (the driver has crashed and been reloaded since this file was
/// opened). `ENOTCONN` here is the INTERNAL pre-refresh transport signal, NOT
/// a terminal userspace error: the VFS dispatch wrapper intercepts it and
/// routes the mismatch into the lazy open-fd revalidation layer
/// (`vfs_revalidate_open_file()`,
/// [Section 14.1](#virtual-filesystem-layer--open-file-descriptor-recovery-generation-refresh)),
/// which re-runs `FileOps::open()` on the reloaded instance and refreshes
/// `open_generation`. Userspace observes `EIO` only if revalidation
/// permanently fails (latched in `OpenFile.reopen_errno`); a transient
/// second-crash retries. (The earlier "userspace must close and re-open"
/// contract predated the revalidation layer and no longer holds.)
#[inline(always)]
fn vfs_check_open_generation(file: &OpenFile) -> Result<(), KernelError> {
    let current_gen = file.inode.i_sb.driver_generation.load(Ordering::Acquire);
    if file.open_generation.load(Ordering::Acquire) != current_gen {
        return Err(KernelError::ENOTCONN);
    }
    Ok(())
}

/// Select the VFS ring for the current CPU — the first step of the
/// producer dispatch protocol (ring selection + slot reservation).
///
/// This is the hot-path entry point — called on every VFS operation that
/// crosses the domain boundary. The full protocol:
///
/// ```
/// preempt_disable();
/// let ring = select_ring(ring_set)?;           // ENXIO if RECOVERING/QUIESCING
/// let core_side = &sb.ring_core[ring_idx];     // this ring's Core-only companion
/// let (slot_idx, seq) = reserve_slot(ring, core_side, ring_set)?; // claims slot; counts
///                                              // inflight; RE-CHECKS state after the
///                                              // count (quiescence race closure, below)
/// preempt_enable();
/// // Enroll in the Core-resident in-flight table BEFORE publishing —
/// // the entry must exist before any response can arrive. For a demand
/// // ReadPage fill the entry carries `fill: Some(FillCompletion)` — the
/// // obligation the faulter moved off its FillLease (`lease.into_inflight()`,
/// // [Section 4.4](04-memory.md#page-cache)); the completion/claim path discharges it exactly once.
/// // `None` for Readahead batches and every non-fill opcode.
/// core_side.inflight.store(request_id, entry); // [Section 14.2](#vfs-ring-buffer-protocol)
/// // --- preemption safe zone: fill data, may fault ---
/// if let Err(e) = fill_slot_data(ring, slot_idx, &request, user_src) {
///     // copy_from_user failed (EFAULT) or a fatal signal interrupted
///     // the fault. The RESERVED slot MUST NOT leak (it would pin `tail`
///     // forever) and the inflight count MUST balance:
///     abort_slot(ring, core_side, slot_idx, seq); // fill-with-Nop + complete
///     let _ = core_side.inflight.remove(request_id); // un-enroll (claim rule)
///     /* free caller-owned DMA buffer, if any */
///     return Err(e);
/// }
/// complete_slot(ring, core_side, slot_idx, seq); // store(FILLED, Release) + advance published
/// ```
///
/// **Preemption note**: `preempt_disable()` is held only around
/// `select_ring()` + `reserve_slot()` (~3-8 instructions, ~4-20 cycles).
/// After reservation, `preempt_enable()` — the slot is owned by the
/// reserving task regardless of which CPU it runs on. Data fill and
/// `complete_slot()` (mark slot as FILLED) can happen from any CPU,
/// including after migration or page fault. The preempt_disable window
/// is ~nanoseconds (slot reservation only), not the entire ring operation.
///
/// **Producer model**: Under `PerCpu` granularity (the primary mode), each
/// ring has exactly one producer — the `preempt_disable` window guarantees
/// no other task on this CPU can interleave. This is pure SPSC: no CAS on
/// `head`, just a relaxed load + store. Under `PerNuma`/`PerLlc`/`Fixed`
/// granularity, multiple CPUs share a ring. The `reserve_slot()` function
/// claims `head` via the guarded position claim (`reserve_head_claim()`).
/// See `reserve_slot()` below for both paths.
///
/// **Inline write advisory**: For shared-ring modes (`PerNuma`/`PerLlc`/
/// `Fixed`), inline writes that may trigger page faults during
/// `copy_from_user()` can hold a RESERVED slot for the duration of the
/// fault (~1-50 us minor, ~1-10 ms major). This causes head-of-line
/// blocking for the consumer on that ring. For shared rings, the VFS
/// dispatch path SHOULD prefer the DMA buffer path (pre-copy before
/// reservation) for inline-eligible writes if the ring's pending count
/// exceeds `ring.size / 2`. This is a performance heuristic, not a
/// correctness requirement — the consumer handles RESERVED slots correctly
/// by waiting (see consumer algorithm below).
///
/// Cost (PerCpu): 1 atomic load (Relaxed) + bounds check + 1 store. ~3-5 cycles.
/// Cost (shared): 1 CAS loop (~5-20 cycles under contention) + 1 CAS on slot state.
fn select_ring(ring_set: &VfsRingSet) -> Result<&VfsRingPair, KernelError> {
    // Fast-path rejection during crash recovery or live evolution.
    // This Relaxed load is a FILTER, not the synchronization point. A
    // producer can pass this check with a stale ACTIVE value and be
    // arbitrarily delayed (interrupt, NMI, inter-CPU store latency)
    // before `reserve_slot()` increments `inflight_ops` — during that
    // window it is invisible to the recovery/evolution quiescence scan.
    // The race is closed INSIDE `reserve_slot()`: after the
    // `inflight_ops` increment it executes a SeqCst fence and RE-CHECKS
    // this state (store-buffering/Dekker pairing with the SeqCst fence
    // that `set_all_rings_disconnected()` — and evolution Phase A' step
    // 1 — issues between the state store and the first counter read).
    // Exactly one of the following therefore always holds:
    //   (a) the producer's increment is visible to the quiescence scan
    //       (it is counted and waited for), or
    //   (b) the producer's re-check observes non-ACTIVE and it backs out
    //       via `abort_slot()`.
    // A producer that proceeds past the re-check is guaranteed visible;
    // one the scan misses is guaranteed to back out.
    //
    // False negatives here (reading RECOVERING after U17 stored ACTIVE
    // with Release) are harmless — the producer gets ENXIO and retries;
    // the window is nanoseconds to microseconds.
    if ring_set.state.load(Ordering::Relaxed) != VFSRS_ACTIVE {
        return Err(KernelError::ENXIO);
    }
    let cpu = current_cpu().index();
    let ring_idx = if cpu < ring_set.cpu_to_ring_len as usize {
        // SAFETY: cpu < cpu_to_ring_len, validated above. cpu_to_ring points
        // to a slab-allocated array of cpu_to_ring_len AtomicU16 elements,
        // valid for the lifetime of the mount.
        unsafe { (*ring_set.cpu_to_ring.add(cpu)).load(Ordering::Relaxed) } as usize
    } else {
        0 // Fallback for CPUs beyond the table (should not happen).
    };
    // SAFETY: ring_idx is in bounds — cpu_to_ring values are validated
    // at mount time to be < ring_count. Bounds check is redundant but
    // present for defense-in-depth.
    debug_assert!(ring_idx < ring_set.ring_count as usize);
    let idx = if ring_idx < ring_set.ring_count as usize {
        ring_idx
    } else {
        0
    };
    // SAFETY: rings is a valid, non-null pointer to ring_count VfsRingPair
    // elements, allocated from kernel slab at mount time and valid for the
    // lifetime of the mount. The pointer is set during mount initialization
    // and never modified afterward.
    debug_assert!(!ring_set.rings.is_null());
    Ok(unsafe { &*ring_set.rings.add(idx) })
}

/// Reserve a slot on the given ring. Returns `(slot_index, sequence_number)`.
/// The `sequence_number` is the `head` value at reservation time — passed to
/// `complete_slot()` so it can correctly advance the `published` watermark
/// without reading a potentially stale `head`.
///
/// Must be called with preemption disabled (caller holds PreemptGuard).
///
/// **PerCpu mode** (single producer per ring): No CAS on `head`. The caller
/// is the sole producer under `preempt_disable`, so `head` load + store is
/// safe. Cost: ~3-5 cycles.
///
/// **Shared-ring mode** (PerNuma/PerLlc/Fixed — multiple CPUs share a ring):
/// Claims the next slot via `reserve_head_claim()` — the guarded
/// position-counter claim discipline
/// ([Section 3.1](03-concurrency.md#rust-ownership-for-lock-free-paths--guarded-position-claim)): the
/// commit is conditioned on an unbroken claim window and the fullness
/// validation (the `tail` load) is issued INSIDE that window, never against
/// a value captured before the anchor. A lost claim retries with the
/// updated `head` (no false RingFull). Cost: ~5-20 cycles depending on
/// contention.
///
/// After `head` is successfully advanced, the producer CAS's the slot state
/// from EMPTY → RESERVED as defense-in-depth (should always succeed because
/// the consumer transitions CONSUMED → EMPTY before `tail` advances past
/// the slot, and `head - tail < size` was checked).
///
/// **Inflight tracking + quiescence race closure**: On successful
/// reservation, increments `ring.request_ring.inflight_ops` (AtomicU32,
/// Relaxed), then executes `fence(SeqCst)` and RE-CHECKS
/// `ring_set.state`. This counter is decremented by `complete_slot()`
/// (including the `abort_slot()` back-out path). Crash recovery and
/// evolution quiescence wait for all per-ring `inflight_ops` to reach
/// zero before touching the rings; the fence pairing (see
/// `select_ring()` comment and `set_all_rings_disconnected()`) makes
/// that wait sound: a producer the quiescence scan does not count is
/// guaranteed to observe the non-ACTIVE state here and back out.
enum ReserveError {
    /// Ring is full (backpressure — caller sleeps/retries per protocol).
    RingFull,
    /// `ring_set.state` was not ACTIVE at the post-increment re-check
    /// (crash recovery or live evolution began concurrently). The slot
    /// was already released via `abort_slot()` and `inflight_ops` is
    /// balanced. Caller maps this to ENXIO for BOTH RECOVERING and
    /// QUIESCING — the dispatch wrapper parks ENXIO on `sb.recovery_wait`
    /// and retries after re-activation (crash Step U17 / evolution Phase C
    /// both store ACTIVE and wake the queue), same as the `select_ring()`
    /// gate. This is an INTERNAL sleep-and-retry; no errno reaches userspace.
    NotActive,
}

fn reserve_slot(
    ring: &VfsRingPair,
    core_side: &VfsRingCoreSide,
    ring_set: &VfsRingSet,
) -> Result<(u32, u64), ReserveError> {
    if ring.shared_ring {
        // --- Shared-ring path: guarded head claim ---
        // Single-word ring POSITION claim — squarely in scope for the
        // guarded-claim ruling
        // ([Section 3.1](03-concurrency.md#rust-ownership-for-lock-free-paths--guarded-position-claim)).
        // The generic `guarded_claim` primitive decides claimability by a
        // per-slot sequence EQUALITY; a head/tail ring decides by the tail
        // INEQUALITY (fullness), so the claim is spelled per-lowering in
        // `reserve_head_claim()` below with the SAME two lowerings and the
        // same soundness arguments as the primitive. The former shape here —
        // a raw value-compare CAS on `head` with the tail check performed
        // before (outside) the claim window — was exactly the ruled-away
        // hazard: on the 32-bit-counter leg a claimant stalled across a full
        // counter wrap back to the same numeric `head` could commit against
        // stale fullness state, and the later slot-state swap cannot make
        // that commit conditional on an unbroken reservation.
        let head = match reserve_head_claim(&ring.request_ring) {
            Ok(head) => head,
            Err(ClaimUnavailable) => return Err(ReserveError::RingFull),
        };
        // We own slot `head`. Swap slot state as defense-in-depth (never
        // the correctness argument — that is the guarded claim above).
        let idx = (head & (ring.request_ring.inner.size as u64 - 1)) as u32;
        let prev = ring.request_ring.slot_state(idx as usize).swap(
            RingSlotState::Reserved as u8,
            Ordering::AcqRel,
        );
        debug_assert_eq!(prev, RingSlotState::Empty as u8,
            "Slot {} should be EMPTY after guarded head claim, was {}",
            idx, prev);
        // Publish this task as the producer holding a RESERVED
        // slot on THIS ring, BEFORE it becomes in-flight — this
        // is the pointer sigkill_stuck_producers() matches to
        // find and unstick a wedged producer during recovery /
        // evolution quiescence. Release: ordered ahead of the
        // inflight_ops increment and the SeqCst re-check below;
        // cleared in complete_slot() (which every exit path,
        // including abort_slot(), funnels through).
        unsafe { &*current_task() }.current_vfs_ring.store(
            core::ptr::from_ref(ring).cast_mut(),
            Ordering::Release,
        );
        // Track in-flight operation for quiescence.
        // AcqRel: the release half publishes the current_vfs_ring
        // store to any thread whose Acquire load of inflight_ops
        // observes this increment (the U5 SIGKILL scan) — the
        // SeqCst fence in reserve_recheck covers only the
        // state-vs-count Dekker pairing, not this pointer-vs-count
        // edge.
        ring.request_ring.inflight_ops.fetch_add(1, Ordering::AcqRel);
        reserve_recheck(ring, core_side, ring_set, idx, head)?;
        Ok((idx, head))
    } else {
        // --- PerCpu path: single producer, no CAS on head ---
        let head = ring.request_ring.inner.head.load(Ordering::Relaxed);
        let tail = ring.request_ring.inner.tail.load(Ordering::Acquire);
        if head.wrapping_sub(tail) >= ring.request_ring.inner.size as u64 {
            return Err(ReserveError::RingFull);
        }
        let idx = (head & (ring.request_ring.inner.size as u64 - 1)) as u32;
        // Unconditional swap(RESERVED). Under preempt_disable, we are the
        // sole producer on this per-CPU ring. If head - tail < size, the
        // slot at `head` MUST be EMPTY — any other state indicates a broken
        // invariant (a bug), not ring congestion. Using swap instead of CAS
        // avoids silently returning RingFull on invariant violations.
        let prev = ring.request_ring.slot_state(idx as usize).swap(
            RingSlotState::Reserved as u8,
            Ordering::AcqRel,
        );
        debug_assert_eq!(prev, RingSlotState::Empty as u8,
            "PerCpu ring invariant violation: slot {} should be EMPTY, was {}",
            idx, prev);
        ring.request_ring.inner.head.store(
            head.wrapping_add(1), Ordering::Release,
        );
        // Publish this task as the producer holding a RESERVED slot on THIS
        // ring, BEFORE it becomes in-flight — the pointer
        // sigkill_stuck_producers() matches to unstick a wedged producer.
        // Release: ordered ahead of the inflight_ops increment and the
        // SeqCst re-check; cleared in complete_slot() (every exit path,
        // including abort_slot(), funnels through it).
        unsafe { &*current_task() }.current_vfs_ring.store(
            core::ptr::from_ref(ring).cast_mut(),
            Ordering::Release,
        );
        // Track in-flight operation for quiescence.
        // AcqRel: the release half publishes the current_vfs_ring store to any
        // thread whose Acquire load of inflight_ops observes this increment
        // (the U5 SIGKILL scan) — the SeqCst fence in reserve_recheck covers
        // only the state-vs-count Dekker pairing, not this pointer-vs-count
        // edge.
        ring.request_ring.inflight_ops.fetch_add(1, Ordering::AcqRel);
        reserve_recheck(ring, core_side, ring_set, idx, head)?;
        Ok((idx, head))
    }
}

/// Guarded claim of the shared request ring's `head` position word.
///
/// This is the head/tail-ring realization of the guarded position-claim
/// discipline ([Section 3.1](03-concurrency.md#rust-ownership-for-lock-free-paths--guarded-position-claim)).
/// It cannot be routed through the generic `guarded_claim` primitive verbatim
/// — that primitive's deciding load is a per-slot sequence compared for
/// EQUALITY (`dif == 0` claims), while a head/tail ring's claimability is the
/// fullness INEQUALITY `head - tail < size` — so this function carries the
/// discipline's two lowerings itself, with the primitive's own soundness
/// arguments:
///
/// - **Legs with 64-bit atomics** (all but the one 32-bit-atomic leg): anchor
///   `head` (`Acquire`), load `tail` (`Acquire`) INSIDE the window, reject
///   `ClaimUnavailable` when full, then `compare_exchange_weak(head, head+1)`.
///   A value match on a 64-bit counter implies an unwrapped counter (a `2^64`
///   wrap is physically unreachable — the primitive's width-vacuity argument),
///   so the value CAS is a sound commit and no reservation is held.
/// - **32-bit-atomic leg (PPC32)**: the ring position words are
///   `ClaimPos`-width on this leg (the claim-counter width rule,
///   [Section 3.1](03-concurrency.md#rust-ownership-for-lock-free-paths--guarded-position-claim)). The
///   claim uses the arch reservation seam exactly as `guarded_claim`'s ll/sc
///   lowering does: `arch::current::atomic::load_reserved(head)` anchors the
///   reservation; the `tail` `Acquire` load inside the window is an ordinary
///   load and leaves the reservation intact; the full exit first calls
///   `arch::current::atomic::kill_reservation()` (every non-committing window
///   exit consumes the reservation); and
///   `arch::current::atomic::store_conditional(head, head+1)` commits ONLY on
///   an unbroken reservation. A claimant stalled across a full `2^32` wrap
///   back to the same numeric `head` therefore CANNOT commit — the wrap's
///   stores to the granule already broke the reservation. Structural, not
///   probabilistic.
///
/// **Both lowerings issue the deciding `tail` load inside the claim window**
/// — never against a value captured before the anchor — which is the ruled
/// requirement this function exists to satisfy. A lost claim (CAS failure /
/// broken reservation) re-anchors and retries; `Err(ClaimUnavailable)` means
/// the ring is genuinely full at an unbroken observation.
fn reserve_head_claim(
    request_ring: &RingBuffer<VfsRequest>,
) -> Result<u64, ClaimUnavailable> {
    let size = request_ring.inner.size as u64;
    #[cfg(target_has_atomic = "64")]
    loop {
        let head = request_ring.inner.head.load(Ordering::Acquire);   // anchor
        let tail = request_ring.inner.tail.load(Ordering::Acquire);   // inside window
        if head.wrapping_sub(tail) >= size {
            return Err(ClaimUnavailable);
        }
        match request_ring.inner.head.compare_exchange_weak(
            head,
            head.wrapping_add(1),
            Ordering::AcqRel,
            Ordering::Relaxed,
        ) {
            Ok(_) => return Ok(head),
            Err(_) => core::hint::spin_loop(),                        // lost race → re-anchor
        }
    }
    #[cfg(not(target_has_atomic = "64"))]
    loop {
        // Anchor a reservation on the (ClaimPos-width) head word.
        let head = arch::current::atomic::load_reserved(&request_ring.inner.head);
        // Deciding fullness load INSIDE the reservation window (ordinary
        // load — leaves the reservation intact).
        let tail = request_ring.inner.tail.load(Ordering::Acquire);
        if head.wrapping_sub(tail) as u64 >= size {
            // Non-committing window exit: consume the reservation.
            arch::current::atomic::kill_reservation();
            return Err(ClaimUnavailable);
        }
        // Commits ONLY on an unbroken reservation; either outcome consumes it.
        if arch::current::atomic::store_conditional(
            &request_ring.inner.head, head.wrapping_add(1),
        ) {
            return Ok(head as u64);
        }
        // Reservation lost → re-anchor.
        core::hint::spin_loop();
    }
}

/// Post-increment state re-check — the producer half of the
/// store-buffering (Dekker) pairing that makes quiescence sound.
///
/// Producer order:  inflight_ops.fetch_add  →  fence(SeqCst)  →  state load.
/// Recovery order:  state store (Release)   →  fence(SeqCst)  →  inflight_ops load.
/// With both fences, at least one side observes the other: either the
/// scan sees the increment (and waits for this producer), or this
/// producer sees the non-ACTIVE state (and backs out here). Without the
/// re-check, a producer delayed between `select_ring()`'s filter and the
/// increment could write into a ring that recovery is concurrently
/// resetting.
#[inline]
fn reserve_recheck(
    ring: &VfsRingPair,
    core_side: &VfsRingCoreSide,
    ring_set: &VfsRingSet,
    idx: u32,
    seq: u64,
) -> Result<(), ReserveError> {
    core::sync::atomic::fence(Ordering::SeqCst);
    if ring_set.state.load(Ordering::Relaxed) != VFSRS_ACTIVE {
        // Back out: RESERVED must not leak (it would pin the consumer's
        // `tail` forever, including across a live-evolution swap where
        // the rings are NOT reset). abort_slot() completes the slot as a
        // Nop and decrements inflight_ops — ring invariants and the
        // quiescence count both balance. The Nop is discarded by the
        // ring reset (crash recovery) or answered as a side-effect-free
        // Ok(0) by the surviving/new consumer (evolution) — either way
        // no in-flight table entry exists for it, so the response drain
        // drops the answer.
        abort_slot(ring, core_side, idx, seq);
        return Err(ReserveError::NotActive);
    }
    Ok(())
}

/// Release a RESERVED slot that its producer can no longer fill.
///
/// Used by: (a) the `reserve_recheck()` back-out above, (b) the
/// `fill_slot_data()` error arm (EFAULT / fatal signal — this is the
/// unwind leg the SIGKILL escalation in `sigkill_stuck_producers()`
/// relies on: a killed producer's fault handler returns `EINTR`, the
/// fill fails, and this path balances the slot and the count on the way
/// to `exit_task()`).
///
/// A RESERVED slot cannot revert to EMPTY: in shared-ring modes later
/// producers may already have FILLED slots past it, and in every mode an
/// unpublished EMPTY hole inside `[tail..published)` would permanently
/// stall the consumer's tail advance. Abort therefore = fill-with-Nop:
/// write a minimal `VfsRequest { opcode: Nop, args: VfsRequestArgs::Nop {},
/// .. }` (no DMA handle, no side effects — driver contract in
/// [Section 14.2](#vfs-ring-buffer-protocol)) and complete the slot normally. Cost:
/// one wasted slot on an error path.
fn abort_slot(ring: &VfsRingPair, core_side: &VfsRingCoreSide, idx: u32, seq: u64) {
    // SAFETY: the producer owns the RESERVED slot exclusively; writing
    // the entry is the same access `fill_slot_data()` would perform.
    unsafe {
        vfs_write_slot(ring, idx, &VfsRequest {
            request_id: 0, // never enrolled; response (if any) matches no entry
            opcode: VfsOpcode::Nop,
            _pad_opcode: 0,
            ino: 0,
            fh: u64::MAX,
            args: VfsRequestArgs::Nop {},
        });
    }
    // Funnel through complete_slot so the inflight_ops decrement (and its
    // quiescence wake, when armed) happens on the abort path too — a reserve-
    // time back-out or a fill error that drives this ring to zero must wake a
    // parked wait_until_quiesced, not leave it sleeping to its timeout.
    complete_slot(ring, core_side, idx, seq); // FILLED + published + inflight_ops balance
}

/// Write a fully-constructed `VfsRequest` into a RESERVED slot the
/// caller owns. Infallible (kernel-memory copy only).
///
/// # Safety
/// `idx` must identify a slot in RESERVED state owned by the calling
/// producer (returned by `reserve_slot()` and not yet completed).
unsafe fn vfs_write_slot(ring: &VfsRingPair, idx: u32, request: &VfsRequest) {
    // Entry pointer via the typed accessor; cast away const — the
    // producer's exclusive slot ownership makes the write race-free.
    let dst = ring.request_ring.entry_ptr(idx as usize) as *mut VfsRequest;
    core::ptr::write(dst, core::ptr::read(request));
}

/// Fill a RESERVED slot with request data. This is the step that may
/// FAULT: for inline small writes it performs the `copy_from_user()`
/// into `VfsRequestArgs::Write::inline_data`, which can fail with
/// `EFAULT` (bad user pointer) or `EINTR` (a fatal signal — including
/// the SIGKILL that `sigkill_stuck_producers()` sends — interrupted the
/// page fault). Runs OUTSIDE the preempt-disabled window; the slot is
/// owned by the reserving task regardless of CPU migration.
///
/// # Error contract (normative)
/// On `Err`, the caller MUST run the abort ladder from the protocol
/// block above, in order: `abort_slot(ring, core_side, idx, seq)` (releases the
/// RESERVED slot and balances `inflight_ops`), remove the request's
/// in-flight table entry if already enrolled (claim-by-remove), free any
/// caller-owned DMA buffer, and propagate the error. Leaking the
/// RESERVED slot pins the consumer's `tail` forever; leaking the
/// increment wedges every future quiescence — both are bugs this
/// contract exists to prevent.
fn fill_slot_data(
    ring: &VfsRingPair,
    idx: u32,
    request: &VfsRequest,
    user_src: Option<(UserPtr<u8>, usize)>, // Some for inline small writes
) -> Result<(), KernelError> {
    // Write header + args (kernel memory — cannot fault).
    // SAFETY: caller owns the RESERVED slot at `idx`.
    unsafe { vfs_write_slot(ring, idx, request); }

    // Inline small-write path: copy user bytes DIRECTLY into the slot's
    // `VfsRequestArgs::Write::inline_data` region — this is the whole
    // point of the split protocol (single copy, fault-tolerant, outside
    // the preempt-disabled window). copy_from_user() can fail with
    // EFAULT (bad user pointer) or EINTR (fatal signal interrupted the
    // page fault).
    if let Some((uptr, len)) = user_src {
        debug_assert!(len <= INLINE_IO_MAX);
        let slot = ring.request_ring.entry_ptr(idx as usize) as *mut VfsRequest;
        // SAFETY: exclusive RESERVED-slot ownership; `inline_data` is a
        // fixed [u8; INLINE_IO_MAX] field inside the Write variant.
        let dst = unsafe { inline_data_ptr(slot) };
        copy_from_user(dst, uptr, len)?; // Err → caller runs the abort ladder
    }
    Ok(())
}

/// Address of `VfsRequestArgs::Write::inline_data` within a slot.
/// A fixed offset from the slot base: `VfsRequest.args` starts at byte
/// 32 (const_assert'ed in [Section 14.2](#vfs-ring-buffer-protocol)) and the
/// `#[repr(C, u32)]` Write-variant layout places `inline_data` at a
/// compile-time-constant offset within it (stable across compilation
/// units — that is what the repr guarantees).
///
/// # Safety
/// `slot` must point to a RESERVED slot the caller owns whose header
/// `opcode` is `Write`.
unsafe fn inline_data_ptr(slot: *mut VfsRequest) -> *mut u8;

/// Mark a reserved slot as filled (data is ready for consumer).
/// Called after data fill is complete. May be called from any CPU —
/// the slot is owned by the reserving task, not bound to a CPU.
///
/// `seq` is the sequence number (head value) returned by `reserve_slot()`.
/// It is used to advance the `published` watermark: `published` tracks
/// `seq + 1` (i.e., the slot just filled is now visible). The consumer
/// uses `published` as a doorbell hint — it tells the consumer that new
/// slots may be ready up to `published`, but the consumer still checks
/// per-slot state to determine which slots are actually FILLED.
///
/// **Out-of-order completion**: When slots are completed out of order
/// (e.g., slot 6 fills before slot 5 because slot 5's copy_from_user
/// faulted), `published` may temporarily lag behind the highest FILLED
/// slot. This is correct: `published` is a *lower bound* on the highest
/// filled slot. The consumer scans from `tail` to `published` and
/// processes only FILLED slots. A RESERVED slot at position `tail` blocks
/// `tail` advancement (head-of-line blocking), but the consumer can
/// process FILLED slots ahead of it (scan-and-process model, see
/// consumer algorithm below).
///
/// **Memory ordering**: The `Release` store on `slot_state` (FILLED)
/// happens before the `Release` in `fetch_max` on `published`. The
/// consumer loads `published` with `Acquire`, establishing a happens-before
/// relationship: the consumer is guaranteed to see the FILLED state and
/// all data written by the producer.
#[inline]
fn complete_slot(ring: &VfsRingPair, core_side: &VfsRingCoreSide, idx: u32, seq: u64) {
    ring.request_ring.slot_state(idx as usize)
        .store(RingSlotState::Filled as u8, Ordering::Release);
    // Advance published watermark. Uses the reservation-time sequence
    // number (seq + 1), NOT the current head. This prevents advertising
    // slots that were reserved after this one but not yet filled.
    ring.request_ring.inner.published.fetch_max(
        seq.wrapping_add(1),
        Ordering::Release,
    );
    // Clear this task's producer-tracking pointer BEFORE publishing
    // quiescence via the inflight_ops decrement below. sigkill_stuck_producers()
    // matches a task by this pointer; if the decrement were ordered first, a
    // concurrent recovery scan could observe inflight_ops reach zero for this
    // ring while the pointer still matched, and SIGKILL a task whose operation
    // has already completed — and preemption between the two atomics makes
    // that window unbounded. Clearing ahead of the decrement closes it: the
    // decrement is the store that PUBLISHES quiescence (fetch_sub Release),
    // and the null store is sequenced before it, so any thread that observes
    // the decrement (Acquire load in wait_until_quiesced) has, through the
    // release sequence the fetch_sub heads, also observed the cleared pointer.
    // A pointer the scan still matches therefore always implies the in-flight
    // count still includes this task, making the SIGKILL legitimate rather
    // than spurious. Every producer exit path reaches here — normal
    // completion, the reserve_recheck back-out, and the fill-error unwind all
    // funnel through complete_slot() via abort_slot(). Release mirrors the
    // reserve-side store (pointer set BEFORE the increment) and the exit_task
    // clear.
    unsafe { &*current_task() }.current_vfs_ring.store(
        core::ptr::null_mut(),
        Ordering::Release,
    );
    // Decrement the in-flight counter LAST — this is the store that publishes
    // quiescence. Crash recovery and evolution wait for it to reach zero
    // before draining (ensures no producer is mid-fill). Release: the null
    // store above is sequenced before it, so an observer of the decrement
    // also observes the cleared pointer.
    let prev = ring.request_ring.inflight_ops.fetch_sub(1, Ordering::Release);
    // Sleep-not-spin quiescence wake: on the 1→0 edge (prev == 1, this ring is
    // now idle) AND only when a quiescence wait is armed on this ring, wake the
    // parked waiter. The Acquire load pairs with wait_until_quiesced's Release
    // arm; the wake is skipped in the (overwhelming) common case where nothing
    // is quiescing, so steady-state completion pays one Relaxed-cost Acquire
    // load and no wake. wait_event's post-queue predicate re-check closes the
    // arm-vs-decrement race, so this is lost-wake-proof.
    if prev == 1 && core_side.quiesce_waiters.load(Ordering::Acquire) != 0 {
        core_side.producer_quiesce_wq.wake_up_all();
    }
}

14.3.3.3 Consumer-Side Algorithm

The consumer (filesystem driver) processes slots from tail towards published. The algorithm handles out-of-order completion (RESERVED slots between FILLED slots) by using a scan-and-process model with strict tail advancement rules.

/// Consumer-side ring drain algorithm. Called by the driver's consumer
/// thread(s) when the doorbell fires or on polling wakeup.
///
/// **Invariants maintained by this function**:
/// - `tail` advances only past contiguous slots that have been processed
///   (CONSUMED → EMPTY transition). `tail` never skips a RESERVED slot.
/// - FILLED slots ahead of a RESERVED slot ARE processed (dispatched to
///   the driver's internal work queue), but `tail` does not advance past
///   the RESERVED blocker until it becomes FILLED and is processed.
/// - After processing a FILLED slot, the consumer stores EMPTY directly
///   (single owner — no race; no intermediate state is visible). A
///   consequence: while `tail` is pinned behind a RESERVED blocker,
///   `[tail..published)` legally contains EMPTY slots (already
///   processed) — the scan skips them.
///
/// **Head-of-line blocking**: A slow RESERVED slot (e.g., producer stuck
/// in a page fault during copy_from_user) blocks `tail` advancement but
/// does NOT block processing of later FILLED slots. The ring's effective
/// capacity is reduced by the number of RESERVED slots between `tail` and
/// the first unprocessed FILLED slot. Under PerCpu mode (the primary
/// mode), the producer IS the blocked task, so no other slots can be
/// reserved on this ring during the fault — head-of-line blocking is
/// moot. Under shared-ring modes, other CPUs can reserve and fill slots
/// past the blocked one; those slots are processed but `tail` stays
/// pinned until the blocked slot completes.
///
/// **Livelock prevention**: The scan window is bounded by `published`.
/// The consumer does not scan beyond `published` (which is updated
/// atomically by producers). Each scan pass has bounded work: at most
/// `published - tail` slots. If no FILLED slots are found in a pass,
/// the consumer returns and waits for the next doorbell.
fn drain_ring(ring: &VfsRingPair) {
    let mask = ring.request_ring.inner.size as u64 - 1;

    loop {
        let tail = ring.request_ring.inner.tail.load(Ordering::Acquire);
        let published = ring.request_ring.inner.published.load(Ordering::Acquire);

        if tail == published {
            return; // Ring is empty (no new slots to process).
        }

        let mut processed_any = false;

        // Scan from tail to published, processing FILLED slots.
        // Use `!=` instead of `<` for wrapping-safe comparison: pos starts
        // at tail and increments toward published; the ring size guarantees
        // published - tail <= ring.size, so the scan always terminates.
        let mut pos = tail;
        while pos != published {
            let idx = (pos & mask) as usize;
            let state = ring.request_ring.slot_state(idx)
                .load(Ordering::Acquire);

            match state {
                s if s == RingSlotState::Filled as u8 => {
                    // Process this slot.
                    // SAFETY: slot is FILLED and we are the sole consumer.
                    let request = unsafe { ring.request_ring.read_entry(idx) };
                    dispatch_to_work_queue(request);

                    // FILLED → EMPTY directly (single owner — no CAS,
                    // no intermediate state).
                    ring.request_ring.slot_state(idx)
                        .store(RingSlotState::Empty as u8, Ordering::Release);
                    processed_any = true;
                }
                s if s == RingSlotState::Reserved as u8 => {
                    // Slot not yet filled by its producer (e.g., blocked
                    // in copy_from_user page fault). Skip this slot and
                    // continue scanning — FILLED slots ahead of a RESERVED
                    // slot ARE processed (dispatched to the driver's work
                    // queue). `tail` cannot advance past this RESERVED
                    // blocker until it becomes FILLED and is processed,
                    // but processing later FILLED slots reduces latency
                    // for those requests. Under PerCpu mode, only one
                    // producer exists per ring, so a RESERVED slot means
                    // the producer is mid-fill and no later FILLED slots
                    // exist — the continue is effectively a no-op. Under
                    // shared-ring mode, other CPUs may have filled later
                    // slots that can be processed now.
                    pos = pos.wrapping_add(1);
                    continue;
                }
                s if s == RingSlotState::Empty as u8 => {
                    // LEGAL, not an invariant violation: this slot was
                    // FILLED, processed, and reset to EMPTY on an earlier
                    // pass while `tail` stayed pinned behind a RESERVED
                    // blocker before it. Skip it. (An earlier revision
                    // debug-asserted here and broke out of the scan —
                    // which wedged the consumer on every pass whenever a
                    // processed slot sat behind a blocker.)
                    pos = pos.wrapping_add(1);
                    continue;
                }
                _ => {
                    // CONSUMED (value 3) is never stored by any code
                    // path — observing it IS an invariant violation
                    // (memory corruption or a version-skewed producer).
                    debug_assert!(false,
                        "Unexpected slot state {} at pos {} (tail={}, published={})",
                        state, pos, tail, published);
                    // Release builds: skip the slot rather than break —
                    // breaking would stall tail behind a corrupt byte
                    // forever; skipping bounds the damage to one slot
                    // and the per-request timeout surfaces the loss.
                    pos = pos.wrapping_add(1);
                    continue;
                }
            }
            pos = pos.wrapping_add(1);
        }

        // Advance tail past contiguous EMPTY slots from the old tail.
        // Slots we processed above were set to EMPTY, so tail advances
        // past all of them (up to the first non-EMPTY slot).
        let mut new_tail = tail;
        while new_tail != published {
            let idx = (new_tail & mask) as usize;
            let state = ring.request_ring.slot_state(idx)
                .load(Ordering::Acquire);
            if state != RingSlotState::Empty as u8 {
                break;
            }
            new_tail = new_tail.wrapping_add(1);
        }
        if new_tail != tail {
            ring.request_ring.inner.tail.store(new_tail, Ordering::Release);
        }

        if !processed_any {
            return; // No progress this pass — wait for next doorbell.
        }
        // Loop to check if new slots were published while we were processing.
    }
}

Work-queue dispatch: drain_ring hands each FILLED request to the driver's named work queue so the (potentially blocking) filesystem operation runs off the consumer scan loop:

/// Hand a decoded VFS request to the serving domain's named work queue for
/// execution. The request's response is delivered asynchronously on the
/// completion ring keyed by `request.request_id` — this function does NOT
/// return a result to `drain_ring`.
///
/// **Borrow contract**: `request` borrows the ring slot, which `drain_ring`
/// resets to EMPTY immediately after this call returns. The implementation
/// MUST copy every value it needs (into the enqueued work item) before
/// returning — it may NOT retain the borrow past the call. `VfsRequest` is
/// `Copy`-cloned by value into the work item (owned DMA/handle references are
/// duplicated with their refcount incremented; see the slot-teardown notes).
///
/// The target work queue is the serving domain's `umkad-vfs-<domain>` pool
/// ([Section 3.4](03-concurrency.md#cumulative-performance-budget)); enqueue is a bounded, non-allocating
/// push onto a per-domain MPSC work list. Execution context on the worker is
/// process context (may sleep, may do I/O).
fn dispatch_to_work_queue(request: &VfsRequest);

Tail advancement guarantee: tail advances strictly monotonically and only past slots in EMPTY state (meaning they were FILLED, processed, and reset to EMPTY by the consumer). A RESERVED slot at position tail pins tail until that slot completes its lifecycle (RESERVED → FILLED → process → EMPTY). This ensures no slot is ever skipped or double-processed.

Ring capacity under head-of-line blocking: If one RESERVED slot pins tail, the ring's usable capacity is reduced by the number of EMPTY slots between tail and the RESERVED blocker. In the worst case (one RESERVED slot at tail, all other slots EMPTY), the ring has size - 1 usable slots — functionally identical to a normal SPSC ring. Under shared-ring modes with high contention, multiple RESERVED slots can accumulate, temporarily reducing effective capacity. The negotiation protocol's auto- selection heuristic (Section 14.3) accounts for this by allocating more slots per ring in shared modes (default depth 256 for PerNuma/PerLlc vs 64-128 for PerCpu with many rings).

---

14.3.4 Request ID Generation

Request IDs must be unique within a mount to support response matching. The baseline protocol uses per-ring monotonic IDs. With N rings, two strategies are possible:

Chosen strategy: Global atomic counter per mount.

/// Generate a mount-globally unique request ID.
///
/// All rings on this mount share a single AtomicU64 counter. This ensures
/// that request IDs are unique across all rings without per-ring namespacing.
///
/// Cost: one atomic fetch_add(1, Relaxed) per VFS operation. On x86-64,
/// this is a LOCK XADD (~15-20 cycles under contention). On AArch64
/// with LSE atomics (ARMv8.1+), LDADD is ~10-30 cycles uncontended,
/// ~40-80 cycles under 64-core contention. On ARMv8.0 without LSE or
/// RISC-V without Zacas, the LL/SC retry loop costs ~10-20 cycles per
/// attempt with ~O(N) expected attempts under N-way contention. Under
/// 64-core saturation (worst case): ~640-1280 cycles per request_id
/// allocation. Phase 3 optimization: per-CPU pre-allocated ID ranges
/// can eliminate this contention for non-LSE ARM targets.
/// Under 64-core contention, the cache line bounces — but this is a
/// DIFFERENT cache line from the ring's head/published, so it does not
/// compound with ring contention.
///
/// The alternative (per-ring monotonic + ring_index prefix) was rejected
/// because it complicates response matching in the driver and breaks the
/// existing assumption that request_id is a simple monotonic u64.
#[inline]
fn alloc_request_id(ring_set: &VfsRingSet) -> u64 {
    ring_set.next_request_id.fetch_add(1, Ordering::Relaxed)
}

Longevity analysis: At 10 billion requests per second (far beyond any conceivable filesystem workload — 100 Gbps NVMe at 4KB would be ~25M IOPS), a u64 counter wraps after ~58 years. At realistic rates (1M IOPS sustained), wrap time is ~584,000 years. Safe for 50-year uptime.

Why not per-ring monotonic IDs? Per-ring IDs would avoid the global atomic but create two problems: (a) the driver must track which ring a response belongs to, adding complexity to the response matching path; (b) the cancellation side-channel uses request_id to identify requests — with per-ring IDs, cancel tokens would need a (ring_index, request_id) pair, breaking the existing CancelToken struct layout (Section 14.2). The global counter adds ~15-80 cycles under contention (< 0.5% of minimum VFS operation latency). Per-ring IDs were rejected because: (1) breaks CancelToken uniqueness without ring_index, (2) makes driver response matching ring-aware, (3) complicates crash recovery log replay (per-ring ordering must be reconstructed from ring sequence numbers rather than using a single global ID sequence). Relaxed ordering on the counter is sufficient: the ring's Release/Acquire on published provides the happens-before guarantee between the writer (who wrote the request including its ID) and the reader. The CacheLinePadded wrapping ensures no false sharing.

---

14.3.5 Doorbell Coalescing Across N Rings

With N rings, naively ringing each ring's doorbell independently would cause N doorbell interrupts per batch of operations. The coalesced doorbell mechanism aggregates notifications across all rings in a mount.

/// Coalesced doorbell for a VfsRingSet. Instead of each ring having an
/// independent doorbell, a single coalesced doorbell aggregates pending
/// work across all rings.
///
/// The producer (VFS) sets a per-ring "pending" bit and then decides
/// whether to ring the shared doorbell based on coalescing policy.
/// The consumer (driver) checks all rings with pending bits set.
// kernel-internal, not KABI
#[repr(C, align(64))]
pub struct CoalescedDoorbell {
    /// Bitmask of rings that have new entries since the last doorbell.
    /// Bit i is set when ring i has new entries. The driver clears bits
    /// as it drains rings.
    ///
    /// AtomicU256 is not available on all architectures. For MAX_VFS_RINGS
    /// = 256, this is implemented as an array of 4 AtomicU64 values.
    /// Each AtomicU64 covers 64 rings.
    ///
    /// **32-bit leg note** (keyed on `target_has_atomic = "64"`, NOT on pointer
    /// width): ARMv7-A HAS a native 64-bit atomic (LDREXD/STREXD), slower than
    /// a native-width atomic (~10-20 cycles vs ~3-5 cycles). This is
    /// acceptable: called per-VFS-operation (hot path), not per-mount — the
    /// ~10-20 cycle overhead is <8% of minimum VFS metadata operation latency
    /// (~200-500 ns). PPC32 is the sole supported leg with NO native 64-bit
    /// atomic: 32-bit PowerPC has no 64-bit reservation (there is no paired
    /// `lwarx`/`stwcx.` 64-bit RMW), so `AtomicU64` does not exist there and a
    /// logically-64-bit value is governed by the classification rule of the
    /// 64-bit-atomic semantic family
    /// ([Section 3.5](03-concurrency.md#locking-strategy--64-bit-atomics-on-32-bit-legs-the-semantic-family)).
    /// Functional correctness is maintained on all architectures.
    pub pending_mask: [AtomicU64; 4],

    /// The actual doorbell register. Writing any non-zero value wakes
    /// the driver's consumer thread(s).
    pub doorbell: DoorbellRegister,

    /// Coalescing state (producer-side). Tracks entries since last doorbell
    /// for adaptive coalescing.
    pub coalescer: DoorbellCoalescer,
}
// CoalescedDoorbell (#[repr(C, align(64))]): pending_mask [AtomicU64; 4] = 32
// (offset 0..32), then doorbell: DoorbellRegister — itself #[repr(C, align(64))]
// occupying one cache line = 64 B ([Section 12.8](12-kabi.md#kabi-domain-runtime)) — placed at the
// next 64-aligned offset (64..128), then coalescer: DoorbellCoalescer
// (pending_count/max_batch/coalesce_timeout_us u32 ×3 + batch_start_ns u64 = 24,
// offset 128..152). Rounded up to the struct's 64-byte alignment → 192. The
// size is arch-independent: every member size (32, 64, 24) is pointer-width-
// independent.
const_assert!(core::mem::size_of::<CoalescedDoorbell>() == 192);
// kernel-internal, not KABI — CoalescedDoorbell contains DoorbellRegister and
// DoorbellCoalescer which have platform-dependent layout. Accessed only within
// Tier 0 Core via VfsRingSet.

impl CoalescedDoorbell {
    /// Mark a ring as having pending entries and optionally ring the doorbell.
    ///
    /// Called by the VFS after enqueueing a request on a specific ring.
    /// The doorbell is rung when:
    /// (a) the coalescer's pending_count reaches max_batch, OR
    /// (b) the coalescer's timeout expires (first entry in batch is older
    ///     than coalesce_timeout_us), OR
    /// (c) the request is a synchronous high-priority operation (fsync,
    ///     mount, unmount) — these bypass coalescing entirely.
    ///
    /// Cost: one atomic OR (~5-10 cycles) + conditional doorbell write.
    #[inline]
    pub fn notify(&self, ring_index: u16, force: bool) {
        let word = ring_index as usize / 64;
        let bit = ring_index as u64 % 64;
        self.pending_mask[word].fetch_or(1u64 << bit, Ordering::Release);

        if force || self.coalescer.should_ring() {
            self.doorbell.ring();
            self.coalescer.reset();
        }
    }

    /// Read and clear pending ring mask. Called by the driver consumer.
    ///
    /// Returns a snapshot of which rings have pending entries, then clears
    /// those bits. The driver iterates the set bits and drains each ring.
    #[inline]
    pub fn take_pending(&self) -> [u64; 4] {
        let mut result = [0u64; 4];
        for i in 0..4 {
            result[i] = self.pending_mask[i].swap(0, Ordering::AcqRel);
        }
        result
    }
}

Coalescing policy for VFS operations:

Operation class	Coalescing behavior	Rationale
Synchronous metadata (Fsync, Mount, Unmount, Freeze, Thaw, SyncFs)	Force doorbell immediately (force = true)	Caller is blocked waiting for completion. Coalescing adds latency with no throughput benefit.
Readahead (Readahead, ReadPage)	Coalesce up to batch-32 or 50 us timeout	Readahead is speculative; latency tolerance is high. Batch amortizes doorbell cost.
Regular I/O (Read, Write, Lookup, Create, etc.)	Coalesce up to batch-8 or 20 us timeout	Balance between latency and throughput.
Batched metadata (StatxBatch)	Coalesce entire batch into one doorbell	Already batched by io_uring path.

Performance impact: With N=64 rings and coalesced doorbells, the PostgreSQL checkpoint scenario goes from 64 independent doorbells per batch to 1 coalesced doorbell. This saves ~63 * doorbell_cost (~5-150 cycles depending on whether the doorbell is an MMIO write or a memory write with interrupt). Combined with the elimination of cache line bouncing on the ring head, the total saving per fsync batch is ~200-4500 cycles.

---

14.3.6 Mount-Time Negotiation

Ring count is negotiated between the VFS and the filesystem driver during the mount() sequence. The driver advertises its capability; the VFS selects the actual count based on system topology and mount options.

14.3.6.1 Driver Capability Advertisement

The KABI driver manifest (Section 12.6) is extended with a VFS ring count field:

/// Extension to KabiDriverManifest for VFS drivers.
/// Placed in section `.kabi_vfs_caps` adjacent to `.kabi_manifest`.
#[repr(C)]
pub struct KabiVfsCapabilities {
    /// Magic: 0x56465343 ("VFSC") — identifies a valid VFS capability block.
    pub magic: u32,
    /// Structure version (currently 1).
    pub version: u32,

    /// Maximum number of VFS request rings this driver can consume.
    /// 1 = legacy single-ring mode (backward compatible).
    /// N > 1 = driver supports multi-ring mode with up to N rings.
    ///
    /// The driver must be prepared to handle any ring_count in [1, ring_count_max].
    /// The VFS selects the actual count and communicates it during mount.
    pub ring_count_max: u16,

    /// Driver's preferred ring granularity hint. The VFS may override this
    /// based on system topology and mount options.
    pub preferred_granularity: RingGranularity,

    /// Reserved for future use. Must be zero.
    pub _reserved: [u8; 5],
}

const_assert!(size_of::<KabiVfsCapabilities>() == 16);

/// Default VFS capabilities for drivers that do not include a `.kabi_vfs_caps`
/// section (backward compatibility).
pub const KABI_VFS_CAPS_DEFAULT: KabiVfsCapabilities = KabiVfsCapabilities {
    magic: 0x56465343,
    version: 1,
    ring_count_max: 1,         // Single-ring mode (legacy).
    preferred_granularity: RingGranularity::Single,
    _reserved: [0; 5],
};

14.3.6.2 Negotiation Protocol

Mount sequence (extended from [Section 14.2](#vfs-ring-buffer-protocol)):

1. VFS reads driver's KabiVfsCapabilities from the .kabi_vfs_caps ELF section.
   If absent, use KABI_VFS_CAPS_DEFAULT (ring_count_max = 1).

2. VFS determines the desired ring count:
   a. If mount option `vfs_ring_count=N` is specified: use min(N, driver.ring_count_max).
   b. If mount option `vfs_ring_granularity=<mode>` is specified: compute ring count
      from topology (per-cpu, per-numa, per-llc).
   c. If no mount options: auto-select based on online CPU count:
      - 1-4 CPUs: 1 ring (single-ring mode, no overhead).
      - 5-16 CPUs: min(nr_numa_nodes, driver.ring_count_max) rings (PerNuma).
      - 17-64 CPUs: min(nr_llc_groups, driver.ring_count_max) rings (PerLlc).
      - 65+ CPUs: min(nr_cpus, driver.ring_count_max) rings (PerCpu).

3. VFS allocates ring_count VfsRingPair structures in shared memory.
   Each ring has independent head/tail/published/size fields.
   Ring entry size and depth are uniform across all rings (same as the
   per-mount mount options `vfs_ring_depth=N`).

4. VFS builds the cpu_to_ring mapping table.

4a. VFS allocates the Core-only per-ring companion state
   (`SuperBlock.ring_core: Box<[VfsRingCoreSide]>`, one per ring —
   in-flight table + response_drain_lock, [Section 14.2](#vfs-ring-buffer-protocol);
   NOT mapped into the driver's domain), writes the initial
   `ring_set.driver_generation` mirror (= `sb.driver_generation`),
   spawns the per-mount Core response worker (`vfs_response_worker`),
   and registers the mount's `SubsystemRecoveryDescriptor` on the
   provider's domain
   (`register_recovery_descriptor(domain_id, &sb.vfs_recovery_desc)`,
   ctx = the SuperBlock — see
   [Section 11.9](11-drivers.md#crash-recovery-and-state-preservation--subsystem-recovery-descriptors)).
   Umount reverses all four (descriptor unregistered FIRST, before the
   SuperBlock can be freed; the worker is signaled to exit via
   `sb.ring_worker_exit.store(1, Release)` + a completion-doorbell kick,
   and umount waits for the worker's task exit before freeing
   `ring_core`).

5. VFS passes the ring set to the driver during mount initialization.
   For T1 transport: the KabiT1EntryFn signature is extended to accept
   a ring array:
     entry_ring(ksvc, rings: *mut [RingBuffer], ring_count: u16) -> u32
   For backward compatibility: if ring_count == 1, this is identical to
   the existing single-ring entry point.

6. Driver initializes its consumer side for all ring_count rings.
   The driver may spawn multiple consumer threads or use a single thread
   with round-robin polling — this is an internal driver decision.

14.3.6.3 Mount Options

New mount options for per-CPU rings:

Mount option	Type	Default	Description
vfs_ring_count=N	u16	auto	Number of VFS rings. 0 = auto (topology-based). 1 = force single-ring.
vfs_ring_granularity=<mode>	string	auto	One of: auto, per-cpu, per-numa, per-llc, fixed. auto selects based on CPU count (step 2c above).
vfs_ring_depth=N	u32	256	Depth of each individual ring. Same as existing mount option. Applies uniformly to all rings.

---

14.3.7 Driver-Side Multiplexing

The filesystem driver must consume requests from N rings instead of one. Three consumer strategies are supported:

14.3.7.1 Strategy 1: Single-Thread Round-Robin (Simple Drivers)

/// Simple round-robin consumer for multi-ring VFS.
/// Suitable for filesystem drivers with a single consumer thread.
///
/// The thread iterates all rings in round-robin order, draining each ring
/// before moving to the next. The pending_mask from the coalesced doorbell
/// guides which rings to check, avoiding wasted iteration over empty rings.
///
/// **Disconnect check (mandatory)**: after EVERY doorbell wake and before
/// touching any ring contents, the loop loads the per-ring
/// `inner.state` — this is the VFS analog of the generic KABI consumer
/// loop's Phase 1.5 check (which applies only to `CrossDomainRing`, a
/// ring type VFS does not use). Recovery Step U3 stores
/// `RING_STATE_DISCONNECTED` (Release) and signals the doorbells; the
/// woken consumer's Acquire load here observes it and the thread EXITS
/// via `kthread_exit()` — Step U5a waits for exactly these task exits.
/// A consumer loop without this check can never leave the loop, U5a
/// times out (or worse, races U7's resets), and recovery aborts.
fn vfs_consumer_round_robin(
    rings: &[VfsRingPair],
    doorbell: &CoalescedDoorbell,
) {
    loop {
        // Wait for doorbell notification.
        doorbell.doorbell.wait();

        // Set-level disconnect check: ring 0's state is always valid
        // (ring_count >= 1) and U3 stores DISCONNECTED on ALL rings
        // before signaling, so one representative Acquire load suffices
        // to detect a recovery in progress after the wake.
        if rings[0].request_ring.inner.state.load(Ordering::Acquire)
            == RING_STATE_DISCONNECTED
        {
            kthread_exit(); // U5a observes the task exit
        }

        // Read which rings have pending work.
        let pending = doorbell.take_pending();

        // Iterate set bits — each bit corresponds to a ring with work.
        for word_idx in 0..4 {
            let mut bits = pending[word_idx];
            while bits != 0 {
                let bit = bits.trailing_zeros() as u16;
                let ring_idx = (word_idx as u16 * 64) + bit;
                bits &= bits - 1; // Clear lowest set bit.

                // Per-ring disconnect re-check inside the pending loop
                // (a crash can land mid-iteration).
                let ring = &rings[ring_idx as usize];
                if ring.request_ring.inner.state.load(Ordering::Acquire)
                    == RING_STATE_DISCONNECTED
                {
                    kthread_exit();
                }
                // Drain this ring completely before moving to the next.
                drain_ring(ring);
            }
        }
    }
}

14.3.7.2 Strategy 2: Per-Ring Consumer Threads (High-Performance Drivers)

/// High-performance consumer model: one kthread per ring.
///
/// Each kthread is affinity-bound to the NUMA node (or LLC group) that
/// the ring serves. This maximizes cache locality — the ring's head/published
/// cache lines stay in the consumer thread's L1/L2.
///
/// Suitable for high-IOPS filesystem drivers (ext4, XFS, btrfs) that can
/// process requests independently per ring.
///
/// The per-ring doorbell is embedded in each VfsRingPair (the existing
/// doorbell field). The coalesced doorbell is used only when strategy 1
/// is active. In strategy 2, each ring uses its own doorbell independently.
///
/// **Disconnect check (mandatory)** — same contract as Strategy 1: the
/// state is checked after EVERY wake, BEFORE touching ring contents;
/// observing DISCONNECTED exits the kthread so recovery Step U5a can
/// proceed. U3's Release store / this Acquire load is the ordering edge.
fn vfs_consumer_per_ring(
    ring: &VfsRingPair,
    ring_index: u16,
) {
    loop {
        ring.doorbell.wait();
        if ring.request_ring.inner.state.load(Ordering::Acquire)
            == RING_STATE_DISCONNECTED
        {
            kthread_exit(); // U5a observes the task exit
        }
        drain_ring(ring);
    }
}

/// **Consumer thread registration (normative, both strategies)**: VFS
/// consumer threads are created by the driver inside `vfs_init()` via
/// `kernel_services.create_kthread(entry, arg, KthreadRole::RingConsumer)`
/// — a kernel-services KABI method that executes in Core. When the
/// `KthreadRole::RingConsumer` role is passed, Core pushes the new
/// kthread's TaskId onto the calling domain's
/// `DomainDescriptor.consumer_threads` (under `domain.entry_lock`) and
/// removes it on kthread exit. This is what makes Step U5a's
/// consumer-exit barrier real (it iterates exactly that list) and keeps
/// the crash-ejection targeting statement in [Section 12.8](12-kabi.md#kabi-domain-runtime)
/// truthful for VFS domains. Threads created with other roles
/// (domain-internal workers) are NOT registered — U5a does not wait on
/// them; the U4/P4 rendezvous parks any that are on-CPU in the domain
/// and they eject at resume. (`KthreadRole` is a `#[repr(u32)]` parameter of the
/// kernel-services `create_kthread` KABI method:
/// `Worker = 0` (default, unregistered), `RingConsumer = 1`.)

14.3.7.3 Strategy 3: Adaptive (Recommended)

Drivers detect the ring count at mount time and choose: - ring_count == 1: Use the existing single-consumer model (no change). - ring_count <= 4: Use round-robin (one thread, minimal overhead). - ring_count > 4: Use per-ring consumer threads (maximum parallelism).

The threshold (4) is a heuristic — with 4 rings, one thread can drain all rings without significant latency. Above 4, the round-robin cycle time exceeds the typical operation latency and per-ring threads become worthwhile.

14.3.7.4 Response Routing

With N request rings, the driver sends responses on the corresponding response ring — ring index i's request ring has a paired response ring i. The VFS consumer for responses is already per-ring (each VfsRingPair has its own response_ring and completion WaitQueue). A thread blocked on a synchronous VFS operation sleeps on the specific ring's WaitQueue, not a global one. When the response arrives on ring i's response_ring, only threads waiting on ring i are woken.

Request flow:
  CPU 7 → cpu_to_ring[7] = ring 2 → ring_set.rings[2].request_ring → enqueue

Response flow:
  Driver dequeues from ring 2's request_ring
  Driver enqueues response on ring 2's response_ring
  VFS consumer for ring 2 wakes threads on ring 2's completion WaitQueue

This ensures that the response arrives on the same ring where the request was submitted. The thread that submitted the request is sleeping on that ring's WaitQueue and is woken directly — no global WaitQueue scanning.

---

14.3.8 Driver-Side Ring Entry Prefetch

When a filesystem driver's consumer thread dequeues a request from the ring, the ring protocol specifies that the consumer prefetches the next N ring entries into CPU cache while processing the current request. This hides the memory latency of reading ring entries behind the computation/I/O latency of processing the current request.

/// Prefetch the next `prefetch_count` ring entries after the current dequeue
/// position. Called by the driver's consumer loop immediately after dequeuing
/// a request, before beginning I/O processing for that request.
///
/// The prefetch is a cache hint (software prefetch instruction); it does not
/// modify the ring state or advance the consumer pointer. If the prefetched
/// entries are not yet published (tail has not advanced that far), the
/// prefetch is a harmless no-op (prefetching an unpublished slot reads
/// stale data that will be overwritten before the consumer reaches it).
///
/// **Architecture mapping**:
/// - x86-64: `_mm_prefetch(ptr, _MM_HINT_T0)` — prefetch into L1.
/// - AArch64: `PRFM PLDL1KEEP, [ptr]` — prefetch for load, keep in L1.
/// - ARMv7: `PLD [ptr]` — data prefetch.
/// - RISC-V: no standard prefetch instruction (Zicbop extension adds
///   `prefetch.r`; fallback is no-op on cores without Zicbop).
/// - PPC32/PPC64LE: `dcbt` — data cache block touch.
/// - s390x: no user-accessible prefetch; no-op.
/// - LoongArch64: `preld` — prefetch for load.
///
/// **Prefetch count**: 4 entries is the default. This covers the typical
/// pipeline depth where the driver has dispatched the current request to
/// a work queue and is about to dequeue the next. At ~320 bytes per
/// `VfsRequest` entry, 4 entries = 1,280 bytes = ~20 cache lines. This
/// fits comfortably in L1 on all architectures (minimum L1 = 16 KiB on
/// ARMv7 Cortex-A15). Configurable per-driver via the KABI manifest
/// field `ring_prefetch_count` (default 4, range 0-16, 0 = disabled).
///
/// **Cost**: ~1-4 cycles per prefetch instruction (overlapped with
/// current request processing — effectively zero additional latency).
///
/// **Benefit**: Eliminates ~50-100 cycles of L2/L3 read latency per
/// ring entry dequeue (the entry is already in L1 when the consumer
/// reaches it). Over a batch of 8 dequeues, this saves ~400-800 cycles.
#[inline(always)]
fn prefetch_ring_entries(
    ring: &RingBuffer<VfsRequest>,
    current_tail: u64,
    prefetch_count: u32,
) {
    let mask = ring.inner.size as u64 - 1; // ring size is power-of-2
    for i in 1..=prefetch_count {
        let idx = (current_tail.wrapping_add(i as u64)) & mask;
        // entry_ptr() returns a raw pointer WITHOUT dereferencing —
        // exactly what prefetch needs (the slot may be unpublished).
        let entry_ptr = ring.entry_ptr(idx as usize);
        // SAFETY: entry_ptr is within the ring's allocated data region
        // (bounded by mask). The prefetch is a hint and does not dereference
        // the pointer; no memory safety violation even if the slot is
        // unpublished.
        unsafe { arch::current::cpu::prefetch_read(entry_ptr as *const u8); }
    }
}

Integration point: The prefetch call is inserted into the consumer loop between "dequeue current entry" and "dispatch to work queue":

loop {
    let entry = ring.dequeue();           // Read current request
    prefetch_ring_entries(ring, tail, 4); // Prefetch next 4 while processing
    dispatch_to_work_queue(entry);        // Dispatch (may involve I/O)
    tail = tail.wrapping_add(1);
}

---

14.3.9 Completion Coalescing (Response Direction)

The doorbell coalescing mechanism (above) batches request-direction notifications (Tier 0 VFS → Tier 1 driver). An analogous mechanism is needed for the response direction (Tier 1 driver → Tier 0 VFS) to avoid waking the VFS consumer on every individual completion.

Problem: Without completion coalescing, a driver that completes 8 requests in rapid succession (e.g., 8 readahead pages arriving from NVMe in a single interrupt) generates 8 separate WaitQueue wakeups on the VFS side. Each wakeup involves an IPI to the waiting CPU (~200-500 cycles on x86-64 cross-core) plus the WaitQueue wake protocol (~50-100 cycles). For 8 completions: ~2,000-4,800 cycles of wakeup overhead.

Solution: The driver batches completions on the response ring and signals the VFS with a single completion doorbell after writing N responses (or after a configurable timeout).

/// Completion coalescing state, embedded in each VfsRingPair.
/// The driver side accumulates completions and signals the VFS consumer
/// only after the batch threshold is reached or the coalescing timeout
/// expires.
///
/// **Classification**: Nucleus (data structure), Evolvable (threshold/timeout
/// parameters are ML-tunable via ParamId 0x0309 and 0x030A).
///
/// Interior-mutability fields (`pending_completions`, `first_unsignaled_cycles`)
/// allow `should_signal` to take `&self`, which is required because the call
/// site holds only `&VfsRingPair` (a shared reference into the ring-set).
/// Both fields are accessed exclusively by the single driver consumer thread
/// (SPSC discipline); Relaxed ordering suffices — there are no ordering
/// dependencies between these fields and the response_ring's Release/Acquire
/// publish protocol. Config fields (`batch_threshold`, `coalesce_timeout_cycles`)
/// are immutable after initialization and need no interior mutability.
pub struct CompletionCoalescer {
    /// Number of responses written since the last VFS wakeup.
    /// Incremented by the driver on each response_ring enqueue.
    /// Reset to 0 after signaling the VFS.
    pub pending_completions: AtomicU32,

    /// Batch threshold: signal VFS after this many completions.
    /// Default: 8 for regular I/O, 1 for synchronous operations
    /// (Fsync, Mount — these bypass coalescing because the caller
    /// is blocked waiting for exactly one response).
    pub batch_threshold: u32,

    /// Timestamp (in TSC or arch-equivalent monotonic cycles) of the
    /// first unsignaled completion. If the time since first unsignaled
    /// completion exceeds `coalesce_timeout_cycles`, signal the VFS
    /// regardless of batch size. This bounds worst-case latency for
    /// sparse completion streams.
    ///
    /// **CPU migration note**: If the consumer thread migrates between
    /// CPUs, TSC on the new CPU may differ from the old CPU (pre-Zen3
    /// AMD, non-constant-TSC platforms). The `wrapping_sub` check in
    /// `should_signal()` treats a negative delta as a very large positive
    /// value, causing immediate signal — a false positive (extra wakeup),
    /// not a missed wakeup. This is benign: one extra wakeup per
    /// migration event (~once per seconds at most). On AArch64 (CNTVCT_EL0
    /// is globally synchronized) and x86 with constant_tsc + nonstop_tsc,
    /// migration has no effect on cycle counter monotonicity.
    ///
    /// **Recommendation**: Driver consumer threads SHOULD be affinity-pinned
    /// to a NUMA node or LLC group (see Strategy 2: Per-Ring Consumer Threads).
    /// Pinned threads avoid this edge case entirely.
    pub first_unsignaled_cycles: AtomicU64,

    /// Coalescing timeout in cycles. Default: ~10 us worth of cycles
    /// (e.g., ~30,000 cycles at 3 GHz). Converted from the ML-tunable
    /// parameter `vfs_completion_coalesce_timeout_us` (ParamId 0x030A)
    /// at mount time and on parameter update.
    pub coalesce_timeout_cycles: u64,
}

impl CompletionCoalescer {
    /// Called by the driver after writing a response to the response ring.
    /// Returns `true` if the VFS should be signaled (wake the completion
    /// WaitQueue), `false` if the completion should be coalesced.
    ///
    /// Takes `&self` because the call site holds `&VfsRingPair` (shared
    /// reference). Interior mutability via AtomicU32/AtomicU64 provides
    /// the required mutation under `&self`.
    ///
    /// **Synchronous bypass**: If the response is for a synchronous
    /// operation (Fsync, Mount, Unmount, Freeze, Thaw, SyncFs), this
    /// function always returns `true` — the caller is blocked and must
    /// be woken immediately.
    #[inline]
    pub fn should_signal(&self, is_sync_op: bool) -> bool {
        if is_sync_op {
            self.pending_completions.store(0, Ordering::Relaxed);
            return true;
        }

        let pending = self.pending_completions.fetch_add(1, Ordering::Relaxed) + 1;

        if pending == 1 {
            self.first_unsignaled_cycles.store(
                arch::current::cpu::read_cycle_counter(),
                Ordering::Relaxed,
            );
        }

        if pending >= self.batch_threshold {
            self.pending_completions.store(0, Ordering::Relaxed);
            return true;
        }

        let now = arch::current::cpu::read_cycle_counter();
        if now.wrapping_sub(self.first_unsignaled_cycles.load(Ordering::Relaxed))
            >= self.coalesce_timeout_cycles
        {
            self.pending_completions.store(0, Ordering::Relaxed);
            return true;
        }

        false
    }
}

Driver integration: After writing each VfsResponseWire to the response ring:

// response.driver_generation was stamped from
// ring_set.driver_generation.load(Acquire) at construction (see
// VfsRingSet.driver_generation).
ring.response_ring.enqueue(response);
ring.response_ring.inner.published.store(new_published, Release);
if ring.completion_coalescer.should_signal(is_sync_op) {
    // Wake sync waiters on this ring (they opportunistically drain) ...
    ring.completion.wake_up_all();
    // ... and the per-mount Core response worker, which is what drains
    // async completions (ReadPage/Readahead/WritePage) that have no
    // parked waiter. See the response matching protocol in
    // [Section 14.2](#vfs-ring-buffer-protocol).
    ring_set.completion_doorbell.notify(ring_index, /* force */ true);
}

Performance impact: With completion coalescing at batch=8, the 8-readahead scenario generates 1 wakeup instead of 8. Savings: ~1,750-4,200 cycles per readahead batch. Combined with request-side doorbell coalescing, a full readahead cycle (8 requests batched into 1 doorbell + 8 responses batched into 1 wakeup) saves ~2,000-5,000 cycles total vs. the uncoalesced baseline.

---

14.3.10 Crash Recovery

Crash recovery (Section 11.9) must drain ALL N rings when a filesystem driver crashes.

14.3.10.1 Unified VFS Driver Crash Recovery Sequence

The base protocol (Section 14.2) defines VFS-specific Steps 1-5.5. The general recovery protocol (Section 11.9) defines Steps 1-9. This section specifies the canonical merged sequence — the single authoritative ordering of all steps. Both base protocol and general recovery descriptions are normative for their individual step content, but THIS section defines the step ordering and interleaving. An implementing agent follows this sequence top-to-bottom.

Invocation — how the U-steps get called. The generic crash-recovery machinery has ZERO VFS knowledge. The connection is the SubsystemRecoveryDescriptor registration pattern (Section 11.9) — the same descriptor-registration shape as CheckerDescriptor and TypeDescriptor: the Evolvable subsystem registers a data block of function pointers plus an opaque ctx, and the generic worker walks the registry at fixed hook points.

At MOUNT time — when the mount binds the filesystem provider's domain and allocates the VfsRingSet — the VFS registers one descriptor on that domain (register_recovery_descriptor(domain_id, &sb.vfs_recovery_desc), under domain.entry_lock), with ctx = sb as *const SuperBlock. This registration IS the DomainId → SuperBlock mapping: no global domain→superblock table exists or is needed — the descriptor walk hands each hook its own superblock back. The descriptor is unregistered at umount, before the SuperBlock is freed. One descriptor per mount; a domain hosting N mounts (one provider serving several superblocks, or grouped providers) carries N descriptors and the worker walks all of them at each hook point. The domain registry is runtime-scaled (Section 11.9); registration failure at mount time is ENOMEM through the normal mount error path.

The VFS descriptor's hook bindings (each hook's step bracket is marked in the sequence below):

Hook	Context	Runs
isolate_fn	Exception, immediately after general Step 2' (crash_lock held; stores + doorbell signals only — no sleeping, no ordered locks)	U3
quiesce_and_drain_fn	Process (recovery worker, recovery_mutex held; may sleep), between general Steps 3 and 4	U5, U5a, U6, U7, U8, U9, U10, U10a, U10b
reload_fn	Process, inside general Step 8 — after module load + Hello + key re-arm (U14(c')), before the driver is declared ready	U14(d)-(e), U15
resume_fn	Process, at general Step 9, after the Step 9 Recovering → Active CAS succeeds	U17, U18 gating

An Err from quiesce_and_drain_fn or reload_fn aborts the recovery per Reload Failure Handling (Section 11.9), leaving ring_set.state = RECOVERING (producers keep getting ENXIO) — identical to U5a's abort arm. The Post-Reload event bus (device_recovery_subscribe, Step 9) is NOT this mechanism: the event bus serves subsystems that DEPEND on a recovered device (e.g., VFS on a recovered block driver); the descriptor serves the subsystem whose OWN provider lives in the crashed domain.

UNIFIED VFS DRIVER CRASH RECOVERY SEQUENCE

Step U1.  [General 1]    FAULT DETECTED
          Hardware exception / watchdog / ring corruption in Tier 1 domain.

Step U1a. [General 1a]   TIER CHECK
          If effective_tier() == IsolationTier::Tier0: panic (no isolation).
          If IsolationTier::Tier1 or IsolationTier::Tier2: proceed.

Step U2.  [General 2]    ISOLATE
          Revoke domain (PKRU AD bit / POR_EL1 / DACR). Mask interrupts.

Step U3.  [isolate_fn — exception ctx, after General 2']
          SET RING STATE + WAKE SLEEPING CONSUMERS
          Invoked by the exception handler's descriptor walk right
          after general Step 2' (VFS rings are NOT CrossDomainRings on
          the DomainDescriptor ring arrays — Step 2' cannot reach them;
          this hook is what does).
          ring_set.state = RECOVERING (blocks select_ring()).
          fence(SeqCst) — the recovery half of the store-buffering
          pairing with reserve_slot()'s post-increment re-check.
          Per-ring inner.state = RING_STATE_DISCONNECTED.
          Then signal every per-ring doorbell and the ring set's
          CoalescedDoorbell + completion_doorbell. This wake is
          MANDATORY, not hygiene: a consumer kthread sleeping in
          doorbell.wait() slept through the Core brackets with NO open
          window — U4's crash ejection covers only parked/open windows
          (which eject at resume) and can never touch a legally
          sleeping thread. The signal wakes each sleeping
          consumer, whose per-wake DISCONNECT CHECK (Strategy 1/2
          consumer loops, [Section 14.3](#vfs-per-cpu-ring-extension--driver-side-multiplexing))
          observes DISCONNECTED (Release store above / Acquire load)
          and exits via kthread_exit() WITHOUT entering the revoked
          domain. (The generic kabi Phase 1.5 check applies only to
          CrossDomainRing consumer loops — VFS consumer loops carry
          their own equivalent check; citing Phase 1.5 here was a
          category error in an earlier revision.) Without this wake, a
          consumer could sleep through U3-U7, wake after U7(f) restores
          per-ring ACTIVE, and attempt switch_domain() into the
          domain revoked in U2 — a cascading secondary crash.
          See set_all_rings_disconnected() below.

Step U4.  [General 2a]   NMI RENDEZVOUS (PARK + ACKNOWLEDGE)
          NMI IPI to all CPUs: entry stubs park any open window's
          transit words; handlers ack. In-domain threads eject AT
          RESUME (deny-all unpark -> fault -> crash trampoline;
          [Section 11.9](11-drivers.md#crash-recovery-and-state-preservation)).

Step U5.  [quiesce_and_drain_fn; VFS Step 1]   QUIESCE PRODUCERS
          First step of the process-context descriptor hook (recovery
          worker, recovery_mutex held, between general Steps 3 and 4).
          Wait for all per-ring inflight_ops == 0 (5s timeout).
          This ensures no producer is mid-copy_from_user with a
          RESERVED slot. On timeout: SIGKILL stuck producers (a REAL
          kill — queue SIGKILL, then wake), RE-ENTER the wait for a
          second 5s window (killed producers unwind through the
          fill_slot_data error ladder → abort_slot(), which balances
          inflight_ops), and if the count is STILL nonzero, ABORT the
          recovery (Err(QuiesceTimeout) from quiesce_and_drain_fn —
          same terminal state as U5a's abort arm; BOTH abort arms run
          abort_terminal_rescue() first, so already-parked page-lock and
          writeback waiters reach their EIO terminal state even though
          U6/U7(g)/U10b never run). NEVER proceed to U6
          with a nonzero count.
          See wait_for_producers_quiesced() below.

Step U5a. [quiesce_and_drain_fn; VFS Step 1a]  WAIT FOR OLD CONSUMER EXIT
          For each TaskId in DomainDescriptor.consumer_threads, wait
          until the kthread has exited (task exit state). VFS consumer
          TaskIds are ON that list because
          kernel_services.create_kthread(.., KthreadRole::RingConsumer)
          registers them at creation time (see "Consumer thread
          registration" under Driver-Side Multiplexing) — without that
          registration this barrier would pass VACUOUSLY and U6/U7
          would race live consumers. Every old consumer exits by
          exactly one of THREE paths: (i) it was executing in the
          domain at crash time — parked/ejected at resume (U4's
          rendezvous + at-resume ejection) and self-terminated via
          kthread_exit() in domain_crash_trampoline()
          ([Section 11.9](11-drivers.md#crash-recovery-and-state-preservation)); (ii) it was
          sleeping in doorbell.wait() — woken by U3's doorbell signal,
          observed DISCONNECTED at its per-wake disconnect check, and
          exited via kthread_exit(); or (iii) it was blocked in a
          NESTED sync kabi_call into a healthy third domain (a
          dispatched method blocking inside a producer_core_section) —
          a LEGAL, SELF-RESOLVING hold-out bounded by the NESTED
          call's timeout_ns (DEFAULT_KABI_TIMEOUT_NS = 5s, or the
          nested service's manifest override, e.g. 60s SCSI tape;
          [Section 12.8](12-kabi.md#kabi-domain-runtime)): when the nested wait completes,
          the consumer's next disconnect check / deny-all re-entry
          fault ejects it.
          TWO-TIER TIMEOUT — path (iii) is why the previous flat "5s
          defensive timeout" was wrong: it exactly RACED the nested
          call's 5s default, so U5a could declare "kernel bug" and
          abort (device offline) a recovery whose hold-out was about
          to resolve by itself. The bound must DOMINATE the nested
          worst case (see the derivation note on
          DEFAULT_KABI_TIMEOUT_NS, [Section 12.8](12-kabi.md#kabi-domain-runtime)):
            - WARN at 5s: emit an FMA warning naming the ring set and
              the hold-out TaskId; KEEP polling (the generic Teardown
              Liveness step-5 posture).
            - ABORT only at 2 x max(timeout_ns across the crashed
              domain's resolved outbound service bindings,
              DEFAULT_KABI_TIMEOUT_NS), computed by the recovery
              worker from the domain's module descriptors' handles
              (process context, recovery_mutex held — cold path).
              Past that bound no legal nested wait can still be
              pending, so a live consumer indicates a kernel bug
              (missed wake, ejection failure), not a stuck userspace:
              emit an FMA event, run abort_terminal_rescue() (defined
              with the Step U5 functions below — the abort skips
              U6/U7(g)/U10b, the only wakers for already-parked
              page-lock and writeback sleepers, so their EIO terminal
              wakes must happen on this arm; no waiter sleeps without a
              waker), then abort the recovery sequence with an
              error (device goes offline via Reload Failure Handling,
              [Section 11.9](11-drivers.md#crash-recovery-and-state-preservation)), and leave
              ring_set.state = RECOVERING so producers keep getting
              ENXIO — never proceed to U7(f) with a live old consumer.
          This barrier MUST precede U7: U7(e') resets the doorbells
          (discarding the U3 wake latch) and U7(f) restores per-ring
          ACTIVE — both are only safe once no old consumer can still
          observe them. It also strengthens U6/U7's exclusive-access
          safety argument: after U5 (producers quiesced) and U5a
          (consumers exited), the recovery worker is the sole thread
          touching the rings.

Step U5b. [quiesce_and_drain_fn; VFS Step 1b]  RESPONSE-DRAINER BARRIER
          For each ring, acquire then immediately release
          core_side.response_drain_lock. ring_set.state==RECOVERING is
          already published (U3); every response drainer
          (vfs_drain_response_ring, opportunistic sync-waiter drains, the
          per-mount response worker) holds this lock for its WHOLE pass
          and re-checks ring_set.state under it at the top of the pass, so
          this acquire+release handoff is a full barrier: any in-progress
          drain pass has finished, and no later pass can proceed past its
          state check. After U5b the recovery worker is provably the only
          actor performing terminal side effects on the in-flight table.
          U6's exclusive-access claim rests on U5+U5a+U5b.

Step U6.  [quiesce_and_drain_fn; VFS Step 2a]  SWEEP THE IN-FLIGHT TABLE
          Sweep each ring's Core-resident in-flight table
          (core_side.inflight — [Section 14.2](#vfs-ring-buffer-protocol)), NOT
          the ring memory: claim-by-remove every entry. For each claimed
          ReadPage/Readahead entry, perform its terminal page state
          INLINE, per claimed entry, BEFORE freeing the entry: for each
          page index in entry.page_index..+entry.nr_pages, look up the
          page in entry.inode.i_mapping.page_cache under RCU and, if
          present, set PageFlags::ERROR and unlock_page() (waking any
          wait_on_page_locked() waiter -> EIO). The inode pin
          (entry.inode: Arc<Inode>, captured at submit time) keeps the
          embedded i_mapping AddressSpace live across this deref — there
          is no raw page_cache_id pointer to dangle. Then free the
          Core-captured DMA handle and rcu-free the entry.
          The ring contents are UNTRUSTED (driver-writable shared
          memory) and INCOMPLETE (the consumer empties slots at
          dispatch, so dispatched-but-uncompleted requests are absent
          from [tail..published) — but never absent from the table).
          Exactly-once terminal page state: claim-by-remove (atomic
          XArray remove) yields ONE winner per entry; U5b guarantees no
          response drainer completes an entry concurrently; a completed
          ReadPage's entry was already removed (and its page already
          unlocked) by the pre-crash response drain, so it is not in the
          table; waiters claim only their OWN sync entries and perform no
          page unlocks; and a locked page cannot be evicted, so the RCU
          lookup's None arm is pure defense-in-depth. No fixed cap, no
          "manual intervention" arm: every page reaches a terminal
          (unlocked, ERROR) state in this one pass.
          See drain_all_vfs_rings() below.

Step U7.  [quiesce_and_drain_fn; VFS Step 2b + General 3]  DRAIN AND RESET RINGS
          For each ring:
            (a) Acquire core_side.response_drain_lock and hold it across
                (d)(e)(e')(f) ONLY. This is now uncontended
                defense-in-depth: U5b already excluded every response
                drainer (acquire+release barrier while state==RECOVERING),
                so no concurrent consumer remains — the lock here simply
                keeps the reset atomic against a mis-ordered late drainer.
                Response-ring contents are DISCARDED wholesale by the
                pointer reset — pre-crash completions are suspect; their
                table entries were already reaped in U6.
            (b) Semantics only (no wake here): pending requests fail —
                waiters observe state != ACTIVE, claim-or-lose their
                entry, and return EIO. The wake itself is step (g),
                issued AFTER the drain lock is released (two-phase
                discipline: response_drain_lock is a leaf, every wakeup
                happens outside it). After U6 + the reset, each ring's
                in-flight table is EMPTY — no stale pre-crash entry
                survives recovery (u64 request_ids never repeat, so a
                leaked entry would be permanent; this purge is the
                50-year no-leak guarantee for the table).
            (c) (REMOVED — DMA-handle freeing moved into U6's table
                sweep, which frees from Core-captured copies instead of
                trusting ring memory. Letter retained so (d)-(f)
                references stay stable.)
            (d) Reset all slot_states to EMPTY.
            (e) Reset ring pointers (head=tail=published=0).
            (e') Reset doorbells: ring.doorbell.reset() and the ring
                set's CoalescedDoorbell AND completion_doorbell
                (doorbell.reset() + clear pending_mask + coalescer
                counters, each). DoorbellRegister is
                level-latched ([Section 12.8](12-kabi.md#kabi-domain-runtime--doorbellregister-cross-domain-signaling-primitive)):
                a signal latched by the crashed driver (or by a producer
                just before U3) would otherwise survive the ring reset
                and spuriously wake the NEW consumer's first wait() for
                work that no longer exists. A missed reset is benign
                (consumer re-checks tail..published, finds nothing,
                re-sleeps) — this step is hygiene, and it also
                guarantees the new consumer's doorbell starts from a
                known-quiescent state so post-recovery wake accounting
                (coalescer batching) is not skewed by a phantom first
                signal.
            (f) Reset per-ring inner.state to RING_STATE_ACTIVE.
                NOTE: Between U7(f) and U17, individual ring states are ACTIVE
                while ring_set.state remains RECOVERING. This mixed state is
                safe and intentional:
                - Producers: blocked by ring_set.state == RECOVERING at
                  select_ring() — no new requests can be enqueued.
                - Consumer: the new driver (loaded in U14) needs ACTIVE
                  per-ring state to start its consumer loop. If ring.state
                  were still RING_STATE_DISCONNECTED, the consumer loop's
                  per-wake disconnect check would immediately exit it.
                - Old consumers: cannot observe this ACTIVE — the U5a
                  barrier guaranteed every old consumer thread exited
                  before U7 began.
                The Core response worker still backs off until U17
                (its gate is ring_set.state, not the per-ring state).
            (g) Release the drain lock, THEN ring.completion.wake_up_all()
                — UNCONDITIONAL, matching the two-phase discipline (every
                wakeup happens OUTSIDE response_drain_lock) and the
                already-fixed executable snippet in drain_all_vfs_rings().
                A blocked synchronous waiter's liveness is tracked by the
                Core in-flight table, not request-ring occupancy, so the
                wake must NOT be gated on ring state/occupancy: a ring
                with tail == published can still hold a waiter that U6
                just made completable. Woken waiters re-evaluate
                (state != ACTIVE || completed) and return EIO; a spurious
                wake of an empty queue is free.
          See drain_all_vfs_rings() below.

Step U8.  [quiesce_and_drain_fn; VFS Step 2c]  (RENUMBERING-STABILITY NOTE)
          Orphaned-page terminal state is now performed INLINE in U6, per
          claimed entry (set PageFlags::ERROR + unlock_page() as each
          ReadPage/Readahead entry is claimed), NOT as a separate wake
          pass. There is no orphaned-page collection and no fixed cap:
          the exactly-once proof lives in U6 (claim-by-remove yields one
          winner; U5b excludes concurrent drainers; a completed
          ReadPage's entry was already removed and its page unlocked by
          the pre-crash response drain; waiters unlock no pages; a locked
          page cannot be evicted, so the None arm is defense-in-depth).
          The step number is retained so U9+ references stay stable.

Step U9.  [quiesce_and_drain_fn; VFS Step 2.5] PAGE CACHE INTEGRITY CHECK
          Walk XArray trees for corruption detection.

Step U10. [quiesce_and_drain_fn; VFS Step 3]   DIRTY PAGE DETECTION
          Pin ALL dirty pages for deferred writeback (U15). Core does
          NOT consult journal-transaction state — that state lives in
          the crashed domain; see the Core-only rule in
          [Section 14.2](#vfs-ring-buffer-protocol). The committed-extent fast
          flush runs in U10b on the Core-resident DirtyIntentEntry
          substrate.

Step U10a. [quiesce_and_drain_fn; VFS Step 5.5] GENERATION COUNTER BUMP
          (moved before U11)
          sb.driver_generation.fetch_add(1, Release), THEN copy the new
          value into the driver-readable mirror:
          ring_set.driver_generation.store(new, Release). The sb field
          is authoritative; the mirror is what the reloaded driver
          stamps into every VfsResponseWire (it cannot read the
          SuperBlock). Single writer for both: this recovery worker
          (mount init wrote the initial values).
          **This MUST happen BEFORE the U11 io_uring CQE drain AND before
          driver reload (U14).** Two independent reasons:
          (1) Before U11: U11 publishes failure CQEs to the userspace
              io_uring CQ ring (res = -EIO for dispatched in-flight
              SQEs, -ECANCELED for their never-dispatched linked
              successors — the canonical assignment stated at U11).
              Userspace can observe such a CQE and
              immediately submit a new SQE on the same fd via
              io_uring_enter(). With the bump published BEFORE the CQE
              stores, the Release publication of each CQE carries the
              bump with it: any submission path that observes the CQE
              (Acquire) and then loads sb.driver_generation (Acquire)
              sees the NEW generation, so vfs_check_open_generation()
              fails with ENOTCONN instead of admitting the request at
              the stale generation. With the old ordering (bump after
              the drain), weak-memory architectures (AArch64, RISC-V,
              PPC) permitted a remote CPU to observe the CQE before the
              bump — a resubmission window closed only by the
              ring_set.state check. This ordering makes generation
              staleness a second, INDEPENDENT line of defense; the
              select_ring() ENXIO rejection (ring_set.state ==
              RECOVERING until U17) remains the first.
          (2) Before U14 reload: the new driver instance and all its
              responses (including writeback completions in U15) must
              carry the NEW generation. If the bump were after U15 (the
              old U16 position), writeback responses from U15 would
              carry the OLD generation and be discarded by the VFS
              consumer's `response.driver_generation ==
              sb.driver_generation` check, causing **silent data loss**
              (dirty pages completed by the driver but not marked clean).

Step U10b. [quiesce_and_drain_fn; Writeback]  DRAIN WRITEBACK RING +
          BYPASS FLUSH
          The writeback subsystem talks to the filesystem provider over
          its OWN ring (WritebackRequest/WritebackResponse,
          [Section 4.6](04-memory.md#writeback-subsystem--writeback-cross-domain-dispatch))
          — a ring this sequence would otherwise never touch, leaving
          fsync callers blocked in filemap_write_and_wait_range phase 2
          on pages whose WRITEBACK flag nobody clears.
          (a) Drain/discard the writeback response ring with the same
              EIO-discard semantics as U7(a); reset its pointers.
          (b) Run the writeback CRASH RECOVERY BYPASS steps 1-3
              ([Section 4.6](04-memory.md#writeback-subsystem--writeback-cross-domain-dispatch)):
              direct block writes for committed DirtyIntentEntry
              records (Core-resident block_addr/sb_dev/block_dev — no
              driver cooperation). Legal here: the block path is
              operational — either the block driver is in a different,
              healthy domain, or (same-domain crash) block recovery ran
              to completion before VFS recovery started per the
              interleaving constraint below. U10b < U14(e) orders these
              direct writes BEFORE the journal replay — closing the
              torn-journal hazard of an unordered bypass.
          (c) Complete writeback on the pages of the FLUSHED committed
              extents via the shared `writeback_page_epilogue(page, 0)`
              (bypass step 5). The
              epilogue clears WRITEBACK AND calls `page.wake_waiters()`:
              clearing the flag ALONE does not wake an already-parked
              `wait_on_page_writeback()` sleeper (`wait_event` re-evaluates
              its predicate only on a wakeup), so the explicit
              `wake_waiters()` inside the epilogue — not the bare flag
              clear — is the ordered wake for fsync phase-2 waiters. Each
              fully-flushed committed entry is then REMOVED from the
              intent list via its defined terminal API,
              `vfs_flush_extent_complete()`
              ([Section 14.1](#virtual-filesystem-layer--dirty-page-handling-on-vfs-crash)
              names Core's bypass as its second sanctioned caller).
          (c') CRASH-STRANDED WRITEBACK REQUEUE (Phase-1 and
              failed-committed pages — bypass step 4). The cross-domain
              writeback submitter clears DIRTY and sets WRITEBACK on
              every dispatched page BEFORE the provider runs
              ([Section 4.6](04-memory.md#writeback-subsystem--writeback-cross-domain-dispatch)
              step 2), so a dispatched page whose intent never reached
              Phase 2 (`block_addr` still `None`) — and any committed
              page whose direct write in (b) failed — is PageFlags::WRITEBACK,
              NOT dirty, at this point. Such a page has no completion
              coming (the crashed driver will never respond) and would
              be INVISIBLE to U15's DIRTY-list iteration. For each such
              page (enumerated from the intent entries: the Phase-1
              entries' `[file_offset, file_offset+len)` ranges plus the
              failed committed ranges — every dispatched page is covered
              by an intent entry, reserved at dirtying time), run the
              shared ERROR epilogue `writeback_page_epilogue(page, -EIO)`:
              its error arm re-dirties the page (restoring both
              `nr_dirty` counters on the 0→1 edge), records the error in
              the mapping's ErrSeq (`wb_err.set_err`), clears WRITEBACK
              (decrementing both writeback counters), and wakes waiters
              ([Section 15.2](15-storage.md#block-io-and-volume-management--writeback-io-completion-callback)).
              A parked fsync phase-2 waiter thus wakes and TRUTHFULLY
              reports EIO via its ErrSeq check — its data did not reach
              stable storage during the outage; the data itself is NOT
              lost: the page is DIRTY again and U15's re-write through
              the reloaded driver flushes it. Phase-1 intent ENTRIES
              stay in the list (they still describe dirty data for any
              subsequent crash).
              POSTCONDITION (load-bearing for U15): after (c'), NO page
              of this superblock remains in crash-stranded PageFlags::WRITEBACK
              — every dispatched page is either clean (committed,
              flushed, epilogued in (c)) or DIRTY again (requeued here);
              the only WRITEBACK pages that exist while U15 runs are
              U15's own submissions.
          Trigger unification: fault-initiated recovery reaches this
          step via the descriptor walk; the KABI health monitor
          (missed heartbeat / ring overflow) is a DETECTOR that
          enqueues a CrashRecoveryRequest — it does not run the bypass
          independently, so the bypass executes exactly once, at this
          ordered slot. Placement after U10a is deliberate: bypass
          completions and their CQE interactions observe the new
          generation.

Step U11. [General 4]    DRAIN PENDING I/O
          Complete all remaining user requests with EIO.
          Post io_uring failure CQEs under the CANONICAL errno
          assignment (normative — this specific rule OVERRIDES the
          blanket "EIO" wording of the general crash protocol for
          io_uring CQEs, and [Section 19.3](19-sysapi.md#io-uring-subsystem) states the same
          assignment at the io_uring end; the two sites are one rule):
          an SQE that was DISPATCHED and whose I/O was killed in flight
          by the crash completes with res = -EIO — its I/O genuinely
          failed, and the Linux io_uring contract reports the
          underlying I/O error for a request whose I/O failed, while
          -ECANCELED is reserved for cancelled, never-executed requests
          (Linux `io_uring/io_uring.c:req_fail_link_node`;
          Linux `io_uring/timeout.c:io_fail_links` / `io_req_tw_fail_links`). A
          linked-chain successor that was NEVER
          dispatched completes with res = -ECANCELED (the Linux
          Linux `io_fail_links` contract).
          The generation was already bumped in U10a, so a CQE-triggered
          resubmission cannot be admitted at the old generation even if
          it races the remainder of recovery.
          **io_uring path invariant (normative)**: VFS SQE submission
          from io_uring ([Section 19.3](19-sysapi.md#io-uring-subsystem)) MUST go through the
          same vfs_check_open_generation() + select_ring() entry used by
          synchronous VFS syscalls. There is NO io_uring-specific fast
          path that bypasses either check — any future io_uring dispatch
          optimization must preserve both checks or re-derive this
          recovery sequence's safety argument.

Step U12. [General 4a]   EMIT FMA EVENT
          fma_emit(FaultEvent::DriverCrash { ... })

Step U13. [General 5-7 + DMA]  DEVICE RESET + RELEASE LOCKS + UNLOAD
          FLR, KABI lock release, driver memory free.
          DMA quiescence (FLR + IOTLB invalidation + wait_dma_quiesce) was
          initiated between U2-U4 as part of the unified interleaving
          specified in [Section 11.9](11-drivers.md#crash-recovery-and-state-preservation--dma-quiescence-during-crash-recovery).
          By this step, DMA is fully quiesced and IOMMU entries are revoked.

Step U13a. (REMOVED — generation bump moved to Step U10a, before the
          U11 io_uring CQE drain. See U10a rationale above. No-op
          placeholder to preserve step numbering.)

Step U13b. [Per-CPU ext; VFS-module-domain crash ONLY]  DENTRY SLOT RECLAIM
          When the crashed domain is the VFS MODULE domain (not a mere
          provider-only crash), reclaim the dead dcache's tracked dentry
          slots so their storage budget is not leaked across crashes.
          PRECONDITIONS (satisfied by the ordering here): U5/U5a guarantee
          no old producer/consumer thread survives, and U13's module unload
          has completed the RCU-callback drain (`rcu_barrier()` — every
          `dentry_free_rcu` queued before the crash has run, so its slot has
          already left the live registry and cannot be double-dropped). One
          additional `rcu_synchronize()` then covers any reference-less
          Core-side reader. The pass calls the Nucleus primitive
          `tracked_reclaim_owner_refs::<Dentry>()`
          ([Section 13.18](13-device-classes.md#live-kernel-evolution--generic-tracked-allocator)); the full
          reclamation design (pinned-survivor rule, cascade frees, FMA
          accounting) is specified at
          [Section 14.1](#virtual-filesystem-layer--dirty-page-handling-on-vfs-crash).
          A provider-only crash SKIPS this step — the surviving VFS module
          prunes its own dentries through the normal dentry-cache invalidation path.

Step U14. [VFS Step 4 + General 8; (d)-(e) run in reload_fn]
          RELOAD DRIVER AND REMOUNT
          Load new driver binary from CrashRecoveryPool or buddy allocator.
          The new driver instance goes through the standard KABI module Hello
          protocol ([Section 12.8](12-kabi.md#kabi-domain-runtime--module-hello-protocol)):
          (a) Register with the domain service.
          (b) Declare dependencies (block device, DMA allocator, etc.).
          (c) Domain service resolves dependencies and hands out handles.
              **KABI→VFS ring handoff**: The Hello protocol creates
              `CrossDomainRing` objects for generic KABI service bindings.
              However, VFS rings use a different ring type (`VfsRingPair`
              with 320-byte `VfsRequest` entries and `VfsOpcode`-based
              dispatch). The handoff works as follows:
              - The generic KABI Hello protocol creates `CrossDomainRing`
                objects for the driver's non-VFS dependencies (DMA, crypto,
                etc.) — these use the standard 64-byte `T1CommandEntry`.
              - For the VFS-specific ring, the domain service does NOT
                create a `CrossDomainRing`. Instead, it passes the existing
                `VfsRingSet` pointer (which survived the crash — the ring
                memory ALLOCATION is kernel-owned: umka-nucleus owns its
                lifetime, and it is untouched by the crash and reload of
                the driver domain. Domain ACCESS to it is a separate,
                revocable grant: U2's `revoke_domain_permissions()` set
                the domain key to deny-all, and (c') below re-arms it for
                the reloaded instance) directly to the driver's
                `vfs_init()` entry point. The `VfsRingSet` was reset in
                Steps U7-U10 and its rings are ready for reuse.
          (c') RE-ARM DOMAIN-KEY PERMISSIONS FOR THE RELOADED INSTANCE.
                Crash recovery reuses the domain descriptor IN PLACE: the
                `DomainId` and `isolation_key` are RETAINED across reload
                ([Section 11.9](11-drivers.md#crash-recovery-and-state-preservation), "Domain
                descriptor disposition: REUSE IN PLACE"). But U2's
                `revoke_domain_permissions()` set that key to deny-all in
                the domain image table — no `switch_domain()` call
                can re-enable it. Before handing the `VfsRingSet` pointer
                to `vfs_init()` (and before (d) spawns consumer threads),
                the recovery worker performs the ONE authorized reset:
                re-arm the retained key from deny-all back to the
                driver's normal permission set (x86 MPK: clear the AD/WD
                bits for the PKEY in the domain image table; AArch64
                POE: restore the overlay index's POR field; ARMv7: restore
                the DACR field; AArch64 mainstream: allocate/validate a
                fresh ASID for the reused page-table root). This is safe
                only HERE because U13 has already freed all old-instance
                memory in the domain and U5a guaranteed no old thread can
                run under this key.
                Because the key is retained, the mount-time page-level
                grant is still correct — the ring data regions' PTEs are
                still stamped with this key (the PTE stamping was never
                undone; U2 revoked the KEY's permissions, not the pages'
                key assignment) — so no per-page re-grant is needed: ring
                access is restored atomically by the key-level re-arm.
                Without (c'), the reloaded driver's first ring access
                (consumer thread reading the ring header) takes a
                protection fault against the still-revoked key and
                cascades into a secondary crash.
                Tier 2 reload differs: the replacement driver is a NEW
                process, so the ring pages are re-mapped into its address
                space (same physical pages, fresh mmap + IOMMU domain
                setup), as at mount time.
              - The driver's `vfs_init()` receives both the generic KABI
                handles (from Hello) and the VFS-specific `VfsRingSet`
                pointer (passed separately by the domain service).

          **VFS initialization KABI interface**: The `vfs_init()` function
          is declared in the filesystem KABI `.kabi` definition as an
          optional initialization method (present only for filesystem
          drivers, not for all Tier 1 drivers):
          /// Filesystem-specific initialization. Called by the domain
          /// service after the Hello protocol completes and generic KABI
          /// handles are resolved. Receives the VfsRingSet for this mount.
          ///
          /// The driver creates consumer threads (one per ring pair) using
          /// `kernel_services.create_kthread(entry, arg,
          /// KthreadRole::RingConsumer)` — a KABI kernel-services
          /// method, NOT a direct kthread_create() syscall. Tier 1 drivers
          /// cannot create kthreads directly; they request creation via
          /// the kernel-services KABI handle obtained during Hello.
          /// The RingConsumer role makes Core push each new TaskId onto
          /// DomainDescriptor.consumer_threads (under entry_lock) and
          /// remove it on kthread exit — the registration that Step
          /// U5a's consumer-exit barrier iterates. Passing a different
          /// role for a ring-consumer thread is a driver bug that voids
          /// the recovery sequence's exclusive-access argument.
          ///
          /// GENERATION STAMPING CONTRACT: the driver MUST stamp
          /// `response.driver_generation =
          /// ring_set.driver_generation.load(Ordering::Acquire)` when
          /// constructing EVERY VfsResponseWire. Responses stamped with
          /// any other value are discarded by the Core response drain
          /// (stale-response filter). The mirror is maintained by Core
          /// (mount init + recovery Step U10a); the driver only reads it.
          ///
          /// Ring memory permissions (PRECONDITION): this driver's domain
          /// key has read-write access to the VfsRingSet and its ring data
          /// regions — granted at mount time for a fresh mount, and
          /// re-armed via step (c') above for a crash-recovery reload
          /// (the retained key is deny-all from U2's revocation until
          /// (c') resets it). The ring control structures
          /// (head/tail/published) are AtomicU64 — interior mutability
          /// through shared references is safe.
          fn vfs_init(
              &self,
              ring_set: &VfsRingSet,
              kernel_services: &KernelServicesHandle,
          ) -> Result<(), KabiError>;
          (d) The new driver inherits the existing sb.ring_set: the domain
              service passes the VfsRingSet pointer to the driver's init
              function. The driver starts N consumer threads (one per ring
              in ring_set), each bound to the corresponding VfsRingPair.
              The per-ring inner.state was reset to ACTIVE in U7(f), so
              the consumer loop's per-wake disconnect check passes.
          (e) Mount RO, run fsck_fast() (fast metadata consistency check),
              remount RW if fsck_fast passes.

Step U15. [reload_fn; VFS Step 5]   FLUSH DEFERRED DIRTY PAGES
          writeback_deferred_dirty(sb)
          Runs inside reload_fn, after U14(e)'s journal replay — the
          new instance's writeback completions carry the NEW generation
          (bumped in U10a), so the response drain accepts them.

Step U16. (REMOVED — generation bump moved to Step U10a, before the U11
          CQE drain and the U14 driver reload.
          See U10a rationale above. This step is now a no-op placeholder
          to preserve step numbering.)

Step U17. [resume_fn; Per-CPU ext]  RING SET ACTIVE
          ring_set.state.store(VFSRS_ACTIVE, Release)
          This is the LAST step. The Release ordering ensures all
          prior ring resets, slot_states clears, and driver init are
          visible to producers before they observe ACTIVE.
          Immediately AFTER the ACTIVE store, wake the SuperBlock's
          `recovery_wait` WaitQueue (`sb.recovery_wait.wake_up_all()`) so the
          boundary-blocked dispatchers parked during recovery (Step U3-era
          producers that took ENXIO and slept, [Section 14.1](#virtual-filesystem-layer))
          re-run `select_ring()` and now observe ACTIVE. The wake is ordered
          after the Release store, so a woken producer cannot re-park on a
          state it already passed. The wake contract is normatively stated on
          the `SuperBlock.recovery_wait` field.
          resume_fn runs ONLY after general Step 9's
          compare_exchange(Recovering -> Active) on the domain state
          SUCCEEDS ([Section 11.9](11-drivers.md#crash-recovery-and-state-preservation)). If a
          second crash intervened (Step 9 observes Crashed), resume_fn
          is skipped and ring_set.state stays RECOVERING — the second
          recovery pass owns the mount and will reach its own U17.

Step U18. [resume_fn; General 9]    DRIVER READY
          Driver announces readiness to domain service.

VFS/Block Recovery Interleaving (Same-Domain Crash)

When a Tier 1 filesystem driver crashes and it shares a domain with the block driver (common on platforms with limited isolation domains), TWO crash recovery sequences are triggered for the SAME domain crash event:

1. Block I/O recovery (Section 15.2): Drains block request queues, completes in-flight bios with EIO, resets the block device's hardware queues.
2. VFS recovery (this section): The unified sequence U1-U18 above.

Ordering constraint: Block I/O recovery MUST complete before VFS Step U14 (RELOAD DRIVER AND REMOUNT). The filesystem driver's vfs_init() submits block I/O (for fsck_fast metadata reads), which requires the block device to be operational. The crash recovery worker (Section 12.8) serializes these via the domain-level recovery_mutex — block recovery runs first (it is faster: ~10-50ms for queue drain), then VFS recovery starts.

PERSISTENT bio replay coordination: block recovery's completion includes the replay of captured BioFlags::PERSISTENT bios (journal commits, superblock writes) to the new block driver instance, in submission order from the per-device FIFO retry list (Section 15.2). Because the "block recovery before U14" constraint above covers ALL of block recovery, the PERSISTENT replay is guaranteed to have completed — every replayed bio's end_io fired, success or per-bio error — before U14(e) runs fsck_fast()/journal replay. This ordering is load-bearing: the filesystem's journal replay reads journal metadata from disk, and a journal commit block still sitting in the retry list (not yet replayed) would make the journal appear uncommitted, causing the FS replay to roll back a transaction whose commit bio then lands afterwards — torn journal state. Replay-before-U14(e) makes the on-disk journal state stable before the filesystem interprets it.

Bio completion callback domain: When the VFS submits a bio to the block layer, the bio completion callback (bio.end_io) executes in the context of the block device's interrupt handler — which runs in the Core domain, not in the VFS driver's isolation domain. This is by design: bio completions update Core-resident page cache state (PG_writeback, PG_uptodate, PG_error flags) and wake Core-resident waitqueues. The bio completion callback NEVER enters the VFS driver's isolation domain — it only touches Core data structures. This ensures bio completions continue to work even during VFS driver recovery.

For an ISOLATED (effective Tier 1/2) filesystem driver, this is not merely convention but a structural consequence of the KABI boundary: the driver cannot hand the block layer a raw function pointer into its own domain (cross-domain calls go through rings, never raw pointers), so end_io for driver-submitted bios is always one of the Core-resident completion shims (Section 15.2) — for a Tier 1/2 filesystem, the shim posts a typed completion message to the driver's VFS response ring rather than calling into the driver. Consequently a VFS driver crash CANNOT dangle an in-flight bio's end_io: the pointer targets Core/block-layer code that U13's driver unload never unmaps (end_io stability across module lifecycle is specified in Section 15.2). What the crash DOES orphan is the ring-delivery leg: completion messages for the crashed driver's in-flight bios have nowhere to go. These are handled by the sequence above — completions already in the response ring are drained as -EIO in U7(a); completions arriving between U3 and U17 are rejected at the ring (state != ACTIVE) and the shim's Core-side page cache updates still run; and any journal/filesystem state machine progress lost with the crashed driver's memory is reconstructed by U14(e)'s journal replay, exactly as after a power failure. (Only a Tier 0 filesystem may install a direct end_io into filesystem code, and Tier 0 has no crash recovery — a Tier 0 fault is a kernel panic, U1a.)

14.3.10.2 Recovery Functions

Step U3: Set ring state

fn set_all_rings_disconnected(ring_set: &VfsRingSet) {
    // Block new select_ring() calls.
    ring_set.state.store(VFSRS_RECOVERING, Ordering::Release);

    // Store-buffering (Dekker) pairing with reserve_slot()'s
    // post-increment re-check: state store → fence(SeqCst) here;
    // inflight fetch_add → fence(SeqCst) → state load there. Guarantees
    // every producer is either counted by wait_for_producers_quiesced()
    // or backs out via abort_slot() — no third possibility.
    core::sync::atomic::fence(Ordering::SeqCst);

    for i in 0..ring_set.ring_count as usize {
        // SAFETY: rings is valid for ring_count elements.
        let ring = unsafe { &*ring_set.rings.add(i) };
        ring.request_ring.inner.state.store(RING_STATE_DISCONNECTED, Ordering::Release);
        ring.response_ring.inner.state.store(RING_STATE_DISCONNECTED, Ordering::Release);
    }

    // Wake all consumer kthreads so they observe DISCONNECTED (the
    // per-wake disconnect check in the Strategy 1/2 consumer loops)
    // and exit before U7(f) restores per-ring ACTIVE. A kthread
    // sleeping in wait_for_entries() is not on any CPU and is NOT ejected
    // by U4's NMI — this explicit wake is the ONLY thing that unblocks it.
    // Signaled AFTER the DISCONNECTED stores above: the doorbell wake
    // provides the release/acquire edge, so a woken consumer is guaranteed
    // to observe DISCONNECTED. U5a then waits for the exits to complete.
    for i in 0..ring_set.ring_count as usize {
        // SAFETY: rings is valid for ring_count elements.
        let ring = unsafe { &*ring_set.rings.add(i) };
        ring.doorbell.signal(); // wake the driver-side consumer for this ring
    }
    ring_set.coalesced_doorbell.doorbell.signal(); // coalesced-wait consumers
    // Also kick the Core response worker so it observes the non-ACTIVE
    // gate and parks instead of sleeping through recovery.
    ring_set.completion_doorbell.doorbell.signal();
}

The mount-level ring_set.state is checked at the top of select_ring() — if not ACTIVE, the VFS operation returns ENXIO immediately without touching any individual ring. This provides a fast-path rejection of new operations during recovery, avoiding the need to check each ring's state individually.

Step U5: Wait for producer quiescence

/// Wait for all in-flight producers to complete their current operations.
///
/// After set_all_rings_disconnected(), no NEW operations can enter via
/// select_ring() (returns ENXIO). But producers that already passed
/// select_ring() and reserve_slot() may be mid-copy_from_user with a
/// RESERVED slot. This function waits for those producers to finish.
///
/// The inflight_ops counter is incremented in reserve_slot() and
/// decremented in complete_slot(). When it reaches zero for all rings,
/// no producer is between reserve and complete — safe to drain.
///
/// Timeout: 5 seconds (matches the per-sb quiescence timeout in the
/// base protocol). After timeout: SIGKILL processes with stuck operations.
///
/// **Sleep, not spin**: this is a cold path (crash recovery / evolution
/// quiescence). It SLEEPS on each ring's Core-only `producer_quiesce_wq`
/// (`wait_until_quiesced` below), woken by `complete_slot()` on the
/// `inflight_ops` 1→0 edge — it does NOT busy-spin. The former
/// `core::hint::spin_loop()` poll was a SCHED_FIFO-90 spinner that starved
/// normal-priority producers (and even SIGKILLed ones) on a single CPU, so a
/// healthy recovery could burn both 5 s windows and abort. `complete_slot()`
/// IS the signal the producer emits when it drains a ring to zero, so a
/// WaitQueue is not just possible but strictly better: sleeping yields the CPU
/// to the very producers that must run to unwind.
fn wait_for_producers_quiesced(
    sb: &SuperBlock,               // FMA attribution on the abort path
    ring_set: &VfsRingSet,
) -> Result<(), CrashRecoveryError> {
    // Window 1: wait for voluntary completion.
    if wait_until_quiesced(sb, ring_set, Duration::from_secs(5)) {
        return Ok(());
    }

    // Escalation: SIGKILL stuck producers, then RE-ENTER the wait.
    // Returning immediately after the kill (as an earlier revision did)
    // let recovery reach the ring reset while a "killed" producer was
    // still alive holding a RESERVED slot: its eventual complete_slot()
    // would store FILLED into a reset slot_states array and fetch_sub an
    // inflight_ops that U7 had zeroed — an AtomicU32 underflow to ~4e9
    // that made EVERY subsequent quiescence (recovery AND evolution
    // Phase A') spin its full timeout and SIGKILL innocents, forever.
    sigkill_stuck_producers(ring_set);

    // Window 2: the killed producers unwind — the fatal signal
    // interrupts their copy_from_user page fault, fill_slot_data()
    // returns Err(EINTR), and the abort ladder (abort_slot()) balances
    // both the slot and the count on the way to exit_task(). Each unwinding
    // producer's abort funnels through complete_slot(), which wakes this
    // sleeping wait on the 1→0 edge.
    if wait_until_quiesced(sb, ring_set, Duration::from_secs(5)) {
        return Ok(());
    }

    // Still nonzero after kill + unwind window: a kernel bug (a producer
    // wedged where even SIGKILL cannot reach). NEVER proceed to the
    // drain with a live producer — abort exactly like U5a's timeout arm:
    // FMA event, recovery aborts (device offline per Reload Failure
    // Handling), ring_set.state stays RECOVERING so producers keep
    // getting ENXIO.
    fma_emit(FaultEvent::RecoveryQuiesceTimeout { sb_dev: sb.s_dev });
    // Terminal waiter rescue BEFORE going offline: this abort skips
    // U6 / U7(g) / U10b — the only wakers for already-parked page-lock
    // and writeback sleepers — so their EIO terminal wakes must happen
    // here. No waiter sleeps without a waker, even on the abort path.
    abort_terminal_rescue(sb, ring_set);
    Err(CrashRecoveryError::QuiesceTimeout)
}

/// Terminal waiter rescue on a recovery ABORT (the U5 wedged-producer
/// arm and the U5a consumer-exit-failure arm). An aborted sequence never
/// reaches U6 (page unlock), U7(g) (blocked-submitter wake), or U10b
/// (writeback epilogue) — the ONLY wakers for tasks already parked in
/// `wait_on_page_locked()` / `wait_on_page_writeback()` / the per-ring
/// completion queues. The device is going permanently offline
/// (ring_set.state stays RECOVERING, Reload Failure Handling), so every
/// such waiter's truthful terminal state is EIO — delivered HERE,
/// because no later step will run. Contract: no waiter sleeps without a
/// waker, even on the abort path.
///
/// Exclusion argument (WEAKER than U6's, sufficient for these effects):
/// - The crashed domain was revoked in U2 and its rings DISCONNECTED in
///   U3, so no driver-side actor can touch pages or rings in either
///   abort case.
/// - Step (1) is U5b's response-drainer barrier — it needs NO producer
///   or consumer quiescence; after it, no response drainer completes an
///   entry (drainers re-check ring_set.state at the top of a pass).
/// - The table sweep's exactly-once is the claim-by-remove rule itself
///   (one winner per key): a race with a woken waiter claiming its OWN
///   entry, or with a wedged producer's later unwind, resolves to a
///   single terminal actor — U6's own argument, which does not depend on
///   the failed quiescence.
/// - The ring RESET (U7 d-f) is deliberately NOT performed: no reload is
///   coming, and resetting pointers under a possibly-live wedged
///   producer is neither needed nor safe. `inflight_ops` is left intact
///   (it still truthfully counts the wedged producer).
fn abort_terminal_rescue(sb: &SuperBlock, ring_set: &VfsRingSet) {
    for i in 0..ring_set.ring_count as usize {
        // ring_core is Some whenever ring_set is Some (mount step 4a).
        let core_side = &sb.ring_core.as_ref().unwrap()[i];
        // (1) U5b barrier: exclude every response drainer, then
        // (2) EIO-sweep the in-flight table — the page-LOCK waiter
        //     rescue (fills discharged via grant revoke +
        //     complete_err(EIO); readahead pages ERROR + unlocked; DMA
        //     handles freed by the claimer).
        drop(core_side.response_drain_lock.lock());
        sweep_inflight_table_eio(core_side);
    }
    // (3) Writeback-waiter rescue — see abort_writeback_rescue() below.
    abort_writeback_rescue(sb);
    // (4) U7(g)'s wake, without the reset: UNCONDITIONALLY wake blocked
    //     submitters on each ring's completion queue — outside every
    //     drain lock (two-phase discipline). Woken waiters observe
    //     (state != ACTIVE || completed) and return EIO.
    for i in 0..ring_set.ring_count as usize {
        // SAFETY: rings valid for ring_count elements.
        let ring = unsafe { &*ring_set.rings.add(i) };
        ring.completion.wake_up_all();
    }
}

/// Abort-path writeback rescue: discard the writeback response ring
/// (U10b(a) semantics — its pre-crash completions are suspect), then run
/// `writeback_page_epilogue(page, -EIO)` on every page of `sb` still
/// under WRITEBACK, enumerated from the Core-resident dirty-intent
/// entries (`DIRTY_INTENT_INDEX`) exactly as U10b(c') enumerates them —
/// committed AND Phase-1 ranges alike, because on an abort no direct
/// write happened, so ALL dispatched pages take the error epilogue:
/// re-dirty + ErrSeq EIO + WRITEBACK clear + waiter wake
/// ([Section 4.6](04-memory.md#writeback-subsystem--writeback-cross-domain-dispatch),
/// [Section 15.2](15-storage.md#block-io-and-volume-management--writeback-io-completion-callback)).
/// A parked fsync phase-2 waiter wakes and truthfully reports EIO; the
/// dirty data remains cached (re-dirtied) — nothing is silently dropped.
/// Intent ENTRIES are retained (the device may return via manual
/// intervention; the entries still describe the dirty data).
fn abort_writeback_rescue(sb: &SuperBlock);

/// Sleep until every ring's producers have quiesced (`inflight_ops == 0`) or
/// `window` elapses; returns true iff fully quiesced. Cold path (crash recovery
/// / evolution quiescence), process context — it SLEEPS. Each ring is waited on
/// its Core-only `producer_quiesce_wq` via the canonical `wait_event_timeout`
/// protocol, woken by `complete_slot()` on that ring's `inflight_ops` 1→0 edge.
/// Replaces the former `spin_until_quiesced()` `core::hint::spin_loop()` poll.
///
/// Per-ring waits under ONE shared deadline: a ring already at zero satisfies
/// its predicate immediately (no sleep); the total wait is bounded by `window`
/// regardless of ring count.
///
/// Lost-wake-proof: `quiesce_waiters` is armed (Release) BEFORE the wait, so a
/// `complete_slot()` whose Acquire load observes the arm will wake this queue;
/// `wait_event_timeout` re-checks the predicate after enqueuing, so a decrement
/// that raced the arm (already at zero) returns without sleeping. `quiesce_waiters`
/// is disarmed after each ring so steady-state `complete_slot()` skips the wake.
fn wait_until_quiesced(sb: &SuperBlock, ring_set: &VfsRingSet, window: Duration) -> bool {
    let deadline = monotonic_now() + window;
    for i in 0..ring_set.ring_count as usize {
        // SAFETY: rings pointer valid for ring_count elements.
        let ring = unsafe { &*ring_set.rings.add(i) };
        // ring_core is Some whenever ring_set is Some (mount step 4a).
        let core_side = &sb.ring_core.as_ref().unwrap()[i];
        // Arm this ring's quiescence wait before checking/sleeping.
        core_side.quiesce_waiters.fetch_add(1, Ordering::Release);
        let remaining_ns = deadline.saturating_sub(monotonic_now()).as_nanos() as u64;
        let quiesced = core_side.producer_quiesce_wq.wait_event_timeout(
            remaining_ns,
            || ring.request_ring.inflight_ops.load(Ordering::Acquire) == 0,
        );
        core_side.quiesce_waiters.fetch_sub(1, Ordering::Release);
        if !quiesced {
            return false; // this ring did not quiesce within the window
        }
    }
    true
}

/// Identify and SIGKILL processes that have operations stuck in the VFS ring.
///
/// Called when `wait_for_producers_quiesced()` times out after 5 seconds.
/// A producer is "stuck" if it called `reserve_slot()` (incrementing
/// `inflight_ops`) but never called `complete_slot()` (decrementing it).
/// This can happen if the producer's thread is blocked in an uninterruptible
/// sleep between reserve and complete (e.g., page fault during `copy_from_user`
/// that blocks on I/O to the now-crashed filesystem — a deadlock).
///
/// **Mechanism**: Each ring maintains a per-ring `WaitQueue` that producers
/// sleep on when the ring is full (`reserve_slot()` calls `wq.wait_event`).
/// After the ring state is set to RING_STATE_DISCONNECTED, producers woken from this
/// waitqueue check the state and return ENXIO. For producers stuck in
/// `copy_from_user()` (not sleeping on the waitqueue), SIGKILL is the only
/// option — it interrupts the page fault handler and causes the thread to
/// enter `exit_task()`.
///
/// The function iterates all tasks in the system and sends SIGKILL to any
/// task that has a pending VFS operation on a ring belonging to this ring_set.
/// This is identified by checking if the task's `current_vfs_ring` pointer
/// (set in `reserve_slot()`, cleared in `complete_slot()`, and cleared in
/// `exit_task()` via the VFS exit hook) points to a ring in this ring_set.
/// The scan's `current_vfs_ring` load is `Relaxed`, yet it is correctly
/// ordered: the producer stores the pointer with `Release` BEFORE the
/// `AcqRel` `inflight_ops` increment (`reserve_slot()`), and the recovery
/// worker reached this scan only after an `Acquire` `inflight_ops` load in
/// `wait_until_quiesced()` observed that increment (a nonzero count). The
/// release/acquire edge on `inflight_ops` therefore transitively publishes
/// the pointer store, so a stuck producer counted in `inflight_ops` always
/// has its ring pointer visible to the scan.
///
/// **Cross-subsystem invariant**: `exit_task()` MUST execute
/// `task.current_vfs_ring.store(core::ptr::null_mut(), Ordering::Relaxed)`
/// before the task enters zombie state. Without this clear, a zombie task
/// with a stale `current_vfs_ring` pointer could match a newly-allocated
/// `VfsRingSet` whose backing VA was reused from the freed set, causing a
/// spurious SIGKILL against a task that never issued an operation on the
/// new ring. See [Section 8.2](08-process.md#process-lifecycle-teardown) (task exit) for the implementation site.
fn sigkill_stuck_producers(ring_set: &VfsRingSet) {
    // Iterate the task table (RCU-protected read) looking for tasks
    // with current_vfs_ring pointing into this ring_set's ring array.
    let ring_base = ring_set.rings as usize;
    let ring_end = ring_base + ring_set.ring_count as usize
        * core::mem::size_of::<VfsRingPair>();

    rcu_read_lock();
    for_each_task(|task| {
        let ring_ptr = task.current_vfs_ring.load(Ordering::Relaxed) as usize;
        if ring_ptr >= ring_base && ring_ptr < ring_end {
            // This task has an in-flight VFS operation on one of our
            // rings. ACTUALLY send SIGKILL: send_signal_to_task() queues
            // SIGKILL in the per-task pending set, sets TIF_SIGPENDING,
            // and performs the fatal signal_wake_up(task, true) wake per
            // the delivery-wakeup protocol
            // ([Section 8.6](08-process.md#signal-handling--sending-a-signal);
            // signal_wake_up()'s contract REQUIRES the pending bit to be
            // set first — an earlier revision called bare
            // signal_wake_up(task, true), which queues nothing: the
            // woken task found no pending signal and went back to
            // sleep, making the "kill" a no-op).
            send_signal_to_task(task, SIGKILL, &SigInfo::kernel(SIGKILL));
        }
    });
    rcu_read_unlock();
}

Steps U6-U8: Drain all VFS rings (unified 4-phase function)

/// Shared U6 sweep body: claim-by-remove every entry of ONE ring's
/// Core-resident in-flight table and publish its EIO terminal state
/// (grant revoke + fill complete_err for demand ReadPage; ERROR +
/// unlock for readahead pages; DMA handle freed by the claimer).
/// Called from Phase 1 of `drain_all_vfs_rings()` (the completed
/// recovery sequence) AND from `abort_terminal_rescue()` (the recovery
/// abort arms). Exactly-once per entry is carried by the atomic
/// claim-by-remove itself — one winner per key — not by the caller's
/// exclusion strength; both callers first exclude response drainers via
/// the response_drain_lock barrier.
fn sweep_inflight_table_eio(core_side: &VfsRingCoreSide) {
    loop {
        // Collect a bounded batch of keys under RCU (XArray ordered
        // iteration from key 0), then claim each key outside the
        // read section. Re-loop until the table is empty.
        let mut batch: ArrayVec<u64, 256> = {
            let _rcu = rcu_read_lock();
            core_side.inflight.iter_keys_from(0).take(256).collect()
        };
        if batch.is_empty() { break; }
        for request_id in batch.drain(..) {
            let Some(entry) = core_side.inflight.remove(request_id) else {
                continue; // a waiter won the claim — it cleans up
            };

            // Terminal page state INLINE, per claimed ReadPage/
            // Readahead entry — reached through the entry's inode pin
            // (entry.inode: Arc<Inode>), never a raw pointer. The pin
            // keeps the embedded i_mapping AddressSpace live across
            // the deref; dropping `entry` (below) drops the pin. No
            // fixed cap, no deferral: every page reaches ERROR +
            // unlocked here, in one pass.
            match entry.opcode {
                VfsOpcode::ReadPage | VfsOpcode::Readahead => {
                    // Exactly-once entry disposition ⇒ exactly-once fill
                    // consumption: this sweep already REMOVED the entry, so it
                    // is the sole terminal actor for it. A demand ReadPage
                    // carries a linear FillCompletion — discharge it via
                    // complete_err (recovery outcome is EIO): the SINGLE place
                    // ERROR is published on Page.flags + LOCKED cleared + wake
                    // ([Section 4.4](04-memory.md#page-cache)).
                    if let Some(fc) = entry.fill.take() {
                        // Revoke the FS domain's DMA grant for this page
                        // BEFORE the terminal completion — the crash-drain
                        // arm of the grant contract ("revokes the grant and
                        // completes the fill with EIO",
                        // [Section 14.2](#vfs-ring-buffer-protocol--ringmount-addressspaceops-provider)),
                        // mirroring vfs_complete_read_pages. Without the
                        // revoke, the grant would outlive the buffer free
                        // below: after U14 re-arms the domain key, the
                        // reloaded instance would inherit DMA access to a
                        // page whose slot dma_pool_free() has made
                        // reallocatable — a crashed-domain DMA window into
                        // a reused frame. Revoke drops ACCESS (generation
                        // bump); the dma_pool_free below returns the SLOT —
                        // orthogonal, both required, in this order.
                        // Same-domain providers have no grant table
                        // (`dma_grant_table()` is `None`) and skip this.
                        if let Some(grant) =
                            entry.inode.i_mapping.ops.dma_grant_table()
                        {
                            grant.revoke(entry.dma_handle);
                        }
                        fc.complete_err(Errno::EIO);
                    } else {
                        // Readahead batch (no per-entry obligation): publish
                        // ERROR directly on each physical Page.flags. RCU: a
                        // page could race eviction between lookup and flag
                        // update, but a LOCKED page cannot be evicted — the
                        // None arm is pure defense-in-depth.
                        let _rcu = rcu_read_lock();
                        if let Some(page_cache) =
                            entry.inode.i_mapping.page_cache.as_ref()
                        {
                            for pg in 0..entry.nr_pages as u64 {
                                if let Some(page_entry) =
                                    page_cache.pages.load(entry.page_index + pg)
                                {
                                    // Content ERROR on the PHYSICAL frame's
                                    // `Page.flags` (single authority); the
                                    // dirty-scan index is the XArray's own
                                    // per-node `XA_TAG_DIRTY` tag bitmap, not
                                    // a slot field.
                                    page_entry.page.page().flags_fetch_or(
                                        PageFlags::ERROR,
                                        Ordering::Release,
                                    );
                                    // Clears LOCKED and wakes
                                    // wait_on_page_locked() waiters -> EIO.
                                    unlock_page(page_entry.page.page());
                                }
                            }
                        }
                    }
                }
                _ => {}
            }
            // Claimer owns the DMA free (exactly once — see the
            // claim rule in [Section 14.2](#vfs-ring-buffer-protocol)).
            if entry.dma_handle != DmaBufferHandle::ZERO {
                dma_pool_free(entry.dma_handle);
            }
            rcu_free_inflight_entry(entry);
        }
    }
}

/// Drain all VFS rings during crash recovery.
///
/// This function implements Steps U6 (table sweep, including inline
/// terminal page state) and U7 (DRAIN+RESET). U8 is a renumbering note
/// only: orphaned-page wakes are performed INLINE in U6, per claimed
/// entry — there is no separate wake pass and no orphaned-page
/// collection (the fixed-cap `MAX_ORPHANED_PAGES` / `OrphanedPageEntry`
/// buffer and the `wake_orphaned_pages()` helper are DELETED; pages may
/// never remain locked pending "manual intervention").
///
/// **Phases per ring** (rings processed sequentially):
///
/// Phase 1 (U6): SWEEP THE IN-FLIGHT TABLE — claim-by-remove every
///   entry in core_side.inflight; for each claimed ReadPage/Readahead
///   entry, publish its terminal page state INLINE (set PageFlags::ERROR
///   + unlock_page() per page, the mapping reached through the entry's
///   inode pin — `entry.inode.i_mapping.page_cache`); then free the
///   entry's Core-captured DMA handle and rcu-free it. Ring memory is
///   NEVER read: it is driver-writable (untrusted after a crash) and
///   incomplete (the consumer empties slots at dispatch —
///   dispatched-but-uncompleted requests exist only in the table).
///   Exclusion + exactly-once: U5 quiesced producers, U5a exited
///   consumers, and U5b's response_drain_lock barrier excluded every
///   response drainer, so the recovery worker is the sole actor
///   performing terminal side effects. Claim-by-remove has exactly one
///   winner per key; a concurrently woken waiter claims only its OWN
///   sync entry and unlocks no pages; a completed ReadPage's entry was
///   already removed and its page unlocked by the pre-crash response
///   drain. A locked page cannot be evicted, so the RCU lookup's None
///   arm is pure defense-in-depth. No fixed cap, no "manual
///   intervention": every page reaches a terminal (unlocked, ERROR)
///   state in this one pass.
///
/// Phase 2 (U7 a): acquire the ring's response_drain_lock (now
///   uncontended defense-in-depth — U5b already excluded every response
///   drainer). The blocked-submitter wake (U7 g) is NOT issued here:
///   response_drain_lock is a LEAF (two-phase discipline — every wakeup
///   happens OUTSIDE it), so the wake is DEFERRED to the tail of Phase 3,
///   after the lock is dropped.
///
/// Phase 3 (U7 d-f, then g): RESET — clear ring state for replacement
///   driver.
///   Reset all slot_states to EMPTY.
///   Reset ring pointers (head = tail = published = 0) — this discards
///   the response-ring contents wholesale (pre-crash completions are
///   suspect; their table entries were reaped in Phase 1).
///   Reset doorbells; reset per-ring inner.state to RING_STATE_ACTIVE.
///   THEN drop response_drain_lock and UNCONDITIONALLY wake blocked
///   submitters on ring.completion (U7 g): a synchronous waiter's
///   liveness is tracked by the Core in-flight table, NOT request-ring
///   occupancy, so the wake must NOT be gated on ring state/occupancy —
///   a ring with tail == published can still hold a waiter that Phase 1
///   just made completable. Woken waiters observe `state != ACTIVE ||
///   completed` and return EIO.
///
/// Order: rings are drained sequentially (ring 0, ring 1, ..., ring N-1).
/// No parallelism needed — crash recovery is a cold path with a 500ms
/// latency target.
fn drain_all_vfs_rings(
    sb: &SuperBlock,
    ring_set: &VfsRingSet,
) {
    for i in 0..ring_set.ring_count as usize {
        // SAFETY: rings pointer valid for ring_count elements.
        let ring = unsafe { &*ring_set.rings.add(i) };
        // ring_core is Some whenever ring_set is Some (mount step 4a).
        let core_side = &sb.ring_core.as_ref().unwrap()[i];

        // Phase 1 (U6): sweep the Core-resident in-flight table.
        // Producers are quiesced (U5: inflight_ops == 0, else recovery
        // aborted), consumers exited (U5a), AND every response drainer
        // was excluded by U5b's response_drain_lock barrier, so no new
        // entries can appear and no drainer completes an entry
        // concurrently; the only remaining concurrent mutators are
        // waiters doing claim-by-remove on their OWN entries — the atomic
        // remove resolves every such race with exactly one winner.
        // The sweep body is shared with the recovery-abort waiter rescue
        // (`abort_terminal_rescue()` below), whose weaker exclusion
        // argument rests on the same claim-by-remove exactly-once rule.
        sweep_inflight_table_eio(core_side);

        // Phase 2 (U7 a): take exclusive consumership of the response ring
        // for the reset. The drain lock excludes the per-mount Core response
        // worker and any opportunistic waiter drain for the remainder of the
        // reset; wait is bounded (a drain pass is bounded by ring depth, and
        // holders back off on the non-ACTIVE state gate). The response
        // ring's CONTENTS are not read at all — pre-crash completions
        // are suspect and their table entries were reaped in Phase 1;
        // the Phase 3 pointer reset discards them wholesale. (An earlier
        // revision walked the response ring here reading each
        // VfsResponseWire — deleted: nothing trusted remained to read.)
        // The pending-submitter WAKE (U7 g) is NOT issued under this lock:
        // response_drain_lock is a LEAF (all wakeups happen outside it —
        // two-phase discipline), so it is deferred until after the reset and
        // the explicit lock drop, below.
        let _drain_guard = core_side.response_drain_lock.lock();

        // Phase 3: RESET — clear ring state for replacement driver.
        // Reset all slot_states to EMPTY FIRST (before ring pointers).
        // Relaxed ordering is safe here because the downstream barrier
        // chain guarantees visibility: the ring_set.state store to
        // RING_STATE_ACTIVE (Step U17) uses Release ordering. The
        // replacement driver's consumer thread observes RING_STATE_ACTIVE
        // via Acquire load. This Release/Acquire pair establishes a
        // happens-before relationship: all Relaxed stores to slot_states
        // (done before the Release store) are visible to the consumer
        // thread (which reads after the Acquire load). No intermediate
        // Release is needed on the individual slot_state stores.
        for slot_idx in 0..ring.request_ring.inner.size as usize {
            // slot_state() encapsulates the raw-pointer arithmetic
            // (`*const AtomicU8` does not support `[]` indexing) — the
            // same accessor the hot-path producer/consumer functions use.
            ring.request_ring.slot_state(slot_idx)
                .store(RingSlotState::Empty as u8, Ordering::Relaxed);
        }
        for slot_idx in 0..ring.response_ring.inner.size as usize {
            ring.response_ring.slot_state(slot_idx)
                .store(RingSlotState::Empty as u8, Ordering::Relaxed);
        }
        // inflight_ops is ALREADY zero — U5 aborts the whole recovery if
        // quiescence fails, so control cannot reach this point with a
        // nonzero count. Do NOT "defensively" store 0 here: an earlier
        // revision did, which converted a detectable stall (nonzero
        // count) into a silent AtomicU32 underflow when a late
        // complete_slot() decremented past the reset — poisoning every
        // future quiescence on this ring.
        debug_assert_eq!(
            ring.request_ring.inflight_ops.load(Ordering::Acquire), 0,
            "U7 reached with live producers — U5 must abort first");
        // Reset ring pointers.
        ring.request_ring.inner.head.store(0, Ordering::Release);
        ring.request_ring.inner.published.store(0, Ordering::Release);
        ring.request_ring.inner.tail.store(0, Ordering::Release);
        ring.response_ring.inner.head.store(0, Ordering::Release);
        ring.response_ring.inner.published.store(0, Ordering::Release);
        ring.response_ring.inner.tail.store(0, Ordering::Release);
        // Reset the doorbell (U7(e')): DoorbellRegister is level-latched
        // — a signal latched by a producer just before U3, by the
        // crashed driver's final actions, or by U3's own consumer-exit
        // wake (unconsumed when the consumer had already been
        // crash-ejected) survives the pointer reset above and would
        // spuriously wake the new consumer's first wait() for work that
        // no longer exists. Discard the latch. Safe only because the
        // U5a barrier ran: no old consumer can still be sleeping on
        // this doorbell when its latch is discarded.
        // (The producer-side wake path is `ring.completion`, a
        // WaitQueue, not a doorbell — a WaitQueue holds no latched
        // state, so this doorbell reset does not concern its waiters;
        // they are woken UNCONDITIONALLY with EIO in U7(g) below, AFTER
        // the drain lock is dropped — not here, and not in Phase 2.)
        ring.doorbell.reset();
        // Reset per-ring state to Active for the replacement driver.
        ring.request_ring.inner.state.store(RING_STATE_ACTIVE, Ordering::Release);
        ring.response_ring.inner.state.store(RING_STATE_ACTIVE, Ordering::Release);

        // U7(g): drop the drain lock BEFORE waking. response_drain_lock is a
        // LEAF (nothing is acquired under it; all wakeups happen OUTSIDE it —
        // two-phase discipline), and completion.wake_up_all() takes the
        // WaitQueue lock and enters the scheduler. The wake is UNCONDITIONAL:
        // a blocked synchronous waiter's liveness is tracked by the Core
        // in-flight table, NOT by request-ring occupancy. The driver can
        // advance the request ring's tail up to published (FILLED → EMPTY +
        // tail bump) while the operation is still outstanding in the in-flight
        // table, so a ring with tail == published can still have a waiter that
        // Phase 1 (U6) just made completable by removing its entry. Waking
        // every ring's completion queue guarantees each such waiter
        // re-evaluates its condition (`state != VFSRS_ACTIVE || completed`)
        // and returns EIO; a wake gated on `published - tail > 0` would
        // suppress exactly this wake and the waiter would sleep indefinitely.
        // A spurious wake of an empty queue is free.
        drop(_drain_guard);
        ring.completion.wake_up_all();
    }

    // Reset the ring set's coalesced doorbell: discard the latched
    // signal, clear pending_mask (all four AtomicU64 words to 0), and
    // zero the coalescer's batch-tracking fields. The crashed driver may
    // have died between a producer's pending_mask OR and the doorbell
    // write — this brings the coalescing state back in sync with the
    // (now empty) rings.
    ring_set.coalesced_doorbell.doorbell.reset();
    for w in ring_set.coalesced_doorbell.pending_mask.iter() {
        w.store(0, Ordering::Relaxed); // visibility via U17's Release store
    }
    // Same reset for the response-direction doorbell (the Core response
    // worker's wake source): a latch from the crashed driver's final
    // enqueue must not phantom-wake the worker into an empty ring set.
    ring_set.completion_doorbell.doorbell.reset();
    for w in ring_set.completion_doorbell.pending_mask.iter() {
        w.store(0, Ordering::Relaxed);
    }
    // DoorbellCoalescer's tracking fields (pending_count, batch_start_ns —
    // see [Section 5.1](05-distributed.md#distributed-kernel-architecture)) are plain integers.
    // SAFETY: exclusive access — ring_set.state == RECOVERING since U3
    // blocks all producers at select_ring(), and this recovery worker is
    // the sole thread touching the ring set until U17.
    unsafe {
        let c = &ring_set.coalesced_doorbell.coalescer
            as *const DoorbellCoalescer as *mut DoorbellCoalescer;
        (*c).pending_count = 0;
        (*c).batch_start_ns = 0;
    }
}

// NOTE: the former `free_request_dma_handles(entry: &VfsRequest)` and
// `drain_spsc_response_ring(...)` recovery helpers are DELETED. DMA
// handles are freed from the Core-captured `VfsInflightEntry.dma_handle`
// during the U6 table sweep (the dispatch path that allocates a DMA
// buffer records the handle in the entry at enroll time — adding a
// VfsRequestArgs variant with a DmaBufferHandle field WITHOUT capturing
// it in the entry is a DMA-pool leak, the same 50-year-leak rule the
// deleted per-variant match enforced), and the response ring is never
// read during recovery — its contents are discarded by the U7 pointer
// reset under the response_drain_lock.

// NOTE: the former `wake_orphaned_pages(orphaned_pages: &[OrphanedPageEntry])`
// helper — a separate Step-U8 wake pass over a collected orphaned-page list,
// resolving each page from a raw `page_cache_id` pointer — is DELETED. Its
// terminal page state (PageFlags::ERROR + unlock_page) is now performed INLINE
// in the U6 table sweep above, per claimed ReadPage/Readahead entry, via the
// entry's `inode` pin (`entry.inode.i_mapping.page_cache`). The pin makes the
// AddressSpace deref structurally sound (no raw pointer to dangle), and inline
// processing removes the fixed `MAX_ORPHANED_PAGES` cap and the
// "pages remain locked until manual intervention" arm entirely.

writeback_deferred_dirty() Definition (Step U15)

/// Resumable ascending-by-`ino` cursor over the dirty inodes of ONE superblock,
/// starting at `ino >= from_ino`. Cold path (crash recovery). Dirty-inode
/// tracking lives on the BDI (`BdiWriteback.b_dirty` / `b_io` / `b_more_io`,
/// [Section 4.6](04-memory.md#writeback-subsystem)), which are BDI-WIDE and in dirtying order, so each
/// `next()` MIN-SCANS the three lists for the smallest matching `ino` above the
/// cursor — O(n) per step, O(n²) total, bounded by the sb's dirty inode count.
/// Ascending order is what makes the caller's batch-resume correct: on a full
/// batch the caller resumes INCLUSIVELY at the ino of the yielded-but-unpushed
/// inode (`resume_ino = inode.i_ino`), so that inode is retried while the
/// already-batched inodes — all with strictly smaller inos — are not repeated;
/// no inode is skipped or repeated across the RCU drop / re-acquire between
/// batches. Callers hold `rcu_read_lock`.
pub struct DirtyInodeCursor<'a> {
    /// The superblock whose dirty inodes are iterated.
    sb: &'a SuperBlock,
    /// Yield only inodes with `i_ino >= floor`; advanced past each yielded ino.
    floor: u64,
}

impl<'a> Iterator for DirtyInodeCursor<'a> {
    type Item = &'a Inode;

    fn next(&mut self) -> Option<&'a Inode> {
        let bdi = self.sb.s_bdi.as_ref()?; // BDI-less sb → no dirty inodes
        let wb = &bdi.wb;
        // Min-scan all three BDI dirty lists for the smallest ino >= floor that
        // belongs to THIS superblock.
        let mut best: Option<&'a Inode> = None;
        let all = wb
            .b_dirty
            .iter()
            .chain(wb.b_io.iter())
            .chain(wb.b_more_io.iter());
        for inode in all {
            // Belongs to this sb? (Inode.i_sb: Arc<SuperBlock>.)
            if !core::ptr::eq(Arc::as_ptr(&inode.i_sb), self.sb as *const SuperBlock) {
                continue;
            }
            if inode.i_ino < self.floor {
                continue;
            }
            if best.map_or(true, |b| inode.i_ino < b.i_ino) {
                best = Some(inode);
            }
        }
        if let Some(inode) = best {
            self.floor = inode.i_ino + 1; // advance past this ino
        }
        best
    }
}

impl SuperBlock {
    /// Resumable ascending-`ino` iteration over this superblock's dirty inodes,
    /// starting at `ino >= from_ino`. Used by `writeback_deferred_dirty` to drain
    /// deferred dirty pages in bounded batches across RCU sections; ascending
    /// order plus the caller's inclusive full-batch resume (resume at the
    /// yielded-but-unpushed inode's ino) guarantees the resume cursor neither
    /// skips nor repeats an inode. Cold path (crash recovery).
    pub fn dirty_inodes_iter_from(&self, from_ino: u64) -> DirtyInodeCursor<'_> {
        DirtyInodeCursor { sb: self, floor: from_ino }
    }
}

/// Flush dirty pages that were deferred during crash recovery.
///
/// During the driver outage (Steps U1-U14), dirty pages accumulated in
/// the page cache with no backing driver to write them to disk. After
/// the replacement driver remounts (Step U14), this function flushes
/// those dirty pages via the standard writeback path.
///
/// The function iterates the superblock's dirty inode list
/// (`sb.s_dirty` / `sb.s_io`) and submits writeback work items for
/// each dirty inode. The writeback is synchronous: this function
/// blocks until all deferred dirty pages are written (or error).
///
/// # Arguments
///
/// - `sb`: The superblock of the remounted filesystem. The replacement
///   driver is already loaded and accepting writeback requests.
///
/// # Preconditions
///
/// U10b(c') already re-dirtied every crash-stranded PageFlags::WRITEBACK page of
/// this superblock (Phase-1 and failed-committed intent ranges) and
/// cleared its WRITEBACK flag, so by the time U15 runs NO page of this
/// superblock is in crash-stranded PageFlags::WRITEBACK. The DIRTY-list
/// iteration below is therefore COMPLETE: the writepages contract's
/// mandatory skip of already-PageFlags::WRITEBACK pages
/// ([Section 14.1](#virtual-filesystem-layer)) can only skip U15's own concurrent
/// submissions — never a page stranded by the crash, which would
/// otherwise be invisible here (not DIRTY) and permanently strand its
/// fsync waiters.
///
/// # Errors
///
/// Individual page writeback errors are logged via FMA but do not abort
/// the recovery. Pages that fail writeback retain the DIRTY flag and
/// are retried on the next periodic writeback cycle — NOT by this pass:
/// the recovery pass visits each dirty inode AT MOST ONCE, because the
/// resume cursor advances past every processed inode regardless of
/// outcome. (An earlier revision advanced the cursor only on a full
/// batch; a persistently failing inode in a final partial batch was then
/// re-collected from the unchanged cursor forever, and recovery never
/// reached U17 — every boundary-blocked operation stayed asleep.) The
/// function returns the count of failed pages for diagnostic purposes.
///
/// # Performance
///
/// This is a cold path (runs once per crash recovery). The writeback
/// rate is bounded by the replacement driver's throughput. On a typical
/// NVMe device, flushing 100 MB of deferred dirty pages takes ~20-50ms.
pub fn writeback_deferred_dirty(sb: &SuperBlock) -> u64 {
    let mut failed_count: u64 = 0;

    // Phase 1: Collect dirty inodes under RCU read lock.
    // We MUST NOT perform blocking I/O (WritebackSyncMode::Wait) inside an RCU
    // read-side critical section — blocking with tree-RCU prevents
    // grace period completion, causing RCU stalls and potential deadlock.
    // Instead, collect inode references into a bounded ArrayVec, drop
    // the RCU lock, then writeback outside RCU.
    //
    // Capacity 1024: sufficient for most crash recovery scenarios.
    // If the superblock has more than 1024 dirty inodes, the function
    // iterates in batches (collect 1024, drop RCU, writeback, re-acquire
    // RCU for the next batch). This is correct because new dirty inodes
    // cannot be created during crash recovery (ring_set.state == RECOVERING,
    // so no new I/O is accepted). The dirty inode list only shrinks
    // (as writeback completes) or stays the same.
    const BATCH_SIZE: usize = 1024;
    let mut batch: ArrayVec<Arc<Inode>, BATCH_SIZE> = ArrayVec::new();
    let mut resume_ino: u64 = 0;

    loop {
        batch.clear();
        // Phase 1: Collect dirty inodes under RCU protection.
        {
            let _rcu = rcu_read_lock();
            for inode in sb.dirty_inodes_iter_from(resume_ino) {
                if batch.is_full() {
                    // This inode was YIELDED but NOT pushed (the full check
                    // precedes the push below). Resume INCLUSIVELY at its ino
                    // so the next batch retries it: dirty_inodes_iter_from
                    // yields `i_ino >= from_ino`, and every already-batched
                    // inode has a strictly smaller ino (ascending order), so
                    // this neither skips this inode nor repeats the batched
                    // ones. `inode.i_ino + 1` here would SKIP it entirely.
                    resume_ino = inode.i_ino;
                    break;
                }
                // Pin the inode against eviction by taking an owned reference
                // from the superblock's inode cache (`igrab`). An inode absent
                // from the cache is concurrently being torn down — its dirty
                // pages are handled by the eviction path, so skipping it here
                // is correct.
                if let Some(pinned) = inode.igrab() {
                    batch.push(pinned);
                }
            }
        }
        // _rcu dropped here — RCU lock released before blocking I/O.

        if batch.is_empty() {
            break; // No more dirty inodes.
        }

        // Cursor advance PAST this batch, success or failure. Ascending
        // order makes this exact: every batched inode has i_ino <
        // `resume_ino`-after-a-full-batch (the yielded-but-unpushed
        // inode's ino), so `max()` preserves the full-batch resume point
        // while a final PARTIAL batch — where the collection loop left
        // `resume_ino` untouched — advances past its last member. A
        // failed inode is NOT revisited by this pass (its pages retain
        // DIRTY; the periodic writeback cycle owns the retry): advancing
        // on the failure arm is what bounds this loop.
        let advance = batch.last().unwrap().i_ino + 1;

        // Phase 2: Writeback each dirty inode OUTSIDE RCU.
        for inode_ref in batch.iter() {
            let mapping = &inode_ref.i_mapping;
            let wbc = WritebackControl {
                sync_mode: WritebackSyncMode::Wait, // fsync-class: block until stable
                nr_to_write: i64::MAX, // Write all dirty pages
                pages_written: 0,
                range_start: 0,
                range_end: u64::MAX,
                range_cyclic: false,
                cyclic_start: 0,
                for_kupdate: false,
                for_background: false,
                for_reclaim: false,
                tagged_writepages: true, // finite write set (crash-recovery sweep)
            };
            if let Err(_e) = mapping.writeback_range(&wbc) {
                // Count and log, do NOT abort and do NOT revisit: the
                // cursor advance below moves past this inode either way.
                // Its still-DIRTY pages are the periodic writeback
                // cycle's retry work, not this pass's.
                failed_count += 1;
                fma_emit(FaultEvent::WritebackError {
                    inode: inode_ref.i_ino,
                    sb_dev: sb.s_dev,
                });
            }
        }
        // Advance the resume cursor past every inode processed in this
        // batch (success or failure) — see the `advance` comment above.
        // Without this, a failing inode in a final partial batch is
        // re-collected from an unchanged cursor forever.
        resume_ino = resume_ino.max(advance);
        // `Arc<Inode>` drops release the pinned references.
    }
    failed_count
}

Step U17: Restore ring set state to ACTIVE

After the generation counter is bumped (Step U10a), the replacement driver remounts (Step U14), and dirty pages are flushed (Step U15), the ring set is re-activated:

// Step U10a already bumped generation (before CQE drain and driver reload).
// Steps U13a and U16 are now no-ops (generation bump moved to U10a).

// `sb` is the resume_fn context (registered as `ctx = sb`, the DomainId →
// SuperBlock mapping); `ring_set` is the SuperBlock's ring set — always
// present during recovery (the driver re-binds to the existing set, so the
// `Option` is `Some`; U17 is unreachable otherwise).
let ring_set: &VfsRingSet =
    sb.ring_set.as_ref().expect("ring_set present across crash recovery");

// Step U17: re-activate the ring set. This is the LAST step.
// The Release ordering ensures all prior ring resets, slot_states clears,
// driver initialization, and generation bump are visible to producers
// before they observe ACTIVE and begin enqueuing new requests.
ring_set.state.store(VFSRS_ACTIVE, Ordering::Release);
// Wake dispatchers parked on the SuperBlock's recovery_wait during the
// boundary-blocking window (producers that took ENXIO and slept). Ordered
// AFTER the Release store so a woken producer re-runs select_ring() and
// observes ACTIVE. The wake contract lives on SuperBlock.recovery_wait.
sb.recovery_wait.wake_up_all();

This transition from RECOVERING to ACTIVE is the final gate. Without it, the mount remains permanently stuck in RECOVERING state and all VFS operations return ENXIO indefinitely.

Recovery latency impact: Draining N rings sequentially adds O(N) to recovery time. Each ring's cost is O(in-flight entries) for the U6 table sweep plus O(ring_depth) for the U7 slot/pointer reset — with depth 256, ~256+256 iterations per ring worst case. For N=64 rings: ~32,768 iterations, each ~50-200 ns (includes DMA handle freeing and orphaned page collection) = ~1.6-6.6 ms. Well within the 500 ms recovery latency target. The producer quiescence wait (Step U5) adds at most 5 seconds in the worst case (copy_from_user on a major page fault), but typically completes in microseconds (most copy_from_user operations are cache-hot).

---

14.3.11 Live Evolution

Live kernel evolution (Section 13.18) replaces a running filesystem driver with a new version. The evolution protocol interacts with per-CPU rings as follows:

Phase A' (Quiescence) — extended for N rings:

1. Set ring_set.state = QUIESCING (Release) — new VFS operations then take the same path as any non-ACTIVE state: select_ring() returns ENXIO and the dispatch wrapper parks the caller on sb.recovery_wait (an INTERNAL sleep-and-retry, NOT an errno surfaced to userspace), to be woken by Phase C's re-activation below — then fence(SeqCst) BEFORE the first counter read in step 2. This is the same store-buffering pairing as crash recovery's set_all_rings_disconnected(): a producer that raced past select_ring()'s Relaxed filter either has its inflight_ops increment visible to step 2's scan, or observes QUIESCING at reserve_slot()'s post-increment re-check and backs out via abort_slot() (the Nop it leaves in the ring is consumed by the surviving/new consumer, answered Ok(0), and dropped by the response drain — no in-flight entry matches it). Without the fence + re-check, a delayed producer could write into the rings after step 2 declared them quiescent.
2. Wait for all N rings' inflight_ops counters to reach zero. Each RingBuffer<T> has an independent inflight_ops: AtomicU32 counter (incremented in reserve_slot(), decremented in complete_slot()). The same counter is used by crash recovery (Step U5 above).
3. Drain all N response rings to process any final completions from the old driver (via the normal response drain under each ring's response_drain_lock — the state gate exempts QUIESCING here because evolution, unlike crash recovery, wants these completions DELIVERED: vfs_evolution_final_drain() takes each ring's drain lock and runs one drain pass with the gate check suppressed; the old driver is quiesced, so no new responses can race it).

Phase B (Atomic Swap) — the ring pointers are unchanged during vtable swap. The new driver inherits the same ring set. Ring count does not change during live evolution (changing ring count requires unmount/remount).

Phase C (Post-Swap Cleanup) — the new driver re-initializes its consumer threads for all N rings. If the new driver supports a different ring_count_max than the old driver, the ring count remains unchanged until the next remount.

Immediately after all N replacement consumer loops are ready to dequeue, the ring set is re-activated — the identical ACTIVE-store-then-wake ordering as crash recovery Step U17. This is the step that clears QUIESCING; without it the state stays QUIESCING forever and every dispatcher that took ENXIO during quiescence and parked on sb.recovery_wait sleeps until an UNRELATED crash-U17 or umount happens to wake the queue (a lost wakeup on evolution completion):

// `sb` is the evolution context's SuperBlock; `ring_set` is its ring set.
// Release: all replacement consumer-thread setup is visible to producers
// before they observe ACTIVE and begin enqueuing again.
ring_set.state.store(VFSRS_ACTIVE, Ordering::Release);
// Wake dispatchers that took ENXIO from select_ring() during QUIESCING and
// parked on the SuperBlock's recovery_wait. Ordered AFTER the Release store
// so a woken producer re-runs select_ring() and observes ACTIVE. The wake
// contract lives on SuperBlock.recovery_wait.
sb.recovery_wait.wake_up_all();

---

14.3.12 CPU Hotplug

When a CPU comes online or goes offline, the cpu_to_ring mapping must be updated.

14.3.12.1 CPU Online

/// Called by the CPU hotplug framework when a new CPU comes online.
/// Updates the cpu_to_ring mapping for all mounted filesystems.
fn vfs_rings_cpu_online(cpu: CpuId) {
    for sb in all_superblocks() {
        // Skip in-kernel (Tier 0 direct-dispatch) filesystems that have no
        // ring set — only ring-backed mounts carry a cpu_to_ring table.
        let Some(ring_set) = sb.ring_set.as_deref() else { continue };
        if cpu.index() < ring_set.cpu_to_ring_len as usize {
            // Assign the new CPU to a ring based on the mount's granularity.
            let ring_idx = compute_ring_for_cpu(cpu, ring_set);
            // SAFETY: cpu < cpu_to_ring_len, validated above. cpu_to_ring
            // points to a slab-allocated array valid for the mount's lifetime.
            unsafe { (*ring_set.cpu_to_ring.add(cpu.index())).store(ring_idx, Ordering::Release) };
        }
        // If cpu >= cpu_to_ring_len (CPU ID exceeds table allocated at mount
        // time), the fallback in select_ring() routes to ring 0. This can
        // occur if CPUs are hot-added beyond the boot-time num_possible_cpus().
        // A remount would rebuild the table with the new num_possible_cpus().
    }
}

14.3.12.2 CPU Offline

/// Called by the CPU hotplug framework when a CPU goes offline.
/// The cpu_to_ring entry for the offline CPU is NOT cleared — it becomes
/// stale but harmless (any thread migrated off the offline CPU will call
/// select_ring() on its new CPU and get the correct ring). No ring is
/// removed or deallocated on CPU offline — ring count is fixed for the
/// lifetime of the mount.
fn vfs_rings_cpu_offline(cpu: CpuId) {
    // No action required. The ring assigned to this CPU continues to exist
    // and may still have in-flight operations. The driver's consumer thread
    // for this ring continues to drain it.
    //
    // If the offline CPU was the ONLY CPU assigned to a particular ring,
    // that ring becomes idle — the driver's consumer thread for it will
    // find no new work. This is benign.
}

Ring count is immutable for the lifetime of a mount. Rings are allocated at mount time and freed at unmount. CPU hotplug changes the CPU-to-ring mapping but never adds or removes rings. Adding rings would require the driver to reinitialize its consumer side (equivalent to a mini-remount); removing rings would orphan in-flight requests. Both are too disruptive for a hotplug event. If the operator wants to adjust ring count after a topology change, they must unmount and remount.

---

14.3.13 Performance Analysis

14.3.13.1 Cache Line Contention Elimination

The primary performance gain is eliminating cross-CPU cache line contention on the request ring's head/published cache line.

Before (single ring, 64-CPU PostgreSQL checkpoint):

Metric	Value
Producer reservation contention per fsync	~63 contenders on single-ring CAS
Reservation CAS cost (x86-64, 64-way contention)	~3,150-4,410 cycles (~1.3-1.8 us)
Ring head cache line bounces per produce	~1 (lock holder writes, lock release bounces)
Total contention overhead per fsync	~3,200-4,500 cycles
Total for 1000-file checkpoint	~1.3-1.8 ms contention overhead

After (per-CPU rings, 64-CPU PostgreSQL checkpoint):

Metric	Value
Producer reservation contention	0 (each CPU is sole SPSC producer on its ring)
Ring head cache line bounces per produce	0 (ring head is CPU-local)
Global request_id bounce	~1 per fsync (~15-20 cycles)
Total contention overhead per fsync	~15-20 cycles (~6-8 ns)
Total for 1000-file checkpoint	~6-8 us contention overhead

Speedup on contention (PerCpu mode): ~200x reduction in per-fsync contention overhead. The single-ring CAS contention is eliminated entirely — each CPU owns its ring and reserves slots without cross-CPU contention.

Shared-ring mode (PerNuma with 16 CPUs/node): The guarded head claim (reserve_head_claim()) has O(N) expected retries under N-way contention, where N is the number of CPUs sharing the ring. With 16 CPUs per NUMA node: ~15 contenders on the head claim, ~750-1,050 cycles per reservation (vs ~3,150-4,410 for the global single-ring case). This is a ~4x improvement over single-ring, but ~50x worse than PerCpu. PerNuma is appropriate when the memory overhead of PerCpu is unacceptable but some contention reduction is needed.

14.3.13.2 Memory Overhead

Each VfsRingPair occupies: - Ring headers: 2 * 128 bytes = 256 bytes (two cache lines per ring, request + response) - Ring data: 2 * (ring_depth * entry_size). VfsRequest is a Rust enum whose size is dominated by the largest VfsRequestArgs variant (SetXattr contains a KernelString at 256 bytes plus DmaBufferHandle, value_len, and flags). With the enum discriminant, alignment padding, and the VfsRequest header fields (request_id, opcode, ino, fh), the actual entry size is ~320 bytes. VfsCompletion is smaller (~64 bytes). With default depth 256: request ring = 256 * 320 = 80 KB, response ring = 256 * 64 = 16 KB, total ring data ≈ 96 KB per ring pair. - Doorbell + WaitQueue: ~128 bytes - Total per ring: ~96.4 KB

Ring count	Total memory per mount	Notes
1 (legacy)	~96 KB	Baseline
4 (per-NUMA, 4-socket)	~386 KB	Typical server
16 (per-LLC, AMD EPYC)	~1.5 MB	High-core-count server
64 (per-CPU)	~6.2 MB	Maximum parallelism
256 (per-CPU, 256-core)	~24.7 MB	Extreme case

Note: If the per-ring memory for high ring counts is excessive, the ring depth can be reduced proportionally. With 64 per-CPU rings at depth 64 (instead of 256), total per-mount memory drops to ~1.5 MB while still providing sufficient queue depth for typical VFS workloads.

Memory is allocated from the kernel slab at mount time (warm path). For the common case (4-16 rings), memory overhead is modest. The 64-ring and 256-ring cases are opt-in via explicit mount options and appropriate only for high-IOPS storage servers.

14.3.13.3 Per-CPU Ring Depth Reduction

With N rings, each ring serves fewer CPUs and therefore needs fewer slots. The effective queue depth per CPU remains the same or better:

Configuration	Rings	Depth/ring	Effective depth/CPU	Total slots
Single ring, 64 CPUs	1	256	4	256
Per-NUMA (4 nodes), 64 CPUs	4	256	16	1024
Per-LLC (16 groups), 64 CPUs	16	128	128	2048
Per-CPU, 64 CPUs	64	64	64	4096

For per-CPU mode, the per-ring depth can be reduced (via vfs_ring_depth mount option) without reducing effective per-CPU capacity. A depth of 64 per ring with 64 rings provides 64 in-flight operations per CPU — far more than any single CPU can sustain.

14.3.13.4 Impact on Performance Budget

The per-CPU ring extension does NOT increase the per-I/O domain crossing cost. The ring protocol (SPSC produce, domain switch, consume) is identical per operation. The changes affect only:

Cost component	Change	Impact
Ring selection (select_ring())	+1-3 cycles (atomic load + bounds check)	+0.001% on 10 us op
Request ID generation	+15-20 cycles (global atomic fetch_add)	+0.006-0.008% on 10 us op
Doorbell coalescing mask update	+5-10 cycles (atomic OR)	+0.002-0.004% on 10 us op
Cache line contention elimination	-3,150-4,410 cycles (under 64-CPU contention)	-1.3-1.8% saved
Net impact under contention	-3,100-4,370 cycles saved	-1.26-1.78% improvement
Net impact without contention	+21-33 cycles added	+0.009-0.013% overhead

The extension is a net performance win under any multi-CPU workload and negligibly more expensive (~0.01%) for single-CPU workloads. The overhead is well within the existing 2.5% headroom under the 5% budget (Section 3.4).

14.3.13.5 Amortization Math: Negative Overhead Analysis

The design target is NEGATIVE overhead — UmkaOS filesystem I/O must be FASTER than Linux despite the Tier 1 domain switch (Section 1.1). This section presents the amortization math against a production Linux kernel baseline (CONFIG_PROVE_LOCKING=n, CONFIG_LOCK_STAT=n).

Linux baseline for a read() cache miss (measured path, no isolation):

Linux path component	Cost (x86-64, cycles)	Notes
Syscall entry (SYSCALL + kernel stack setup)	~40	Shared with UmkaOS
VFS vfs_read() dispatch (function call, no vtable)	~10-15	Direct call
Linux filesystem ext4_file_read_iter() (indirect call through f_op)	~5-8	Indirect call + branch predictor
filemap_get_pages() (page cache XArray lookup)	~30-50	XArray walk, cache miss case
ext4_readahead() (extent tree lookup + bio build)	~100-300	Varies with extent depth
bio submission (block layer dispatch)	~50-100	Request queue + scheduling
Total Linux in-kernel overhead	~235-513	Before device I/O

UmkaOS path for the same read() cache miss (with per-CPU ring, N=8 batch):

UmkaOS path component	Cost (x86-64, cycles)	Notes
Syscall entry (SYSCALL + kernel stack setup)	~40	Identical to Linux
VFS ring enqueue (write entry + advance published)	~15-20	SPSC produce, no lock
Ring selection (select_ring())	~1-3	Atomic load + bounds check
Request ID generation	~15-20	Atomic fetch_add
Doorbell coalescing mask update	~5-10	Atomic OR (amortized)
Doorbell (domain switch) — amortized over N	~23/N	WRPKRU, amortized
Driver dequeue (ring entry already prefetched in L1)	~5-10	L1 hit from prefetch
Filesystem processing (same as Linux)	~100-300	Extent tree + bio
Response enqueue + completion coalescing	~10-15	SPSC produce + coalesce check
Completion wakeup — amortized over N	~300/N	IPI + WaitQueue, amortized (see note)
Total UmkaOS in-kernel overhead (N=16)	~220-433	Before device I/O

Completion wakeup cost (300 cycles) derivation: The 300-cycle figure assumes same-NUMA-node IPI (~200 cycles on x86-64 Intel Xeon Scalable, measured via rdtsc across APIC_ICR write to handler entry) plus WaitQueue wake overhead (~50-100 cycles for priority-ordered wake + rescheduling check). Cross-NUMA IPI costs ~500-1000 cycles; if the VFS consumer runs on a different NUMA node than the filesystem driver, completion wakeup rises to ~600-1100 cycles. The 300-cycle figure applies to PerLlc and PerNuma ring granularities where producer and consumer share a NUMA node. Cross-NUMA worst case is documented in the per-architecture table below.

Per-operation domain crossing cost — production Linux baseline:

The per-operation overhead that UmkaOS must amortize is compared against the Linux function-call chain overhead that UmkaOS eliminates by replacing indirect calls with a ring protocol.

Linux production per-operation function-call overhead eliminated by UmkaOS:
  vfs_read() dispatch:  ~10-15 cycles (direct call)
  f_op->read_iter:      ~5-8 cycles  (indirect call, x86-64 retpoline)
  Lock stat accounting: ~5-10 cycles (CONFIG_LOCK_STAT=n: 0 cycles;
                                      production kernels vary, ~5 cycles
                                      for inline static key check residual)
  TOTAL (production):   ~20-28 cycles

Note: Linux debug kernels with CONFIG_PROVE_LOCKING=y add ~25 cycles of lockdep
checking per lock acquisition. The baseline above uses PRODUCTION builds only.

Total domain crossing overhead = doorbell_cost + completion_cost
                               = 23 + 300 = 323 cycles (uncoalesced, x86-64)

Breakeven batch size = 323 / 28 = ~12 operations (production Linux)

Why UmkaOS achieves savings despite the domain switch:

1. No lockdep overhead: Linux's lockdep (lock dependency checker) adds ~20-30 cycles to every lock acquisition on debug kernels. Production kernels with Linux production kernels with CONFIG_PROVE_LOCKING=n and CONFIG_LOCK_STAT=n have zero lockdep overhead, but still pay ~5 cycles for inline static key checks on lock-stat-capable paths. UmkaOS's compile-time lock ordering eliminates all runtime lock validation: 0 cycles on all builds.

2. No indirect call overhead: Linux's VFS dispatches through f_op->read_iter — an indirect call through a function pointer. On x86-64 with Spectre v2 mitigations (retpoline/IBRS), indirect calls cost ~15-25 cycles. UmkaOS's ring protocol avoids indirect calls on the hot path — the opcode is a match on a u32, which the compiler converts to a jump table (direct branch).

3. Cache-friendlier ring layout: The ring buffer is a contiguous array with predictable access pattern (sequential consume). Linux's VFS path walks multiple non-contiguous data structures (file -> dentry -> inode -> superblock -> f_op -> address_space -> page tree). The ring's sequential layout produces fewer L1 cache misses (~2-3 vs ~5-8 for the pointer-chasing VFS path).

4. Prefetch hides latency: The driver-side ring entry prefetch (see "Driver-Side Ring Entry Prefetch" above) loads the next 4 entries into L1 while processing the current request. Linux has no equivalent — each VFS function call must load its arguments from wherever they happen to reside in the cache hierarchy.

Summary table — per-operation overhead at different batch sizes (x86-64, production Linux):

Batch size (N)	Domain crossing per-op	Linux prod. overhead saved	Net	Verdict
1 (uncoalesced)	323 cycles	28 cycles	+295 cycles	1.2% overhead on 10us op
2	162 cycles	28 cycles	+134 cycles	0.54% overhead
4	81 cycles	28 cycles	+53 cycles	0.21% overhead
8	40 cycles	28 cycles	+12 cycles	0.048% overhead
12	27 cycles	28 cycles	-1 cycle	Breakeven
16	20 cycles	28 cycles	-8 cycles	NEGATIVE overhead
32	10 cycles	28 cycles	-18 cycles	Negative (bonus)

Breakeven is at N=12 against production Linux. At N>=12, UmkaOS per-operation cost is less than Linux's equivalent function-call path. At N=16 (typical io_uring depth), the saving is ~8 cycles/op. At N=32 (common for high-IOPS NVMe workloads), the saving is ~18 cycles/op.

N=8 (the default regular I/O coalescing batch) is NOT negative-overhead against production Linux — it adds ~12 cycles/op (+0.048% on a 10us operation). This is well within the 2.5% headroom under the 5% budget. The negative-overhead threshold requires N>=12, which is achieved by: - io_uring workloads (typical depth 32-128): always negative overhead. - PostgreSQL checkpoint (fsync storm on 64 backends): N>>12. - Batched readahead (sequential reads trigger 4-32 page prefetch): N>=16.

When N < 12: For single-threaded sequential reads (effective batch size 1-4), the domain crossing adds ~0.2-1.2% overhead — well within the 5% budget. The page cache absorbs >95% of reads without any domain crossing (cache hits are served entirely in Tier 0), so the effective overhead across all operations is much lower than the per-miss figure.

Inline small I/O path (N=1, negative overhead): For reads/writes where count <= INLINE_IO_MAX (192 bytes) — covering >90% of procfs/sysfs accesses — data is carried inline in the ring entry (Section 14.2). No DMA buffer allocation, no IOMMU map/unmap. This eliminates ~150-300ns per small I/O:

Path	Linux cost	UmkaOS inline cost	Delta
read("/proc/self/status", buf, 128)	~280-400 cycles (Linux VFS + seq_file + copy_to_user)	~180-260 cycles (ring + inline_data + copy_to_user)	-100 to -140 cycles
read("/sys/class/net/eth0/mtu", buf, 8)	~250-380 cycles	~160-240 cycles	-90 to -140 cycles

For metadata-heavy workloads (container startup reading hundreds of small files), this is measurable throughput improvement. The inline path achieves negative overhead at N=1 — no batching required. This is the strongest negative-overhead argument for procfs/sysfs workloads that dominate container and monitoring scenarios.

---

14.3.14 Backward Compatibility

14.3.14.1 Single-Ring Drivers

Drivers that do not include a .kabi_vfs_caps ELF section (or set ring_count_max = 1) operate in single-ring mode. The VfsRingSet is allocated with ring_count = 1 and the cpu_to_ring table maps all CPUs to ring 0. This is functionally identical to the baseline protocol — no behavioral change.

14.3.14.2 VfsRingPair Preservation

The VfsRingPair struct is unchanged. The extension wraps it in VfsRingSet without modifying the per-ring structure. The request/response ring layout, entry format, opcodes, and all existing fields remain identical.

14.3.14.3 Cancellation Protocol

The cancellation protocol (Section 14.2) is unchanged per ring. CancelToken.request_id uses the mount-global ID, so the driver can match it against any ring's pending requests. The cancellation side-channel is per ring — each ring has its own cancellation channel, and the cancel token is enqueued on the ring where the original request was submitted: vfs_cancel(request_id) resolves the ring via VfsInflightEntry.ring_index, recorded in the Core-resident in-flight table at enroll time (this table lookup is the request_id → ring mapping; no per-task ring bookkeeping is needed for cancellation).

14.3.14.4 Timeout Handling

Per-request timeouts (Section 14.2) are unchanged. Each request has an independent timer regardless of which ring it was submitted on. The timer callback cancels the request on the specific ring where it was submitted.

---

14.3.15 Cross-References

• VFS ring buffer protocol (base): Section 14.2
• IPC ring buffer design: Section 11.8
• Crash recovery: Section 11.9
• Live kernel evolution: Section 13.18
• KABI transport classes: Section 12.6
• Performance budget: Section 1.3
• Cumulative performance budget: Section 3.4
• Core provisioning: Section 7.11
• Doorbell coalescing: Section 5.1
• fsync end-to-end flow: Section 14.4
• Page cache: Section 4.4

---

14.3.16 Phase Assignment

Component	Phase	Rationale
VfsRingSet struct and single-ring allocation	Phase 2	Replaces VfsRingPair allocation at mount; backward compatible with ring_count=1.
Mount option parsing (vfs_ring_count, vfs_ring_granularity)	Phase 2	Mount option infrastructure exists; adding new options is incremental.
CPU-to-ring mapping table and select_ring()	Phase 2	Core hot-path change; must be correct from first multi-ring mount.
Global request_id counter	Phase 2	Replaces per-ring counter; simple atomic.
Coalesced doorbell	Phase 2	Required for multi-ring to avoid doorbell storms.
Driver-side round-robin consumer (Strategy 1)	Phase 2	Minimum viable multi-ring consumer.
KabiVfsCapabilities ELF section and negotiation	Phase 2	Must exist before any driver can advertise multi-ring support.
Crash recovery for N rings	Phase 2	Must be correct before any production use of multi-ring mode.
Per-ring consumer threads (Strategy 2)	Phase 3	Optimization; round-robin is sufficient for Phase 2.
CPU hotplug integration	Phase 3	Hotplug is uncommon in production; Phase 2 mapping is static.
Live evolution for N rings	Phase 3	Live evolution is Phase 3 feature.
Adaptive granularity auto-selection	Phase 3	Requires topology discovery infrastructure.
Driver-side ring entry prefetch	Phase 2	Trivial to implement (one prefetch intrinsic per dequeue); significant L1 cache benefit.
Completion coalescing (response direction)	Phase 2	Required for negative-overhead target; mirrors request-side doorbell coalescing.

---

14.4 fsync / fdatasync End-to-End Flow

The fsync(2) and fdatasync(2) syscalls guarantee that file data (and optionally metadata) reach stable storage. This section documents the complete call path from syscall entry to disk write completion, crossing VFS, page cache, filesystem, and block layer boundaries.

Syscall entry → VFS dispatch:

fsync(fd) / fdatasync(fd)
  → sys_fsync() / sys_fdatasync()
  → vfs_fsync_range(file, start=0, end=LLONG_MAX, datasync)

vfs_fsync_range(file, start, end, datasync) is the canonical entry point. The sync_file_range(2) syscall also calls it with a sub-range.

Step 1 — Writeback dirty pages:

/// VFS-level fsync implementation.
///
/// `file` is the canonical open-file description (`OpenFile`,
/// [Section 14.1](#virtual-filesystem-layer--openfile-open-file-description)) — held by the caller as
/// `Arc<OpenFile>` and passed by shared reference. There is no separate
/// `File` type in UmkaOS. All per-fd mutable state touched here
/// (`f_wb_err`) is interior-mutable (atomic), so `&OpenFile` suffices
/// even when the description is shared across threads via `dup(2)`/`fork(2)`.
fn vfs_fsync_range(
    file: &OpenFile,
    start: i64,
    end: i64,
    datasync: bool,
) -> Result<(), IoError> {
    // Data plane: the mapping to flush is the RESOLVED data inode's
    // (`OpenFile::data_inode`, [Section 14.1](#virtual-filesystem-layer)), which equals
    // `file.inode` for every non-stacking filesystem.
    let mapping = &file.data_inode.i_mapping;

    // (1) Flush all dirty pages in [start, end] to the block layer.
    //     The writeback engine iterates dirty pages, calling
    //     AddressSpaceOps::writepage() for each one, which builds bios
    //     and submits them. Does NOT wait for I/O completion yet.
    //     Hold the result — do NOT early-`?` it, so step (3)'s ErrSeq advance
    //     still runs even on a synchronous flush error (FLOW-16 J12).
    let flush_res = filemap_write_and_wait_range(mapping, start, end);

    // (1b) DSM-aware writeback: for MS_DSM_COOPERATIVE superblocks, after
    //      local writeback completes, wait for DSM home node acknowledgment.
    //      dsm_sync_pages() sends dirty DSM pages via RDMA and blocks until
    //      PutAck is received from each home node, ensuring data durability
    //      on the remote node before fsync returns.
    //      See [Section 6.12](06-dsm.md#dsm-subscriber-controlled-caching--fsync-semantics).
    //      Gated on a successful flush (as before), but its result is HELD
    //      rather than early-returned so step (3) still advances the snapshot.
    // `mapping.host` is `Weak<Inode>` — must upgrade before field access.
    // Upgrade CANNOT return None for an open file: the fd keeps the Inode alive.
    // Surface a clean EBADF if the invariant is somehow violated.
    let dsm_res: Result<(), IoError> = if flush_res.is_ok() {
        match mapping.host.upgrade() {
            Some(host_inode) => {
                if host_inode.i_sb.s_flags.load(Relaxed) & MS_DSM_COOPERATIVE != 0 {
                    dsm_sync_pages(&*host_inode, start, end)
                } else {
                    Ok(())
                }
            }
            None => Err(IoError::from_raw(EBADF as i32)),
        }
    } else {
        Ok(())
    };

    // (2) Dispatch to filesystem-specific fsync (journal commit, etc).
    //     For KABI Tier 1/2 drivers: sends Fsync through VFS ring buffer.
    //     For in-kernel filesystems: calls FileOps::fsync() directly.
    //     Gated on flush+DSM success (matching the original short-circuit),
    //     result HELD for step (3).
    //     `Inode` stores the raw inode number as `i_ino: u64`; the FileOps
    //     interface uses the `InodeId` newtype — wrap explicitly.
    //
    //     GENERATION RECHECK between the phase-2 wake and this dispatch
    //     (load-bearing): step (1) can PARK this task across an entire
    //     driver crash recovery — the U10b writeback rescue is the waker
    //     for phase-2 sleepers, and it runs BEFORE Step U13 frees the
    //     crashed instance's heap and BEFORE the U14/U17 reload+resume
    //     ([Section 14.3](#vfs-per-cpu-ring-extension--crash-recovery)). A
    //     `private_data` token loaded without this recheck would name
    //     per-open state in that freed heap; dispatching it — directly,
    //     or via a ring submission that parks until U17 — hands the NEW
    //     driver instance a token it never issued (use-after-free
    //     class). The syscall-entry generation check does not cover this
    //     window: it ran BEFORE the park, against the pre-crash
    //     generation. `vfs_ensure_current_generation()` (below) re-runs
    //     the dispatch wrapper's own composition; only after it passes
    //     is the token loaded — the passing Acquire generation compare
    //     is ordered after revalidation's Release store of the fresh
    //     token, so the pre-crash token cannot be observed here.
    let fs_res: Result<(), IoError> = if flush_res.is_ok() && dsm_res.is_ok() {
        let inode_id = InodeId(file.inode.i_ino);
        match vfs_ensure_current_generation(file) {
            Ok(()) => {
                let private = file.private_data.load(Ordering::Relaxed) as u64;
                file.f_ops.fsync(inode_id, private, start as u64, end as u64, datasync as u8)
            }
            // HELD, not early-returned: step (3)'s ErrSeq advance must
            // still run (the J12 rule below).
            Err(e) => Err(e),
        }
    } else {
        Ok(())
    };

    // (3) SINGLE per-fd writeback-error report point. `check_and_advance` runs
    //     UNCONDITIONALLY — it is never skipped by an early `?` on steps
    //     (1)/(1b)/(2). That skip was the defect (FLOW-16 J12): a writeback
    //     error set both the AS flags (consumed+returned by
    //     filemap_write_and_wait_range) and `wb_err`; the early `?` exited
    //     before this advance, so the fd snapshot never moved and the NEXT
    //     fsync re-reported the same error via `wb_err`. Advancing the fd's
    //     snapshot here reports each writeback error EXACTLY ONCE per fd;
    //     `wb_err` (ErrSeq) is the authoritative fd channel, so the AS flags
    //     already consumed by filemap for non-fd callers stay coherent.
    //     `f_wb_err` is an atomic snapshot: dup'd/forked fds share one
    //     OpenFile, so check_and_advance() CAS-advances — exactly one racer
    //     reports (once-per-fd; see the ErrSeq contract below).
    let wb_errno = mapping.wb_err.check_and_advance(&file.f_wb_err);

    // Report precedence: a synchronous flush / DSM / fs error first (these are
    // NOT recorded in `wb_err`, so returning them here does not re-report), else
    // the once-per-fd writeback error just consumed above.
    flush_res?;
    dsm_res?;
    fs_res?;
    if let Some(errno) = wb_errno {
        return Err(IoError::from_raw(errno));
    }

    Ok(())
}

/// Ensure `file` is usable at the CURRENT driver generation before a
/// driver dispatch that consumes its `private_data` token OUTSIDE the
/// VFS dispatch wrapper (vfs_fsync_range step (2) is the caller). Runs
/// the wrapper's own pre-`select_ring()` composition
/// ([Section 14.1](#virtual-filesystem-layer--open-file-descriptor-recovery-generation-refresh)):
/// the `vfs_check_open_generation()` Acquire fast compare; on ENOTCONN,
/// `vfs_revalidate_open_file()` (re-open against the live instance —
/// Release-stores the fresh token BEFORE the generation); on a
/// second-recovery ENXIO, an interruptible park on `sb.recovery_wait`
/// and retry. Terminal errors are mapped to `IoError` for the fsync
/// context: dead fd → EIO, allocation failure → ENOMEM, interrupted
/// park → the wait's standard syscall-restart disposition.
fn vfs_ensure_current_generation(file: &OpenFile) -> Result<(), IoError>;

filemap_write_and_wait_range(mapping, start, end) performs two phases:

1. Write phase: Walk the page cache (XArray range scan) for pages in [start, end] with PageFlags::DIRTY set. For each dirty page, call AddressSpaceOps::writepage(mapping, page, wbc). The filesystem maps the page to physical block(s), builds a Bio, and submits it to the block layer (Section 15.2). Set WRITEBACK flag on the page.

DIRTY / WRITEBACK ownership (submitter-side protocol): the writeback SUBMISSION path — the filesystem's writepages(), the per-page writepage() fallback below, and the cross-domain dispatch step 2 (Section 4.6) — clears PageFlags::DIRTY and the XArray dirty tag under the page lock AND sets WRITEBACK + increments nrwriteback, all at dispatch time. The COMPLETION path never clears DIRTY, and it performs NO nr_dirty decrement: it clears only WRITEBACK, decrements nrwriteback (and the per-BDI BdiWriteback.nr_writeback in lock-step), then wakes waiters. nr_dirty/BdiWriteback.nr_dirty accounting belongs entirely to the submission path (decrement-at-submission, D1-A): the submitter cleared DIRTY and decremented both dirty counters on the 1→0 edge at dispatch, so the completion owns no DIRTY edge to account for. Clearing DIRTY at submission still lets the completion tell a genuine-clean page (DIRTY==0) from a concurrent redirty (DIRTY set again), but that distinction drives only the re-queue decision — preserve the redirty for the next writeback cycle — never an nr_dirty decrement. For EVERY deployment the completion funnels through the shared tier-agnostic epilogue writeback_page_epilogue() (Section 15.2): - Co-located filesystems (filesystem driver shares the Core domain): writeback_end_io() (deferred via writeback_end_io_deferred on the blk-io workqueue) calls writeback_page_epilogue() per bio segment. - Cross-domain filesystems (Tier 1/2 deployment): the Core-side WritebackResponse handler (steps 11-12 in Section 4.6) calls writeback_page_epilogue() per page — on success AND on error. Exactly one path calls the epilogue per page per deployment. This single-owner design prevents the double-decrement bug that would occur if two completion paths each decremented nrwriteback/nr_dirty for one submission.

1. Wait phase: Walk the same range again. For each page with WRITEBACK set, block until the bio completion callback clears the flag. If any page has AddressSpaceFlags::EIO / ENOSPC error state, return the error and clear it (one-shot error reporting — see below).

Provider crash while waiting: if the filesystem provider's domain crashes with this fsync's WritebackRequests in flight on the writeback ring, the wake that unblocks this phase comes from recovery Step U10b (Section 14.3): the writeback crash-recovery bypass flushes committed extents directly to the block device and then runs writeback_page_epilogue() on the affected pages, which clears WRITEBACK AND calls page.wake_waiters(). The explicit wake — not the bare flag clear — is what unblocks these parked waiters: wait_on_page_writeback() sleeps in wq.wait_event, which re-checks its predicate only on a wakeup, so clearing the flag without waking would strand the sleeper for the whole (untimed) wait. The wake is bounded by the recovery sequence rather than by an independent detector. Error state (uncommitted extents deferred to journal replay + U15 re-write) propagates through ErrSeq exactly as for any writeback error.

Dual error reporting (mapping flags + ErrSeq): The AddressSpaceFlags::EIO/ENOSPC bits are the quick-check mechanism. ErrSeq (wb_err) is the primary error reporting mechanism: it provides per-fd error visibility (each open fd sees the error exactly once via check_and_advance()). Both mechanisms are set together by writeback_end_io() for consistency, but ErrSeq is authoritative for fsync() error returns. The mapping flags are used by filemap_write_and_wait_range for callers that check AddressSpace state directly.

/// Flush all dirty pages in [start, end] to the block layer and wait
/// for completion. This is the core implementation behind fsync step 1.
///
/// # Algorithm
/// Phase 1 (write): iterate dirty pages and submit writeback bios.
/// Phase 2 (wait): block until all submitted bios complete.
///
/// # Locking
/// Does NOT acquire and does NOT require `i_rwsem` — callers invoke it
/// AFTER releasing the inode write lock (`page_cache_write_iter` drops
/// `i_rwsem` before its O_SYNC flush; `vfs_fsync_range` never takes it).
/// Waiting for storage I/O under an exclusive inode lock would block all
/// readers for the I/O duration.
/// Acquires page locks individually via write_begin/end protocol.
///
/// # Error handling
/// Collects errors from both phases. Returns the first error encountered.
/// All pages are processed even after an error (no short-circuit) to
/// maximize data written to disk before returning failure.
fn filemap_write_and_wait_range(
    mapping: &AddressSpace,
    start: i64,
    end: i64,
) -> Result<(), IoError> {
    let mut first_err: Result<(), IoError> = Ok(());

    // --- Phase 1: Write dirty pages ---
    let start_idx = (start as u64) >> PAGE_SHIFT;
    let end_idx = (end as u64) >> PAGE_SHIFT;

    // Resolve page cache once — DAX mappings have no page cache and require
    // no flush (persistent memory is always coherent). Return Ok(()) early
    // so subsequent phases don't attempt to iterate a non-existent cache.
    let pc = match &mapping.page_cache {
        Some(pc) => pc,
        None => return Ok(()), // DAX mapping — no page cache to flush.
    };

    // Full WritebackControl init ([Section 4.6](04-memory.md#writeback-subsystem) — canonical
    // struct; there is no `WbSyncMode`, the sync-mode enum is
    // `WritebackSyncMode` and fsync uses `Wait`). `tagged_writepages`
    // guarantees a finite write set (pages dirtied after this point are
    // NOT chased — they belong to the next fsync).
    let wbc = WritebackControl {
        sync_mode: WritebackSyncMode::Wait,
        nr_to_write: i64::MAX,
        pages_written: 0,
        range_start: start as u64,
        range_end: end as u64,
        range_cyclic: false,
        cyclic_start: 0,
        for_kupdate: false,
        for_background: false,
        for_reclaim: false,
        tagged_writepages: true,
    };

    // Prefer writepages() for batch I/O if the filesystem supports it.
    // writepages() is all-or-nothing (see the contract on
    // AddressSpaceOps::writepages in [Section 14.1](#virtual-filesystem-layer)): on
    // error it rolls back PageFlags::WRITEBACK/nrwriteback for any page it touched
    // but did not actually dispatch, so the per-page fallback below never
    // double-submits a page or double-increments nrwriteback.
    if mapping.ops.writepages(mapping, &wbc).is_err() {
        // Fall back to per-page writepage() over the RESIDENT dirty entries via
        // XArray tagged iteration (`xa_for_each_tagged(XA_TAG_DIRTY)` — the same
        // O(k)-in-dirty-pages scan writeback_inode_pages and
        // truncate_inode_pages_range use). This REPLACES the old scalar
        // point-load per index over [start_idx, end_idx]: with fsync passing
        // end = LLONG_MAX (end_idx ≈ 2^51) that loop was ~2^51 probes and never
        // completed, even on an empty or sparse file — a whole-file fsync hang
        // (FLOW-16 J1). Tagged iteration skips whole untagged radix nodes, so an
        // empty/absent range yields nothing and returns at once, and a bounded
        // [start,end] range (O_SYNC, sync_file_range) stays O(k) in the dirty
        // pages it covers.
        let mut iter = pc.pages.iter_tagged(start_idx, XA_TAG_DIRTY);
        while let Some((index, entry)) = iter.next() {
            if index > end_idx {
                break; // ascending iteration — past the requested range.
            }
            // Explicit PageEntry → Page projection: pc.pages stores PageEntry (a
            // PageRef plus a content-generation counter), NOT a Page. Content
            // state (DIRTY/WRITEBACK) lives on Page.flags, reached via
            // entry.page.page() (the slot's PageRef, then PageRef::page());
            // shadow/refault entries live in a SEPARATE XArray and never appear
            // here, so this is a real resident page — never a cast of the entry
            // to a Page (FLOW-16 J4).
            let page = entry.page.page();
            page.lock();
            // Skip pages already under writeback: per the all-or-nothing
            // writepages() contract ([Section 14.1](#virtual-filesystem-layer)), a page the
            // failed writepages() left with WRITEBACK set has a bio in flight;
            // re-submitting would double-submit and double-count nrwriteback.
            // Phase 2 waits for it like any other.
            if page.flags_load(Acquire).contains(PageFlags::WRITEBACK) {
                page.unlock();
                continue;
            }
            // Confirm-step + submitter protocol (shared with
            // writeback_inode_pages and the cross-domain path,
            // [Section 4.6](04-memory.md#writeback-subsystem)): XA_TAG_DIRTY is an INDEX, not the
            // authority — it is set under xa_lock but cleared here under the
            // page lock, so a tag can outlive the frame's DIRTY bit. Under the
            // page lock, clear DIRTY on the frame (Page.flags is the sole
            // authority, H1) and read the RMW's old value as the edge guard. A
            // stale tag (old lacked DIRTY) is a harmless skip. On a genuine 1→0
            // edge, decrement nr_dirty at BOTH aggregation levels, drop the tag,
            // set WRITEBACK, and increment the writeback counters — all at
            // SUBMISSION. The completion epilogue clears WRITEBACK and decrements
            // the writeback counters; it NEVER touches DIRTY or nr_dirty, so a
            // redirty during writeback is a fresh 0→1 edge that survives for the
            // next cycle (FLOW-16 J7).
            let old = page.flags_fetch_and(!PageFlags::DIRTY, AcqRel);
            pc.pages.clear_tag(index, XA_TAG_DIRTY);
            if !old.contains(PageFlags::DIRTY) {
                page.unlock();
                continue; // stale tag — page cleaned concurrently.
            }
            pc.nr_dirty.fetch_sub(1, Relaxed);
            pc.bdi().wb.nr_dirty.fetch_sub(1, Relaxed);
            page.flags_fetch_or(PageFlags::WRITEBACK, Release);
            mapping.nrwriteback.fetch_add(1, Relaxed);
            pc.bdi().wb.nr_writeback.fetch_add(1, Relaxed);
            page.unlock();
            if let Err(e) = mapping.ops.writepage(mapping, page, &wbc) {
                // Synchronous submission failure (no bio in flight → no
                // completion will run the epilogue): UNDO this page's submission
                // state — re-set DIRTY + the dirty tag (the page is still dirty
                // and must be retried; clearing them would silently lose the
                // data) and RE-INCREMENT both nr_dirty counters on that 0→1
                // re-dirty edge (they were decremented at submission), clear
                // WRITEBACK, and decrement the writeback counters (nrwriteback +
                // per-BDI nr_writeback). Same submitter-protocol undo as
                // writeback_inode_pages ([Section 4.6](04-memory.md#writeback-subsystem))
                // (FLOW-16 J8-family).
                page.lock();
                let old = page.flags_fetch_or(PageFlags::DIRTY, AcqRel);
                if !old.contains(PageFlags::DIRTY) {
                    pc.nr_dirty.fetch_add(1, Relaxed);
                    pc.bdi().wb.nr_dirty.fetch_add(1, Relaxed);
                }
                {
                    let _xa = pc.pages.xa_lock();
                    pc.pages.set_tag_locked(index, XA_TAG_DIRTY);
                }
                page.flags_fetch_and(!PageFlags::WRITEBACK, Release);
                mapping.nrwriteback.fetch_sub(1, Relaxed);
                pc.bdi().wb.nr_writeback.fetch_sub(1, Relaxed);
                page.unlock();
                if first_err.is_ok() { first_err = Err(e); }
            }
        }
    }

    // --- Phase 2: Wait for WRITEBACK completion ---
    // Range-bounded scan with the SAME (index, entry) yield shape as Phase 1
    // above, but UNTAGGED. XA_TAG_DIRTY was cleared at each page's submission
    // (before WRITEBACK was set), so a dirty-tag scan would skip exactly the
    // pages now under writeback; there is no writeback tag, so iterate the plain
    // [start_idx, end_idx] index range directly. This is O(resident-pages-in-
    // range), not O(file-resident): a bounded sync_file_range no longer pays a
    // whole-file scan, and it is never a scalar point-load per index over
    // [start_idx, end_idx] (~2^51 probes for a whole-file fsync, FLOW-16 J1).
    // WRITEBACK is content state on Page.flags, reached via entry.page.page() —
    // the entry is a PageEntry, not a Page (J4). No per-page PageFlags::ERROR
    // test here: a transient writeback error is delivered exactly once per fd
    // via ErrSeq at the single report point in vfs_fsync_range (and via the AS
    // flags below for non-fd callers), so reading the sticky ERROR bit — which
    // is never cleared on a later successful retry — returned spurious EIO from
    // every future fsync (FLOW-16 J10).
    for (_index, entry) in pc.pages.range(start_idx..=end_idx) {
        let page = entry.page.page();
        // Sleep until the completion epilogue clears WRITEBACK and wakes us.
        wait_on_page_writeback(page);
    }

    // Check mapping-level error bits (set by mapping_set_error on I/O error).
    let err_bits = AddressSpaceFlags::EIO | AddressSpaceFlags::ENOSPC;
    if mapping.flags_load(Acquire).intersects(err_bits) {
        let flags = mapping.flags_fetch_and(!err_bits, Release);
        if first_err.is_ok() {
            if flags.contains(AddressSpaceFlags::ENOSPC) {
                first_err = Err(IoError::new(Errno::ENOSPC));
            } else {
                first_err = Err(IoError::new(Errno::EIO));
            }
        }
    }

    first_err
}

Page Wait Queue Infrastructure (used by wait_on_page_writeback and wait_on_page_locked):

/// Global page wait hash table. Hashed by page address to reduce memory
/// overhead (one WaitQueueHead per hash bucket, not per page).
/// Size: 256 buckets (matches Linux's PAGE_WAIT_TABLE_BITS = 8).
/// Warm path: accessed on every fsync and every page fault that waits
/// for I/O completion.
// Inline-const array repeat: WaitQueueHead is not Copy (interior atomics),
// so a plain `[WaitQueueHead::new(); 256]` repeat expression is illegal —
// same gotcha as the workqueue BoundedMpmcRing slot array.
static PAGE_WAIT_TABLE: [WaitQueueHead; 256] =
    [const { WaitQueueHead::new() }; 256];

/// Map a Page reference to its hash bucket in the page wait table.
fn page_waitqueue(page: &Page) -> &'static WaitQueueHead {
    // Hash by page struct address (NOT physical address) — the page struct
    // is pinned in MEMMAP and has a stable address for the kernel lifetime.
    let hash = (page as *const Page as usize >> PAGE_SHIFT) & 0xFF;
    &PAGE_WAIT_TABLE[hash]
}

/// Sleep until the WRITEBACK flag is cleared on the page.
/// Called by fsync Phase 2 and by the page fault path when a page is
/// undergoing writeback. The matching wake is in `Page::wake_waiters()`,
/// called from `writeback_end_io()` after clearing WRITEBACK.
fn wait_on_page_writeback(page: &Page) {
    // Fast check: if WRITEBACK is already clear, return immediately.
    if page.flags.load(Acquire) & PageFlags::WRITEBACK.bits() == 0 {
        return;
    }
    let wq = page_waitqueue(page);
    wq.wait_event(|| page.flags.load(Acquire) & PageFlags::WRITEBACK.bits() == 0);
}

impl Page {
    /// Wake all waiters sleeping on this page's hash bucket.
    /// Called from writeback completion (after clearing WRITEBACK) and
    /// from unlock_page (after clearing PageFlags::LOCKED).
    pub fn wake_waiters(&self) {
        page_waitqueue(self).wake_up_all();
    }
}

/// Set writeback error on an AddressSpace. Called from writeback completion
/// paths (writeback_end_io, end_page_writeback) when a bio completes with
/// an I/O error.
///
/// Wraps ErrSeq::set_err() and sets the AddressSpaceFlags::EIO/ENOSPC bits.
/// Both mechanisms are updated together for consistency — ErrSeq is
/// authoritative for fsync() error reporting, the mapping flags serve
/// callers that check AddressSpace state directly.
fn mapping_set_error(mapping: &AddressSpace, errno: Errno) {
    // Increment ErrSeq generation and store the error code.
    mapping.wb_err.set_err(errno);
    // Set the mapping-level quick-check bits.
    if errno == Errno::ENOSPC {
        mapping.flags_fetch_or(AddressSpaceFlags::ENOSPC, Release);
    } else {
        mapping.flags_fetch_or(AddressSpaceFlags::EIO, Release);
    }
}

/// Add an inode to the BDI's dirty list if not already present.
/// Called from set_page_dirty() when a page first transitions to dirty
/// ([Section 4.4](04-memory.md#page-cache)). The caller passes the owning inode directly
/// (`PageCache::inode()` returns `Arc<Inode>` via the AddressSpace's
/// `host: Weak<Inode>` upgrade) — a bare `InodeId` would be ambiguous
/// here, since `InodeId` is only unique per superblock and this function
/// has no superblock context to resolve it against.
/// Equivalent to mark_inode_dirty(inode, I_DIRTY_PAGES) — checks the
/// I_DIRTY_PAGES flag to avoid duplicate list insertions.
fn bdi_dirty_inode(bdi: &BackingDevInfo, inode: Arc<Inode>) {
    // Check I_DIRTY_PAGES — idempotent, no action if already set.
    if inode.i_state.fetch_or(InodeStateFlags::I_DIRTY_PAGES, AcqRel) & InodeStateFlags::I_DIRTY_PAGES != 0 {
        return; // Already on the dirty list.
    }
    // Add to BDI's b_dirty list under writeback_lock.
    bdi.wb.push_dirty_inode(inode);
}

Step 2 — Filesystem-specific sync (journaled filesystems):

For journaled filesystems (ext4, XFS, btrfs), fsync() does more than flush pages:

Filesystem	fsync action after writeback
ext4 (data=ordered)	Force the journal transaction to stable storage and issue a cache flush
ext4 (data=journal)	Commit journal transaction containing both data and metadata
XFS	Force the log through the LSN covering the inode's metadata
btrfs	Flush the per-root log tree, then the superblock
tmpfs	No-op (no backing store)
NFS	Send a COMMIT RPC to the server
FUSE (Section 14.11)	Send FUSE_FSYNC opcode through /dev/fuse
KABI Tier 1/2	Send VfsRequest::Fsync { datasync, start, end } through ring buffer

/// ext4 fsync implementation. Called via FileOps::fsync() dispatch.
/// Ensures all data and metadata for the inode reach stable storage
/// by forcing the JBD2 journal to commit.
///
/// Linux equivalent: ext4_sync_file() in fs/ext4/fsync.c.
fn ext4_fsync(
    inode_id: InodeId,
    private: u64,
    start: u64,
    end: u64,
    datasync: u8,
) -> Result<()> {
    // `private` is ext4's per-open context: ext4's `FileOps::open()`
    // returns `inode.i_private as u64` — the `*const Ext4InodeInfo` for
    // this inode ([Section 15.6](15-storage.md#filesystem-ext4)). A bare `InodeId` cannot be
    // resolved here: `InodeId` is unique only within a superblock
    // (`inode_cache_lookup(sb, ino)` needs the sb), and the static
    // `FileOps` vtable carries no superblock. The Ext4InodeInfo
    // back-pointer supplies the VFS inode (and through it `i_sb`).
    //
    // SAFETY: the VFS caller (`vfs_fsync_range`) holds
    // `file.inode: Arc<Inode>` across this call; `Ext4InodeInfo` is
    // allocated with the ext4 inode and freed only in
    // `evict_inode()`, so it outlives every open file on the inode.
    let ext4_info = unsafe { &*(private as *const Ext4InodeInfo) };
    // SAFETY: `vfs_inode` is set once when the ext4 inode is allocated and valid
    // for the Inode's lifetime (see Ext4InodeInfo in [Section 15.6](15-storage.md#filesystem-ext4)).
    let inode = unsafe { &*ext4_info.vfs_inode };
    debug_assert_eq!(inode.i_ino, inode_id.0);
    let sbi = ext4_sb_info(&inode.i_sb);
    let journal = &sbi.journal;

    // For data=journal mode, all data is already in the journal.
    // Force the transaction containing this inode's data to commit.
    //
    // For data=ordered mode, data pages were already flushed by
    // filemap_write_and_wait_range() (step 1 in vfs_fsync_range).
    // We only need to commit the metadata transaction.

    // If fdatasync and only timestamps changed (I_DIRTY_TIME without
    // I_DIRTY_DATASYNC), skip the journal commit entirely.
    if datasync != 0
        && inode.i_state.load(Acquire) & InodeStateFlags::I_DIRTY_DATASYNC == 0
        && inode.i_state.load(Acquire) & InodeStateFlags::I_DIRTY_TIME != 0
    {
        return Ok(());
    }

    // Force the journal transaction containing this inode's metadata
    // to disk. journal_force_commit() waits for the commit I/O to
    // complete, including the commit block written with
    // BioFlags::PREFLUSH | BioFlags::FUA (JBD2 commit protocol step 5,
    // [Section 15.6](15-storage.md#filesystem-ext4)). PREFLUSH forces the transaction's
    // descriptor/metadata/revoke (and ordered-mode data) blocks out of the
    // device's volatile cache BEFORE the commit block, and FUA makes the
    // commit block itself durable — so no separate post-commit flush bio is
    // needed here.
    // i_datasync_tid is in the ext4-specific inode info (`ext4_info`,
    // already resolved from `private` above), not the generic Inode.
    let tid = ext4_info.i_datasync_tid.load(Acquire);
    journal.force_commit(tid)?;

    // The commit block's PREFLUSH is what guarantees ordering: it makes every
    // preceding write of the transaction durable before the commit block lands,
    // and the commit block's FUA (or its post-write-flush emulation on
    // FUA-incapable devices) makes the commit block itself durable. FUA WITHOUT
    // PREFLUSH would leave the preceding data/metadata in volatile cache — a
    // power loss would then discard the transaction (CRC mismatch) or lose
    // ordered-mode data despite fsync success (FLOW-16 J2).

    Ok(())
}

Step 3 — Block layer flush (inside filesystem-specific fsync):

The device cache flush described below is performed inside the filesystem-specific fsync() implementation (step 2), not as a separate VFS-level post-fsync action. For ext4, the BioFlags::PREFLUSH | BioFlags::FUA on the journal commit block serves this purpose — PREFLUSH drains the cache of the preceding transaction writes and FUA (or its post-write-flush emulation on FUA-incapable devices) makes the commit block durable. For other filesystems:

After the filesystem commits its journal/log, it issues a cache flush to the storage device to ensure write-back caches are drained:

bio_submit_and_wait(bio with BioFlags::PREFLUSH | BioFlags::FUA)
  → block device request queue
  → NVMe: FLUSH command (opcode 0x00) / SATA: FLUSH CACHE EXT (0xEA)
  → completion interrupt → bio_complete() → wake waiters

Tier 1 crash recovery: Journal commit bios MUST set BioFlags::PERSISTENT (Section 15.2) so they are preserved across Tier 1 storage driver crash recovery. The block layer's pending bio list retains PERSISTENT bios during domain teardown and replays them to the new driver instance after reload. Without this flag, a Tier 1 driver crash between journal write submission and completion would lose the journal commit — corrupting the filesystem on the next mount (journal replay would be incomplete).

For fdatasync: metadata-only changes (atime, mtime) are NOT flushed. The filesystem skips journal commit if only timestamps changed (I_DIRTY_TIME flag without I_DIRTY_DATASYNC).

Error reporting — one-shot semantics:

/// Error state per AddressSpace. Encapsulates a monotonic sequence counter
/// and the most recent errno. On writeback I/O failure, call `set_err(errno)`.
/// On fsync(), compare the file's snapshot with `sample()` to detect new errors.
///
/// Provides the same "each error seen exactly once per fd" semantics as
/// Linux's errseq_t. Errno and counter are packed into a single atomic word
/// to prevent torn reads between errno and sequence counter.
///
/// **64-bit architectures** (x86-64, AArch64, RISC-V 64, PPC64LE, s390x,
/// LoongArch64): uses `AtomicU64` — errno in low 16 bits, counter in high
/// 48 bits. No counter wrap concern.
///
/// **32-bit architectures** (ARMv7, PPC32): uses packed `AtomicU32` matching
/// Linux's errseq_t layout — bits [11:0] = errno, bit [12] = seen flag,
/// bits [31:13] = counter.
/// Longevity: 19-bit counter wraps after 524,288 errors. At 1 error/sec
/// (extreme), wraps in ~6 days. Matches Linux errseq_t layout (ABI-constrained).
/// 32-bit targets (ARMv7, PPC32) only. 64-bit targets use 47-bit counter
/// (140T values, safe for 50-year uptime at any realistic error rate).
/// Wrap behavior: false "no new error" on check_and_advance — acceptable
/// because (a) errno field still carries the last error code, (b) filesystems
/// with 500K+ errors have already been marked for fsck.
///
/// Single atomic operation for both `set_err()` and `check_and_advance()` —
/// no torn reads between errno and counter.
#[cfg(target_pointer_width = "64")]
pub struct ErrSeq {
    /// Packed: bits [15:0] = errno (unsigned, max 4095),
    /// bit [16] = seen flag, bits [63:17] = counter.
    /// Single AtomicU64 prevents torn reads.
    inner: AtomicU64,
}

#[cfg(target_pointer_width = "32")]
pub struct ErrSeq {
    /// Packed: bits [11:0] = errno, bit [12] = seen flag,
    /// bits [31:13] = counter. Matches Linux errseq_t layout.
    inner: AtomicU32,
}

#[cfg(target_pointer_width = "64")]
impl ErrSeq {
    // 64-bit ERRNO_BITS is 16 (not 12) to simplify the bit layout; values > 4095
    // are kernel bugs caught by debug_assert. 47-bit counter provides ample headroom.
    const ERRNO_BITS: u32 = 16;
    const SEEN_BIT: u64 = 1 << Self::ERRNO_BITS;
    const CTR_INC: u64 = 1 << (Self::ERRNO_BITS + 1);
    const ERRNO_MASK: u64 = Self::SEEN_BIT - 1;

    pub const fn new() -> Self { Self { inner: AtomicU64::new(0) } }

    /// Record a writeback error. The generation counter advances ONLY when the
    /// current error has already been observed (SEEN set) or the field is empty;
    /// an as-yet-unseen error is overwritten in place, its generation unchanged
    /// (the advance is OBSERVATION-GATED — the ErrSeq discipline confirmed
    /// against lib/errseq.c). Two consequences (FLOW-16 J11/J16):
    /// - A single errored K-page bio whose completion epilogue calls set_err()
    ///   once per page advances the generation AT MOST ONCE — same-generation
    ///   idempotence by construction, so the ILP32 19-bit counter is not burned
    ///   per-page.
    /// - A fresh error AFTER an observation gets a new generation (SEEN was set
    ///   by check_and_advance in the shared word), so it is reported.
    /// The advance clears SEEN, marking the fresh generation unobserved.
    /// No backoff needed: writeback_lock serializes concurrent writeback per
    /// inode; the only contention is a rare set_err/check_and_advance race.
    pub fn set_err(&self, errno: i32) {
        debug_assert!(errno.unsigned_abs() <= 4095, "errno exceeds MAX_ERRNO");
        let errno_val = (errno.unsigned_abs() as u64) & Self::ERRNO_MASK;
        loop {
            let old = self.inner.load(Acquire);
            // Advance the generation only past a SEEN (observed) error, or when
            // the field carries no error yet (first-ever error). Otherwise
            // refresh the errno in place, leaving SEEN clear and the counter
            // unchanged.
            let advance = old & Self::SEEN_BIT != 0 || old & Self::ERRNO_MASK == 0;
            let new_val = if advance {
                (old & !Self::ERRNO_MASK & !Self::SEEN_BIT)
                    .wrapping_add(Self::CTR_INC) | errno_val
            } else {
                (old & !Self::ERRNO_MASK & !Self::SEEN_BIT) | errno_val
            };
            if new_val == old {
                return; // idempotent: this unseen errno is already recorded.
            }
            match self.inner.compare_exchange_weak(old, new_val, AcqRel, Acquire) {
                Ok(_) => break,
                Err(_) => continue,
            }
        }
    }

    /// Snapshot current value with "seen" bit set.
    pub fn sample(&self) -> u64 { self.inner.load(Acquire) | Self::SEEN_BIT }

    /// Check for new errors since the snapshot in `since`. Returns errno
    /// if changed. The returned errno is always POSITIVE (e.g., 5 for EIO,
    /// not -5). `set_err()` accepts both positive and negative errnos (via
    /// `unsigned_abs()`), but the stored and returned value is always the
    /// absolute (positive) errno. Callers that need a negative errno for
    /// syscall returns must negate: `Err(-(errno as i32))`.
    ///
    /// `since` is the fd's ATOMIC snapshot (`OpenFile::f_wb_err`). An
    /// `OpenFile` is shared across threads by `dup(2)`/`fork(2)`, so two
    /// threads can call `fsync()` on the same open file description
    /// concurrently — a plain `&mut u64` snapshot would be a data race
    /// (and is unreachable through the `Arc<OpenFile>` the callers hold).
    ///
    /// Report-once mechanics (FLOW-16 J11): the SEEN bit is recorded in the
    /// SHARED word (`self.inner`), not only in the fd snapshot. The old code
    /// set SEEN in the snapshot alone, so `snap == current` NEVER held again
    /// (the snapshot carried SEEN, `inner` did not) and every subsequent fsync
    /// re-reported the same error forever. Recording SEEN in `inner` makes the
    /// generation converge — the advancing store copies that same value
    /// (SEEN included) into the snapshot, so the next fsync sees
    /// `snap == current` and returns None — and gates `set_err`'s generation
    /// advance, so a genuinely new error still gets a fresh, reportable
    /// generation. The snapshot CAS keeps once-per-fd under a dup'd/forked-fd
    /// race: exactly one racer advances the snapshot and reports; the loser
    /// reloads, finds the snapshot equal to `current`, and returns None.
    pub fn check_and_advance(&self, since: &AtomicU64) -> Option<i32> {
        let mut current = self.inner.load(Acquire);
        loop {
            let snap = since.load(Acquire);
            if current == snap {
                return None; // fd up to date (equality includes the SEEN bit).
            }
            // Record the observation in the SHARED word so (a) any other fd's
            // next check converges once advanced past this generation and (b)
            // set_err advances to a NEW generation on the next error rather than
            // overwriting this one. Best-effort: if the CAS loses to a
            // concurrent set_err (new generation) or another observer, reload
            // and re-evaluate against the latest shared value.
            if current & Self::SEEN_BIT == 0 {
                match self.inner.compare_exchange(
                    current, current | Self::SEEN_BIT, AcqRel, Acquire,
                ) {
                    Ok(_) => current |= Self::SEEN_BIT,
                    Err(actual) => { current = actual; continue; }
                }
            }
            // Advance THIS fd's snapshot to the observed generation. CAS so a
            // dup'd/forked-fd race resolves to exactly one reporter.
            match since.compare_exchange(snap, current, AcqRel, Acquire) {
                Ok(_) => {
                    let errno = (current & Self::ERRNO_MASK) as i32;
                    return if errno == 0 { None } else { Some(errno) };
                }
                Err(_) => { current = self.inner.load(Acquire); continue; }
            }
        }
    }
}

#[cfg(target_pointer_width = "32")]
impl ErrSeq {
    const ERRNO_BITS: u32 = 12; // ilog2(MAX_ERRNO=4095) + 1
    const SEEN_BIT: u32 = 1 << Self::ERRNO_BITS;
    const CTR_INC: u32 = 1 << (Self::ERRNO_BITS + 1);
    const ERRNO_MASK: u32 = Self::SEEN_BIT - 1;

    pub const fn new() -> Self { Self { inner: AtomicU32::new(0) } }

    /// Observation-gated generation advance — same discipline as the 64-bit
    /// variant (FLOW-16 J11/J16): advance the counter only past a SEEN (observed)
    /// error or an empty field; otherwise refresh the errno in place. Per-page
    /// set_err over a K-page errored bio advances the 19-bit ILP32 counter at
    /// most once (same-generation idempotence), preserving its wrap horizon.
    pub fn set_err(&self, errno: i32) {
        debug_assert!(errno.unsigned_abs() <= 4095, "errno exceeds MAX_ERRNO");
        let errno_val = (errno.unsigned_abs()) & Self::ERRNO_MASK;
        loop {
            let old = self.inner.load(Acquire);
            let advance = old & Self::SEEN_BIT != 0 || old & Self::ERRNO_MASK == 0;
            let new_val = if advance {
                (old & !Self::ERRNO_MASK & !Self::SEEN_BIT)
                    .wrapping_add(Self::CTR_INC) | errno_val
            } else {
                (old & !Self::ERRNO_MASK & !Self::SEEN_BIT) | errno_val
            };
            if new_val == old {
                return; // idempotent: this unseen errno is already recorded.
            }
            match self.inner.compare_exchange_weak(old, new_val, AcqRel, Acquire) {
                Ok(_) => break,
                Err(_) => continue,
            }
        }
    }

    pub fn sample(&self) -> u32 { self.inner.load(Acquire) | Self::SEEN_BIT }

    /// Returns positive errno. Records SEEN in the SHARED word so the generation
    /// converges (report-once) and set_err advances on the next error — see the
    /// 64-bit variant doc comment for the full mechanics (FLOW-16 J11).
    pub fn check_and_advance(&self, since: &AtomicU32) -> Option<i32> {
        let mut current = self.inner.load(Acquire);
        loop {
            let snap = since.load(Acquire);
            if current == snap {
                return None; // fd up to date (equality includes the SEEN bit).
            }
            if current & Self::SEEN_BIT == 0 {
                match self.inner.compare_exchange(
                    current, current | Self::SEEN_BIT, AcqRel, Acquire,
                ) {
                    Ok(_) => current |= Self::SEEN_BIT,
                    Err(actual) => { current = actual; continue; }
                }
            }
            match since.compare_exchange(snap, current, AcqRel, Acquire) {
                Ok(_) => {
                    let errno = (current & Self::ERRNO_MASK) as i32;
                    return if errno == 0 { None } else { Some(errno) };
                }
                Err(_) => { current = self.inner.load(Acquire); continue; }
            }
        }
    }
}

/// Width-dispatched type alias for a writeback-error snapshot stored in
/// `OpenFile::f_wb_err`. Must match the `since` parameter type expected by
/// `ErrSeq::check_and_advance` on each architecture:
/// - 64-bit targets (x86-64, AArch64, RISC-V 64, PPC64LE, s390x, LoongArch64):
///   `ErrSeq` stores in `AtomicU64` → snapshot is `AtomicU64`.
/// - 32-bit targets (ARMv7, PPC32): `ErrSeq` stores in `AtomicU32` → snapshot
///   is `AtomicU32`.
///
/// The snapshot is ATOMIC (not a plain integer) because an `OpenFile` is a
/// shared open file description: `dup(2)` and `fork(2)` hand the same
/// `Arc<OpenFile>` to multiple threads/processes, and concurrent `fsync()`
/// calls on those fds both read-modify-write this snapshot. A plain field
/// would be (a) a data race and (b) unwritable through the `&OpenFile`
/// shared reference the fsync path holds. Initialized at `open()` time via
/// `WbErrSnapshot::new(mapping.wb_err.sample())`.
///
/// Using this alias ensures `&file.f_wb_err` matches the `&AtomicU64` /
/// `&AtomicU32` parameter of `ErrSeq::check_and_advance` on all eight
/// supported architectures.
#[cfg(target_pointer_width = "64")]
pub type WbErrSnapshot = AtomicU64;
#[cfg(target_pointer_width = "32")]
pub type WbErrSnapshot = AtomicU32;

fdatasync vs fsync decision matrix:

Condition	fdatasync action	fsync action
Dirty data pages exist	Writeback + wait	Writeback + wait
File size changed (truncate/append)	Metadata flush (size is data-relevant)	Metadata flush
Only timestamps changed	Skip metadata flush	Metadata flush
Permissions/ownership changed	Skip metadata flush	Metadata flush
Journal commit needed	Yes (for data blocks)	Yes (for all)
Device cache flush	Yes	Yes

Cross-references: - AddressSpaceOps::writepage(): Section 14.1 - Page cache dirty tracking: Section 4.2 - Block I/O layer and bio submission: Section 15.2 - Journal write barrier: Section 15.5 - VFS ring buffer protocol (Tier 1/2 fsync dispatch): Section 14.2 - Writeback thread organization: Section 4.6 - Copy-on-Write / Redirect-on-Write infrastructure: below

14.4.1 Copy-on-Write and Redirect-on-Write Infrastructure

Modern filesystems fall into three write models. Linux treats all three identically at the VFS level — each filesystem independently manages its own write path, extent sharing, and snapshot interaction. This means the VFS cannot optimize writeback scheduling, cannot share page cache pages between reflinked files, and cannot accurately predict free space costs for dirty page flushes.

UmkaOS's VFS distinguishes these models explicitly, enabling generic optimizations that benefit all CoW/RoW filesystems without filesystem-specific code in the VFS.

14.4.1.1 Write Mode Declaration

/// Write mode declared by each filesystem via `FileSystemOps::write_mode()`.
/// Cached in `SuperBlock.write_mode` at mount time. Informs the VFS writeback
/// path, page cache sharing strategy, and free space accounting.
#[derive(Clone, Copy, PartialEq, Eq)]
pub enum WriteMode {
    /// Traditional in-place overwrite (ext4 without reflinks, tmpfs, ramfs).
    /// Writeback reuses the same block address. No extent sharing awareness
    /// needed. Free space cost of flushing dirty pages: zero (no new blocks).
    InPlace,

    /// Copy-on-Write for shared extents (XFS with reflinks, ext4 with reflinks).
    /// Non-shared extents are overwritten in place. Shared extents (refcount > 1
    /// due to reflinks, snapshots, or dedup) require new block allocation on
    /// write. The writeback path queries `ExtentSharingOps::is_extent_shared()`
    /// to decide: shared → allocate new block; unshared → overwrite in place.
    CopyOnWrite,

    /// Redirect-on-Write: the filesystem NEVER overwrites data blocks (Btrfs,
    /// ZFS, bcachefs, UPFS). All writes allocate new blocks; old blocks are
    /// freed only when no snapshot, clone, or active reference retains them.
    /// Consistency is achieved by atomic metadata root pointer updates (e.g.,
    /// ZFS uberblock, Btrfs tree root, UPFS checkpoint record) rather than
    /// journaling.
    ///
    /// Writeback always requests a new block address from the filesystem.
    /// Free space accounting must reserve space for pending redirections.
    RedirectOnWrite,
}

Why three modes matter:

Aspect	InPlace	CopyOnWrite	RedirectOnWrite
Writeback block allocation	Reuse existing	Conditional (check sharing)	Always new
Free space cost of dirty flush	Zero	Zero (unshared) or one block (shared)	One new block per dirty block
Sequential writeback batching	Not useful (scattered overwrites)	Not useful (scattered)	Highly useful (batch new allocations into sequential runs)
Page cache sharing for reflinks	N/A (no reflinks)	Yes (shared extents)	Yes (all data may be snapshot-shared)
Journal needed	Typically yes	Depends on FS	No (atomic root update suffices)
Snapshot integration	External (LVM/dm-snapshot)	Per-extent refcount	Native (tree root versioning)

14.4.1.2 Extent Sharing Operations

/// Trait implemented by filesystems that support extent sharing (reflinks,
/// snapshots, clones, dedup). Optional — only `CopyOnWrite` and
/// `RedirectOnWrite` filesystems implement this.
///
/// The VFS queries these methods during:
/// - Writeback: to decide CoW vs in-place for `CopyOnWrite` filesystems.
/// - Page cache lookup: to enable shared-extent page cache.
/// - Free space accounting: to estimate true cost of flushing dirty pages.
///
/// All methods are called from the writeback workqueue context (not the page
/// fault hot path). Implementations may acquire filesystem-internal locks.
pub trait ExtentSharingOps: Send + Sync {
    /// Returns true if the extent covering `[file_offset, file_offset + len)`
    /// in the given inode is shared (refcount > 1 due to reflinks, snapshots,
    /// or dedup). Returns false for holes, unallocated ranges, and unshared
    /// extents.
    ///
    /// For `CopyOnWrite` filesystems: determines whether writeback must
    /// allocate a new block or can overwrite in place.
    /// For `RedirectOnWrite` filesystems: always returns true conceptually
    /// (all blocks are "shared" with the previous tree version), but the
    /// implementation may optimize by returning false for blocks that are
    /// guaranteed unshared (e.g., newly allocated since the last checkpoint).
    fn is_extent_shared(&self, inode: InodeId, file_offset: u64, len: u64) -> bool;

    // NOTE: the physical-extent hook `extent_phys_addr` was PROMOTED off this
    // trait to an Option-returning `AddressSpaceOps` method
    // (`fn extent_phys_addr(&self, mapping: &AddressSpace, index: u64) ->
    // Option<PhysExtent>`, default `None`; [Section 14.1](#virtual-filesystem-layer)), so
    // the page-cache fill primitive `page_cache_get_or_fill()` consults it via
    // `mapping.ops` on the miss path — BEFORE allocating a frame — for BOTH the
    // buffered read path and the mmap fault path (the same consultation, one
    // resident copy per physical extent). The `PHYS_EXTENT_CACHE` lookup+pin
    // that consumes it is `phys_extent_cache_pin()` below. Only
    // `is_extent_shared` and `cow_allocate` (writeback-side decisions) remain
    // on `ExtentSharingOps`.

    /// Allocate a new block for a CoW write. Called by the writeback path when
    /// `is_extent_shared()` returns true (CopyOnWrite mode) or unconditionally
    /// (RedirectOnWrite mode). The filesystem allocates a new physical extent,
    /// updates its internal mapping, and returns the new extent descriptor.
    ///
    /// The old extent's refcount is decremented. If it drops to zero and no
    /// snapshot retains it, the filesystem may free it (deferred to the
    /// filesystem's own garbage collection or checkpoint cycle).
    fn cow_allocate(
        &self,
        inode: InodeId,
        file_offset: u64,
        len: u64,
    ) -> Result<PhysExtent>;
}

/// A physical extent descriptor — identifies a contiguous range on a block device.
pub struct PhysExtent {
    /// Block device that owns this extent.
    pub bdev: DevId,
    /// Physical byte offset on the block device.
    pub phys_offset: u64,
    /// Length of the extent in bytes.
    pub len: u64,
}

14.4.1.3 Shared-Extent Page Cache

Problem (Linux limitation): Linux's page cache is indexed solely by (address_space, file_offset). When two files share a physical extent via reflink, each file gets a separate page cache entry for the same data — doubling memory consumption. This is a known limitation acknowledged by Linux developers, unfixed because the assumption "one page = one mapping pointer" is deeply embedded in the Linux MM.

UmkaOS design: dual-indexed page cache with physical extent awareness.

UmkaOS has no such legacy constraint. The page cache uses a two-level lookup:

1. Primary index (unchanged): per-inode PageCache keyed by file_offset — the standard per-inode page tree used for all I/O operations (Section 4.4).

2. Secondary index (new): PhysExtentCache — a global RcuHashMap keyed by (DevId, phys_offset) that maps to Page references. Populated only for filesystems with WriteMode::CopyOnWrite or WriteMode::RedirectOnWrite.

/// Global cache of pages indexed by physical extent location.
/// Enables page sharing between files that reference the same physical blocks
/// (reflinks, snapshots, clones). RCU-protected for lock-free read-side lookups.
///
/// This cache is consulted by the page-cache fill primitive
/// `page_cache_get_or_fill()` on its miss path ([Section 4.4](04-memory.md#page-cache)): when the
/// mapping's `AddressSpaceOps` supplies `extent_phys_addr` (CoW/RoW
/// filesystems), the fill primitive looks up `(bdev, phys_offset)` here
/// BEFORE allocating a frame — so the buffered read path and the mmap fault
/// path share one consultation and one resident copy per physical extent.
///
/// Lookup path (read): RCU read lock → hash lookup → Page refcount increment.
/// Insert path (miss): filesystem provides PhysExtent via `extent_phys_addr` →
///   read from disk → insert into PhysExtentCache → insert into per-inode PageCache.
/// Eviction: when a Page's refcount (across all address_spaces) drops to zero,
///   the page is removed from PhysExtentCache and per-inode PageCaches.
/// The value is a `&'static Page` — a handle to the PERMANENT MEMMAP frame
/// descriptor (frames outlive the map; the map indexes them, it does not own
/// `Page` by value). The RCU guard bounds only the map-node borrow; the
/// `&'static Page` read out of a node is genuinely `'static` and needs a
/// `page_get_speculative` before its frame DATA may be trusted (see
/// `phys_extent_cache_pin`).
pub static PHYS_EXTENT_CACHE: BootOnceCell<RcuHashMap<PhysExtentKey, &'static Page>> = BootOnceCell::new();

/// Key for the physical extent cache: (block device, physical byte offset).
/// Uses the page-aligned offset (physical offset rounded down to page size).
#[derive(Hash, Eq, PartialEq, Clone, Copy)]
pub struct PhysExtentKey {
    pub bdev: DevId,
    pub phys_offset: u64, // page-aligned
}

Read path (for CoW/RoW filesystems). Steps 2-3 below are exactly the consultation the page-cache fill primitive page_cache_get_or_fill() performs on its miss path (Section 4.4); the mmap fault path reaches the same primitive, so a reflinked file faulted via mmap hits a resident shared extent instead of re-reading it:

read(inode_A, index_X):   // index_X = file_offset_X / PAGE_SIZE; mapping = inode_A.i_mapping
  1. Look up (mapping, index_X) in per-inode PageCache.
     → Hit: return page (standard fast path, no change from InPlace mode).
     → Miss: continue to step 2 (inside page_cache_get_or_fill's miss path).
  2. Call mapping.ops.extent_phys_addr(mapping, index_X)
     → PhysExtent { bdev, phys_offset } (None for non-sharing filesystems → skip to a normal frame alloc + read).
  3. Look up (bdev, phys_offset) in PHYS_EXTENT_CACHE.
     → Hit: page is already cached (another file sharing this extent loaded it).
       Insert a reference into inode_A's PageCache. Return page.
     → Miss: read from disk. Insert into PHYS_EXTENT_CACHE and inode_A's PageCache.

Write path (CoW-on-write for shared pages):

write(inode_A, file_offset_X, data):
  1. Look up page in inode_A's PageCache.
  2. If page is in PHYS_EXTENT_CACHE AND has references from multiple address_spaces:
     → Allocate a new private page for inode_A.
     → Copy old page contents to new page.
     → Apply the write to the new page.
     → Remove inode_A's reference from the old page in PHYS_EXTENT_CACHE.
     → Insert new page into inode_A's PageCache (not into PHYS_EXTENT_CACHE —
       it's now private to inode_A until the filesystem assigns it a new
       physical location during writeback).
     → Mark new page dirty.
  3. If page is NOT shared (single reference):
     → Modify in place, mark dirty (standard path).

This design eliminates the 2x memory penalty for reflinked files. The cost is one additional hash lookup on cache miss (step 2-3), which is cold-path (disk I/O dominates). Hot-path reads (step 1 hit) are unchanged.

Memory savings: For a 10 GB dataset reflinked to 5 containers, Linux uses 50 GB of page cache (5 copies). UmkaOS uses 10 GB (one copy, 5 references). This is particularly significant for container-dense workloads where the same base image is reflinked across hundreds of containers.

14.4.1.4 Shared-Extent Fill Consultation (phys_extent_cache_pin)

phys_extent_cache_pin() is the shared-extent side of the page-cache admission primitive: page_cache_get_or_fill() calls it on its miss path, BEFORE allocating a frame (Section 4.4), so both the buffered read path and the mmap fault path share ONE consultation and ONE resident copy per physical extent. Its signature is declared page-cache-side ([Section 4.4](04-memory.md#page-cache)); the body lives here because the AddressSpaceOps::extent_phys_addr hook, PHYS_EXTENT_CACHE, and the dual-index sharing/eviction rules are defined in this section.

It returns a PagePin by PUBLISHING a shared per-inode PageEntry for (mapping, index) that references the already-resident shared frame, then pinning it through the canonical PageEntry::try_pin path — there is no private PagePin construction. Non-CoW mappings (no hook) return None at one branch; a shared frame that is not resident returns None (the caller then allocates and reads).

/// Consult the shared-extent cache for the page-cache miss path. Returns
/// `Some(pin)` when `mapping` exposes `extent_phys_addr` (reflink/CoW/RoW) AND
/// the resolved extent is already resident in `PHYS_EXTENT_CACHE` under another
/// inode: it publishes a shared `PageEntry` for `(mapping, index)` referencing
/// that frame and returns a pin on it. `None` on a non-sharing mapping or a
/// shared-cache miss.
pub fn phys_extent_cache_pin(mapping: &AddressSpace, index: PageIndex) -> Option<PagePin> {
    // Non-sharing filesystems inherit the AddressSpaceOps default (None).
    let ext = mapping.ops.extent_phys_addr(mapping, index)?;
    let key = PhysExtentKey {
        bdev: ext.bdev,
        phys_offset: ext.phys_offset & !(PAGE_SIZE as u64 - 1), // page-aligned
    };
    let cache = PHYS_EXTENT_CACHE.get()?;                 // initialized at boot
    let pc = mapping.page_cache.as_ref()?;                // regular-file mapping

    loop {
        {
            let rcu = rcu_read_lock();
            // A racer may already have published our per-inode entry — pin it.
            if let Some(entry) = pc.pages.load(index) {
                if let Some(pin) = entry.try_pin(&rcu, pc, index) {
                    return Some(pin);
                }
            }
            // Is the physical extent resident under some OTHER inode? The map
            // stores a `&'static Page` (permanent frame handle); copy it OUT of
            // the RCU-bounded node borrow — the pointer value is `'static`, so
            // this is a read of a stored handle, not a coercion of the borrow.
            let shared: &'static Page = *cache.get(&key)?;  // miss → None: caller reads
            // Speculatively pin the shared frame so it cannot be freed while we
            // publish a per-inode reference to it — increment-if-nonzero + a
            // same-key revalidation, the `PageEntry::try_pin` contract
            // ([Section 4.4](04-memory.md#page-cache), [Section 4.2](04-memory.md#physical-memory-allocator)).
            if !page_get_speculative(shared) {
                continue;                                  // frame hit 0 — retry lookup
            }
            if !cache.get(&key).is_some_and(|cur| core::ptr::eq(*cur, shared)) {
                page_put_rcu(shared);                      // key re-pointed — retry
                continue;
            }
            // Publish the shared PageEntry into THIS inode's page cache. The
            // speculative reference above is the reference the new per-inode
            // entry owns: the frame is now dual-indexed (PHYS_EXTENT_CACHE +
            // this PageCache), and eviction removes both indexes only when the
            // last address_space reference drops (the eviction rule above).
            match pc.pages.try_store(index, PageEntry::new(shared)) {
                Ok(()) => { /* published — pin it below, outside RCU */ }
                Err(_existing) => {
                    page_put_rcu(shared);                  // racer won — retry (→ pin)
                    continue;
                }
            }
        } // rcu dropped

        // Pin the freshly-published shared entry through the canonical path.
        let rcu = rcu_read_lock();
        if let Some(entry) = pc.pages.load(index) {
            if let Some(pin) = entry.try_pin(&rcu, pc, index) {
                return Some(pin);
            }
        }
        // Evicted between publish and pin (we held a reference, so this is
        // vanishingly rare): retry the whole consultation.
    }
}

14.4.1.5 Reflink Operations and Ioctl Dispatch

The remap_file_range() method in FileOps (Section 14.1) is the filesystem-level backend. The VFS generic layer handles validation and dispatches via ioctls:

/// Flags for remap_file_range().
pub struct RemapFlags(u32);
impl RemapFlags {
    /// Only remap if source and destination byte ranges are identical.
    /// Used by FIDEDUPERANGE ioctl for deduplication.
    pub const REMAP_FILE_DEDUP: Self = Self(1 << 0);

    /// Caller accepts a shorter remap than requested. The filesystem may
    /// return fewer bytes than `len` if the source extent ends early.
    /// Used by FICLONE/FICLONERANGE (always set) and copy_file_range.
    pub const REMAP_FILE_CAN_SHORTEN: Self = Self(1 << 1);
}

/// Clone range descriptor for FICLONERANGE ioctl.
/// Matches Linux's `struct file_clone_range` layout for binary compatibility.
#[repr(C)]
pub struct FileCloneRange {
    /// Source file descriptor.
    pub src_fd: i64,
    /// Source file offset.
    pub src_offset: u64,
    /// Length to clone (0 = clone to EOF).
    pub src_length: u64,
    /// Destination file offset.
    pub dest_offset: u64,
}
// Layout: 8 + 8 + 8 + 8 = 32 bytes.
const_assert!(size_of::<FileCloneRange>() == 32);

Ioctl definitions (Linux ABI-compatible):

Ioctl	Number (x86-64)	Argument	Semantics
FICLONE	0x40049409 (_IOW(0x94, 9, i32))	Source fd	Clone entire file into destination fd
FICLONERANGE	0x4020940d (_IOW(0x94, 13, FileCloneRange))	FileCloneRange	Clone specified byte range
FIDEDUPERANGE	0xC0189436 (_IOWR(0x94, 54, FileDeduperangeHdr))	Variable-length	Dedup if content matches

VFS ioctl dispatch (generic, not per-filesystem):

ioctl(dst_fd, FICLONE, src_fd):
  1. Validate: both fds open, dst writable, same superblock, same filesystem type.
  2. Acquire the source and destination `i_rwsem` write guards in ascending inode
     number order to prevent deadlock when concurrent calls swap source and destination.
  3. Call dst.file_ops.remap_file_range(src, 0, dst, 0, src.size,
     REMAP_FILE_CAN_SHORTEN).
  4. Invalidate dst's affected range in PHYS_EXTENT_CACHE (new shared extents will
     be populated lazily on next read).
  5. Return 0 on success, -errno on failure.

14.4.1.6 copy_file_range() VFS Dispatch

copy_file_range(2) (syscall 326 on x86-64) is the general-purpose server-side copy interface. The VFS dispatch prioritizes zero-copy where possible:

copy_file_range(fd_in, off_in, fd_out, off_out, len, flags=0):
  1. If same filesystem AND filesystem implements remap_file_range():
     → Try reflink (remap_file_range with REMAP_FILE_CAN_SHORTEN).
     → If EOPNOTSUPP: fall through to step 2.
  2. If same filesystem AND filesystem implements a dedicated copy_file_range handler:
     → Use filesystem-specific server-side copy (e.g., NFS server-side copy,
       CIFS CopyChunk).
  3. Fallback: splice-based copy through page cache (generic, works cross-filesystem).
     This reads source pages into the page cache, then writes them to the
     destination — no userspace round-trip, but does consume page cache memory.

Note: Unlike Linux (which uses syscall 326 on x86-64), UmkaOS uses the same syscall number for ABI compatibility. The flags parameter is reserved (must be 0).

14.4.1.7 CoW/RoW-Aware Writeback

The writeback thread (Section 4.6) uses SuperBlock.write_mode to adapt its behavior:

Write mode	Writeback behavior
InPlace	Standard: for each dirty page, issue write to the page's existing block address. The block address is already known (stored in the iomap).
CopyOnWrite	Before writing each dirty page, call is_extent_shared(). If shared: call cow_allocate() to get a new block address, write to the new address, update the filesystem's extent mapping. If unshared: write in place (same as InPlace).
RedirectOnWrite	For ALL dirty pages, call cow_allocate() to get new block addresses. The filesystem's allocator can batch these requests to produce sequential physical layouts, reducing seek overhead on rotational media and improving flash write amplification. The old block addresses are not reused until the filesystem's checkpoint/commit cycle confirms the new tree root.

RoW writeback batching optimization: For RedirectOnWrite filesystems, the writeback thread collects all dirty pages for a given inode before requesting block allocations. This allows the filesystem allocator to assign a contiguous physical range (one large extent) rather than many scattered single-block allocations. The batch size is bounded by BDI_MAX_WRITEBACK_BATCH (default: 1024 pages = 4 MB). This produces sequential I/O patterns even when dirty pages were written at random file offsets — a significant advantage for RoW filesystems on both rotational and flash storage.

14.4.1.8 Free Space Accounting for CoW/RoW Filesystems

Traditional statfs() reports free blocks = total - used. For CoW/RoW filesystems, this is misleading because:

• Pending CoW: Dirty shared pages will consume new blocks on writeback. The "true" free space is lower than statfs() reports.
• Snapshot overhead: Deleting a file doesn't free its blocks if snapshots reference them.
• RoW garbage: Old blocks from previous tree versions occupy space until garbage collection reclaims them.

UmkaOS adds an extended space accounting interface:

/// Extended filesystem space information, reported by CoW/RoW-aware
/// filesystems in addition to the standard StatFs. Optional — InPlace
/// filesystems return None.
pub struct ExtendedSpaceInfo {
    /// Bytes reserved for pending CoW allocations (dirty shared pages that
    /// will need new blocks on writeback).
    pub cow_reserved_bytes: u64,
    /// Bytes reclaimable by snapshot deletion (blocks held only by snapshots,
    /// not by live files).
    pub snapshot_reclaimable_bytes: u64,
    /// Bytes occupied by stale RoW tree versions pending garbage collection.
    pub gc_pending_bytes: u64,
    /// Effective free bytes = statfs.free - cow_reserved - gc_pending.
    /// This is the "true" free space available for new writes.
    pub effective_free_bytes: u64,
}

This information is exposed via the statfs extended attributes (STATX_ATTR_* flags) and through the UmkaOS-specific /ukfs/kernel/fs/<mount>/space_info umkafs interface.

Cross-references: - WriteMode declaration: FileSystemOps::write_mode() (Section 14.1) - remap_file_range(): FileOps trait (Section 14.1) - Writeback thread organization: Section 4.6 - Page cache and AddressSpace: Section 4.4 - Block I/O submission: Section 15.2 - Btrfs (RedirectOnWrite): Section 15.8 - XFS (CopyOnWrite with reflinks): Section 15.7 - ZFS (RedirectOnWrite): Section 15.10 - FICLONE/FICLONERANGE Linux compat: Section 19.1 - Dirty extent pre-registration: Section 14.1 (VFS crash recovery)

14.5 Character and Block Device Node Framework

All device classes that expose character or block device files under /dev register through a unified device node framework. This framework manages major/minor number allocation, the global device registry, and automatic /dev node lifecycle via devtmpfs.

14.5.1 Character Device Region Registration

/// Character device region registration. All device classes (TTY, evdev, ALSA,
/// DRM, watchdog, SPI, RTC, etc.) register through this unified interface.
/// A region reserves a contiguous range of minor numbers under a single major.
pub struct ChrdevRegion {
    /// Major device number. Either a well-known major from the allocation
    /// table below, or dynamically allocated via `alloc_chrdev_region()`.
    /// Valid range: 1-4095 (0 is reserved). Linux ABI uses 12-bit major (MKDEV
    /// encoding: bits 31:20), so the hard limit is 4095. Dynamic allocation
    /// uses the range 234-254, then 384-511.
    pub major: u16,

    /// First minor number in this region.
    pub minor_base: u32,

    /// Number of minor numbers reserved (contiguous from `minor_base`).
    /// Must be >= 1. The range `minor_base..minor_base+minor_count` must
    /// not overlap with any other registered region under the same major.
    ///
    /// **Overflow check**: `register_chrdev_region()` validates:
    /// ```
    /// minor_base.checked_add(minor_count).ok_or(EINVAL)?;
    /// assert!(minor_base + minor_count <= MINORMASK + 1);
    /// ```
    /// Without this check, a `minor_base + minor_count` overflow wraps
    /// to a valid range, causing silent overlap with unrelated device
    /// regions under the same major. `MINORMASK` is `0xFFFFF` (20 bits,
    /// matching Linux's `MINORBITS = 20`).
    pub minor_count: u32,

    /// File operations for all devices in this region. Called by the VFS
    /// when userspace opens, reads, writes, ioctls, or closes a device node
    /// with a matching major:minor pair.
    pub fops: &'static dyn FileOps,

    /// Human-readable name for this region (e.g., "ttyS", "input/event",
    /// "snd/pcmC"). Used in `/proc/devices` output and diagnostic logging.
    /// Max 31 bytes (null-terminated).
    pub name: &'static str,
}

/// Global character device registry. Indexed by a composite key of
/// `(major << 20 | minor_base)` for O(1) lookup during `open()`.
///
/// `XArray` provides:
/// - O(1) lookup by composite key on the `open()` hot path.
/// - RCU-protected reads: `open()` does not acquire any lock — readers
///   call `rcu_read_lock()` + `xa_load()` for lockless lookup.
/// - Ordered iteration for `/proc/devices` enumeration.
///
/// Writers (register/unregister) acquire `CHRDEV_WRITE_LOCK` to serialize
/// mutations, then modify the XArray under its internal lock.
/// Registrations happen at subsystem init and driver probe (warm path).
static CHRDEV_TABLE: XArray<Arc<ChrdevRegion>> = XArray::new();

/// Writer-side serialization for `CHRDEV_TABLE`. Readers never touch this lock.
/// Held only during `register_chrdev_region` / `unregister_chrdev_region`.
static CHRDEV_WRITE_LOCK: SpinLock<()> = SpinLock::new(());

/// Register a character device region. Called by subsystems during init
/// (e.g., TTY layer registers major 4 for serial, input layer registers
/// major 13 for evdev).
///
/// Returns `Ok(())` on success. Returns `Err(DeviceError::RegionConflict)`
/// if the requested major:minor range overlaps with an existing registration.
/// Returns `Err(DeviceError::MajorExhausted)` if dynamic allocation is
/// requested (`major == 0`) and no free major numbers remain.
pub fn register_chrdev_region(region: ChrdevRegion) -> Result<(), DeviceError>;

/// Dynamically allocate a major number and register a region. Used by
/// device classes that do not have a well-known major (UIO, RTC, etc.).
/// The kernel selects the lowest available major in the dynamic range
/// (234-254, then 384-511).
///
/// Returns the allocated major number on success.
pub fn alloc_chrdev_region(
    name: &'static str,
    minor_base: u32,
    minor_count: u32,
    fops: &'static dyn FileOps,
) -> Result<u16, DeviceError>;

/// Unregister a character device region. Called during driver unload or
/// crash recovery. After this call, `open()` on device nodes with matching
/// major:minor returns `ENODEV`.
///
/// Does NOT remove `/dev` nodes — that is handled by `devtmpfs_remove_node()`.
/// The two operations are decoupled because a crash recovery sequence may
/// unregister the old region before registering the replacement.
pub fn unregister_chrdev_region(major: u16, minor_base: u32);

/// Resolve the character-device region `FileOps` whose
/// `[minor_base, minor_base+minor_count)` range contains `dev`'s minor, under
/// RCU read lock. `None` if no region owns it (`open()` → `-ENODEV`). O(1) via
/// `CHRDEV_TABLE` keyed by `(major << 20 | …)`; returns the region's `fops`,
/// which the driver's `open()` runs through.
pub fn chrdev_lookup(dev: DevId) -> Option<&'static dyn FileOps>;

/// Block-device analogue of `chrdev_lookup`, over `BLKDEV_TABLE`.
pub fn blkdev_lookup(dev: DevId) -> Option<&'static dyn FileOps>;

open() dispatch: When userspace calls open("/dev/foo", ...):

1. VFS resolves the path through the dentry cache to a device inode.
2. The inode's i_rdev field contains the DevId (major:minor).
3. Cgroup device access check: The VFS calls cgroup_bpf_run(BPF_CGROUP_DEVICE, &ctx) where ctx is a BpfCgroupDevCtx { access_type, major, minor } whose access_type PACKS the requested access flags and the device type as (BPF_DEVCG_ACC_* << 16) | BPF_DEVCG_DEV_* (Linux bpf_cgroup_dev_ctx layout — device type is NOT a separate field): the access bits (BPF_DEVCG_ACC_READ, BPF_DEVCG_ACC_WRITE, or both depending on O_RDONLY/O_WRONLY/O_RDWR) in the high half and the device type (BPF_DEVCG_DEV_CHAR/BPF_DEVCG_DEV_BLOCK) in the low half, with major/minor from the inode's i_rdev. The BPF program is evaluated bottom-up from the task's cgroup to the root — access is allowed only if every ancestor's program (if any) returns 1. If any program returns 0, open() returns -EPERM immediately. If no BPF program is attached to any ancestor, access is allowed by default. See Section 17.2 for the full enforcement model and the v1 devices.allow/devices.deny translation.
4. The VFS extracts the actual major and minor from the inode's i_rdev (DevId). It looks up the ChrdevRegion in CHRDEV_TABLE (for character devices) or BlkdevRegion in BLKDEV_TABLE (for block devices) under RCU read lock. The lookup iterates entries for the given major to find the region whose range [minor_base, minor_base + minor_count) contains the inode's minor number. The XArray is keyed by (major << 20 | minor_base), so for a major with multiple regions, the lookup walks entries at keys (major << 20 | 0) through (major << 20 | inode_minor) to find the containing range (XArray ordered iteration, typically 1-2 entries per major).
5. If found, the region's fops is attached to the new OpenFile.
6. fops.open() is called with the inode's actual minor number (not the region's minor_base), allowing the driver to compute the device instance index as minor - region.minor_base.

The cgroup check (step 3) applies identically to character- and block-device opens; the BPF context distinguishes the two via the device-type bits packed into the low half of access_type (BPF_DEVCG_DEV_CHAR=2 vs BPF_DEVCG_DEV_BLOCK=1). The mknod() syscall also calls the same hook with the BPF_DEVCG_ACC_MKNOD access bit set (access_type = (BPF_DEVCG_ACC_MKNOD << 16) | dev_type) before creating a device node in the filesystem.

Executable dispatch (device_node_open). Steps 3-6 above are NOT optional prose the VFS may skip: the generic open path (open_and_install, Section 14.1) MUST route every S_IFCHR/S_IFBLK inode through this function instead of calling inode.i_fop.open() directly. Doing so is what enforces the device cgroup and binds the registered driver's region FileOps — a generic i_fop.open() bypasses both (the container device-cgroup isolation and the driver dispatch), which is a correctness defect, not an optimization.

/// Resolved target of a device-node open: the region's `FileOps` (which becomes
/// the `OpenFile.f_ops`) and the driver's per-open private token.
pub struct DeviceOpen {
    /// Region `FileOps` for this major:minor — the correct driver surface,
    /// distinct from the filesystem's generic inode `i_fop`.
    pub fops: &'static dyn FileOps,
    /// Driver `open()` token — the `OpenOutcome::private` the region's
    /// `FileOps::open` returned ([Section 14.1](#virtual-filesystem-layer)), stashed in
    /// `OpenFile.private_data`. The rest of the `OpenOutcome` is not carried:
    /// a device-node region open never rebinds data I/O to another inode, so
    /// its `data_inode` is always `None` and the VFS device arm supplies that
    /// `None` itself.
    pub private: u64,
}

/// Open a character/block device node. Realizes steps 3-6 of the dispatch:
/// (3) the cgroup `BPF_CGROUP_DEVICE` access check, (4) the `CHRDEV_TABLE` /
/// `BLKDEV_TABLE` region lookup by `i_rdev`, (5)-(6) attaching the region
/// `FileOps` and invoking its `open()`. Returns `-EPERM` if the cgroup denies,
/// `-ENODEV` if no region contains the minor. Called by the VFS open path for
/// device inodes ONLY.
pub fn device_node_open(inode: &Arc<Inode>, flags: u32) -> Result<DeviceOpen, Errno> {
    let rdev = inode.i_rdev;                        // DevId (major:minor)
    let is_block = (inode.i_mode.load(Ordering::Relaxed) & 0o170000) == 0o060000; // S_IFBLK
    // (3) cgroup device access check — the executable form of the
    // BPF_CGROUP_DEVICE hook (`cgroup_device_permitted` builds the
    // `BpfCgroupDevCtx` and runs the bottom-up program evaluation,
    // [Section 17.2](17-containers.md#control-groups--device-access-control-bpf-based)). `read`/`write` are
    // derived from O_ACCMODE (Linux ACC_MODE: O_RDONLY→read, O_WRONLY→write only,
    // O_RDWR→both). READ is NOT assumed — a write-only DeviceAllow policy must
    // permit an O_WRONLY open, so an O_WRONLY open presents ACC_WRITE alone.
    let accmode = flags & 0o3;                       // O_ACCMODE
    let read    = accmode == 0 || accmode == 2;      // O_RDONLY | O_RDWR
    let write   = accmode == 1 || accmode == 2;      // O_WRONLY | O_RDWR
    if !cgroup_device_permitted(is_block, rdev.major() as u32, rdev.minor(), read, write) {
        return Err(Errno::EPERM);
    }
    // (4) region lookup by i_rdev under RCU → the region's FileOps.
    let fops = {
        let _rcu = rcu_read_lock();
        if is_block {
            blkdev_lookup(rdev).ok_or(Errno::ENODEV)?
        } else {
            chrdev_lookup(rdev).ok_or(Errno::ENODEV)?
        }
    };
    // (5)-(6) call the region's open. The driver derives its device-instance
    // index from the inode's `i_rdev` minor as `minor - region.minor_base`.
    let outcome = fops.open(InodeId(inode.i_ino),
                            OpenFlags::from_bits_truncate(flags))?;
    Ok(DeviceOpen { fops, private: outcome.private })
}

Step 6's "actual minor" is delivered through the inode: a device inode carries its i_rdev, so the driver's FileOps::open reads the real minor from the inode (resolved via InodeId) and computes its instance index as minor - region.minor_base — the FileOps::open(InodeId, …) signature is unchanged (no global ripple), and the minor is authoritative from i_rdev, not inferred.

14.5.2 Block Device Registration

Block devices use an analogous registration path and a separate BLKDEV_TABLE: XArray<Arc<BlkdevRegion>>. The block layer (Section 15.2) adds additional registration state (request queue, disk geometry, partition table) that character devices do not need.

14.5.3 Major Number Allocation Table

Well-known major numbers are assigned to match Linux for userspace compatibility. Tools like ls -l, stat, udev rules, and container runtimes rely on these values being identical to Linux.

Major	Device Class	Minor Range	Notes
1	mem (null, zero, random, urandom, full)	0-15	/dev/null=1,3; /dev/zero=1,5; /dev/full=1,7; /dev/random=1,8; /dev/urandom=1,9
4	ttyS (serial terminals)	64-255	/dev/ttyS0=4,64; legacy range for 16550-compatible UARTs
5	tty, console, ptmx	0-2	/dev/tty=5,0; /dev/console=5,1; /dev/ptmx=5,2
10	misc (miscellaneous character devices)	varies	/dev/fuse=10,229; /dev/rfkill=10,242; /dev/watchdog=10,130; /dev/loop-control=10,237
13	input (evdev, joydev, mousedev)	0-1023	/dev/input/event0=13,64; mousedev 13,32-63; joydev 13,0-31; evdev 13,64-95; extended evdev 13,256+ (Linux 2.6+)
29	fb (framebuffer)	0-31	/dev/fb0=29,0; legacy interface, DRM preferred
31	mtdblock (MTD block translation)	0-31	/dev/mtdblock0=31,0
90	mtd (raw MTD character access)	0-31	/dev/mtd0=90,0
116	ALSA (snd)	0-255	/dev/snd/pcmC0D0p=116,16; /dev/snd/controlC0=116,0
136	pts (PTY slave devices)	0-1048575	/dev/pts/0=136,0; devpts filesystem allocates minors dynamically
226	DRM (dri)	0-255	/dev/dri/card0=226,0; /dev/dri/renderD128=226,128
239	IPMI device interface	0-31	/dev/ipmi0=239,0
dynamic	UIO, RTC, hwmon, etc.	allocated at registration	Major assigned by alloc_chrdev_region()

14.5.4 Devtmpfs: Automatic /dev Node Lifecycle

Devtmpfs is a kernel-managed tmpfs instance mounted on /dev that automatically creates and removes device nodes in response to device registration and unregistration events. It eliminates the boot-time race between device discovery and userspace udev — device nodes exist before any userspace process runs.

/// Devtmpfs entry describing a device node to create under /dev.
/// Passed to `devtmpfs_create_node()` by the device registry when a
/// device is registered, and to `devtmpfs_remove_node()` on unregistration
/// or crash recovery.
pub struct DevtmpfsEntry {
    /// Path relative to /dev (e.g., "ttyS0", "input/event3", "snd/pcmC0D0p").
    /// Intermediate directories (e.g., "input/", "snd/") are created
    /// automatically if they do not exist. Max 63 bytes.
    pub path: ArrayString<64>,

    /// Device type: character or block.
    pub dev_type: DevType,

    /// Major:minor device identifier.
    pub dev_id: DevId,

    /// File permissions (e.g., 0o666 for /dev/null, 0o620 for TTY devices,
    /// 0o660 for block devices). The owner is always root:root; udev rules
    /// can adjust ownership after boot.
    pub mode: u16,
}

/// Device type discriminant for device nodes.
#[repr(u8)]
pub enum DevType {
    /// Character device (S_IFCHR).
    Char  = 0,
    /// Block device (S_IFBLK).
    Block = 1,
}

/// Major:minor device identifier. Encoded as a single u32 for storage
/// efficiency (matches Linux's `MKDEV(major, minor)` encoding).
///
/// Linux `dev_t` is u32 with MAJOR = top 12 bits (0–4095) and MINOR =
/// bottom 20 bits (0–1048575). The `new()` constructor validates that
/// `major` fits in 12 bits; callers passing a u16 > 4095 get a panic
/// (debug) or truncation would silently produce wrong device numbers.
pub struct DevId {
    /// Encoded as `(major << 20) | (minor & 0xFFFFF)`.
    /// Major occupies bits 31:20 (12 bits, 0–4095).
    /// Minor occupies bits 19:0  (20 bits, 0–1048575).
    pub raw: u32,
}

impl DevId {
    /// Create a `DevId` from separate major and minor numbers.
    ///
    /// # Panics
    /// Panics if `major > 4095` — Linux ABI reserves only 12 bits for major.
    pub fn new(major: u16, minor: u32) -> Self {
        assert!(major <= 0x0FFF, "DevId: major {} exceeds 12-bit Linux ABI limit (max 4095)", major);
        assert!(minor <= 0xFFFFF, "DevId: minor {} exceeds 20-bit limit (max 1048575)", minor);
        DevId { raw: (major as u32) << 20 | (minor & 0x000F_FFFF) }
    }
    pub fn major(&self) -> u16 { (self.raw >> 20) as u16 }
    pub fn minor(&self) -> u32 { self.raw & 0x000F_FFFF }

    /// Encode for stat()/fstat()/newfstatat() `st_dev` and `st_rdev` fields.
    /// This encoding differs from the kernel-internal `DevId` layout.
    /// Matches Linux `new_encode_dev()` in include/linux/kdev_t.h.
    /// The SysAPI layer calls this when filling `struct stat` responses.
    /// For statx() responses, use `major()` and `minor()` directly (statx
    /// has separate `stx_rdev_major`/`stx_rdev_minor` u32 fields).
    pub fn new_encode_dev(&self) -> u32 {
        let major = self.major() as u32;
        let minor = self.minor();
        (minor & 0xff) | ((major & 0xfff) << 8) | ((minor & !0xffu32) << 12)
    }

    /// Decode a stat()/fstat() encoded device number back to DevId.
    /// Inverse of `new_encode_dev()`. Matches Linux `new_decode_dev()`.
    pub fn new_decode_dev(encoded: u32) -> DevId {
        let major = ((encoded & 0xfff00) >> 8) as u16;
        let minor = (encoded & 0xff) | ((encoded >> 12) & 0xfff00);
        DevId::new(major, minor)
    }
}
// Round-trip verification: encode and decode must be inverses.
// Test with boundary values: major 0..4095 (12 bits), minor 0..1048575 (20 bits).
const_assert!({
    let d = DevId::new(0, 0);
    let enc = d.new_encode_dev();
    let dec = DevId::new_decode_dev(enc);
    dec.major() == 0 && dec.minor() == 0
});
const_assert!({
    let d = DevId::new(4095, 1048575);
    let enc = d.new_encode_dev();
    let dec = DevId::new_decode_dev(enc);
    dec.major() == 4095 && dec.minor() == 1048575
});

Lifecycle hooks:

/// Create a device node under /dev. Called by `DeviceRegistry::register()`
/// after a device and its chrdev/blkdev region are successfully registered.
///
/// Creates the inode in the devtmpfs superblock with the specified
/// major:minor, type, and permissions. If intermediate path components
/// do not exist (e.g., "input/" for "input/event3"), they are created as
/// directories with mode 0o755.
///
/// This function is idempotent: if the node already exists with the same
/// major:minor, it is a no-op. If it exists with a different major:minor,
/// the old node is replaced (stale node from a previous driver instance).
pub fn devtmpfs_create_node(entry: &DevtmpfsEntry) -> Result<(), IoError>;

/// Remove a device node from /dev. Called by `DeviceRegistry::unregister()`
/// and by the crash recovery manager when a driver's device is being
/// cleaned up.
///
/// Removes the inode from devtmpfs. Empty parent directories are NOT
/// removed (they may be needed by other devices in the same class).
///
/// Idempotent: removing a non-existent node is a no-op (returns `Ok(())`).
pub fn devtmpfs_remove_node(path: &str) -> Result<(), IoError>;

/// Create a symbolic link inside devtmpfs. `link` and `target` are relative to
/// the devtmpfs root (e.g. `devfs_symlink("watchdog", "watchdog0")` creates
/// `/dev/watchdog -> watchdog0`). Used by device classes that expose a stable
/// alias for the first (index-0) instance of a class — the watchdog core
/// ([Section 13.19](13-device-classes.md#hardware-watchdog-framework)) is one such caller.
///
/// Creates the symlink inode in the devtmpfs superblock. Returns
/// `IoError::EEXIST` if `link` already exists (the caller is responsible for
/// removing a stale alias first). Userspace `udev` may later add its own
/// persistent symlinks; this in-kernel link is the boot-time default.
pub fn devfs_symlink(link: &str, target: &str) -> Result<(), IoError>;

Boot sequence:

Devtmpfs is mounted during boot Phase 5 (after the physical memory allocator, slab allocator, and VFS are initialized, but before the root filesystem is mounted):

Boot Phase 5: devtmpfs initialization
  1. Create an in-kernel tmpfs instance for devtmpfs.
  2. Mount it internally (not yet visible to userspace).
  3. Create standard device nodes:
     - /dev/null     (1, 3)   mode 0o666   — discard sink
     - /dev/zero     (1, 5)   mode 0o666   — zero source
     - /dev/full     (1, 7)   mode 0o666   — always-full sink
     - /dev/random   (1, 8)   mode 0o666   — blocking entropy source
     - /dev/urandom  (1, 9)   mode 0o666   — non-blocking entropy source
     - /dev/console  (5, 1)   mode 0o600   — kernel console
     - /dev/tty      (5, 0)   mode 0o666   — controlling terminal alias
     - /dev/ptmx     (5, 2)   mode 0o666   — PTY master multiplexer
  4. Device discovery (PCI enumeration, platform devices, DT/ACPI) probes
     drivers, which register devices → devtmpfs_create_node() populates
     /dev with hardware-specific nodes.
  5. After rootfs mount: bind-mount devtmpfs onto /dev in the real root.
     Userspace udev starts and may adjust permissions, create symlinks
     (e.g., /dev/disk/by-uuid/...), and apply udev rules.

Crash recovery interaction: When a Tier 1 driver crashes and its device is being recovered (Section 11.9), the crash recovery manager calls devtmpfs_remove_node() for all device nodes owned by the crashed driver. After the replacement driver loads and re-registers its devices, devtmpfs_create_node() recreates the nodes. Userspace processes that had open file descriptors to the old nodes receive EIO on subsequent I/O; they must reopen the device to get a file descriptor backed by the new driver instance.

14.5.4.1 Crash Recovery and Hotplug Event Interaction

When a driver crash overlaps with hotplug events (e.g., a USB hub driver crashes while devices are being enumerated), the following ordering guarantees apply:

1. Event queue freeze: The crash recovery manager acquires the hotplug workqueue's drain lock before beginning recovery. New HotplugEvent::DeviceArrival events for the crashed driver's bus subtree are enqueued but not processed until recovery completes. Events for unrelated bus subtrees continue processing normally.

2. Device node cleanup: devtmpfs_remove_node() is called for each device owned by the crashed driver. The removal is atomic per-node: either the inode is fully removed or the operation has no effect (idempotent).

3. Pending event replay: After the replacement driver loads and its init() returns ProbeResult::Ok, the hotplug workqueue drain lock is released. Queued arrival events for the recovered subtree are replayed in FIFO order. The new driver instance receives DeviceArrival events for any devices that appeared during the recovery window.

4. Stale removal events: DeviceRemoval events for devices that were already cleaned up during crash recovery are silently dropped (the device handle no longer exists in the registry). This is safe because devtmpfs_remove_node() is idempotent.

5. Uevent replay to userspace: After recovery, the netlink translation layer (Section 19.5) emits a synthetic change uevent for each recovered device. This notifies udev/systemd-udevd to re-apply rules (permissions, symlinks) without requiring a full udevadm trigger.

/// Crash recovery hotplug coordination.
///
/// Acquires the drain lock on the hotplug workqueue for the specified bus
/// subtree, preventing event processing until `release_hotplug_drain()`.
/// Events continue to be enqueued — they are replayed on release.
pub fn acquire_hotplug_drain(subtree_root: DeviceHandle) -> HotplugDrainGuard;

/// RAII guard that releases the hotplug drain lock on drop.
/// Queued events for the frozen subtree are replayed in FIFO order.
pub struct HotplugDrainGuard {
    subtree_root: DeviceHandle,
}

impl Drop for HotplugDrainGuard {
    fn drop(&mut self) {
        // Release drain lock. The hotplug workqueue processes all queued
        // events for subtree_root's descendants in FIFO order.
    }
}

14.5.5 Initial Device Naming

The kernel assigns initial device names following Linux conventions for userspace compatibility. Userspace udev may later create persistent symlinks (/dev/disk/by-uuid/, /dev/disk/by-id/, etc.) but the kernel-assigned names must match what Linux tools expect.

Naming rules by device class:

Class	Pattern	Algorithm	Examples
Block (SCSI/NVMe)	sd[a-z]+ / nvme[N]n[M]	SCSI: alphabetic sequence by probe order. NVMe: controller N, namespace M.	sda, sdb, nvme0n1
Block partitions	<disk>N	Partition number from GPT/MBR table.	sda1, nvme0n1p1
Network	eth[N] / wlan[N]	Sequential index per subsystem (Ethernet vs WiFi). udev's predictable naming (ens3, enp0s25) is applied by userspace rules, not the kernel.	eth0, wlan0
TTY serial	ttyS[N]	Port index from UART enumeration (PCI BAR order, DT aliases, ACPI UID).	ttyS0, ttyS1
TTY USB serial	ttyUSB[N]	Sequential index by USB probe order.	ttyUSB0
Input (evdev)	input/event[N]	Sequential index by registration order.	input/event0
ALSA	snd/pcmC[N]D[M]p	Card N (probe order), device M (codec order), p=playback / c=capture.	snd/pcmC0D0p
DRM	dri/card[N] / dri/renderD[128+N]	Sequential by GPU probe order. Render nodes start at minor 128.	dri/card0, dri/renderD128
Framebuffer	fb[N]	Legacy. Sequential by registration.	fb0
Watchdog	watchdog[N]	Sequential by registration.	watchdog0
Loop	loop[N]	Fixed pool of max_loop devices (default 256). Created at boot.	loop0

Implementation: Each device class maintains its own index counter (typically a static AtomicU32). The counter is incremented atomically when a character device or block disk is registered. The generated name is passed to devtmpfs_create_node() and stored in the DeviceNode.dev_name field.

Stability caveat: Kernel-assigned names like sda/sdb depend on probe order, which can vary across boots. This is a known Linux behavior. Persistent naming (/dev/disk/by-uuid/, /dev/disk/by-path/, /dev/disk/by-id/) is handled entirely by userspace udev rules that read device attributes from the uevent/sysfs interface and create stable symlinks. The kernel provides all necessary attributes (serial number, WWN, partition UUID) via the uevent mechanism (Section 19.5).

14.5.6 File Operations Replacement (replace_fops)

Some device classes use a single /dev entry as a multiplexer that switches to a specialized FileOps vtable after open(). Examples:

• ALSA: /dev/snd/controlC0 opens with a generic ALSA control FileOps. PCM device files (/dev/snd/pcmC0D0p) open with PCM-specific FileOps from the start and do NOT use replace_fops. The ALSA replace_fops use case is the control device switching to a specialized monitoring mode via SNDRV_CTL_IOCTL_SUBSCRIBE_EVENTS.
• TTY: A TTY file descriptor switches its line discipline (e.g., from N_TTY to N_SLIP) via TIOCSETD, which replaces the FileOps to reflect the new discipline's read/write/ioctl behavior.
• evdev: EVIOCGRAB transitions an input device to exclusive-grab mode with a specialized FileOps that filters events to the grabbing client.

The OpenFile.f_ops field is declared as &'static dyn FileOps and is normally immutable after creation. replace_fops provides a controlled mechanism to swap it:

/// Atomically replace the FileOps vtable on an open file descriptor.
///
/// This is the mechanism for device classes that multiplex multiple
/// operational modes through a single device node. The caller (device
/// subsystem code, NOT the driver directly) must hold the file's position
/// lock (`fdget_pos()` guard) to prevent concurrent read/write operations
/// from observing a partially-switched state.
///
/// # Safety
///
/// * `new_ops` must be `&'static` — it must outlive the `OpenFile`. In practice
///   this means the new FileOps must be a static vtable defined in the subsystem
///   module (e.g., `static PCM_PLAYBACK_OPS: FileOps = ...`), not a dynamically
///   constructed object.
/// * The caller must ensure no I/O operations are in-flight on the file at the
///   time of the swap. The `fdget_pos()` guard serializes with `read()`/`write()`;
///   `ioctl()` is serialized by the subsystem's own locking (e.g., ALSA's
///   `pcm_stream_lock`, and the TTY subsystem's serialization lock).
/// * The old `FileOps` is not freed (it is `&'static`). No cleanup callback is
///   needed.
///
/// # Implementation
///
/// Uses `AtomicPtr::store(new_ops, Release)` on the internal representation of
/// `f_ops`. Subsequent `read()`/`write()`/`ioctl()` calls load with `Acquire`
/// ordering and dispatch through the new vtable. The `Release`/`Acquire` pair
/// ensures all state mutations made by the caller before calling `replace_fops()`
/// (e.g., initializing PCM hardware parameters, setting up the line discipline
/// buffer) are visible to the next I/O operation through the new vtable.
pub fn replace_fops(
    file: &OpenFile,
    new_ops: &'static dyn FileOps,
    _guard: &FdPosGuard,
) {
    // `_guard` enforces at the type level that the caller holds the
    // fdget_pos() guard, serializing concurrent replace_fops() calls
    // on the same OpenFile. Without this parameter, the UnsafeCell
    // write to f_ops_vtable relies on caller discipline alone.
    // Decompose the fat pointer into data + vtable, store data atomically.
    let (data_ptr, vtable_ptr) = (new_ops as *const dyn FileOps).to_raw_parts();
    // Write vtable FIRST (plain store, ordered by the subsequent Release).
    // SAFETY: all FileOps impl types have 'static vtables. The plain store
    // is safe because the subsequent Release on f_ops_data orders this write
    // relative to any reader's Acquire load.
    unsafe { *file.f_ops_vtable.get() = vtable_ptr; }
    // THEN publish the data pointer with Release. A reader's Acquire load
    // on f_ops_data guarantees visibility of the vtable write above.
    file.f_ops_data.store(data_ptr as *mut (), Release);
}

OpenFile.f_ops representation: To support replace_fops, the internal representation uses two fields rather than a single &'static dyn FileOps: - f_ops_data: AtomicPtr<()> — the data pointer component of the fat pointer, swapped atomically with Release ordering on write, Acquire on read. - f_ops_vtable: UnsafeCell<*const ()> — the vtable pointer component, written BEFORE the f_ops_data Release store. The Release/Acquire pair on f_ops_data guarantees that a reader observing the new data pointer also observes the new vtable pointer. This ordering is correct on all architectures, including weakly ordered ones (AArch64, RISC-V), because Release orders ALL prior writes.

AtomicPtr<dyn FileOps> is NOT valid Rust (dyn FileOps is !Sized, and AtomicPtr<T> requires T: Sized). The two-field decomposition avoids this limitation. The public f_ops() accessor reconstructs the fat pointer from the two components. For the common case (no replacement), this adds zero overhead on x86-64 (Acquire is free under TSO) and a single ldar instruction on AArch64 (~1 cycle). The f_ops field shown in OpenFile (Section 14.1) is the accessor return type, not the storage type.

Subsystem usage constraints: replace_fops is callable only from kernel subsystem code (ALSA core, TTY layer, input core), not from KABI driver callbacks. Tier 1/Tier 2 drivers that need mode-switching behavior must request the switch through their subsystem's control interface (e.g., ALSA snd_pcm_hw_params(), TTY tty_set_ldisc()), which validates the request and calls replace_fops internally.

Cross-references: - Device registry and bus management: Section 11.4 - Crash recovery node cleanup: Section 11.9 - TTY/PTY device nodes: Section 21.1 - ALSA device nodes: Section 21.4 - DRM device nodes: Section 22.1 - Input (evdev) device nodes: Section 21.3

14.6 Mount Tree Data Structures and Operations

The mount tree is the central data structure of the VFS layer that tracks all mounted filesystems, their hierarchical relationships, and their propagation properties. Every path resolution operation traverses the mount tree (via the mount hash table) to cross mount boundaries. This section defines the complete data structures, algorithms, and namespace operations that were previously referenced but unspecified by Section 14.1, Section 14.1, and Section 17.1.

Design principles:

1. RCU for the read path: Mount hash table lookups happen on every path resolution (every open(), stat(), readlink(), execve()). The read path must be completely lock-free. Writers (mount/unmount) serialize through the per-namespace mount_lock and publish changes via RCU.

2. Per-namespace scoping: Unlike Linux, which uses a single global Linux mount_hashtable, UmkaOS scopes the mount hash table per mount namespace. This eliminates contention between namespaces in container-heavy workloads (thousands of namespaces with independent mount trees) and allows mount operations in different namespaces to proceed in parallel with no shared lock. The trade-off is additional memory per namespace; this is acceptable because each namespace already has an independent mount tree and the hash table overhead is proportional to the number of mounts (typically 30-100 per container, well under 1 KiB of hash table memory).

3. Arc-based lifetime management over Nucleus tracked storage: Mount nodes are reference-counted via Arc<Mount>, allocated from Nucleus tracked storage (alloc_tracked + Arc::from_tracked — Mount is a migration-tracked type, Section 13.18). Parent, master, and peer references use Arc (strong) or Weak (where appropriate to break cycles). RCU protects the hash chains and list traversals; Arc protects the Mount node lifetime beyond the RCU grace period. The final Arc drop returns the slot via free_tracked::<Mount>().

4. Capability gating: All mount tree modifications check CAP_MOUNT or CAP_SYS_ADMIN as specified in Section 14.1. The data structures below enforce this at the entry point of each operation, not deep inside the algorithm.

5. 64-bit mount IDs: Per-namespace monotonic counter, never wrapping on any realistic system. Mount IDs are unique within a namespace and are the stable identifier used by statx() (STATX_MNT_ID), the new statmount()/listmount() syscalls, and /proc/PID/mountinfo.

14.6.1 Mount Flags

bitflags! {
    /// Per-mount flags controlling security and access behavior.
    ///
    /// These are distinct from per-superblock options (which control the
    /// filesystem driver's behavior). A single superblock can be mounted
    /// at multiple locations with different per-mount flags (e.g., one
    /// mount point read-write, another read-only via bind mount + remount).
    ///
    /// Bit assignments match Linux's `MNT_*` internal flags
    /// (`include/linux/mount.h`, stable since Linux 2.6.x). These are
    /// NOT the userspace `MS_*` flags (`include/uapi/linux/mount.h`) —
    /// the `mount(2)` and `mount_setattr(2)` compat shims translate
    /// `MS_*`/`MOUNT_ATTR_*` to `MountFlags` at syscall entry.
    #[repr(transparent)]
    pub struct MountFlags: u64 {
        // --- Userspace-visible flags (set via mount/remount/mount_setattr) ---
        //
        // Bit assignments match Linux `include/linux/mount.h` exactly.
        // Verified against torvalds/linux master (2026-03-25).

        /// Do not honor set-user-ID and set-group-ID bits on executables.
        const MNT_NOSUID       = 0x01;       // Linux: MNT_NOSUID = 0x01
        /// Do not allow access to device special files on this mount.
        const MNT_NODEV        = 0x02;       // Linux: MNT_NODEV = 0x02
        /// Do not allow execution of programs on this mount.
        const MNT_NOEXEC       = 0x04;       // Linux: MNT_NOEXEC = 0x04
        /// Do not update access times on this mount.
        const MNT_NOATIME      = 0x08;       // Linux: MNT_NOATIME = 0x08
        /// Do not update directory access times on this mount.
        const MNT_NODIRATIME   = 0x10;       // Linux: MNT_NODIRATIME = 0x10
        /// Update atime only if atime <= mtime or atime <= ctime, or if
        /// the previous atime is more than 24 hours old. Default for most
        /// mounts since Linux 2.6.30 and UmkaOS.
        const MNT_RELATIME     = 0x20;       // Linux: MNT_RELATIME = 0x20
        /// Mount is read-only. Writes return EROFS.
        const MNT_READONLY     = 0x40;       // Linux: MNT_READONLY = 0x40
        /// Do not follow symlinks on this mount. Used by container runtimes
        /// to prevent symlink-based escapes from bind-mounted directories.
        const MNT_NOSYMFOLLOW  = 0x80;       // Linux: MNT_NOSYMFOLLOW = 0x80

        // --- Internal flags (kernel-managed, not settable by userspace) ---

        /// Mount can be expired and automatically unmounted under memory
        /// pressure or after an idle timeout. Used by autofs. The VFS
        /// checks `mnt_count == 0` before expiring a shrinkable mount.
        const MNT_SHRINKABLE   = 0x100;      // Linux: MNT_SHRINKABLE = 0x100
        /// Internal mount (not exposed to userspace). Used for kernel-
        /// internal mounts (pipefs, sockfs, mqueuefs, …), which are
        /// constructed by `internal_mount()`
        /// ([Section 14.6](#mount-tree-data-structures-and-operations--kernel-internal-mounts-mntinternal)).
        const MNT_INTERNAL     = 0x4000;     // Linux: MNT_INTERNAL = 0x4000

        // --- Container namespace lock flags (MNT_LOCK_*) ---
        //
        // These flags prevent unprivileged users in child mount namespaces
        // from changing mount attributes inherited from the parent namespace.
        // Set by the kernel when creating a user namespace or copying a
        // mount namespace. Critical for container security — without these,
        // a container could remount a read-only host path as read-write.

        /// Atime setting is locked (NOATIME/RELATIME/NODIRATIME cannot
        /// be changed by unprivileged mount_setattr in child namespace).
        const MNT_LOCK_ATIME     = 0x040000; // Linux: MNT_LOCK_ATIME = 0x040000
        /// NOEXEC flag is locked.
        const MNT_LOCK_NOEXEC    = 0x080000; // Linux: MNT_LOCK_NOEXEC = 0x080000
        /// NOSUID flag is locked.
        const MNT_LOCK_NOSUID    = 0x100000; // Linux: MNT_LOCK_NOSUID = 0x100000
        /// NODEV flag is locked.
        const MNT_LOCK_NODEV     = 0x200000; // Linux: MNT_LOCK_NODEV = 0x200000
        /// READONLY flag is locked (cannot be remounted read-write by
        /// unprivileged users in child namespace).
        const MNT_LOCK_READONLY  = 0x400000; // Linux: MNT_LOCK_READONLY = 0x400000

        /// Mount is locked and cannot be unmounted by unprivileged
        /// processes. Set on mounts visible in child mount namespaces
        /// created by unprivileged users — prevents a child namespace
        /// from unmounting a mount inherited from the parent. Cleared
        /// only by a process with `CAP_SYS_ADMIN` in the mount's owning
        /// user namespace.
        const MNT_LOCKED         = 0x800000; // Linux: MNT_LOCKED = 0x800000

        /// Mount is in the process of being unmounted. Set by `umount()`
        /// before removing the mount from the hash table. Prevents new
        /// path lookups from entering this mount. Once set, never cleared
        /// (the mount node is freed after the RCU grace period).
        const MNT_DOOMED         = 0x1000000;  // Linux: MNT_DOOMED = 0x1000000
        /// Synchronous unmount requested. Set when MNT_DETACH was NOT
        /// specified and the kernel must wait for all references to drain.
        const MNT_SYNC_UMOUNT    = 0x2000000;  // Linux: MNT_SYNC_UMOUNT = 0x2000000
        /// Mount is being torn down by the umount process.
        const MNT_UMOUNT         = 0x8000000;  // Linux: MNT_UMOUNT = 0x8000000

        // --- UmkaOS extension flags (bits 28+) ---
        //
        // These flags are UmkaOS-original extensions NOT present in Linux's
        // mnt_flags. They occupy high bit positions (28+) to avoid collision
        // with future Linux MNT_* additions. Both are intentional design
        // improvements over Linux.

        /// **UmkaOS extension — not present in Linux mnt_flags.**
        /// Per-mount lazytime: buffer atime updates in memory and flush
        /// lazily. Reduces write I/O for atime-heavy workloads (mail servers).
        /// This is a genuine improvement over Linux's per-superblock
        /// `SB_LAZYTIME`: different bind mounts of the same filesystem can
        /// have different lazytime policies (e.g., `/var/mail` with lazytime,
        /// `/var/log` without, on the same ext4 volume).
        const MNT_LAZYTIME       = 1 << 28;   // UmkaOS extension (bit 28)
        /// **UmkaOS extension — not present in Linux mnt_flags.**
        /// Explicit detached-mount state flag. Set by `fsmount()` before
        /// `move_mount()` attaches the mount to the namespace tree. Detached
        /// mounts are invisible to path resolution and /proc/PID/mountinfo.
        /// Linux tracks this implicitly through namespace tree membership;
        /// UmkaOS makes it an explicit flag used in 10+ places in
        /// fsmount/move_mount/open_tree flows for clarity and correctness.
        const MNT_DETACHED       = 1 << 29;   // UmkaOS extension (bit 29)
    }
}

14.6.1.1 Kernel-Internal Mounts (MNT_INTERNAL)

A kernel-internal mount is a filesystem instance the kernel needs as an object but never publishes in a mount tree: pipefs, sockfs, and the per-IPC-namespace mqueuefs mount (Section 17.3). It has no parent, no mountpoint, and no namespace; path resolution and /proc/PID/mountinfo never see it. internal_mount() is the only constructor for this class.

/// Construct a kernel-internal mount of `fstype`.
///
/// Creates a FRESH superblock for every call — never shared through
/// mount-table reuse; mqueuefs relies on this for its per-IPC-namespace
/// superblocks — runs the type's `populate_super` callback
/// ([Section 14.18](#pseudo-filesystems)), and allocates the `Mount` via `mount_alloc()`
/// with `MountFlags::MNT_INTERNAL` set, `parent = None`, `mountpoint = None`.
///
/// The mount is attached to NO namespace, is invisible to path resolution and
/// `/proc/PID/mountinfo`, and is NOT staged in the shadow registry: like
/// detached mounts (bracket-discipline rule 4,
/// [Section 14.6](#mount-tree-data-structures-and-operations--shadow-registry-reporting)),
/// the returned `Arc` IS the owning reference, and an internal mount does not
/// survive a VFS-module crash. Dropping the last `Arc` tears down the mount
/// and its superblock.
///
/// Userspace can later reach the same filesystem only through its own
/// mount(2)/bind conventions (e.g. binding `/dev/mqueue`); this constructor
/// never touches the mount tree.
///
/// # Errors
///
/// `ENOMEM` — superblock construction or `mount_alloc()` failed.
pub fn internal_mount(fstype: &'static PseudoFsType) -> Result<Arc<Mount>, Errno>;

14.6.2 Propagation Type

/// Mount propagation type. Controls whether mount/unmount events at this
/// mount point are propagated to other mount points, and in which direction.
///
/// Propagation is fundamental to container runtimes: Docker sets the rootfs
/// to MS_PRIVATE by default, Kubernetes uses MS_SHARED for volume mounts
/// that must be visible across pod containers.
///
/// See: Linux kernel Documentation/filesystems/sharedsubtree.rst
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
#[repr(u8)]
pub enum PropagationType {
    /// Mount events propagate bidirectionally within the peer group.
    /// All mounts in the same peer group see each other's mount/unmount
    /// events. This is the Linux default for the initial namespace root.
    Shared = 0,

    /// Mount events are not propagated to or from this mount. This is
    /// the default for new mount namespaces (container isolation).
    Private = 1,

    /// Mount events propagate unidirectionally from the master to this
    /// mount, but not in the reverse direction. Used when a container
    /// should see new mounts from the host but not expose its own mounts
    /// to the host.
    Slave = 2,

    /// Like Private, but additionally prevents this mount from being
    /// used as the source of a bind mount. Used for security-sensitive
    /// mount points that should never be replicated.
    Unbindable = 3,
}

14.6.3 Mount Node

/// A single mount instance in the mount tree.
///
/// Equivalent to Linux's `struct mount` (not `struct vfsmount` — the latter
/// is the subset exposed to filesystem drivers; `struct mount` is the full
/// internal structure). Each `Mount` represents one attachment of a
/// filesystem at a specific point in the directory tree.
///
/// **Lifetime**: `Mount` is a migration-tracked type
/// ([Section 13.18](13-device-classes.md#live-kernel-evolution--generic-tracked-allocator)). Instances are
/// allocated from Nucleus tracked storage via `alloc_tracked::<Mount>()` and
/// bridged to the kernel-wide `Arc<Mount>` handle via `Arc::from_tracked`
/// (strong/weak counts live in the slot's side header; the payload slot holds
/// pure `Mount`, so the evolution orchestrator's migration walk sees clean
/// instances). See "Tracked Allocation — Mount and MountNamespace" below for
/// the allocation helper, type registration, and failure mapping.
///
/// The single tree-side owning `Arc<Mount>` reference is held by the mount's
/// Shadow Mount Registry entry
/// ([Section 14.1](#virtual-filesystem-layer--shadow-mount-registry-and-mount-tree-reconstruction));
/// every other tree link below is a NON-owning intrusive/weak reference that
/// merely threads or locates the mount, and open files pin it separately via
/// `mnt_count`. References are held by:
/// - The Shadow Mount Registry entry — the single tree-side owning
///   `Arc<Mount>` reference.
/// - The mount hash table (NON-owning intrusive link; hash chains are
///   RCU-protected for lockless readers).
/// - The parent mount's `children` list (NON-owning intrusive link).
/// - The peer group's `mnt_share` ring (NON-owning intrusive link).
/// - The master mount's `mnt_slave_list` (NON-owning intrusive link).
/// - Any open file descriptor whose path traversed this mount
///   (a separate pin via the `mnt_count` reference count).
/// - The `MountNamespace.mount_list` (NON-owning intrusive link).
///
/// A mount node is freed when all strong references are dropped, which
/// happens after: (a) removal from the hash table and the other tree links
/// (parent's `children`, `mount_list`, peer/slave lists), (b) the Shadow
/// Mount Registry releasing its single owning `Arc<Mount>` (deferred to an
/// `rcu_call` by the shadow-transaction commit), (c) RCU grace period
/// completion, and (d) all path-resolution references (`mnt_count`) have been
/// released. Ordering of (c) before the final drop is enforced by the
/// teardown paths themselves: `do_umount`/`do_umount_tree`/
/// `destroy_mount_namespace` remove the mount from the hash table (and every
/// other tree link) and then release the Shadow Mount Registry's owning
/// `Arc<Mount>` reference from an `rcu_call` callback (the shadow-transaction
/// commit), so a reference-less RCU reader (path resolution crossing a mount
/// point) can never observe a freed slot.
/// The final `Arc<Mount>` drop releases the embedded `TrackedPtr<Mount>`,
/// invoking `free_tracked::<Mount>()` — returning the slot to the per-type
/// free list and removing the instance from the Nucleus live registry. There
/// is no separate slab-free path.
pub struct Mount {
    // --- Identity ---

    /// Unique mount identifier within the owning namespace. Monotonically
    /// increasing, 64-bit, never reused. This is the value returned by
    /// `statx()` in `stx_mnt_id` (STATX_MNT_ID) and reported in
    /// `/proc/PID/mountinfo` field 1.
    pub mount_id: u64,

    /// Device name string (e.g., "/dev/sda1", "tmpfs", "overlay").
    /// Displayed in `/proc/PID/mountinfo` field 10 (mount source).
    /// Heap-allocated, immutable after mount creation.
    pub device_name: Box<[u8]>,

    // --- Tree structure ---

    /// Parent mount. `None` for the root of the mount namespace.
    /// Uses `Weak` to prevent reference cycles in the mount tree:
    /// parent -> children -> parent would create a cycle with `Arc`.
    /// The parent is always alive while any child exists (the child
    /// holds a position in the parent's hash chain), so the `Weak`
    /// can always be upgraded during normal operation. It fails only
    /// during the teardown of a doomed mount tree, which is expected.
    pub parent: Option<Weak<Mount>>,

    /// Cached parent mount ID. Set at mount time, updated on `move_mount`.
    /// Avoids `Weak::upgrade()` during RCU-walk lookups (the upgrade may
    /// fail during concurrent umount). Helper: `fn mount_id_of_parent(&self)
    /// -> u64 { self.parent_mount_id }`.
    pub parent_mount_id: u64,

    /// The dentry in the parent mount's filesystem where this mount is
    /// attached. For the root mount of a namespace, this is the root
    /// dentry of the parent mount (which is itself).
    ///
    /// Together with `parent`, this pair `(parent_mount, mountpoint_dentry)`
    /// is the key in the mount hash table. Path resolution uses this to
    /// detect mount crossings: when a dentry has `DCACHE_MOUNTED` set,
    /// the VFS calls `mnt_ns.hash_table.lookup(current_mount.mount_id, dentry)` to find the
    /// child mount.
    pub mountpoint: DentryRef,

    /// Root dentry of the mounted filesystem. When path resolution
    /// crosses into this mount, it continues from this dentry.
    pub root: DentryRef,

    /// The superblock of the mounted filesystem. Shared across all
    /// mounts of the same filesystem instance (e.g., bind mounts share
    /// the superblock). The superblock holds the filesystem-specific
    /// state and the `FileSystemOps`/`InodeOps`/`FileOps` trait objects.
    pub superblock: Arc<SuperBlock>,

    /// The mount's cross-domain `AddressSpaceOps` provider instance
    /// (`RingMountAddressSpaceOps`,
    /// [Section 14.2](#vfs-ring-buffer-protocol--ringmount-addressspaceops-provider)),
    /// constructed by `Mount::resolve_aspace_ops()`
    /// ([Section 14.1](#virtual-filesystem-layer)) at bind time when the provider is
    /// cross-domain; `None` for same-domain binds (native ops are
    /// `'static` module vtable instances that need no owner). This slot
    /// is the OWNER that the per-`AddressSpace` `AspaceOpsBinding` cells
    /// point into: a rebind that switches transport replaces it only
    /// AFTER the rebind quiesce has repointed every live binding, and
    /// umount drops it only after the superblock's `AddressSpace`s are
    /// torn down. Written only at mount setup and under the rebind
    /// quiesce ([Section 13.18](13-device-classes.md#live-kernel-evolution)).
    pub ring_aspace_ops: Option<Box<RingMountAddressSpaceOps>>,

    /// Children of this mount — sub-mounts attached at dentries within
    /// this mount's filesystem. Intrusive doubly-linked list for O(1)
    /// insertion and removal. Protected by the namespace's `mount_lock`
    /// for writes; RCU-protected for reads during path resolution.
    pub children: IntrusiveList<Arc<Mount>>,

    /// Link entry for this mount in its parent's `children` list.
    /// Embedded in the `Mount` node to avoid per-child heap allocation.
    pub child_link: IntrusiveListNode,

    // --- Mount flags ---

    /// Per-mount flags (nosuid, nodev, noexec, readonly, noatime, etc.).
    /// Atomically readable for the path-resolution hot path (no lock
    /// needed to check MNT_READONLY or MNT_NOSUID). Modified only under
    /// `mount_lock` via atomic store with Release ordering.
    pub flags: AtomicU64,

    // --- Propagation ---

    /// Propagation type for this mount (Shared, Private, Slave, Unbindable).
    /// Determines how mount/unmount events are forwarded to related mounts.
    /// Modified only under `mount_lock`.
    pub propagation: PropagationType,

    /// Peer group ID for shared mounts. All mounts in the same peer group
    /// have the same `group_id`. Private and unbindable mounts have
    /// `group_id == 0`. Slave mounts retain the `group_id` of their
    /// former peer group (for /proc/PID/mountinfo optional fields).
    ///
    /// Allocated from the namespace's `group_id_allocator`. Unique within
    /// a namespace.
    pub group_id: u64,

    /// Circular linked list of peer mounts (shared propagation).
    /// All mounts in a peer group are linked through `mnt_share`.
    /// When a mount/unmount event occurs on any peer, it is propagated
    /// to all other peers in the ring. For Private/Unbindable mounts,
    /// this list contains only the mount itself (self-loop).
    pub mnt_share: IntrusiveListNode,

    /// Master mount for slave propagation. When this mount is a slave,
    /// `mnt_master` points to the shared mount from which this mount
    /// receives (but does not send) propagation events.
    /// `None` for shared, private, and unbindable mounts.
    pub mnt_master: Option<Weak<Mount>>,

    /// List head for slave mounts of this mount. When this mount is
    /// shared (or was shared), slave mounts derived from it are linked
    /// through `mnt_slave_list`. Each slave's `mnt_slave` node is an
    /// entry in this list.
    pub mnt_slave_list: IntrusiveList<Arc<Mount>>,

    /// Link entry for this mount in its master's `mnt_slave_list`.
    pub mnt_slave: IntrusiveListNode,

    // --- Namespace membership ---

    /// The mount namespace that owns this mount. `Weak` because the
    /// namespace may be destroyed (all processes exited) while detached
    /// mounts or lazy-unmount remnants still exist.
    pub ns: Weak<MountNamespace>,

    /// Link entry in the namespace's `mount_list`. Used for ordered
    /// iteration (e.g., /proc/PID/mountinfo output, umount ordering).
    pub ns_list_link: IntrusiveListNode,

    // --- Reference counting ---

    /// Active reference count. Incremented when path resolution enters
    /// this mount (ref-walk mode) or when an open file descriptor
    /// references a path within this mount. `umount()` checks this
    /// before removing the mount: if `mnt_count > 0`, the mount is
    /// busy and umount returns `EBUSY` (unless `MNT_DETACH` is used).
    ///
    /// Note: this is separate from the `Arc` reference count. `Arc`
    /// tracks the lifetime of the `Mount` struct itself. `mnt_count`
    /// tracks whether the mount is actively *in use* by path lookups
    /// and open files. A mount can have `mnt_count == 0` (not busy)
    /// while still having `Arc` strong count > 0 (struct not yet freed
    /// because it's still in the hash table or child list).
    pub mnt_count: AtomicU64,

    // --- Mount hash chain ---

    /// Link entry in the mount hash table bucket chain. RCU-protected:
    /// readers traverse the chain under `rcu_read_lock()` without any
    /// lock; writers modify the chain under `mount_lock` and publish
    /// via RCU. Uses intrusive linking for zero-allocation hash insertion.
    pub hash_link: IntrusiveListNode,
}

impl Mount {
    /// Cached parent mount ID, avoids Weak::upgrade() during RCU-walk.
    #[inline]
    pub fn mount_id_of_parent(&self) -> u64 {
        self.parent_mount_id
    }

    /// Inode ID of the mountpoint dentry (the dentry in the parent mount
    /// where this mount is attached). Used as the secondary key in the
    /// mount hash table: lookup is `(parent_mount_id, mountpoint_inode_id)`.
    #[inline]
    pub fn mountpoint_inode(&self) -> InodeId {
        self.mountpoint.inode
    }
}

/// Reference to a dentry. Wraps the dentry's inode ID and parent inode ID,
/// which together uniquely identify a dentry in the dentry cache (Section
/// 13.1.2). The VFS resolves this to a cached dentry entry on access.
///
/// This avoids holding a direct pointer into the dentry cache (which is
/// RCU-managed and may be evicted), while still providing O(1) lookup via
/// the dentry hash table.
pub struct DentryRef {
    /// Inode ID of the parent directory containing this dentry.
    pub parent_inode: InodeId,
    /// Name hash of this dentry. For filesystems with a custom
    /// `DentryOps::d_hash()` (case-insensitive filesystems), `name_hash`
    /// stores the result of `d_hash()`, not the default hash. For
    /// filesystems without custom hashing, the default SipHash-1-3 of the
    /// name component is used. Used for O(1) dentry cache lookup without
    /// storing the full name.
    pub name_hash: u64,
    /// Inode ID of the dentry itself (for positive dentries).
    pub inode: InodeId,
}

14.6.4 Mount Hash Table

/// Per-namespace mount hash table. Maps `(parent_mount_id, mountpoint_dentry)`
/// pairs to child `Mount` nodes. This is the data structure consulted on
/// every mount-point crossing during path resolution.
///
/// **Why per-namespace**: Linux uses a single global `mount_hashtable`
/// (Linux: `static struct hlist_head *mount_hashtable __ro_after_init`, sized once
/// at boot by `alloc_large_system_hash("Mount-cache", ...)` — a RAM-scaled
/// bucket count, overridable via the `mhash_entries=` boot parameter, not a
/// fixed number), and ALL writes to it are serialized by one global seqlock —
/// Linux declares `mount_lock` as a `__cacheline_aligned_in_smp DEFINE_SEQLOCK`. The mount
/// Linux helpers `lock_mount_hash()`/`unlock_mount_hash()` are just
/// Linux implements them as `write_seqlock(&mount_lock)`/`write_sequnlock(&mount_lock)`, and the
/// Linux's RCU-walk read side pairs `read_seqbegin(&mount_lock)` with
/// `rcu_read_lock()`. Because that one seqlock guards every bucket, every
/// mount or unmount — in ANY namespace — serializes against every other and
/// bounces the single hot `mount_lock` cache line. In container-heavy
/// environments (thousands of namespaces, each with 30-100 mounts), this
/// global write serialization limits the scalability of concurrent mount
/// operations across otherwise-independent namespaces. UmkaOS gives each
/// namespace its own hash table and its own `mount_lock`, so mounts in
/// different namespaces never contend on a shared lock — cross-namespace
/// contention is eliminated entirely.
///
/// **Sizing**: The hash table is sized to the number of mounts in the
/// namespace, with a minimum of 32 buckets and a maximum of 1024. The table
/// is resized (doubled) when the load factor exceeds 2.0, and shrunk
/// (halved) when the load factor drops below 0.25. Resizing allocates a
/// new bucket array, rehashes under `mount_lock`, and publishes via RCU.
///
/// **Hash function**: SipHash-1-3 of `(parent_mount_id, mountpoint_inode_id)`.
/// The SipHash key is per-namespace, generated from a CSPRNG at namespace
/// creation. This prevents hash-flooding attacks where an adversary crafts
/// mount points that collide in the hash table.
pub struct MountHashTable {
    /// RCU-protected bucket array. Wrapped in `Arc<BucketArray>` because
    /// `RcuCell` requires an atomically-swappable thin pointer — `Box<[T]>`
    /// is a fat pointer (data + length) that cannot be atomically swapped
    /// on any current architecture. `Arc<BucketArray>` is a single thin
    /// pointer that `RcuCell` can swap atomically.
    /// Readers traverse under `rcu_read_lock()`; writers modify under
    /// the namespace's `mount_lock`.
    buckets: RcuCell<Arc<BucketArray>>,

    /// Number of entries in the hash table. Used for load-factor
    /// computation during resize decisions. Modified only under `mount_lock`.
    /// **Bounded**: u32 supports ~4 billion mounts per namespace. Linux's
    /// default `sysctl fs.mount-max` is 100,000; even extreme container
    /// workloads rarely exceed 1 million. u32 is sufficient.
    /// At mount_max=100K, u32 provides ~42,949x headroom. This is a hash
    /// table entry count, not an identifier — the 50-year u64 policy does
    /// not apply.
    count: u32,

    /// SipHash key for this hash table. Per-namespace, generated at
    /// namespace creation from the kernel CSPRNG.
    hash_key: [u64; 2],
}

/// Thin-pointer wrapper for the dynamically-sized bucket array.
/// `Box<[MountHashBucket]>` is a fat pointer (data + length) that cannot be
/// atomically swapped by `RcuCell`. This wrapper provides a thin `Arc` pointer.
// Kernel-internal, not KABI.
struct BucketArray {
    buckets: Box<[MountHashBucket]>,
}

/// A single bucket in the mount hash table. Contains the head pointer
/// of an RCU-protected chain of Mount nodes.
struct MountHashBucket {
    /// Head of the intrusive linked list of Mount nodes hashing to this
    /// bucket. Null if the bucket is empty. Readers follow this chain
    /// under RCU; writers modify under `mount_lock`.
    ///
    /// **Lifecycle**: Hash chain insertion calls `Arc::into_raw()` to obtain
    /// the raw pointer (incrementing the strong count); hash chain removal
    /// under `mount_lock` uses RCU to defer `Arc::from_raw()` (which
    /// decrements the count) until after the grace period. This ensures
    /// RCU readers never access freed memory.
    head: AtomicPtr<Mount>,
}

impl MountHashTable {
    /// Look up a child mount at the given `(parent, dentry)` pair.
    ///
    /// Called during path resolution when a dentry has the `DCACHE_MOUNTED`
    /// flag set. Must be called under `rcu_read_lock()`.
    ///
    /// Returns `Some(&Mount)` if a mount is found at this point, or
    /// `None` if the dentry is not a mount point (stale `DCACHE_MOUNTED`
    /// flag — possible after lazy unmount).
    ///
    /// **Performance**: O(1) expected, O(n) worst-case where n is the
    /// chain length (bounded by load factor < 2.0). No locks, no atomics
    /// beyond the initial `Acquire` load of the bucket head pointer.
    pub fn lookup<'a>(
        &'a self,
        parent_mount_id: u64,
        mountpoint_inode: InodeId,
        _rcu: &'a RcuReadGuard,
    ) -> Option<&'a Mount> {
        let hash = siphash_1_3(
            self.hash_key,
            parent_mount_id,
            mountpoint_inode.0,
        );
        // Readers must obtain the bucket array pointer and compute
        // bucket_count from the same RCU-protected snapshot to avoid
        // OOB access during a concurrent resize.
        let buckets = self.buckets.read(_rcu);
        let bucket_idx = hash as usize % buckets.len();
        let bucket = &buckets[bucket_idx];

        let mut current = bucket.head.load(Ordering::Acquire);
        while !current.is_null() {
            // SAFETY: `current` is a valid Mount pointer within an RCU
            // read-side critical section. The Mount node is not freed
            // until after the RCU grace period.
            let mnt = unsafe { &*current };
            if mnt.mount_id_of_parent() == parent_mount_id
                && mnt.mountpoint_inode() == mountpoint_inode
                && !mnt.is_doomed()
            {
                return Some(mnt);
            }
            current = mnt.hash_link.next.load(Ordering::Acquire);
        }
        None
    }

    /// Transition from RCU-protected `&Mount` to a long-lived reference.
    ///
    /// **Ref-walk mode**: After `lookup()` returns `Some(&Mount)`, the
    /// caller must increment `mnt_count` before dropping the `RcuReadGuard`:
    /// ```
    /// let rcu = rcu_read_lock();
    /// if let Some(mnt) = mount_hash.lookup(parent_id, ino, &rcu) {
    ///     mnt.mnt_count.fetch_add(1, Acquire);
    ///     drop(rcu);
    ///     // `mnt` is now safe to use without RCU protection.
    ///     // Caller must call mnt.mnt_count.fetch_sub(1, Release)
    ///     // when the reference is no longer needed.
    /// }
    /// ```
    ///
    /// **RCU-walk mode**: The caller stays within the RCU critical section
    /// for the entire path resolution and never increments `mnt_count`.
    /// If RCU-walk fails (e.g., dentry seqlock mismatch), the path
    /// resolution restarts in ref-walk mode.
    ///
    /// The `Acquire` on `fetch_add` pairs with the `Release` on
    /// `fetch_sub` to ensure visibility of all mount state modifications
    /// made before the reference was taken.
    pub fn get_counted_ref(mnt: &Mount) {
        mnt.mnt_count.fetch_add(1, Ordering::Acquire);
    }
}

14.6.5 Mount Namespace

/// A mount namespace. Contains an independent mount tree with its own root
/// mount, hash table, and mount list. Created by `clone(CLONE_NEWNS)` or
/// `unshare(CLONE_NEWNS)`.
///
/// The `vfs_root: Capability<VfsNode>` field in `NamespaceSet` (Section 17.1.2)
/// is updated to point to this namespace's root mount:
///
/// ```rust
/// // Updated NamespaceSet field (replaces the previous Capability<VfsNode>):
/// pub mount_ns: Arc<MountNamespace>,
/// ```
///
/// **Relationship to NamespaceSet**: Each task's `NamespaceSet` holds
/// an `Arc<MountNamespace>`. Multiple tasks in the same mount namespace
/// share the same `Arc<MountNamespace>`. When `clone(CLONE_NEWNS)` is called,
/// a new `MountNamespace` is created by cloning the parent's mount tree
/// (via `copy_tree()`).
///
/// **Lifetime**: `MountNamespace` is a migration-tracked type
/// ([Section 13.18](13-device-classes.md#live-kernel-evolution--generic-tracked-allocator)). Instances are
/// allocated via `alloc_tracked::<MountNamespace>()` + `Arc::from_tracked`
/// (see "Tracked Allocation — Mount and MountNamespace" below). Unlike
/// `Mount` and `Dentry`, no RCU-deferred release is needed on the free path:
/// every reader reaches a `MountNamespace` through a strong `Arc` (the
/// task's `NamespaceSet.mount_ns`, an `/proc/PID/ns/mnt` fd, or a bind
/// mount of the namespace file) — there is no reference-less lockless
/// traversal that dereferences a `MountNamespace` pointer. The final
/// `Arc` drop (after `destroy_mount_namespace()` has emptied the tree)
/// releases the embedded `TrackedPtr<MountNamespace>`, invoking
/// `free_tracked::<MountNamespace>()`.
pub struct MountNamespace {
    /// Unique namespace identifier. Used for `/proc/PID/ns/mnt` inode
    /// number and `setns()` namespace comparison.
    pub ns_id: u64,

    /// Root mount of this namespace's mount tree. This is the mount
    /// that corresponds to "/" for all processes in this namespace.
    /// Updated atomically by `pivot_root()`.
    pub root: RcuCell<Arc<Mount>>,

    /// Ordered list of all mounts in this namespace. The ordering is
    /// *normally* topological — parent mounts appear before their children —
    /// but this is NOT a guaranteed invariant: `do_move_mount` (MS_MOVE)
    /// re-parents a mount without relocating it in this list, so a moved
    /// mount can precede its new parent (see the **Maintenance** note below
    /// and `copy_tree` step 4). A consumer that requires a guaranteed
    /// leaf-first or parent-before-child traversal walks `Mount.children`
    /// (postorder), not this list.
    /// This ordering is used by:
    /// - `/proc/PID/mountinfo`: output follows this list order — normally
    ///   parent-before-child, but not a guaranteed topological order;
    ///   consumers that rebuild mount topology use the `parent_id` field
    /// - `umount -a`: a reverse pass over this list normally unmounts
    ///   leaves before parents (a normal-case tendency, not a guarantee)
    /// - Namespace teardown (`destroy_mount_namespace`): unlinks every mount
    ///   in a reverse-list pass whose correctness does NOT depend on the
    ///   order — all mounts are staged then committed atomically
    ///
    /// **Maintenance**: `mount_filesystem` maintains this normal-case ordering by
    /// inserting each new mount immediately after its parent's list entry (`mount_filesystem`
    /// step 4i). `do_move_mount` (MS_MOVE) does NOT currently relocate a
    /// re-parented mount within this list (it only updates the hash table
    /// and `Mount.children` — see `do_move_mount` step 6), so a moved
    /// mount's position may precede its new parent's after a cross-subtree
    /// move (FLOW-19-10). Consumers that need a guaranteed-correct
    /// parent-before-child order regardless of prior moves (e.g.
    /// `copy_tree`) walk `Mount.children` directly instead of relying on
    /// this list — see `copy_tree` step 4.
    pub mount_list: IntrusiveList<Arc<Mount>>,

    /// Number of mounts in this namespace. Used to enforce the
    /// per-namespace mount count limit (default: 100,000 — matching
    /// Linux's `sysctl fs.mount-max`). Prevents mount-storm DoS attacks
    /// where a compromised container creates millions of mounts.
    /// Current-state count bounded by mount_max (~100K). u64 used for
    /// consistency with other AtomicU64 counters in the namespace; u32
    /// would suffice. The 32-bit-leg question is keyed on the compiler
    /// predicate `target_has_atomic = "64"`, NOT on pointer width: ARMv7-A
    /// satisfies it (LDREXD/STREXD) and gets a native 64-bit atomic like the
    /// 64-bit legs; PPC32 is the sole supported leg that does not, and a
    /// logically-64-bit atomic there is governed by the classification rule of
    /// the 64-bit-atomic semantic family
    /// ([Section 3.5](03-concurrency.md#locking-strategy--64-bit-atomics-on-32-bit-legs-the-semantic-family)).
    /// Acceptable on every leg: mount/unmount is a warm path.
    pub mount_count: AtomicU64,

    /// Event counter. Incremented on every mount/unmount/remount
    /// operation. Used by `poll()` on `/proc/PID/mountinfo` to detect
    /// mount tree changes. Container runtimes and systemd use this
    /// to react to mount events without periodic scanning.
    pub event_seq: AtomicU64,

    /// Per-namespace mount hash table. Maps `(parent_mount, dentry)` to
    /// child mount for path resolution mount-point crossings.
    pub hash_table: MountHashTable,

    /// Mutex serializing mount tree modifications (mount, unmount,
    /// remount, pivot_root, bind mount, move mount). Readers (path
    /// resolution) do not acquire this lock — they use RCU.
    /// Lock hierarchy level 20 (MOUNT_LOCK): above DENTRY_LOCK (19),
    /// below EVM_LOCK (22). See [Section 3.5](03-concurrency.md#locking-strategy--lock-hierarchy-summary).
    pub mount_lock: Mutex<()>,

    /// Mount ID allocator. Monotonically increasing 64-bit counter.
    /// IDs are never reused within a namespace. At 1 mount/second
    /// sustained, a 64-bit counter would not wrap for ~584 billion years.
    pub id_allocator: AtomicU64,

    /// Peer group ID allocator. Like mount IDs, monotonically increasing
    /// and never reused. Separate from mount IDs because group IDs are
    /// shared across mounts and have a different lifecycle.
    pub group_id_allocator: AtomicU64,

    /// User namespace that owns this mount namespace. Determines
    /// capability checks for mount operations. A process must have
    /// `CAP_MOUNT` in this user namespace (or an ancestor) to modify
    /// the mount tree.
    pub user_ns: Arc<UserNamespace>,
}

impl MountNamespace {
    /// Return a `PathRef` pointing to the root of this mount namespace.
    ///
    /// The root mount is stored in `self.root` as `RcuCell<Arc<Mount>>`.
    /// This helper loads the current root under an RCU read-side reference
    /// and constructs a `PathRef` at the root dentry of that mount.
    ///
    /// Used by `setns(CLONE_NEWNS)` to reset `task.fs.root` and `task.fs.pwd`
    /// to the new namespace's root when entering the namespace (see
    /// [Section 17.1](17-containers.md#namespace-architecture--container-root-filesystem-pivotroot2)).
    ///
    /// # Return value
    /// `PathRef` whose `.mount` is the namespace root mount and whose `.dentry`
    /// is `root_mount.root_dentry` (the root dentry of the mounted filesystem).
    pub fn root_mount(&self) -> PathRef {
        // SAFETY: RCU read-side — the root is always set before a MountNamespace
        // is published, and pivot_root() swaps it atomically via RcuCell::store().
        let guard = rcu_read_lock();
        let mnt = Arc::clone(&*self.root.read(&guard));
        let dentry = Arc::clone(&mnt.root_dentry);
        PathRef { mount: mnt, dentry }
    }
}

14.6.6 Tracked Allocation — Mount and MountNamespace

Mount and MountNamespace appear on the migration-tracked type list in Section 13.18: both are long-lived structures whose layout must be evolvable over the kernel's operational lifetime (adding propagation metadata, retyping counters, cache-line reordering). Every live instance must therefore be enumerable by the Nucleus evolution orchestrator — which requires allocation through the tracked allocator, never through an untracked slab cache. The conversion follows the Task template (Section 13.18, Section 8.1 step 6).

Path temperature: mount, unmount, bind mount, and clone(CLONE_NEWNS) are warm paths (bounded frequency, process context, may sleep). alloc_tracked's hot path is a per-CPU magazine pop with no lock — cycle-equivalent to the slab fast path (the tracked allocator reuses the slab magazine implementation, Section 13.18) — so the conversion adds zero cost even relative to a hypothetical slab-backed design. The slow path (depot refill) briefly holds TRACKED_REGISTRY_LOCK(135); every allocation site may take it legally — mount_filesystem allocates at step 6d BEFORE acquiring mount_lock, and the worst case at any other site is holding the sleeping mount_lock Mutex, under which a briefly-held SpinLock is permitted by the lock discipline (sleeping mutexes may nest SpinLocks; the reverse is forbidden).

14.6.6.1 Type Registration

// umka-vfs/src/mount/mount_init.rs — Evolvable

impl TrackedType for Mount {
    fn type_id() -> TypeId {
        MOUNT_TYPE_ID.get().copied().expect("Mount not yet registered")
    }
}
impl TrackedType for MountNamespace {
    fn type_id() -> TypeId {
        MOUNT_NS_TYPE_ID.get().copied().expect("MountNamespace not yet registered")
    }
}

static MOUNT_TYPE_ID: BootOnceCell<TypeId> = BootOnceCell::new();
static MOUNT_NS_TYPE_ID: BootOnceCell<TypeId> = BootOnceCell::new();

/// Registered from the VFS module's `#[module_init]` constructor, which runs
/// during boot BEFORE the initial rootfs mount and the creation of the init
/// mount namespace (`INIT_MOUNT_NS`, [Section 17.1](17-containers.md#namespace-architecture)) — so every
/// Mount/MountNamespace ever created, including the boot-time ones, lives in
/// tracked storage. Registration failure (`OutOfStorage`) is a boot-time
/// panic with a diagnostic directing the operator to raise
/// `umka.tracked_storage_size` — deterministic at boot, never a runtime
/// surprise.
#[module_init]
fn mount_register_tracked_types() {
    let mount_template = TypeDescriptorTemplate {
        size: core::mem::size_of::<Mount>() as u32,
        alignment: core::mem::align_of::<Mount>() as u32,
        // Runtime-derived: Mount sub-region budget is 1/2048 of physical
        // memory (16 GiB machine → 8 MiB → ~16K mounts; 256 GiB → ~256K
        // mounts). This scales with the machine the way mount populations
        // do (container density tracks memory). Overridable at boot via
        // `umka.vfs.mount_slots=<count>` for mount-heavy deployments
        // (per the TypeDescriptor contract: max_instances is discovered
        // from hardware capacity and configurable on the command line).
        // Exhaustion surfaces as ENOMEM from mount(2)/fsmount(2)/
        // clone(CLONE_NEWNS) — the same errno Linux returns when kernel
        // allocation fails — never a kernel failure. Note the per-namespace
        // `mount_count` limit (fs.mount-max, default 100,000) bounds a
        // single namespace; this instance budget bounds the SUM across
        // all namespaces, closing the "mount_max × namespace count"
        // multiplication that the per-namespace limit alone leaves open.
        max_instances: boot_param("umka.vfs.mount_slots",
            default = ((total_ram_pages() as u64 * PAGE_SIZE as u64 / 2048)
                / round_up(core::mem::size_of::<Mount>(),
                           core::mem::align_of::<Mount>()) as u64) as u32),
        migration_fn: Some(mount_migrate),
        checker_id: None,
    };
    MOUNT_TYPE_ID.set(register_tracked_type(mount_template)
        .expect("Mount descriptor registration failed"))
        .expect("Mount type_id already set");

    let ns_template = TypeDescriptorTemplate {
        size: core::mem::size_of::<MountNamespace>() as u32,
        alignment: core::mem::align_of::<MountNamespace>() as u32,
        // 0 = unbounded (registry grows on demand). MountNamespace is a
        // slow-changing type — the exact case the TypeDescriptor contract
        // names (alongside NetNamespace) as safe for an unbounded registry.
        // Creation rate is bounded externally by the per-user namespace
        // creation limits ([Section 17.1](17-containers.md#namespace-architecture)) and each instance
        // transitively pins at least one Mount, so the Mount instance
        // budget above indirectly bounds namespace-driven storage growth.
        max_instances: 0,
        migration_fn: Some(mount_ns_migrate),
        checker_id: None,
    };
    MOUNT_NS_TYPE_ID.set(register_tracked_type(ns_template)
        .expect("MountNamespace descriptor registration failed"))
        .expect("MountNamespace type_id already set");
}

mount_migrate and mount_ns_migrate follow the task_migrate field-copy template (Section 13.18): copy preserved fields from the old layout, initialize fields added by the new layout from schema defaults, return Err(MigrationError::Incompatible) on an irreconcilable retype. migration_fn = None (Extension-Array-only) is NOT acceptable for these types: cache-line reordering of Mount's tree-linkage fields and retyping of propagation metadata are anticipated layout changes over a 50-year lifetime, and both require Shadow-and-Migrate. The initially registered functions are identity field-copies; each evolution payload ships its own replacement migration function.

14.6.6.2 Allocation Helpers

Every "Allocate a new Mount node" step in the algorithms below (mount_filesystem step 6d, do_bind_mount step 4a, copy_tree steps 3a/4b, fsmount step 2, open_tree OPEN_TREE_CLONE) allocates through mount_alloc(); copy_tree step 1 allocates the namespace through mount_ns_alloc():

/// Allocate an uninitialized Mount from Nucleus tracked storage and bridge
/// it to the kernel-wide Arc<Mount> handle. The caller MUST initialize every
/// field before publishing the Arc (inserting into the hash table, a
/// children list, or mount_list).
fn mount_alloc() -> Result<Arc<Mount>, Errno> {
    let ptr: TrackedPtr<Mount> = alloc_tracked::<Mount>()
        .map_err(|e| match e {
            // Per-type instance budget exhausted (see registration above).
            AllocError::OutOfInstances => Errno::ENOMEM,
            // Tracked storage region exhausted. The buddy/slab-only variants
            // are unreachable from the tracked allocator but are listed
            // explicitly (no wildcard) so a future AllocError variant forces
            // a compile error rather than a silent ENOMEM.
            AllocError::OutOfStorage
            | AllocError::OutOfMemory
            | AllocError::CgroupLimit
            | AllocError::CacheDraining
            | AllocError::TooLarge
            | AllocError::WouldSleep => Errno::ENOMEM,
        })?;
    // SAFETY: ptr is a valid, exclusive TrackedPtr freshly returned by
    // alloc_tracked; ownership moves into the Arc. The last strong-ref
    // drop invokes free_tracked::<Mount>().
    Ok(unsafe { Arc::from_tracked(ptr) })
}

/// Same pattern for MountNamespace. OutOfInstances is unreachable
/// (max_instances = 0); OutOfStorage maps to ENOMEM.
fn mount_ns_alloc() -> Result<Arc<MountNamespace>, Errno>;

Both failure arms map to ENOMEM deliberately: from userspace, budget exhaustion is indistinguishable from any other kernel memory exhaustion, and mount(2)/clone(2) document ENOMEM for exactly this case. The FMA framework (Section 20.1) records the distinct OutOfInstances cause so operators can tell "raise umka.vfs.mount_slots" apart from genuine memory pressure.

14.6.6.3 Teardown Ordering (RCU)

Path resolution crosses mount points via reference-less RCU lookups in the mount hash table, so a Mount's tracked slot must never be reused while such a reader can still hold its pointer. The teardown paths (do_umount steps 7/13b, do_umount_tree steps 3b/5b, destroy_mount_namespace steps 3b/4b) therefore remove the mount from the mount hash table first — the hash chain links are non-owning, so this only unpublishes the mount (new readers can no longer find it) — and then release the Shadow Mount Registry's owning Arc<Mount> reference from an rcu_call callback at the shadow-transaction commit — the drop, and thus any possible free_tracked::<Mount>(), happens only after a full grace period. Holders of strong references (open files via mnt_count, MountDentry values, lazy-unmount remnants) extend the lifetime beyond that point safely; free_tracked runs at the true final drop.

14.6.7 DCACHE_MOUNTED Integration

The dentry cache (Section 14.1) must track which dentries are mount points. When a filesystem is mounted at a dentry, the VFS sets the DCACHE_MOUNTED flag on that dentry. During path resolution (Section 14.1), when the VFS encounters a dentry with DCACHE_MOUNTED set, it calls mnt_ns.hash_table.lookup() (where mnt_ns is the current task's mount namespace) to find the child mount and continues resolution from the child mount's root dentry.

/// Dentry cache entry flags. Stored in the dentry's `flags: AtomicU32` field.
/// Extended to include DCACHE_MOUNTED for mount-point detection.
bitflags! {
    #[repr(transparent)]
    pub struct DcacheFlags: u32 {
        /// This dentry is a mount point — a filesystem is mounted on it.
        /// Set by `mount_filesystem()` when attaching a mount. Cleared by
        /// `do_umount()` when the last mount at this dentry is removed.
        ///
        /// Path resolution checks this flag on every path component.
        /// When set, `mnt_ns.hash_table.lookup(current_mount.mount_id, dentry)`
        /// is called to find the child mount. This check is a single atomic
        /// load (~1 cycle) — the flag exists specifically to avoid a hash
        /// table lookup on every path component (only mount points need
        /// the lookup).
        const DCACHE_MOUNTED       = 1 << 0;

        /// Dentry has been disconnected from the tree (e.g., NFS stale
        /// handle, deleted directory that is still open).
        const DCACHE_DISCONNECTED  = 1 << 1;

        /// Dentry is a negative dentry (caches a failed lookup).
        const DCACHE_NEGATIVE      = 1 << 2;

        /// Dentry has filesystem-specific operations (d_revalidate, etc.).
        const DCACHE_OP_MASK       = 1 << 3;
    }
}

14.6.8 Filesystem Context (New Mount API)

The new mount API (Linux 5.2+, used increasingly by container runtimes and systemd) separates mount operations into discrete steps: context creation, configuration, superblock creation, and attachment. This provides better error reporting (errors at each step, not a single mount(2) errno) and supports atomic mount configuration changes.

/// Filesystem context for the new mount API.
///
/// Created by `fsopen()`, configured by `fsconfig()`, and consumed by
/// `fsmount()`. The context holds all the state needed to create a new
/// superblock and mount, accumulated through multiple `fsconfig()` calls.
///
/// This is equivalent to Linux's `struct fs_context`.
///
/// **Lifetime**: The context is reference-counted via a file descriptor
/// returned by `fsopen()`. It is destroyed when the file descriptor is
/// closed. If `fsmount()` has not been called, the context is simply
/// freed (no mount created). If `fsmount()` was called, the context's
/// state has been consumed and the mount exists independently.
/// Maximum mount options (key-value pairs) across `options` and
/// `binary_options` combined. `fsconfig()` returns `ENOSPC` when
/// `options.len() + binary_options.len() >= FS_CONTEXT_MAX_OPTIONS`.
pub const FS_CONTEXT_MAX_OPTIONS: usize = 256;

/// Maximum error-log size in bytes for an `FsContext`. This is a UmkaOS
/// internal design parameter; Linux has NO constant of this name. Linux's
/// error log (`struct fc_log` in `include/linux/fs_context.h`) is instead a
/// ring of eight message POINTERS — `char *buffer[8]` with `head`/`tail`
/// ring indices and a `need_free` bitmask marking which entries must be
/// freed — where each message is a separate variable-length
/// Linux formats each allocation with `logfc()` (`fs/fs_context.c`) via
/// Linux `kasprintf(GFP_KERNEL, ...)`, falling back to the static string
/// `"OOM: Can't store error string"` when that allocation fails. UmkaOS
/// instead uses one bounded byte buffer: no per-message heap allocation, and
/// therefore no OOM-fallback message. `fc_log_write()` checks
/// `log.len() + msg.len() <= FS_CONTEXT_LOG_SIZE` before appending; excess
/// bytes are silently truncated. Only the log CONTENT read back via `read()`
/// on the fscontext fd is observable to userspace — the in-kernel storage
/// shape (byte buffer vs pointer ring) is not part of the ABI.
pub const FS_CONTEXT_LOG_SIZE: usize = 4096;

pub struct FsContext {
    /// Filesystem type (e.g., "ext4", "tmpfs", "overlay"). Set at
    /// `fsopen()` time and immutable thereafter.
    pub fs_type: Arc<dyn FileSystemOps>,

    /// Filesystem type name (for diagnostics and /proc/mounts).
    pub fs_type_name: Box<[u8]>,

    /// Source device or path (equivalent to mount(2) `source` parameter).
    /// Set via `fsconfig(FSCONFIG_SET_STRING, "source", ...)`.
    pub source: Option<Box<[u8]>>,

    /// Accumulated mount options as key-value pairs. Each `fsconfig()`
    /// call adds or modifies an entry. The filesystem driver validates
    /// options at `fsconfig(FSCONFIG_CMD_CREATE)` time.
    /// Bounded by FS_CONTEXT_MAX_OPTIONS (256 total across `options` and
    /// `binary_options`). `fsconfig()` returns `ENOSPC` when the combined
    /// count reaches the limit. Cold-path allocation (mount/remount only).
    pub options: Vec<(Box<[u8]>, Box<[u8]>)>,

    /// Binary data options (for filesystems that accept binary mount data).
    /// Set via `fsconfig(FSCONFIG_SET_BINARY, ...)`.
    /// Shares the `FS_CONTEXT_MAX_OPTIONS` limit with `options`.
    pub binary_options: Vec<(Box<[u8]>, Box<[u8]>)>,

    /// Mount flags to apply to the created mount.
    pub mount_flags: MountFlags,

    /// The created superblock. Set by `fsconfig(FSCONFIG_CMD_CREATE)`,
    /// consumed by `fsmount()`.
    pub superblock: Option<Arc<SuperBlock>>,

    /// Error log. Filesystem drivers write diagnostic messages here
    /// during context creation and configuration. Readable by userspace
    /// via `read()` on the fscontext file descriptor.
    /// Bounded to FS_CONTEXT_LOG_SIZE (4096) bytes. Truncated silently when full.
    /// Cold-path allocation (mount error reporting only).
    pub log: Vec<u8>,

    /// Purpose of this context: new mount, reconfiguration, or submount.
    pub purpose: FsContextPurpose,

    /// Lifecycle state of this context. Transitions: New → Configured →
    /// Consumed (by `fsmount()`). Further `fsconfig()` calls on a Consumed
    /// context return `EBUSY`.
    pub state: FsContextState,

    /// User namespace for permission checks. Set at `fsopen()` time
    /// to the caller's user namespace.
    pub user_ns: Arc<UserNamespace>,
}

/// Purpose of a filesystem context, controlling which operations are valid.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
#[repr(u8)]
pub enum FsContextPurpose {
    /// Creating a new mount (from `fsopen()`).
    NewMount = 0,
    /// Reconfiguring an existing mount (from `fspick()`).
    Reconfig = 1,
    /// Internal: creating a submount (e.g., automount).
    Submount = 2,
}

/// Lifecycle state of an `FsContext`.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
#[repr(u8)]
pub enum FsContextState {
    /// Freshly created by `fsopen()` or `fspick()`. Accepting `fsconfig()` calls.
    New         = 0,
    /// Options have been set via `fsconfig()`, but `FSCONFIG_CMD_CREATE`/
    /// `FSCONFIG_CMD_RECONFIGURE` has not yet been called.
    Configuring = 1,
    /// `FSCONFIG_CMD_CREATE` succeeded; superblock is ready. Awaiting `fsmount()`.
    Created     = 2,
    /// `fsmount()` has consumed the superblock. The fsopen fd is still open
    /// for error log retrieval but cannot create another mount.
    Consumed    = 3,
    /// An error occurred during creation. The error log is readable.
    /// Further `fsconfig()` calls return `EBUSY`.
    Failed      = 4,
}

14.6.8.1 FsContext Lifecycle and Error Channel

The new mount API separates mount configuration into discrete, verifiable steps. Each step either advances the context state or returns a structured error. The full lifecycle:

Step 1: fd = fsopen("ext4", FSOPEN_CLOEXEC)
  → Validates "ext4" against the filesystem type registry.
  → Allocates FsContext { fs_type: ext4_ops, purpose: NewMount, state: New, ... }.
  → Returns an O_RDWR file descriptor backed by the FsContext.
  → FsContext state: New.

Step 2: fsconfig(fd, FSCONFIG_SET_STRING, "source", "/dev/sda1", 0)
        fsconfig(fd, FSCONFIG_SET_STRING, "errors",  "remount-ro",  0)
        fsconfig(fd, FSCONFIG_SET_FLAG,   "noatime", NULL,          0)
  → Each call appends to FsContext.options: [("source", "/dev/sda1"), ("errors", "remount-ro"), ...].
  → Returns 0 on success; EINVAL if the key is not recognized by the filesystem type.
  → First `fsconfig()` call transitions state: New → Configuring.
  → FsContext state: Configuring (still accumulating options).

Step 3: fsconfig(fd, FSCONFIG_CMD_CREATE, NULL, NULL, 0)
  → Calls FileSystemOps::mount(source, flags, options) on the configured filesystem type.
  → On success: FsContext.superblock = Some(sb); state → Created.
  → On failure: diagnostic message is written to FsContext.log; state → Failed.
    Caller can read the error via read(fd, buf, len) — see Error Channel below.
  → Returns 0 on success; -errno on failure.

Step 4: mnt_fd = fsmount(fd, FSMOUNT_CLOEXEC, MOUNT_ATTR_NOATIME)
  → Consumes FsContext.superblock (state must be Created; returns EBUSY if
    Consumed, EINVAL if New or Failed).
  → Allocates a MountNode with MNT_DETACHED flag set.
  → Returns an O_PATH fd referencing the detached mount.
  → FsContext state: Consumed (further fsconfig/fsmount calls return EBUSY).

Step 5: move_mount(mnt_fd, "", AT_FDCWD, "/mnt/data", MOVE_MOUNT_F_EMPTY_PATH)
  → Attaches the detached mount to the namespace mount tree at /mnt/data.
  → Clears MNT_DETACHED from the MountNode.
  → Triggers mount propagation to peer/slave mounts (Section 14.2.10).

open_tree(2) — clone or open a mount:

fd = open_tree(dirfd, path, OPEN_TREE_CLONE | AT_RECURSIVE)
  → Resolves path to a mount.
  → OPEN_TREE_CLONE: creates a detached copy of the mount tree rooted at path,
    identical to a recursive bind mount but without modifying the namespace.
    AT_RECURSIVE: the clone includes all submounts below path.
  → The returned O_PATH fd can be passed to move_mount() to attach elsewhere.
  → Without OPEN_TREE_CLONE: returns an O_PATH fd referencing the existing mount
    without cloning (useful for passing a mount reference across namespaces).

mount_setattr(2) — bulk-modify mount tree flags:

mount_setattr(dirfd, path, AT_RECURSIVE, &mount_attr { attr_set, attr_clr }, sizeof)
  → Resolves path to a mount.
  → AT_RECURSIVE: applies to all mounts in the subtree rooted at path.
  → attr_clr: clears these flags from each mount (applied first).
  → attr_set: sets these flags on each mount (applied after attr_clr).
  → The operation is atomic within the subtree: if validation fails for any mount
    (e.g., clearing MNT_READONLY on a superblock-level read-only filesystem), no
    flags are changed on any mount.
  → Requires CAP_MOUNT.

FsContext Error Channel:

When fsconfig(FSCONFIG_CMD_CREATE) or fsmount() encounter a filesystem-level error (e.g., superblock checksum mismatch, missing required option, device I/O error), the error is not conveyed solely via errno. The filesystem driver writes a human-readable diagnostic string to FsContext.log. The caller retrieves it via read(fd, buf, len) on the FsContext file descriptor:

read(fs_context_fd, buf, len):
  if FsContext.log is empty: return 0 (EOF — no error message pending)
  n = min(len, FsContext.log.len())
  copy_to_user(buf, FsContext.log[..n])
  FsContext.log.drain(..n)
  return n

Example error message (readable by system administrators):

ext4: superblock checksum mismatch at block 0: expected 0xdeadbeef, got 0xcafebabe

This approach is superior to the traditional single-errno response: it gives system administrators and container runtimes actionable diagnostic information without requiring a separate diagnostics ioctl or /proc file.

14.6.9 Mount Attribute Structure (mount_setattr)

/// User-visible mount attribute structure for `mount_setattr(2)`.
/// Matches Linux's `struct mount_attr` exactly for ABI compatibility.
///
/// `mount_setattr()` atomically modifies mount properties on a single
/// mount or recursively on an entire mount tree (when `AT_RECURSIVE`
/// is passed). Container runtimes use this for recursive read-only
/// mounts (`MOUNT_ATTR_RDONLY` + `AT_RECURSIVE`).
#[repr(C)]
pub struct MountAttr {
    /// Flags to set on the mount(s). Bits correspond to `MOUNT_ATTR_*`
    /// constants. Applied after `attr_clr` (clear first, then set).
    pub attr_set: u64,

    /// Flags to clear from the mount(s). Applied before `attr_set`.
    pub attr_clr: u64,

    /// Propagation type to set. One of `MS_SHARED`, `MS_PRIVATE`,
    /// `MS_SLAVE`, `MS_UNBINDABLE`, or 0 (no change). Only one
    /// propagation flag may be set; combining them returns `EINVAL`.
    /// The mount_setattr handler validates `attr.propagation` is a valid
    /// PropagationType variant (0-3); returns EINVAL on invalid values.
    pub propagation: u64,

    /// File descriptor of the user namespace to associate with the
    /// mount (for ID-mapped mounts). Set to 0 or omit if not
    /// changing the mount's user namespace mapping.
    pub userns_fd: u64,
}
// Layout: 4 × u64 = 32 bytes.
const_assert!(size_of::<MountAttr>() == 32);

/// MOUNT_ATTR_* flag constants for mount_setattr(2).
/// These map to MountFlags but use a separate constant space matching
/// Linux's UAPI.
pub const MOUNT_ATTR_RDONLY: u64      = 0x00000001;
pub const MOUNT_ATTR_NOSUID: u64      = 0x00000002;
pub const MOUNT_ATTR_NODEV: u64       = 0x00000004;
pub const MOUNT_ATTR_NOEXEC: u64      = 0x00000008;
pub const MOUNT_ATTR_NOATIME: u64     = 0x00000010;
pub const MOUNT_ATTR_STRICTATIME: u64 = 0x00000020;
pub const MOUNT_ATTR_NODIRATIME: u64  = 0x00000080;
pub const MOUNT_ATTR_NOSYMFOLLOW: u64 = 0x00200000;

14.6.10 Mount Operations — Algorithms

All mount tree modification algorithms require holding the namespace's mount_lock (lock hierarchy level 20, Section 3.5). Path resolution (read path) uses only RCU and never acquires mount_lock. The algorithms below describe the kernel-internal implementation; the syscall entry points (mount(2), umount2(2), and the new mount API) perform argument validation and capability checks before calling these internal functions.

14.6.10.1 Shadow Registry Reporting

Every mount-tree transaction is bracketed by a report to the Core-domain Shadow Mount Registry (Section 14.1), which holds the OWNING Arc<Mount> reference of every namespace-attached mount plus the Core-resident record from which crash recovery rebuilds the tree. The registry is updated ONLY through this bracket — authority-confirmed transitions reported by this engine at commit time. It is maintained at EVERY tier: tier assignment is runtime-changeable, so a mount created while the VFS module shares the Core domain must be recoverable after a later demotion and crash. When the engine is co-located with Core, kabi_call! resolves these calls to direct calls; all callers are warm paths.

/// Opaque transaction token. One per mount-tree transaction; monotonic
/// u64, never reused.
pub struct ShadowTxnToken(u64);

/// Staging payload for one mount addition. Field-for-field the
/// `ShadowMountEntry` it becomes at commit
/// ([Section 14.1](#virtual-filesystem-layer--shadow-mount-registry-and-mount-tree-reconstruction)),
/// except that the superblock is named by IDENTITY (`sb`), never by the
/// registry-internal `sb_key` — the engine does not see or track sb_keys.
/// Staging resolves the identity through the registry's `sb_index` and
/// rejects (EINVAL) an identity that is neither live nor staged earlier in
/// the same transaction. Core-internal (never crosses a KABI, wire, or
/// userspace boundary): ordinary Rust struct, exact-copy heap strings,
/// NOT `#[repr(C)]`.
pub struct ShadowMountRecord {
    /// Owning namespace (`MountNamespace.ns_id`).
    pub ns_id: u64,
    /// `Mount.mount_id`; must be fresh within the namespace.
    pub mount_id: u64,
    /// Parent within the same namespace; 0 = namespace root.
    pub parent_mount_id: u64,
    /// `Arc` clone of the engine's new instance; becomes THE owning
    /// tree-side reference when the transaction commits.
    pub mount: Arc<Mount>,
    /// The superblock this mount is a view of, by identity. Resolved to
    /// the shadow record's `sb_key` at staging time.
    pub sb: Arc<SuperBlock>,
    /// Mountpoint path relative to the parent mount's root (≤ PATH_MAX,
    /// exact copy). Empty for the namespace root.
    pub mountpoint: Box<[u8]>,
    /// Mount root relative to the superblock root (non-empty for bind
    /// mounts of subdirectories); empty for a full-filesystem mount.
    pub root_path: Box<[u8]>,
    /// `MountFlags` bits.
    pub flags: u64,
    /// Propagation type.
    pub propagation: PropagationType,
    /// Peer group, `(allocating ns_id, group_id)` encoding; `(0, 0)` = none.
    pub peer_group: (u64, u64),
    /// Master peer group for slaves, same encoding; `(0, 0)` = not a slave.
    pub master_group: (u64, u64),
}

/// Staging payload for one mount modification. `None` = field unchanged;
/// commit overwrites exactly the `Some` fields of the target
/// `ShadowMountEntry`. Core-internal, so `Option` is fine here (this is
/// not a `#[repr(C)]` struct).
pub struct ShadowMountDelta {
    /// Re-parenting (`do_move_mount`, `pivot_root`).
    pub parent_mount_id: Option<u64>,
    /// New mountpoint path, relative to the (possibly new) parent's root.
    pub mountpoint: Option<Box<[u8]>>,
    /// Flag update (`do_remount`, `mount_setattr`).
    pub flags: Option<u64>,
    /// Propagation update (`do_change_propagation`, `mount_setattr`).
    pub propagation: Option<PropagationType>,
    /// Peer-group update, `(allocating ns_id, group_id)` encoding.
    pub peer_group: Option<(u64, u64)>,
    /// Master-group update, same encoding.
    pub master_group: Option<(u64, u64)>,
}

/// Staging payload for one superblock record: the engine-supplied fields
/// of `ShadowSuperblockRecord` (the registry assigns `sb_key` and manages
/// `refs` itself). Staging is IDEMPOTENT by superblock identity: if a
/// record for `sb` already exists — live, or staged earlier in any open
/// transaction — the call is a no-op returning `Ok(())`. The engine
/// therefore stages the superblock UNCONDITIONALLY in `mount_filesystem` (both
/// step-6b arms) and never tracks whether a reused superblock still has a
/// record. That matters: a lazily-unmounted superblock can outlive its
/// shadow record (all shadow entries removed at the MNT_DETACH commit →
/// `refs == 0` → record released, while open fds keep `s_refcount > 0`
/// and the superblock stays in `sb_table`), so the reuse arm of a later
/// mount cannot assume the record exists.
pub struct ShadowSbRecord {
    /// Pinning reference; also the identity key for deduplication.
    pub sb: Arc<SuperBlock>,
    /// Filesystem type string, exact copy (e.g. "ext4", "fuse.sshfs").
    pub fstype: Box<[u8]>,
    /// Backing device (`SuperBlock.s_dev`; anonymous dev for diskless).
    pub device: DevId,
    /// Device name string as passed to mount ("/dev/sda1", "tmpfs", ...).
    pub device_name: Box<[u8]>,
    /// Filesystem-specific mount data, exact copy (≤ one page — the
    /// `copy_mount_options()` bound).
    pub fs_data: Box<[u8]>,
}

/// Open a shadow transaction. Fails only on Core-side allocation failure —
/// the engine maps it to ENOMEM and fails the operation before any tree
/// mutation.
fn shadow_txn_begin() -> Result<ShadowTxnToken, Errno>;

/// Stage one record into an open transaction. Every staging call ALLOCATES
/// on the Core side and VALIDATES structurally (parent exists or is being
/// staged, the add-record's `sb` identity resolves to a live or staged
/// superblock record, path lengths within the syscall bounds, mount_id
/// fresh for adds / present for removes and modifies, `mount` points into
/// live Nucleus tracked storage for the `Mount` type). ENOMEM / EINVAL
/// surface HERE, while the engine can still roll the transaction back —
/// never at commit. Staging converts an accepted `ShadowMountRecord` into
/// the `Box<ShadowMountEntry>` it will link at commit (resolving `sb` →
/// `sb_key`), so commit performs no allocation and no validation — see
/// `ShadowTxn`/`ShadowOp` in
/// [Section 14.1](#virtual-filesystem-layer--shadow-mount-registry-and-mount-tree-reconstruction).
fn shadow_stage_mount_add(t: &ShadowTxnToken, rec: ShadowMountRecord) -> Result<(), Errno>;
fn shadow_stage_mount_remove(t: &ShadowTxnToken, ns_id: u64, mount_id: u64) -> Result<(), Errno>;
fn shadow_stage_mount_modify(t: &ShadowTxnToken, ns_id: u64, mount_id: u64,
                             delta: ShadowMountDelta) -> Result<(), Errno>;
fn shadow_stage_sb_add(t: &ShadowTxnToken, rec: ShadowSbRecord) -> Result<(), Errno>;
fn shadow_stage_ns_add(t: &ShadowTxnToken, ns_id: u64, ns: Arc<MountNamespace>) -> Result<(), Errno>;
fn shadow_stage_ns_remove(t: &ShadowTxnToken, ns_id: u64) -> Result<(), Errno>;

/// Atomically apply all staged records. INFALLIBLE: only links
/// pre-allocated records into the registry, drops removed entries' owning
/// Arcs from an `rcu_call` callback (after the grace period — the RCU
/// teardown ordering contract), and releases superblock records whose
/// reference count reaches zero. Called AFTER the tree mutation has
/// committed, while still holding the lock(s) that serialized the
/// transaction, and BEFORE success returns to the caller.
fn shadow_txn_commit(t: ShadowTxnToken);

/// Drop all staged records (with their Arc clones). Called on every
/// engine-side failure/rollback arm after `shadow_txn_begin()`.
fn shadow_txn_abort(t: ShadowTxnToken);

Bracket discipline (applies to every algorithm in this section):

1. shadow_txn_begin() before the first tree mutation.
2. Stage each created/removed/modified mount AS the transaction constructs it — propagation clones (propagate_mount), recursive-bind clones, and copy_tree clones are each staged individually, so the delta is exactly the set of mounts the transaction touched, computed by the code that touched them. A staging failure (ENOMEM) triggers the operation's normal rollback (propagation is already all-or-nothing) plus shadow_txn_abort().
3. shadow_txn_commit() after the mutation commits, under the serializing lock(s), before returning success.
4. Detached mounts (MNT_DETACHED: fsmount() pre-attach, OPEN_TREE_CLONE) are NOT staged — their owner is the referencing fd's Arc, and they do not survive a VFS-module crash. move_mount() attaching a detached mount stages it then.

Because the shadow entry is the owning reference, a transaction that attaches a mount without staging it leaves that mount ownerless — it is freed at the next grace period and the omission surfaces as an immediate use-after-free under test, not as silent recovery divergence after a crash.

Touchpoints:

Operation	Stages
mount_filesystem (new fs)	sb add (unconditional — idempotent by identity, no-op when the reuse arm's superblock still has a live record) + mount add + one add per propagation clone
do_umount / do_umount_tree	mount remove per unmounted mount (incl. MNT_DETACH — detach removes tree membership) + propagated umounts
do_remount	mount modify (flags) + sb record fs_data/flags refresh
do_bind_mount	mount add per clone (recursive bind: one per cloned sub-mount)
do_move_mount	mount modify (parent_mount_id, mountpoint path)
do_change_propagation	mount modify (propagation, peer_group, master_group) per affected mount
mount_setattr	mount modify per affected mount (recursive: all, atomically staged)
copy_tree	ns add + mount add per cloned mount
pivot_root	mount modify for the two re-parented mounts (root swap)
destroy_mount_namespace	ns remove (drops every entry of the namespace)
autofs expiry (MNT_SHRINKABLE)	mount remove
fsmount / open_tree(OPEN_TREE_CLONE)	nothing (detached — rule 4)
move_mount of a detached mount	mount add

14.6.10.2 mount_filesystem — Mount a Filesystem

mount_filesystem(source, target_path, fs_type, flags, data) -> Result<()>

  0a. Capability check: verify caller holds CAP_MOUNT ([Section 9.1](09-security.md#capability-based-foundation))
      in the target mount namespace. Return EPERM if not held.
  0b. LSM hook: `lsm_call_superblock_security(Mount, cred, sb, &SbOpContext { ... })`.
      If the LSM denies the mount request, return EPERM. This hook fires
      before any path resolution to allow early rejection of unauthorized
      mount operations (e.g., SELinux `mount` permission check against the
      caller's security context and the target path label).
  0c. (Cgroup device-controller check — DEFERRED to step 6a2, not performed
      here.) The check gates access to the SOURCE block device, but `source_dev`
      does not exist until path resolution (step 1) and the FS-type flag lookup
      (step 6a2) derive it from `source`; and an FS_NO_DEV filesystem has NO
      source device to gate at all. Running the check here with an
      undefined/absent `source_dev` was an ordering bug — the device-controller
      gate is applied in step 6a2 immediately after `source_dev` is resolved,
      still BEFORE `FileSystemOps::mount` opens the device (step 6b), so early
      rejection of an inaccessible container device is preserved.

  1. Resolve `target_path` to (mount, dentry) via path resolution (Section 14.1.3).
  2. If `flags` contains MS_REMOUNT, delegate to do_remount() (Section 14.2.9.4).
  3. If `flags` contains MS_BIND, delegate to do_bind_mount() (Section 14.2.9.5).
  4. If `flags` contains MS_MOVE, delegate to do_move_mount() (Section 14.2.9.6).
  5. If `flags` contains MS_SHARED|MS_PRIVATE|MS_SLAVE|MS_UNBINDABLE,
     delegate to do_change_propagation() (Section 14.2.9.7).
  6. Otherwise, this is a new filesystem mount:
     a. Look up the filesystem type by name in the filesystem registry.
        If not registered, return ENODEV.
     a2. **Source resolution by filesystem-type flags** — this is the single
         canonical enforcement site for `FsTypeFlags` FS_REQUIRES_DEV /
         FS_NO_DEV ([Section 17.1](17-containers.md#namespace-architecture--capability-domain-mapping),
         where the flag semantics and per-fs_type mutual exclusion are defined;
         mutual exclusion is rejected when the filesystem type enters the registry):
         - If `fs_type.flags` contains FS_REQUIRES_DEV (ext4, XFS, btrfs, …):
           `source` MUST name a block device. If `source` is absent → ENODEV;
           if it resolves to a path that is NOT a block device → ENOTBLK.
           Resolve it to the block device `source_dev` and pass THAT to the
           mount routine below.
           **Cgroup device-controller gate** (the deferred step-0c check, now
           that `source_dev` is known): if the calling task's cgroup has a device
           controller with a `BPF_CGROUP_DEVICE` program attached, call
           `cgroup_device_permitted(true, source_dev.major(), source_dev.minor(),
           true, true)`; if denied, return EPERM.
           This runs on the RESOLVED `source_dev`, before `FileSystemOps::mount`
           (step 6b) opens it — so an inaccessible container device is rejected
           before any on-disk access.
         - If `fs_type.flags` contains FS_NO_DEV (proc, sysfs, tmpfs, cgroup2,
           devpts, …): `source` is IGNORED — no device is resolved, and any
           `source` string is accepted but unused.
         - Otherwise (neither flag — e.g. a network filesystem): `source` is
           passed through opaquely (the driver interprets it, e.g. as a
           `host:/export` spec).
     b. **Resolve or create the superblock.** This is the canonical
        superblock-deduplication step. Two mounts of the same block device
        MUST share one `SuperBlock`: two independent superblocks over one
        device are two uncoordinated views of the same on-disk metadata —
        journal state, allocation bitmaps, and inode tables diverge and the
        filesystem corrupts. (Linux compat anchor: `fs/super.c`
        Linux `get_tree_bdev()`/`sget_fc()` deduplicate by block device,
        torvalds/linux master.)
        - **FS_REQUIRES_DEV**: look up `source_dev` in the filesystem
          type's superblock table — each filesystem registry entry carries

          /// Live superblocks of this filesystem type, keyed by backing
          /// DevId. Insert/remove under FS_TYPE_SB_LOCK (a per-fs_type
          /// sleeping Mutex; mount/umount are warm paths). FS_NO_DEV
          /// types leave it empty.
          pub sb_table: XArray<Arc<SuperBlock>>,

          - **Found**: verify flag compatibility (`MS_RDONLY` vs an
            existing read-write mount of the same device → EBUSY, matching
            the mount-lifecycle summary
            [Section 14.1](#virtual-filesystem-layer--mount-lifecycle)). On success,
            increment `s_refcount` and REUSE the superblock —
            `FileSystemOps::mount` is NOT called (the filesystem instance
            is already live).
          - **Not found**: call `FileSystemOps::mount(source, flags, data)`
            on the filesystem driver (`source` = the block device resolved
            in step 6a2). This creates and returns a `SuperBlock` with
            `s_refcount = 1`; insert it into `sb_table`. On failure, return
            the error from the driver.
        - **FS_NO_DEV / opaque-source**: always call
          `FileSystemOps::mount(source, flags, data)`; the driver decides
          internally whether to return a fresh instance (tmpfs — every
          mount is a new filesystem) or an existing one (procfs — one
          instance per PID namespace), mirroring Linux `get_tree_nodev()`
          versus Linux `get_tree_keyed()`. A driver returning an existing superblock
          increments `s_refcount` before returning.

        **Superblock release rule** (used by every error arm below and by
        `do_umount` step 16): decrement `s_refcount`; ONLY if it reaches
        zero, remove from `sb_table` (if present), call
        `FileSystemOps::unmount(sb)`, and free the superblock. In the
        reuse arm the superblock is still serving other live mounts — an
        unconditional teardown would rip a filesystem out from under them.
     b2. LSM hook: `lsm_call_superblock_security(superblock, source, flags, data)`.
         If the LSM denies the mount (returns non-zero), release the
         superblock reference (rule above) and return EPERM. This hook
         allows SELinux/AppArmor to enforce mount restrictions based on the
         filesystem type, source device, and mount options.
     c. Check namespace mount count against `mount_max` limit. If exceeded,
        release the superblock reference and return ENOSPC.
     d. Allocate a new `Mount` node via `mount_alloc()` (Nucleus tracked
        storage — see "Tracked Allocation — Mount and MountNamespace").
        On ENOMEM (instance budget or tracked storage exhausted), release
        the superblock reference and return ENOMEM. Initialize every field:
        - `mount_id` from `namespace.id_allocator.fetch_add(1)`
        - `root` = superblock's root dentry
        - `superblock` = the SuperBlock from step 6b
        - `flags` = translate MS_* to MountFlags
        - `propagation` = Private (default for new mounts)
        - `group_id` = 0 (private mount has no peer group)
        - `mnt_count` = 0
        `parent` and `mountpoint` are NOT set here — `attach_mount` sets
        them from the location it is given (step 6e).
     e. Attach the node into the tree:
        `attach_mount(mount, &at)`, where `at` is the (parent mount,
        mountpoint dentry) location resolved in step 1
        ([Section 14.6](#mount-tree-data-structures-and-operations--attachmount-attach-a-constructed-mount-into-the-tree)).
        This performs the entire canonical attachment sequence: the shadow
        bracket, mountpoint accounting, hash/children/`mount_list`
        insertion, namespace counters, propagation to the parent's peers,
        and commit. On error (ENOMEM from staging or from propagation) the
        transaction is already rolled back and the mount is still
        unattached and owned by this caller's `Arc`: release the superblock
        reference (rule in step 6b), drop the node, and return the error.

14.6.10.3 attach_mount — Attach a Constructed Mount into the Tree

attach_mount is the single canonical attachment primitive: every path that inserts a constructed-but-unattached Mount into a namespace goes through it. Callers are mount_filesystem (step 6e), move_mount attaching a detached mount, and follow_automount attaching a kernel-constructed submount (Section 14.10). Because the shadow bracket lives INSIDE this function, it is structurally impossible for a caller to attach a mount without staging it — the ownerless-mount failure mode described under "Bracket discipline" cannot be reached from a conforming caller.

/// Attach a constructed-but-unattached `Mount` into the mount tree at
/// `at` (parent mount + mountpoint dentry), in `at`'s namespace.
///
/// Sets `mount.parent` and `mount.mountpoint` from `at`, then performs the
/// canonical attachment sequence:
///
/// 1. Open the shadow bracket
///    ([Section 14.6](#mount-tree-data-structures-and-operations--shadow-registry-reporting)):
///    `shadow_txn_begin()`, stage the superblock record (unconditionally —
///    staging is idempotent by superblock identity) and the mount-add
///    record (with an `Arc` clone of `mount`, which becomes the owning
///    reference at commit), per bracket-discipline rules 1-3.
/// 2. Acquire `mount_lock`, and under it:
///    - increment the mountpoint dentry's `d_mount_refcount` (Relaxed) and
///      set `DCACHE_MOUNTED` in `d_flags` (Release);
///    - insert into the mount hash table at
///      `bucket(parent_mount_id, mountpoint_inode_id)`;
///    - add to the parent's `children` list;
///    - add to the namespace's `mount_list`, immediately after the parent's
///      list entry;
///    - increment `namespace.mount_count`;
///    - if the parent mount is shared, run `propagate_mount()`
///      ([Section 14.6](#mount-tree-data-structures-and-operations--propagatemount)),
///      staging every clone the propagation walk creates into the same
///      transaction as it is created;
///    - increment `namespace.event_seq`;
///    - `shadow_txn_commit()` — still under `mount_lock`, so per-namespace
///      shadow ordering follows lock order;
///    - release `mount_lock`.
/// 3. Clear `MountFlags::MNT_DETACHED` if it was set.
///
/// # Errors
///
/// `ENOMEM` — shadow staging or propagation failed. The all-or-nothing
/// propagation rollback (`propagate_mount` step 3) plus `shadow_txn_abort()`
/// run first, so on return the mount is UNATTACHED and still owned by the
/// caller's `Arc`, and the tree is exactly as it was on entry.
pub fn attach_mount(mount: Arc<Mount>, at: &MountDentry) -> Result<(), Errno>;

Mountpoint accounting rationale (step 2, first bullet): the d_mount_refcount increment uses Relaxed ordering because it is protected by mount_lock; the flag set uses Release so RCU readers see it during path resolution. d_mount_refcount tracks how many mounts reference this dentry as their mountpoint (see Section 14.1); it is decremented by do_umount(), and DCACHE_MOUNTED is cleared only when it reaches zero. Setting the flag without taking d_lock avoids a lock-ordering violation: mount_lock (level 20) is held, and d_lock (level 19) is lower. Writer-writer safety: both mount and umount hold the namespace's mount_lock before modifying DCACHE_MOUNTED; the atomic fetch_or/fetch_and are for reader visibility under RCU, not for writer synchronization.

mount_list position: parent-before-child ordering is the normal case, not a guaranteed global invariant — see the mount_list field doc under Section 14.6.

Policy flags are the producer's: attach_mount never sets policy bits. A filesystem that wants MNT_SHRINKABLE (expirable automounts) sets it on the Mount before handing it over.

14.6.10.4 do_umount — Unmount a Filesystem

do_umount(target_mount, flags) -> Result<()>

  Capability check: CAP_MOUNT in caller's mount namespace.

  1. If `target_mount` is the namespace root and flags does not contain
     MNT_DETACH, return EBUSY (cannot unmount root).
  2. If `target_mount.flags` has MNT_LOCKED and the caller lacks
     CAP_SYS_ADMIN in the mount's owning user namespace, return EPERM.
  3. If `flags` does not contain MNT_DETACH (not lazy):
     0. Pin-reconciliation gate: if the namespace's shadow entry has
        `pins_unreconciled == 1` (set by post-crash mount tree
        reconstruction), request the Core-side fd reconciliation sweep and
        wait for it — otherwise the step-3a `mnt_count` test would read an
        undercount and wrongly bypass EBUSY. See "Open-file pins after
        reconstruction" in
        [Section 14.1](#virtual-filesystem-layer--shadow-mount-registry-and-mount-tree-reconstruction).
        (The same gate applies to EVERY `mnt_count == 0` teardown test,
        including autofs expiry of MNT_SHRINKABLE mounts.)
     a. Check `target_mount.mnt_count`. If > 0, return EBUSY.
     b. Check that `target_mount.children` is empty. If not, return EBUSY
        (sub-mounts must be unmounted first, unless MNT_DETACH is used).
  4. If `flags` contains MNT_FORCE:
     a. Call `FileSystemOps::force_umount()` if the filesystem supports it.
        This causes in-flight I/O to fail with EIO. NFS uses this for stale
        server recovery.
  5. Acquire `mount_lock`.
  5b. Open the shadow bracket: `shadow_txn_begin()` + stage the removal of
      `target_mount` (and of every mount `propagate_umount` will remove —
      staged by the propagation walk in step 10 as it selects victims).
      On ENOMEM: `shadow_txn_abort()`, release `mount_lock`, return ENOMEM
      (no tree state has changed yet).
  6. Set `MNT_DOOMED` on `target_mount.flags` (atomic OR).
     This prevents new path lookups from entering the mount.
  7. Remove `target_mount` from the mount hash table. Hash chain links are
     NON-owning; the mount's owning `Arc<Mount>` lives in the Shadow Mount
     Registry and is released by the step-13b commit from an `rcu_call`
     callback, not synchronously — reference-less RCU readers crossing this
     mount point may still hold the pointer until a grace period elapses
     (see "Tracked Allocation — Mount and MountNamespace", Teardown
     Ordering).
  8. Remove `target_mount` from the parent's `children` list.
  9. Decrement the mountpoint dentry's mount refcount and conditionally
     clear `DCACHE_MOUNTED`:
     let mountpoint_dentry = target_mount.mountpoint;
     if mountpoint_dentry.d_mount_refcount.fetch_sub(1, AcqRel) == 1 {
         mountpoint_dentry.d_flags.fetch_and(!DCACHE_MOUNTED, Release);
     }
     AcqRel on the decrement: Acquire ensures we see all prior increments
     from other mount operations; Release ensures the flag clear is visible
     to RCU readers only after the refcount reaches zero. Multiple mounts
     can be stacked on the same dentry (cross-namespace or bind mounts);
     `DCACHE_MOUNTED` is cleared only when the last one is removed.
  10. Propagate: if the parent mount is shared, call `propagate_umount()`
      (Section 14.2.10.2) to remove corresponding mounts from peers and slaves.
  11. Remove from `namespace.mount_list`.
  12. Decrement `namespace.mount_count`.
  13. Increment `namespace.event_seq`.
  13b. `shadow_txn_commit()` — the shadow releases each removed mount's
       owning `Arc<Mount>` via `rcu_call` (grace-period-deferred, per the
       Teardown Ordering contract). This covers MNT_DETACH too: a detached
       mount leaves the namespace tree, so it leaves the shadow — from this
       point its owners are the fds that reference it, and it will NOT be
       reconstructed after a VFS-module crash.
  14. Release `mount_lock`.
  15. If `flags` contains MNT_DETACH (lazy unmount):
      a. The mount is now disconnected from the tree but may still be
         referenced by open file descriptors (mnt_count > 0). It will be
         fully freed when the last reference is dropped.
      b. Open files continue to work on the disconnected mount. New path
         lookups cannot reach it.
  16. Release the superblock reference per the release rule in `mount_filesystem`
      step 6b: decrement `s_refcount`; only at zero, remove from
      `sb_table` and call `FileSystemOps::unmount()` — synchronously if
      not lazy, deferred to the final reference drop if lazy (via a
      callback registered on the final `Arc::drop`). Bind mounts and
      multi-mounted superblocks thus release their share without tearing
      down a filesystem other mounts are still using.
      **Ring/recovery teardown at zero refcount (ring-backed mounts —
      `sb.ring_set` is `Some`; NORMATIVE reverse of mount step 4a).**
      After `FileSystemOps::unmount()` returns (the unmount call itself
      may still ride the ring) and BEFORE the `SuperBlock` is freed, the
      final releaser MUST tear down the Core-side ring machinery — the
      teardown the `SuperBlock` field docs promise
      ([Section 14.1](#virtual-filesystem-layer)) but that would otherwise run
      nowhere:
      a. Signal the per-mount Core response worker to exit:
         `sb.ring_worker_exit.store(1, Release)`, kick
         `ring_set.completion_doorbell`, and WAIT for the worker task's
         exit. After this no actor drains the response ring.
      b. Unregister the mount's crash-recovery descriptor:
         `unregister_recovery_descriptor(&sb.vfs_recovery_desc)` — which
         returns only after any in-flight recovery walk has drained
         ([Section 11.9](11-drivers.md#crash-recovery-and-state-preservation--subsystem-recovery-descriptors)),
         so no later domain-fault walk can invoke a descriptor whose
         `ctx` points at freed storage. (A recovery RUNNING at this
         point holds the domain `recovery_mutex`; the unregister's drain
         serializes against it.)
      c. Wake `sb.recovery_wait` (`wake_up_all()`): dispatchers parked
         on a boundary `ENXIO` re-check, observe the unmount, and fail
         with `ENODEV` — the umount arm of the field's wake contract.
      d. Only then free `sb.ring_core` and `sb.ring_set` and proceed to
         the `SuperBlock` teardown (inode eviction ordering per the
         canonical unmount flow in [Section 14.1](#virtual-filesystem-layer)).
      Without a-d, a post-free recovery walk dereferences the dead
      descriptor's `ctx` and the response worker races the `ring_core`
      free — both use-after-free of the `SuperBlock` allocation.

14.6.10.5 do_umount_tree — Recursive Unmount

do_umount_tree(root_mount, flags) -> Result<()>

  Used by MNT_DETACH on a mount with sub-mounts, and by namespace teardown.

  1. Acquire `mount_lock`.
  2. Collect all mounts in the subtree rooted at `root_mount` by traversing
     `root_mount.children` recursively. Collect in reverse topological order
     (leaves first, root last).
  2b. Shadow bracket: `shadow_txn_begin()` + stage the removal of every
      collected mount. On ENOMEM: abort, release `mount_lock`, return
      ENOMEM (nothing mutated yet).
  3. For each mount in the collected list:
     a. Set MNT_DOOMED.
     b. Remove from hash table (non-owning links; the owning `Arc<Mount>`
        is released by the shadow commit in step 5b via `rcu_call`, same
        as `do_umount` steps 7/13b).
     c. Remove from parent's children list.
     d. Decrement the mountpoint dentry's `d_mount_refcount` and
        conditionally clear `DCACHE_MOUNTED` (same protocol as `do_umount`
        step 9: `d_mount_refcount.fetch_sub(1, AcqRel)`; clear flag only
        when refcount reaches 0).
     e. Remove from namespace.mount_list.
     f. Decrement namespace.mount_count.
  4. Propagate umount for each removed mount (propagation victims are
     staged into the same transaction as they are selected).
  5. Increment namespace.event_seq.
  5b. `shadow_txn_commit()`.
  6. Release `mount_lock`.
  7. For each collected mount: release its superblock reference per the
     `mount_filesystem` step-6b release rule (teardown at `s_refcount == 0` only;
     immediate if mnt_count == 0, deferred to the final reference drop if
     lazy).

14.6.10.6 do_remount — Change Mount Flags/Options

do_remount(target_mount, flags, data) -> Result<()>

  Capability check: CAP_MOUNT in caller's mount namespace.

  1. Translate new `flags` to `MountFlags`.
  2. Extract per-superblock options from `data`.
  3. **RW→RO transition: flush dirty pages before flag change.**
     If the remount transitions from read-write to read-only
     (`!(old_flags & MS_RDONLY) && (new_flags & MS_RDONLY)`):
     a. Flush all dirty pages and metadata for `sb`.
        This invokes `writeback_inodes_sb(sb, WritebackSyncMode::Wait)`, which
        writes all dirty pages for this superblock to stable storage.
     b. If any dirty inode cannot be flushed (device error), retry up to
        `REMOUNT_RO_FLUSH_RETRIES` (3) times with a 100ms delay between
        retries. If all retries fail:
        - If `flags & MS_FORCE`: proceed with remount-ro anyway. Dirty
          pages for failed inodes are discarded (data loss accepted —
          the admin explicitly requested force). Log FMA warning.
        - If no MS_FORCE: return `Err(EBUSY)` — cannot remount read-only
          while dirty pages exist that cannot be flushed. The caller
          must either fix the device error or use `mount -o remount,ro,force`.
     c. After successful flush, verify no new dirty pages appeared during
        the flush (a writer may have dirtied pages concurrently). If
        `sb.nr_dirty_inodes > 0`, retry from step 3a (bounded by the
        same 3-retry limit — total retries across all sub-attempts).
     d. Set `sb.s_writers.frozen` to `SbFreezeLevel::Write` to prevent
        new writers from dirtying pages between the final flush and the
        flag change in step 5. Wait for `sb.s_writers.writers[0].sum() == 0`
        (all active writers drain). Released after step 5.
  4. Acquire `mount_lock`.
  5. Update `target_mount.flags` atomically with `Release` ordering.
     Concurrent readers (path walk, statfs) load mount flags with `Acquire`
     ordering to pair with this `Release`, ensuring the flag change is
     visible before subsequent filesystem operations on this mount.
     Note: a remount can change per-mount flags (readonly, nosuid, etc.)
     independently of superblock options. For example, `mount -o remount,ro`
     on a bind mount makes that mount point read-only without affecting
     other mount points of the same filesystem.
  6. If per-superblock options changed, call
     `FileSystemOps::remount(sb, flags, data)`. On failure, restore the
     old flags, release freeze if held, and return the error.
  7. Release freeze: set `sb.s_writers.frozen` to `SbFreezeLevel::Unfrozen`
     and wake blocked writers (step 3d).
  8. Increment `namespace.event_seq`.
  8b. Shadow bracket (opened before step 4's lock acquisition, staged with
      the flag/option delta): `shadow_txn_commit()` — refreshes the entry's
      `flags` and the superblock record's `fs_data` so crash reconstruction
      re-creates the post-remount state, never the original mount options.
  9. Release `mount_lock`.

14.6.10.7 do_bind_mount — Bind Mount (MS_BIND)

do_bind_mount(source_path, target_path, flags) -> Result<()>

  Capability check: CAP_MOUNT + read access to source path.

  1. Resolve `source_path` to (source_mount, source_dentry).
  2. Resolve `target_path` to (target_mount, target_dentry).
  3. If `source_mount.propagation == Unbindable`, return EINVAL.
  4. Clone the source mount:
     a. Allocate a new `Mount` node via `mount_alloc()`.
     b. `superblock` = `source_mount.superblock` (shared — same filesystem
        instance, same data pages).
     c. `root` = `source_dentry` (bind mount's root is the source path,
        not necessarily the source mount's root — this is how bind mounts
        of subdirectories work).
     d. `flags` = copy from source, then apply any new flags from `flags`.
     e. `propagation` = Private (new bind mounts default to Private).
  5. If `flags` contains MS_REC (recursive bind):
     a. For each sub-mount under `source_mount` (descendants of
        `source_dentry`), clone the mount and attach it at the
        corresponding dentry under the new bind mount.
     b. Skip unbindable mounts.
  5b. Shadow bracket: `shadow_txn_begin()` + stage one mount-add per clone
      from steps 4-5 (bind mounts stage NO superblock record — each add
      record's `sb` identity resolves to the source superblock's existing
      record, which is guaranteed live because the source mount is attached
      and therefore holds a shadow entry referencing it; `root_path`
      records the bind root relative to the superblock root). On ENOMEM:
      abort, drop the clones, return ENOMEM.
  6. Acquire `mount_lock`.
  7. Attach the cloned mount(s) at target_path (same steps as
     mount_filesystem steps 6f-6m, including the step-6k staging of any
     propagation clones and the step-6l2 `shadow_txn_commit()`).
  8. Release `mount_lock`.

14.6.10.8 do_move_mount — Move a Mount (MS_MOVE)

do_move_mount(source_mount, target_path) -> Result<()>

  Capability check: CAP_MOUNT in caller's mount namespace.

  1. Resolve `target_path` to (target_parent_mount, target_dentry).
  2. Verify `target_dentry` is not a descendant of `source_mount`
     (moving a mount underneath itself would create a cycle). Return
     EINVAL if it is.
  3. Verify `source_mount` is not the namespace root. Return EINVAL.
  4. Acquire `mount_lock`.
  5. Remove `source_mount` from the old location:
     a. Remove from hash table at old (parent, dentry) key.
     b. Remove from old parent's children list.
     c. Decrement old mountpoint dentry's `d_mount_refcount` and
        conditionally clear `DCACHE_MOUNTED` (same protocol as
        `do_umount` step 9).
  6. Attach at new location:
     a. Update `source_mount.parent` to `target_parent_mount`.
     b. Update `source_mount.mountpoint` to `target_dentry`.
     c. Insert into hash table at new (parent, dentry) key.
     d. Add to new parent's children list.
     e. Increment `target_dentry.d_mount_refcount` and set
        `DCACHE_MOUNTED` (same protocol as `attach_mount`'s mountpoint
        accounting).
  7. Propagation: moving a mount does not trigger propagation
     (matches Linux behavior).
  8. Increment `namespace.event_seq`.
  8b. Shadow bracket (opened before step 4, staged with the new
      `parent_mount_id` and parent-relative mountpoint path):
      `shadow_txn_commit()`.
  9. Release `mount_lock`.

14.6.10.9 do_change_propagation — Set Propagation Type

do_change_propagation(target_mount, type, flags) -> Result<()>

  Capability check: CAP_MOUNT in caller's mount namespace.

  1. Determine the target mount(s):
     - If `flags` contains MS_REC: target mount and all descendants.
     - Otherwise: target mount only.
  2. Acquire `mount_lock`.
  3. For each target mount:
     a. If changing to Shared:
        - Allocate a new `group_id` from `namespace.group_id_allocator`.
        - Set `mount.group_id = new_id`.
        - If the mount was previously a slave, it becomes shared+slave
          (receives from master AND propagates to peers).
     b. If changing to Private:
        - Remove from peer group ring (`mnt_share`).
        - Remove from master's slave list (if slave).
        - Set `mount.group_id = 0`.
        - Set `mount.mnt_master = None`.
     c. If changing to Slave:
        - If the mount is currently shared, it becomes a slave of its
          former peer group. The first remaining peer becomes the master.
        - Remove from peer group ring.
        - Add to master's `mnt_slave_list`.
        - Set `mount.mnt_master` to the former peer group leader.
        - Mount retains its `group_id` (for mountinfo optional fields).
     d. If changing to Unbindable:
        - Same as Private, plus prevents bind mount of this mount.
     e. Update `mount.propagation`.
  4. Increment `namespace.event_seq`.
  4b. Shadow bracket (opened before step 2, one modify record per affected
      mount: `propagation`, `peer_group`, `master_group`):
      `shadow_txn_commit()`.
  5. Release `mount_lock`.

14.6.11 Mount Propagation Algorithms

Mount propagation ensures that mount/unmount events on shared mount points are replicated across all related mount points. This is essential for container volume mounts: when a volume is mounted on a shared host path, all containers that have a slave relationship to that path see the new mount.

14.6.11.1 propagate_mount

propagate_mount(source_mount, new_child_mount) -> Result<()>

  Called under mount_lock when a mount is added to a shared mount point.

  Lock ordering: when the propagation walk must acquire per-mount locks
  (e.g., for mnt_count, children list, or mountpoint hash updates),
  locks are acquired in ascending mnt_id order. This prevents ABBA
  deadlocks when two concurrent propagation walks traverse overlapping
  peer groups. If a lock cannot be acquired in order (e.g., a lower
  mnt_id mount is discovered after a higher one is already locked),
  the higher lock is released and re-acquired after the lower one.

  1. Walk the peer group ring of `source_mount` (via `mnt_share` links).
     For each peer mount (excluding `source_mount` itself):
     a. Clone `new_child_mount` with the peer as parent.
        The clone's mountpoint is the dentry in the peer's filesystem
        that corresponds to `new_child_mount.mountpoint` in the source.
     b. Attach the clone at the peer (insert into hash table, set
        DCACHE_MOUNTED, add to children list, add to mount_list).
     c. If the clone's parent is shared, iteratively propagate to
        that peer group using the tree walk algorithm (matching the iterative
        `propagate_mnt()` walker in Linux). Visited groups are tracked
        via a marker flag on each mount to prevent infinite loops. The
        iterative walker processes the propagation tree in a single loop
        without stack recursion, preventing stack overflow regardless of
        propagation chain depth.
  2. Walk the slave list of `source_mount` (via `mnt_slave_list`).
     For each slave mount:
     a. Clone `new_child_mount` with the slave as parent.
     b. Attach the clone at the slave.
     c. If the slave is also shared (shared+slave), propagate to the
        slave's peer group (step 1 applied to the slave's peers).
  3. If the cloning in any propagation step fails (e.g., ENOMEM for
     the mount count limit), roll back: remove all clones created in
     this propagation pass and return the error. Propagation is
     all-or-nothing within a single mount operation.

14.6.11.2 propagate_umount

propagate_umount(source_mount) -> Result<()>

  Called under mount_lock when a mount is removed from a shared mount point.

  1. Walk the peer group ring of `source_mount.parent` (the parent must
     be shared for propagation to occur).
     For each peer of the parent:
     a. Look up a child mount at the corresponding mountpoint dentry
        in the peer's mount hash table.
     b. If found and the child's superblock matches `source_mount`'s
        superblock (same filesystem), unmount it (do_umount steps 6-12).
     c. If the child mount has its own children, recursively unmount
        the subtree (do_umount_tree).
  2. Walk the slave list of the parent.
     For each slave:
     a. Same as step 1a-1c, applied to the slave.

14.6.12 Namespace Operations

14.6.12.1 copy_tree — Clone Mount Tree for CLONE_NEWNS

copy_tree(
    source_ns:         &Arc<MountNamespace>,   // tree walked under ITS mount_lock
    source_root_mount: &Arc<Mount>,
    source_root_dentry:&Arc<Dentry>,
    owner_user_ns:     &Arc<UserNamespace>,    // the CREATING CREDENTIAL's user ns
    fs_snapshot:       &FsStruct,              // whose root/pwd to translate (step 6)
) -> Result<(Arc<MountNamespace>, FsUpdate)>

  Called by clone(CLONE_NEWNS) — create_task step 10, translating the CHILD's
  private FsStruct (`child_fs`; CLONE_FS|CLONE_NEWNS is EINVAL, so the
  child always has one) — and by unshare(CLONE_NEWNS)/setns-free paths,
  translating the CALLER's FsStruct. The fs to translate is an explicit
  parameter precisely because the two callers differ: the fork path runs
  in the PARENT's context, and translating (or worse, mutating) the
  parent's own fs.root/pwd would retarget a task that STAYS in the old
  namespace onto mounts of the new one — the reverse of the
  namespace-boundary-crossing corruption step 6 exists to prevent.

  `owner_user_ns` is recorded per the owner-assignment rule
  ([Section 17.1](17-containers.md#namespace-architecture--capability-domain-mapping)): the
  post-CLONE_NEWUSER-transform credential's user namespace
  (`child_cred.user_ns` / `pending_cred.user_ns`), NOT the parent
  namespace_set's.

  LOCKING: steps 3-6 execute under the SOURCE namespace's `mount_lock`
  (the per-namespace SLEEPING Mutex — see the MountNamespace field doc;
  sleeping-axis, so allocation inside the walk is legal). Without it, a
  concurrent mount_filesystem/do_umount/do_move_mount in the SHARED source
  namespace (a multi-threaded container runtime with a mount-managing
  thread) mutates `Mount.children`/`parent` mid-walk: a re-parenting
  do_move_mount can make the step-4c `mount_map[source_mount.parent]`
  lookup miss — the "always succeeds" claim in 4c holds ONLY against a
  frozen tree. The NEW namespace's own lock is not needed (unpublished,
  exclusively owned). Fork-path sleeping-axis ordering: THREADGROUP_RWSEM
  (3a, read — held across create_task step 10) → mount_lock; mount_filesystem's own
  allocations already order mount_lock before OOM_LOCK(4).

  1. Allocate a new `MountNamespace` via `mount_ns_alloc()` (Nucleus tracked
     storage — see "Tracked Allocation — Mount and MountNamespace"; bound as
     `new_ns` in steps 5-6 below) with fresh `ns_id`, empty hash table, and
     a new `mount_lock`. On ENOMEM, return ENOMEM.
  2. The new namespace's `user_ns` = `owner_user_ns` (see above). Compute
     `cross_userns = !Arc::ptr_eq(owner_user_ns, &source_ns.user_ns)` —
     true exactly when the clone hands the tree to a DIFFERENT (typically
     unprivileged child) user namespace; steps 4f-x and 6b below apply
     the anti-escape rules for that case.
     Acquire `source_ns.mount_lock` (held through step 6).
  3. Clone the source root mount:
     a. Allocate new `Mount` via `mount_alloc()` with the same superblock
        and root dentry.
     b. Flags are copied. **Propagation is INHERITED from the source root mount**
        (not forced to Private). If the source root is shared, the clone is added
        to the same peer group. If the source root is private, the clone is private.
        This matches Linux's `copy_tree()` behavior and is consistent with step 4e
        logic. The statement "child's mounts are private unless marked shared"
        means shared propagation is PRESERVED, not overridden.
     c. Record old-to-new mount mapping: `mount_map[source_root] = cloned_root`.
        `mount_map` type: `HashMap<*const Mount, Arc<Mount>>` — cold path
        (runs once per `clone(CLONE_NEWNS)` or `unshare(CLONE_NEWNS)`).
        Keys are raw pointers for identity comparison (Arc pointer values
        are not integers, so XArray cannot be used per collection policy).
        HashMap is acceptable on this cold path.
  4. Clone the remaining mounts by walking the SOURCE tree via each mount's
     `children` list (`Mount.children`, [Section 14.6](#mount-tree-data-structures-and-operations--mount-node)),
     starting from `source_root_mount`, depth-first, parent before children.
     **This does NOT iterate `source_ns.mount_list`.** `mount_list` order is
     normally parent-before-child (`mount_filesystem` inserts a new mount
     immediately after its parent's list entry — see `mount_filesystem` step 4i),
     but `do_move_mount` (MS_MOVE, [Section 14.6](#mount-tree-data-structures-and-operations--domovemount-move-a-mount-msmove))
     re-parents a mount (step 6a: `source_mount.parent = target_parent_mount`)
     WITHOUT relocating it within `mount_list` — so after a move, a mount's
     `mount_list` position can precede its new parent's, breaking the
     "topological order" invariant that a single flat pass over `mount_list`
     would depend on (FLOW-19-10). `Mount.children`, by contrast, IS kept
     correct by both `mount_filesystem` (step 4h) and `do_move_mount` (step 6d) —
     it is the authoritative parent-child structure, and a tree walk over
     it is correct regardless of `mount_list` ordering. This also means
     `copy_tree` does not depend on `mount_list` ever being perfectly
     topological — a design choice that avoids having to retrofit
     `do_move_mount` to maintain that invariant.
     For each mount visited (starting with `source_root_mount`'s children;
     `source_root_mount` itself was already cloned in step 3):
     a. Skip unbindable mounts, AND do not recurse into their children —
        an unbindable mount's entire subtree is excluded from the clone
        (matches Linux `copy_tree()`: unbindable mounts and their
        descendants are never cloned into a new namespace).
     b. Clone the mount into the new namespace.
     c. Preserve the parent-child relationship: the cloned mount's parent
        is `mount_map[source_mount.parent]`. This lookup always succeeds:
        the walk visits parents before children (depth-first pre-order),
        and `source_mount.parent` is either `source_root_mount` (mapped in
        step 3) or an ancestor already cloned by this same walk — an
        unbindable ancestor cannot occur here because step 4a already
        excluded that entire subtree from the walk.
     d. Insert into the new namespace's hash table (keyed on the cloned
        parent and the shared mountpoint dentry) and append to the new
        namespace's `mount_list` (`push_back` — the walk's visitation
        order IS parent-before-child, so simple append reconstructs a
        correctly topological `mount_list` in the new namespace, with no
        dependency on the source's `mount_list` order).
     e. Record old-to-new mapping: `mount_map[source_mount] = cloned_mount`.
     f. Set propagation:
        - If the source mount is shared AND `!cross_userns`: the clone is
          added to the same peer group (shared propagation preserved
          across CLONE_NEWNS). This is critical for container runtimes
          that rely on propagation.
        - If the source mount is shared AND `cross_userns`: the clone
          becomes a SLAVE of the source's peer group (`mnt_master` = the
          source peer group; linked into its `mnt_slave_list`). Events
          still propagate INTO the unprivileged namespace, but the
          unprivileged owner can never inject mounts back into the
          privileged peer group. ABI contract: Linux `copy_mnt_ns()`
          passes `CL_SHARED_TO_SLAVE` exactly when
          `user_ns != ns->user_ns` (`fs/namespace.c`, torvalds/linux
          master).
        - If the source mount is a SLAVE: the clone is ALSO a slave of
          the SAME master (`mnt_master = source.mnt_master`; linked into
          that master's `mnt_slave_list`). This is userspace-visible ABI:
          after `unshare(CLONE_NEWNS)` from a slaved tree (kubelet's
          `/var/lib/kubelet`, `mount --make-rslave` setups), mount events
          on the master MUST keep propagating into the new namespace, and
          `/proc/self/mountinfo` MUST keep the `master:X` optional field.
          Converting slaves to Private would silently sever propagation —
          Linux `clone_mnt()` preserves slave linkage on the namespace-
          copy path (no `CL_SLAVE`/`CL_PRIVATE` flag is passed).
        - If the source mount is private or unbindable: the clone is
          Private. (The unbindable arm is reachable only for
          `source_root_mount` itself — step 4a excludes every other
          unbindable subtree from the walk.)
     Then recurse into the (non-unbindable) mount's `children` list.
     **Shadow staging**: each clone (including the step-3 root clone) is
     staged into a shadow transaction opened at step 1
     (`shadow_txn_begin()` + `shadow_stage_ns_add(new_ns.ns_id, ...)`),
     as a mount-add for the NEW namespace referencing the SOURCE mount's
     existing `sb_key`. Committed in step 7 before the lock release; the
     shadow's Arc clones become the new namespace's owning references.
     **Error handling**: If mount cloning fails at step 4b (ENOMEM from
     `mount_alloc()` — instance budget or tracked storage exhausted) or a
     shadow staging call fails, `shadow_txn_abort()`,
     release `source_ns.mount_lock` and drop
     the partially-constructed MountNamespace. All
     previously cloned mounts are freed via their Arc destructors (the
     aborted staging records held the only extra clones). The
     `fs_snapshot` owner's `fs.root` and `fs.pwd` are unchanged — copy_tree
     never mutates any FsStruct (step 6 only COMPUTES the update).
     Return the error to the caller (`ENOMEM`).
  5. Set `new_ns.root` to the clone of `source_root_mount`.
  6. **Compute (do NOT apply) the FsUpdate**: using the `mount_map`, find
     the cloned counterparts of `fs_snapshot.root.mount` and
     `fs_snapshot.pwd.mount`. If the referenced mount was SKIPPED by step
     4a (unbindable — no `mount_map` entry), fall back to the new
     namespace's root — matching Linux's `copy_mnt_ns()` behavior of
     resetting root/pwd to the new namespace root when the original was
     excluded from the clone. Falling through silently (no `else` arm)
     would leave a root/pwd pointing at a `Mount` that belongs to the OLD
     namespace and is absent from the new namespace's `hash_table` —
     subsequent path resolution from that pwd would silently cross the
     namespace boundary, leaking visibility into mounts that CLONE_NEWNS
     was supposed to isolate:
     // Computed against the caller-designated FsStruct SNAPSHOT — the
     // CHILD's private FsStruct in the fork path, the caller's own in
     // unshare. The mount_map stays internal; only the two translated
     // PathRefs escape.
     let fs_update = FsUpdate {
         root: match mount_map.get(&fs_snapshot.root.mount) {
             Some(new_root) => PathRef { mount: Arc::clone(new_root), dentry: fs_snapshot.root.dentry.clone() },
             None => new_ns.root_mount(),
         },
         pwd: match mount_map.get(&fs_snapshot.pwd.mount) {
             Some(new_pwd) => PathRef { mount: Arc::clone(new_pwd), dentry: fs_snapshot.pwd.dentry.clone() },
             None => new_ns.root_mount(),
         },
     };
     The CALLER applies the update at its commit point — create_task applies
     it to the child's (still-local, unwind-safe) `child_fs` immediately
     after copy_tree returns; unshare applies it in its step-5 commit
     section AFTER the last fallible step (`namespace_set_alloc`), so an ENOMEM
     later in namespace assembly leaves the task's path-resolution state
     fully in the old namespace — nothing to roll back. Applying inside
     copy_tree (the previous design) both mutated the WRONG task's fs in
     the fork path and left no way to restore root/pwd when a later
     namespace-creation step failed.
  6b. **Cross-userns lock-down** (`cross_userns` only — the MNT_LOCKED
     anti-escape rule): walk every cloned mount EXCEPT `new_ns.root` and
     set `MNT_LOCKED` in its flags, plus the flag-locks derived from the
     source's state (`MNT_LOCK_READONLY` if the source was read-only,
     `MNT_LOCK_NODEV`/`MNT_LOCK_NOSUID`/`MNT_LOCK_NOEXEC`/`MNT_LOCK_ATIME`
     mirroring the source's nodev/nosuid/noexec/atime flags). Effect: the
     unprivileged namespace owner cannot `umount(2)` a locked child mount
     to REVEAL the files it covers (do_umount's MNT_LOCKED check — step 2
     of do_umount — returns EINVAL), and cannot remount to clear the
     locked security-relevant flags. ABI contract: Linux `copy_mnt_ns()`
     Linux calls `lock_mnt_tree(new)` when `user_ns != ns->user_ns`
     (`fs/namespace.c`, torvalds/linux master); without this, an
     unprivileged userns owner escapes mount-based sandboxing by
     unmounting whatever was stacked over sensitive paths.
  7. `shadow_txn_commit()` (still under `source_ns.mount_lock` — the lock
     that serialized the walk), then release `source_ns.mount_lock`.
     Return `(new_ns, fs_update)`.

/// The deferred fs.root/pwd translation computed by copy_tree() step 6.
/// Carries the two translated PathRefs OUT of copy_tree without exposing
/// the internal mount_map; the caller applies it at its commit point.
pub struct FsUpdate {
    /// Translated root: the new namespace's counterpart of the snapshot's
    /// root mount (same dentry, cloned mount), or the new namespace root.
    pub root: PathRef,
    /// Translated pwd — same rule.
    pub pwd: PathRef,
}

impl FsUpdate {
    /// Apply both PathRefs in ONE FsStruct write section (readers observe
    /// both-old or both-new). `fs` is the same FsStruct the snapshot was
    /// taken from (the fork path passes the child's private copy; unshare
    /// passes the caller's own). FS_STRUCT_LOCK(42) — callers hold no
    /// spin-class lock at or above 42 here (unshare applies under
    /// TASK_LOCK(20): 20 → 42 ascending; fork applies with no lock held).
    pub fn apply(&self, fs: &RwLock<FsStruct>) {
        let mut guard = fs.write();
        guard.root = self.root.clone();
        guard.pwd = self.pwd.clone();
    }
}

14.6.12.2 pivot_root Integration

The pivot_root(2) algorithm specified in Section 17.1 is updated to use the Mount data structure:

pivot_root(new_root_path, put_old_path) -> Result<()>

  Capability check: CAP_SYS_ADMIN in caller's user namespace.
  The caller must be in a mount namespace (not the initial namespace).

  1. Resolve `new_root_path` to (new_root_mount, new_root_dentry).
     Verify `new_root_dentry` is the root of `new_root_mount` (i.e.,
     new_root is a mount point, not just a directory). This resolution is
     lock-free (RCU dentry/mount cache reads).
  2. Resolve `put_old_path` to (put_old_mount, put_old_dentry). Also
     lock-free. Full validation of both paths (steps 4-5 below) happens
     AFTER `mount_lock` is held — if either mount was concurrently
     unmounted or moved between resolution and lock acquisition, the
     under-lock checks catch it (a removed mount fails the "reachable"
     walk; a moved mount is walked in its current, lock-protected
     position).
  3. Acquire `mount_lock`. Steps 4-9 below execute as ONE critical
     section — the invariant checks and the tree mutation are no longer
     split across the lock boundary. This closes a TOCTOU window: without
     the lock held across both the check and the mutation, two concurrent
     `pivot_root` calls in the same mount namespace could each pass their
     own check against the pre-mutation root, then serialize on the lock
     and mutate against a root that no longer matches what they validated.
  4. Verify `new_root_mount` is not the current namespace root
     (i.e., `new_root_mount != namespace.root`, read under `mount_lock`).
     If it is, release `mount_lock` and return `EINVAL` — pivot_root
     is a no-op.
  5. Verify `put_old` is reachable from `new_root` by walking the
     mount tree upward, under `mount_lock`. This ensures `put_old` is a
     valid location within the new root's subtree for mounting the old
     root. If unreachable, release `mount_lock` and return `EINVAL`.
  6. Let `old_root_mount` = namespace's current root mount.
  7. Detach `new_root_mount` from its current position:
     a. Remove from hash table.
     b. Remove from parent's children.
     c. Clear DCACHE_MOUNTED on its old mountpoint.
  8. Reattach `old_root_mount` at `put_old`:
     a. Set `old_root_mount.parent` = `new_root_mount`.
     b. Set `old_root_mount.mountpoint` = the dentry corresponding to
        `put_old` within `new_root_mount`'s filesystem.
     c. Insert `old_root_mount` into hash table at new position.
     d. Set DCACHE_MOUNTED on the put_old dentry.
  9. Set `new_root_mount` as the namespace root:
     a. `new_root_mount.parent` = None (it is now the root).
     b. `new_root_mount.mountpoint` = `new_root_mount.root` (self-referential
        for the root mount).
     c. `namespace.root.update(new_root_mount, &mount_lock_guard)` (RCU
        publish via RcuCell::update).
  10. Update the CURRENT TASK's fs.root and fs.pwd if they reference the old root:
      // task.fs is ArcSwap<RwLock<FsStruct>> — load, then write-lock.
      let fs_arc = current_task().fs.load();
      let mut fs = fs_arc.write();
      if fs.root.mount == old_root_mount {
          fs.root = PathRef { mount: Arc::clone(&new_root_mount), dentry: new_root_dentry };
      }
      if fs.pwd.mount == old_root_mount {
          fs.pwd = PathRef { mount: Arc::clone(&new_root_mount), dentry: new_root_dentry };
      }
      NOTE: Linux does NOT iterate all tasks in the namespace. Other tasks sharing
      the same `FsStruct` see the update via the shared reference. Tasks with
      different `FsStruct` instances that reference the old root will see the old
      root moved to `put_old` on their next path resolution — this is correct
      behavior (they can then chdir to the new root if desired).
  11. Increment `namespace.event_seq`.
  12. Release `mount_lock`.

  Note: Steps 4-9 are now a single atomic critical section under
  `mount_lock` — both the invariant checks (steps 4-5) and the tree
  mutation (steps 7-9) are inside it, eliminating the TOCTOU window
  between validation and mutation. In-flight path lookups that started
  before step 9 see the old root via RCU (the old `RcuCell` value remains
  valid until the grace period). New lookups after step 9 see the new
  root. This matches the atomicity guarantee specified in Section 17.1.3.
  A concurrent `pivot_root` in the same namespace now serializes entirely
  at step 3 (lock acquisition): the second caller's step 4 check re-reads
  `namespace.root` under the lock and observes whatever the first caller
  just published, rather than a stale pre-lock snapshot.

14.6.12.3 Namespace Teardown

When a mount namespace is destroyed (all processes exited, all /proc/PID/ns/mnt file descriptors closed, all bind mounts of the namespace file unmounted):

destroy_mount_namespace(ns) -> ()

  1. Acquire `mount_lock`.
  1b. Shadow bracket: `shadow_txn_begin()` +
      `shadow_stage_ns_remove(ns.ns_id)` — a namespace removal implies
      removal of every entry it holds; no per-mount staging is needed and
      the staging cannot fail structurally (removal records for a live
      namespace).
  2. Iterate `ns.mount_list` in reverse list order. Intra-pass order is
     immaterial to correctness here: every mount is unlinked under
     `mount_lock` and the owning `Arc<Mount>`s are all dropped together by
     the atomic shadow commit at step 4b (grace-period deferred), so no
     mount is freed mid-pass and no sub-step below reads a child before its
     parent — each sub-step merely unlinks THIS mount from shared structures
     (hash table, parent's `children`, peer/slave lists), which is
     order-independent. A guaranteed-topological (leaf-first) traversal is
     therefore unnecessary here, unlike `do_umount_tree`, which walks
     `Mount.children` postorder because it must collect a subtree from the
     authoritative parent-child structure.
  3. For each mount:
     a. Set MNT_DOOMED.
     b. Remove from hash table (non-owning links; each mount's owning
        `Arc<Mount>` is released by the step-4b shadow commit via
        `rcu_call`, same as `do_umount` steps 7/13b).
     c. Remove from parent's children.
     d. Remove from peer group and slave lists.
  4. Increment `namespace.event_seq`.
  4b. `shadow_txn_commit()` — drops every entry's owning Arc (grace-period
      deferred) and the `ShadowNamespace` record with its `Arc<MountNamespace>`.
  5. Release `mount_lock`.
  6. For each removed mount (in reverse order): release its superblock
     reference per the `mount_filesystem` step-6b release rule —
     a. If `mnt_count == 0` and `s_refcount` reaches zero, call
        `FileSystemOps::unmount()` now.
     b. If `mnt_count > 0` (lazy unmount remnants still referenced by
        open file descriptors), defer to the final reference drop.
  7. Drop the hash table and mount list heads (the entries themselves are
     freed by the deferred owning-Arc drops).

14.6.13 New Mount API Syscalls

UmkaOS implements the Linux 5.2+ mount API syscalls for compatibility with modern container runtimes (containerd, CRI-O) and systemd. These are thin wrappers around the internal mount operations described above.

Syscall	Purpose	Capability
fsopen(fs_type, flags)	Create a filesystem context	CAP_MOUNT
fspick(dirfd, path, flags)	Create a reconfiguration context for an existing mount	CAP_MOUNT
fsconfig(fd, cmd, key, value, aux)	Configure a filesystem context	CAP_MOUNT
fsmount(fs_fd, flags, mount_attr)	Create a detached mount from a configured context	CAP_MOUNT
move_mount(from_dirfd, from_path, to_dirfd, to_path, flags)	Attach a detached mount or move an existing mount	CAP_MOUNT
open_tree(dirfd, path, flags)	Open or clone a mount point as a file descriptor	CAP_MOUNT (if OPEN_TREE_CLONE)
mount_setattr(dirfd, path, flags, attr, size)	Modify mount attributes, optionally recursively	CAP_MOUNT

fsopen flow: 1. Validate fs_type against the filesystem registry. 2. Allocate FsContext with purpose = NewMount. 3. Return a file descriptor referencing the context.

fsconfig flow (selected commands): - FSCONFIG_SET_STRING: set a key-value option string. - FSCONFIG_SET_BINARY: set a binary option blob. - FSCONFIG_SET_FD: set an option to a file descriptor (e.g., source device). - FSCONFIG_CMD_CREATE: validate all options and create the superblock by calling FileSystemOps::mount(). On success, the superblock is stored in FsContext.superblock. On failure, diagnostic messages are written to the context's error log. - FSCONFIG_CMD_RECONFIGURE: for fspick contexts, apply new options to the existing superblock via FileSystemOps::remount().

fsmount flow: 1. Consume the superblock from the FsContext. The FsContext is marked consumed (state = FsContextState::Consumed); further fsconfig() calls on this fd return EBUSY. The fsopen fd remains open for error log retrieval but cannot be used to create another mount. Closing the fsopen fd releases the FsContext; double-release is prevented by the consumed state flag. 2. Allocate a Mount node via mount_alloc() with MNT_DETACHED flag set. 3. The mount is not yet attached to any namespace or visible to path resolution. It exists only as a detached object referenced by the returned file descriptor. 4. Return an O_PATH file descriptor referencing the detached mount.

move_mount flow: 1. Resolve the source (detached mount fd or existing mount path). 2. Resolve the target path. 3. If the source is detached (MNT_DETACHED): a. Clear MNT_DETACHED. b. Attach to the namespace via attach_mount() (Section 14.6), which stages the previously-unstaged detached mount (bracket-discipline rule 4) as part of the attachment transaction. 4. If the source is an existing mount: a. Delegate to do_move_mount() (Section 14.6).

open_tree flow: 1. Resolve the path to a mount. 2. If OPEN_TREE_CLONE: a. Clone the mount (like do_bind_mount without attaching). b. The clone is detached (MNT_DETACHED). c. If OPEN_TREE_CLONE | AT_RECURSIVE: recursively clone the subtree. 3. Return an O_PATH file descriptor.

mount_setattr flow: 1. Resolve the path to a mount. 2. Validate attr_set and attr_clr do not conflict. 3. Acquire mount_lock. 4. If AT_RECURSIVE: a. Collect all mounts in the subtree. b. Validate the changes are valid for all mounts (e.g., clearing MNT_READONLY on a mount whose superblock is read-only is invalid). c. If validation fails for any mount, return error (no partial changes). d. Apply attr_clr then attr_set to all mounts atomically. 5. If not recursive: apply to the single mount. 6. If attr.propagation != 0: change propagation type (Section 14.6). 7. Increment namespace.event_seq. 7b. Shadow bracket (opened before step 3, one modify record per affected mount): shadow_txn_commit(). 8. Release mount_lock.

14.6.14 Mount Introspection Syscalls

Linux 6.8 introduced statmount(2) and listmount(2) as structured replacements for parsing /proc/PID/mountinfo. UmkaOS implements both for container introspection tools and future-compatible userspace.

Syscall	Purpose	Capability
statmount(req, buf, bufsize, flags)	Query detailed mount information by mount ID	None (own namespace)
listmount(req, buf, bufsize, flags)	List child mount IDs of a given mount	None (own namespace)

statmount: Returns a struct statmount containing the mount's ID, parent ID, mount flags, propagation type, peer group ID, master mount ID, filesystem type, mount source, mount point path, and superblock options. The request specifies which fields to populate via a bitmask, avoiding unnecessary work (e.g., path resolution for mount point is skipped if STATMOUNT_MNT_POINT is not requested).

listmount: Returns an array of 64-bit mount IDs for the child mounts of a given mount. Supports cursor-based iteration: the caller passes the last seen mount ID, and listmount returns mount IDs after that cursor. This handles concurrent mount/unmount gracefully (mounts added after the cursor are seen; mounts removed are skipped).

14.6.15 /proc/PID/mountinfo Format

Each process exposes its mount namespace's mount tree through /proc/PID/mountinfo and /proc/PID/mounts. These files are read by systemd, Docker, findmnt, df, mountpoint, and other tools.

mountinfo line format (one line per mount, matching Linux exactly):

<mount_id> <parent_id> <major>:<minor> <root> <mount_point> <mount_options> <optional_fields> - <fs_type> <mount_source> <super_options>

Field	Source	Example
mount_id	Mount.mount_id	36
parent_id	Mount.parent.mount_id (self for root)	35
major:minor	SuperBlock.dev major:minor	98:0
root	Path of mount root within the filesystem	/ or /subdir
mount_point	Path of mount point relative to process root	/mnt/data
mount_options	Per-mount flags as comma-separated options	rw,noatime,nosuid
optional fields	Propagation: shared:N, master:N, propagate_from:N	shared:1 master:2
separator	Literal hyphen	-
fs_type	Filesystem type name	ext4
mount_source	Mount.device_name	/dev/sda1
super_options	From FileSystemOps::show_options()	rw,errors=continue

Implementation: The VFS iterates the namespace's mount_list under rcu_read_lock() and formats each line. Because mount_filesystem inserts each new mount immediately after its parent's list entry (step 4i), the output is normally parent-before-child — but the spec does NOT guarantee a topological line order: do_move_mount (Section 14.6, MS_MOVE) re-parents a mount without relocating it within mount_list, so a moved mount can appear before its new parent (the same caveat documented on the MountNamespace.mount_list field and relied on by copy_tree, which walks Mount.children rather than mount_list). Line order is therefore NOT a stable contract. Linux behaves the same way: its /proc/PID/mountinfo iterator (m_start/m_next in fs/namespace.c) walks the mount-namespace rbtree ns->mounts in ascending mnt_id_unique order — likewise not topological, since a mount recreated or re-parented after its neighbours can hold a unique id that places a child before its parent. The per-line FIELD FORMAT matches Linux exactly; consumers that reconstruct mount topology (e.g. systemd, findmnt) key off the mount_id/parent_id fields rather than relying on line order.

/proc/PID/mounts: A simplified view matching the old /etc/mtab format: <device> <mount_point> <fs_type> <options> 0 0. Generated from the same mount_list, omitting mount IDs and propagation fields.

14.6.16 Path Resolution Integration

This section details how the mount tree integrates with the path resolution algorithm described in Section 14.1.

Mount crossing in RCU-walk (fast path):

resolve_component_rcu(current_mount, current_dentry, name):
  1. Look up `name` in the dentry cache: dentry = dcache_lookup(current_dentry, name).
  2. If dentry is not found: fall through to ref-walk (cache miss).
  3. If dentry.flags has DCACHE_MOUNTED:
     a. Call mnt_ns.hash_table.lookup(current_mount.mount_id, dentry.inode, &rcu_guard).
     b. If a child mount is found:
        - current_mount = child_mount
        - current_dentry = child_mount.root
        - If child_mount.root also has DCACHE_MOUNTED, repeat step 3
          (stacked mounts — rare but legal).
     c. If no child mount found: DCACHE_MOUNTED is stale (race with
        umount). Clear the flag lazily and continue with the dentry.
  4. Return (current_mount, dentry).

Mount crossing in ref-walk (slow path):

resolve_component_ref(current_mount, current_dentry, name):
  1. Same as RCU-walk step 1, but takes a dentry reference count.
  2. Same DCACHE_MOUNTED check.
  3. If mount crossing:
     a. Call mnt_ns.hash_table.lookup() under rcu_read_lock().
     b. If found: increment child_mount.mnt_count (atomic add).
     c. Decrement current_mount.mnt_count.
     d. current_mount = child_mount; current_dentry = child_mount.root.
  4. Return (current_mount, dentry).

".." traversal across mount boundaries:

resolve_dotdot(current_mount, current_dentry):
  1. Chroot boundary check: if current_dentry == task.fs.root.dentry
     AND current_mount == task.fs.root.mnt, return (current_mount,
     current_dentry). The process is at its chroot root — ".." must
     not escape the jail.
  2. If current_dentry == current_mount.root:
     - We are at the root of this mount. ".." should cross into the parent
       mount.
     - If current_mount.parent is None: we are at the namespace root.
       ".." resolves to the root itself (cannot go above /).
     - Otherwise: current_mount = current_mount.parent.
       current_dentry = current_mount.mountpoint.
       (Continue resolving ".." from the parent mount's mountpoint.)
  3. If current_dentry != current_mount.root:
     - Normal ".." within the mount's filesystem.
     - current_dentry = current_dentry.parent.
  4. Return (current_mount, current_dentry).

14.6.17 Performance Characteristics

Operation	Cost	Notes
Mount hash lookup (RCU read)	~5-15 ns	SipHash + 1-2 pointer chases, no locks, no atomics. Occurs on every mount-point crossing during path resolution.
DCACHE_MOUNTED check	~1 ns	Single atomic load of dentry flags. Occurs on every path component — the gate that avoids hash lookup on non-mount-point dentries.
Mount (new filesystem)	~1-10 us	Dominated by filesystem driver's mount() (superblock creation). Mount tree insertion is ~200 ns under lock.
Unmount	~500 ns - 5 us	Hash removal + propagation. Filesystem unmount() cost varies (ext4 journal flush vs. tmpfs instant).
Bind mount	~300 ns	Mount node clone + hash insertion. No filesystem I/O.
Bind mount (recursive, N sub-mounts)	~300*N ns	Linear in subtree size.
Propagation (mount, M peers)	~300*M ns	One clone per peer. Propagation to slaves adds per-slave overhead.
/proc/PID/mountinfo generation	~50 ns/mount	One line per mount. 100-mount namespace: ~5 us total.
copy_tree (CLONE_NEWNS, N mounts)	~500*N ns	Clone all mounts. 100-mount namespace: ~50 us.
pivot_root	~1 us	Two hash table mutations + RCU publish.

Memory overhead per mount: ~320 bytes for the Mount struct (including all intrusive list nodes and propagation fields) plus ~16 bytes for the hash table entry. A container with 100 mounts consumes ~33 KiB of mount tree metadata. A system with 10,000 containers (1 million mounts total) consumes ~330 MiB — proportional to the actual number of mounts, not pre-allocated.

14.6.18 Cross-References

• Section 3.5 (Lock Hierarchy): MOUNT_LOCK at level 20, between DENTRY_LOCK (19) and EVM_LOCK (22).
• Section 9.1 (Capabilities): CAP_MOUNT (bit 70) gates all mount operations. CAP_SYS_ADMIN (bit 21) required for pivot_root and MNT_LOCKED override.
• Section 14.1 (VFS Architecture): FileSystemOps::mount() creates the superblock consumed by mount_filesystem(). FileSystemOps::unmount() is called by do_umount() after tree removal.
• Section 14.1 (Dentry Cache): DCACHE_MOUNTED flag triggers mount hash table lookup during path resolution.
• Section 14.1 (Path Resolution): RCU-walk and ref-walk mount crossing detailed in Section 14.6.
• Section 14.1 (Mount Namespace and Capability-Gated Mounting): The capability table and propagation type summary specified there are implemented by the data structures in this section.
• Section 14.8 (overlayfs): OverlayFs::mount() creates an OverlaySuperBlock consumed via the standard mount_filesystem() path.
• Section 17.1 (Namespace Implementation): NamespaceSet.mount_ns: Arc<MountNamespace> provides access to the full mount tree rather than just a capability handle to the root VFS node. The NamespaceSet is per-task (Task.namespace_set), not per-process.
• Section 17.1 (pivot_root): The step-by-step algorithm there is superseded by the precise Mount-struct-based algorithm in Section 14.6.
• Section 17.1 (Namespace Inheritance): CLONE_NEWNS triggers copy_tree() (Section 14.6).

14.7 Distribution-Aware VFS Extensions

When filesystems are shared across cluster nodes (Section 15.14), the VFS must handle cache validity, locking granularity, and metadata coherence across node boundaries. Linux's VFS was designed for local filesystems with network filesystem support bolted on afterward, resulting in several systemic performance problems. UmkaOS's VFS addresses these by integrating with the Distributed Lock Manager (Section 15.15).

Linux Problem	Impact	UmkaOS Fix
Dentry cache assumes local validity	Remote rename/unlink leaves stale dentries on other nodes	Callback-based invalidation: DLM lock downgrade (Section 15.15) triggers targeted dentry invalidation for affected directory entries only
d_revalidate() on every lookup for network FS	Extra round-trip per path component on NFS/CIFS/GFS2	Lease-attached dentries: dentry is valid while parent directory DLM lock is held (Section 15.15); zero revalidation cost during lease period
Inode-level locking forces false sharing	Two nodes writing to different byte ranges of the same file serialize on the inode lock	Range locks in VFS: DLM byte-range lock resources (Section 15.15) allow concurrent operations on different ranges of the same file
No concurrent directory operations	mkdir and create in the same directory serialize globally	Per-bucket directory locks: hash-based directory formats (ext4 htree, GFS2 leaf blocks) use separate DLM resources per hash bucket
readdir() + stat() = 2N round-trips for N files	ls -l on a 1000-file remote directory requires 2001 operations	getdents_plus() returning attributes with directory entries (analogous to NFS READDIRPLUS but in-kernel, avoiding the userspace/kernel boundary per entry). getdents_plus() is an UmkaOS VFS-internal operation (not a new syscall): the VFS's readdir implementation populates both the directory entry and its InodeAttr in a single filesystem callback, caching the attributes for immediate use by a subsequent getattr() / stat() call. Userspace accesses this via the standard getdents64(2) + statx(2) syscalls — the optimization is transparent, eliminating redundant disk or DLM round-trips inside the kernel.
Full inode cache invalidation on lock drop	Dropping a DLM lock on an inode discards all cached metadata, even fields that haven't changed	Per-field inode validity: mtime/size read from DLM Lock Value Block (Section 15.15); permissions and ownership from local capability cache; only stale fields refreshed on lock reacquire

Integration with Section 15.15 DLM:

• Dentry lease binding: When the VFS caches a dentry for a clustered filesystem, it records the DLM lock resource that protects the parent directory. The dentry remains valid as long as that lock is held at CR (Concurrent Read) mode or stronger. When the DLM downgrades or releases the lock (due to contention from another node), the VFS receives a callback and invalidates only the affected dentries — not the entire dentry subtree.

/// Per-dentry lease tracking for distributed VFS.
/// Stored in the dentry's filesystem-private data for clustered filesystems.
pub struct DentryLeaseInfo {
    /// Compact DLM resource handle — the 64-bit hash of the DLM
    /// `ResourceName` protecting the parent directory of this dentry. The
    /// full 258-byte `ResourceName` is far too heavy to store per dentry, so
    /// the DLM indexes resources by this `ResourceId`
    /// ([Section 15.15](15-storage.md#distributed-lock-manager)) and callers hold only the handle.
    pub dlm_resource: ResourceId,
    /// Lease sequence counter. Incremented by the DLM callback when the
    /// lease is invalidated (lock downgrade or release). VFS path walk
    /// compares the dentry's cached `lease_seq` against the current
    /// directory DLM lock's `lease_seq`: if they differ, the dentry is
    /// treated as stale and re-validated.
    ///
    /// Type: u64 (50-year rule: at 10M invalidations/sec, wraps in ~58K years).
    pub lease_seq: u64,
    /// The DLM lock mode at which this dentry was validated.
    pub validated_at_mode: LockMode,
}

• Range-aware writeback: When a process holds a DLM byte-range lock and writes to pages within that range, the VFS tracks dirty pages per lock range (not per inode). On lock downgrade, only dirty pages within the lock's range are flushed (Section 15.15). This eliminates the Linux problem where dropping a lock on a 100 GB file requires flushing all dirty pages, even if only 4 KB was modified.

• Attribute caching via LVB: The VFS reads frequently-accessed inode attributes (i_size, i_mtime, i_blocks) from the DLM Lock Value Block (Section 15.15) rather than performing a disk read on every lock acquire. The LVB is updated by the last writer on lock release, so readers always get current values at the cost of a single RDMA operation (~3-4 μs) instead of a disk I/O (~10-15 μs for NVMe).

14.7.1.1 Lease Invalidation and In-Flight I/O Synchronization

When the DLM downgrades a dentry or byte-range lock (due to contention from another node), in-flight I/O operations that depend on the lease must be coordinated to prevent data corruption. The synchronization protocol:

1. DLM blocking callback received: The VFS receives a dlm_ast_blocking() callback indicating that another node requests the lock at a conflicting mode.

2. In-flight I/O barrier: The VFS increments the per-inode invalidation_seq: AtomicU64 counter (Acquire ordering). All new VFS operations targeting this inode check invalidation_seq before proceeding; if it has changed since the operation began, the operation must re-validate its cached dentry/inode state after re-acquiring the lock.

3. Drain in-flight operations: The VFS waits for all in-flight operations that hold a reference to the current lock grant to complete. This uses a per-lock-resource inflight_count: AtomicU32 reference counter:

4. Each VFS operation that depends on a DLM lock increments inflight_count (Acquire) at operation start and decrements it (Release) at completion.

5. The invalidation path waits on a per-lock-resource WaitQueue with a 30-second timeout (matching Linux GFS2's lock-demotion wait timeout). The WaitQueue is signaled by each VFS operation upon completion (after decrementing inflight_count). If the timeout expires, the lock is downgraded forcibly (the remote node's request takes priority to avoid cluster-wide deadlocks).

6. Flush dirty data: For byte-range locks, dirty pages within the lock's range are flushed to disk (Section 15.15) before the lock is downgraded.

7. Invalidate caches: Dentry cache entries protected by the lock are invalidated. Page cache pages within the byte range are invalidated (discarded if clean, flushed then discarded if dirty).

8. Downgrade/release the lock: The DLM lock is downgraded to the requested mode (or released entirely). The dlm_ast_completion() callback notifies the requesting node that the lock is available.

Ordering guarantee: Steps 2-5 are atomic with respect to the lock: no new operation can acquire the lock between the barrier (step 2) and the downgrade (step 6) because the lock's grant state is set to LOCK_INVALIDATING during this window.

---

14.8 overlayfs: Union Filesystem for Containers

Use case: Container image layering. Docker, containerd, Podman, and Kubernetes all use overlayfs as their primary storage driver. A container image is a stack of read-only filesystem layers; overlayfs merges them with a writable upper layer to present a unified view. Without overlayfs, container runtimes fall back to copy-the-entire-layer approaches (VFS copy, naive snapshots), which are orders of magnitude slower for image pull and container startup.

Tier-agnostic: the overlayfs driver manifest declares preferred_tier = 1 (overlayfs runs in the same domain as umka-vfs whenever possible); the loader computes the effective tier at bind time per Section 11.3.

Rationale for preferred_tier = 1: overlayfs is a stacking filesystem — it sits between the VFS and the underlying filesystem drivers (ext4, XFS, btrfs, tmpfs). Every path lookup, readdir, and file open in a container traverses overlayfs. Binding overlayfs at effective Tier 2 (Ring 3, process boundary) would add two domain crossings per VFS operation inside every container, roughly doubling the path resolution overhead. Since overlayfs delegates all storage I/O to the underlying filesystem (which is itself a Tier 1 driver), overlayfs never touches hardware directly — it is a pure VFS client. Its code complexity is moderate (~3,000 SLOC in Linux) and auditable. The crash containment boundary is the VFS domain: if overlayfs panics, the VFS recovery protocol (Section 14.1) handles it.

Container setup ordering: During container creation, the overlayfs mount must complete before pivot_root() changes the container's root filesystem. The sequence is: (1) mount overlayfs at the target path, (2) mount pseudo-filesystems (/proc, /sys, /dev) on top, (3) pivot_root() to switch the container root to the overlayfs mount. Reversing steps (1) and (3) would leave the container with no root filesystem. This ordering matches the OCI runtime specification and is enforced by the umka-sysapi container setup helpers.

pivot_root() namespace validation: pivot_root() validates that new_root is a mount point in the calling task's mount namespace. If new_root was mounted in a different namespace (e.g., parent), pivot_root() returns EINVAL. This prevents namespace-crossing pivots that would create incoherent mount state.

Design: overlayfs implements FileSystemOps, InodeOps, FileOps, and DentryOps from the VFS trait system (Section 14.1). It does not introduce new VFS abstractions — it composes existing ones.

14.8.1 Mount Options and Configuration

/// Mount options parsed from the `data` parameter of `FileSystemOps::mount()`.
/// Encoded as comma-separated key=value pairs in the `data: &[u8]` slice,
/// matching Linux's overlayfs mount option syntax exactly.
///
/// Example mount command:
/// ```
/// mount -t overlay overlay \
///   -o lowerdir=/lower2:/lower1,upperdir=/upper,workdir=/work \
///   /merged
/// ```
///
/// For read-only overlays (no upperdir/workdir), only lowerdir is required.
/// This is used for container image inspection without a writable layer.
pub struct OverlayMountOptions {
    /// Colon-separated list of lower layer paths, ordered from topmost to
    /// bottommost. At least one lower layer is required. Maximum 500 layers
    /// (matching Linux's limit, which Docker/containerd never approach —
    /// typical images have 5-20 layers).
    ///
    /// Each path must be an existing directory on a mounted filesystem.
    /// The VFS resolves each path to an `InodeId` at mount time and holds
    /// a reference to the underlying superblock for the mount's lifetime.
    ///
    /// Heap-allocated rather than inline (`ArrayVec<_, 500>` would be up to
    /// 4000 bytes on the stack). The 500-layer maximum is enforced at mount
    /// validation time. Mount processing is a rare, non-hot-path operation
    /// where heap allocation is acceptable.
    pub lower_dirs: Box<[InodeId]>,

    /// Upper layer directory (read-write). `None` for read-only overlays.
    /// Must reside on a filesystem that supports: xattr (for whiteouts and
    /// metacopy markers), rename with RENAME_WHITEOUT, and mknod (for
    /// character-device whiteouts). The upper filesystem must be writable.
    pub upper_dir: Option<InodeId>,

    /// Work directory for atomic copy-up staging. Required if `upper_dir`
    /// is set. Must be on the **same filesystem** as `upper_dir` (same
    /// superblock) — copy-up uses rename(2) from workdir to upperdir,
    /// which requires same-device semantics. The VFS verifies this at
    /// mount time by comparing `SuperBlock` identity.
    ///
    /// The workdir must be empty at mount time. overlayfs creates a `work/`
    /// subdirectory inside it for staging, and an `index/` subdirectory
    /// for NFS export handles (if enabled).
    pub work_dir: Option<InodeId>,

    /// Enable metadata-only copy-up. When true, operations that modify
    /// only metadata (chmod, chown, utimes, setxattr) copy only the
    /// inode metadata to the upper layer, deferring data copy until the
    /// first write. Dramatically reduces container startup I/O: a
    /// `chmod` on a 200 MB binary copies ~4 KB of metadata instead of
    /// 200 MB of data.
    ///
    /// Default: true (matches Docker/containerd default since Linux 5.11+
    /// with kernel config `OVERLAY_FS_METACOPY=y`).
    ///
    /// Security restriction: this option is silently forced to `false`
    /// when the mount is user-namespace-influenced (i.e., when the caller
    /// does not hold `CAP_SYS_ADMIN` in the initial user namespace). In
    /// such mounts the upper layer uses `user.overlay.*` xattrs, which
    /// are writable by the file owner without privilege; a forged
    /// metacopy xattr could redirect reads to arbitrary lower-layer files.
    /// See [Section 14.8](#overlayfs-union-filesystem-for-containers--metacopy-trust-model-and-security-constraints)
    /// for the complete trust model and enforcement mechanism.
    pub metacopy: bool,

    /// Directory rename/redirect handling.
    ///
    /// - `On`: Enable redirect xattrs for directory renames. Required
    ///   for rename(2) on merged directories to succeed (without this,
    ///   rename of a directory that exists in a lower layer returns EXDEV).
    /// - `Follow`: Follow existing redirect xattrs but do not create new
    ///   ones. Safe for mounting layers created by a trusted system.
    /// - `NoFollow`: Ignore redirect xattrs entirely. Most restrictive.
    /// - `Off`: Disable redirect handling; directory renames return EXDEV.
    ///
    /// Default: `On` (required by Docker/containerd for correct semantics).
    pub redirect_dir: RedirectDirMode,

    /// Volatile mode. When enabled, overlayfs skips all fsync/sync_fs calls
    /// to the upper filesystem. A crash or power loss may leave the upper
    /// layer in an inconsistent state (workdir staging artifacts, partial
    /// copy-ups). The overlay refuses to remount if it detects a previous
    /// volatile session that was not cleanly unmounted.
    ///
    /// Docker uses volatile mode for ephemeral containers where persistence
    /// is not needed (CI runners, build containers, test environments).
    ///
    /// Default: false.
    pub volatile: bool,

    /// Use `user.overlay.*` xattr namespace instead of `trusted.overlay.*`.
    /// Required for unprivileged (rootless) overlayfs mounts where the
    /// calling process lacks CAP_SYS_ADMIN in the initial user namespace.
    /// The `user.*` xattr namespace is writable by the file owner without
    /// special capabilities.
    ///
    /// Default: false (use `trusted.overlay.*`).
    pub userxattr: bool,

    /// Extended inode number mode. Controls how overlayfs composes inode
    /// numbers to guarantee uniqueness across layers.
    ///
    /// - `On`: Compose inode numbers using upper bits for layer index.
    ///   Requires underlying filesystems to use <32-bit inode numbers
    ///   (ext4, XFS with `inode32` mount option).
    /// - `Off`: Use raw underlying inode numbers. Risk of collisions
    ///   across layers (two files on different layers may share an ino).
    /// - `Auto`: Enable if all underlying filesystems have small enough
    ///   inode numbers; disable otherwise.
    ///
    /// Default: `Auto`.
    pub xino: XinoMode,

    /// NFS export support. When enabled, overlayfs maintains an index
    /// directory (inside workdir) that maps NFS file handles to overlay
    /// dentries. Required if the overlay mount will be exported via NFS.
    ///
    /// Default: false (NFS export of container filesystems is uncommon).
    pub nfs_export: bool,

    /// fs-verity digest validation for lower layer files. When enabled,
    /// overlayfs verifies that lower-layer files have valid fs-verity
    /// digests matching the expected values stored in the upper layer's
    /// metacopy xattr. Provides content integrity for container image
    /// layers without requiring dm-verity on the entire block device.
    ///
    /// - `Off`: No verity checking.
    /// - `On`: Verify if digest is present; allow files without digest.
    /// - `Require`: Reject files that lack a valid fs-verity digest.
    ///
    /// Default: `Off`.
    pub verity: VerityMode,
}

/// Redirect directory mode.
#[derive(Clone, Copy, PartialEq, Eq)]
pub enum RedirectDirMode {
    /// Create and follow redirect xattrs.
    On,
    /// Follow existing redirect xattrs but do not create new ones.
    Follow,
    /// Do not follow redirect xattrs.
    NoFollow,
    /// Disable redirect handling; directory renames return EXDEV.
    Off,
}

/// Extended inode number composition mode.
#[derive(Clone, Copy, PartialEq, Eq)]
pub enum XinoMode {
    /// Always compose inode numbers.
    On,
    /// Never compose inode numbers.
    Off,
    /// Compose if underlying inode numbers fit.
    Auto,
}

/// fs-verity enforcement mode for lower layer files.
#[derive(Clone, Copy, PartialEq, Eq)]
pub enum VerityMode {
    /// No verity checking.
    Off,
    /// Verify if digest present; allow files without digest.
    On,
    /// Reject lower files without valid fs-verity digest.
    Require,
}

14.8.2 Core Data Structures

/// Overlay filesystem superblock state. One instance per overlay mount.
/// Created by `OverlayFs::mount()` and stored in the `SuperBlock`'s
/// filesystem-private field.
pub struct OverlaySuperBlock {
    /// Lower layer inodes (topmost first). Index 0 is the highest-priority
    /// lower layer (searched first after upper). These are directory inodes
    /// on the underlying filesystems, held for the mount's lifetime.
    ///
    /// Heap-allocated rather than inline (`ArrayVec<_, 500>` would exceed
    /// the safe stack frame budget — each `OverlayLayer` contains an
    /// `InodeId`, a `SuperBlock` reference, and a `u16` index). The
    /// 500-layer maximum is enforced at mount validation time. Mount
    /// processing is a rare, non-hot-path operation where heap allocation
    /// is acceptable.
    pub lower_layers: Box<[OverlayLayer]>,

    /// Upper layer state. `None` for read-only overlay mounts.
    pub upper_layer: Option<OverlayLayer>,

    /// Work directory inode on the upper filesystem. Used as a staging
    /// area for atomic copy-up operations.
    pub work_dir: Option<InodeId>,

    /// Index directory inode (inside workdir). Used for NFS export file
    /// handle resolution and hard link tracking across copy-up.
    pub index_dir: Option<InodeId>,

    /// Parsed mount options (immutable after mount).
    pub config: OverlayMountOptions,

    /// The xattr prefix used for overlay-private xattrs. Either
    /// `"trusted.overlay."` (privileged) or `"user.overlay."` (userxattr
    /// mode). Stored once to avoid branching on every xattr operation.
    pub xattr_prefix: &'static [u8],

    /// Volatile session marker. If volatile mode is enabled, this is set
    /// to true after creating the `$workdir/work/incompat/volatile`
    /// sentinel directory. On mount, if the sentinel exists from a
    /// previous unclean session, mount fails with EINVAL.
    pub volatile_active: bool,

    /// True if this overlay was mounted from within a user namespace or
    /// if the upper layer's filesystem mount is owned by a non-initial
    /// user namespace. When true, `metacopy` and `redirect_dir=on` are
    /// disabled regardless of mount options, `userxattr` mode is
    /// mandatory, and data-only lower layers are rejected.
    ///
    /// Set once at `OverlayFs::mount()` time by checking whether the
    /// calling process's user namespace is the initial user namespace
    /// (the calling credential's user namespace has level 0). Immutable thereafter.
    ///
    /// See Section 14.4.6.1 for the full security model.
    pub userns_influenced: bool,
}

/// A single layer in the overlay stack.
pub struct OverlayLayer {
    /// Root directory inode of this layer on its underlying filesystem.
    pub root: InodeId,

    /// Superblock of the underlying filesystem. Arc reference held for
    /// the overlay mount's lifetime to prevent the underlying FS from
    /// being unmounted while the overlay is active.
    pub sb: Arc<SuperBlock>,

    /// Layer index (0 = upper or topmost lower; increases downward).
    /// Used for xino composition and for identifying which layer an
    /// overlay inode's data resides on.
    pub index: u16,
}

/// Atomic optional value using a sentinel for the `None` state.
/// `InodeId` of 0 represents `None` (inode 0 is never valid in any filesystem).
/// Provides lock-free read access via `Acquire` load and one-time write
/// via `compare_exchange` (for copy-up transitions from None -> Some).
pub struct AtomicOption<T: Into<u64> + From<u64>> {
    value: AtomicU64,  // 0 = None, non-zero = Some(T)
}

impl AtomicOption<InodeId> {
    pub fn none() -> Self { Self { value: AtomicU64::new(0) } }
    pub fn load(&self) -> Option<InodeId> {
        match self.value.load(Ordering::Acquire) {
            0 => None,
            v => Some(InodeId(v)),
        }
    }
    /// Atomically transition from None to Some. Returns Err if already set.
    pub fn set_once(&self, val: InodeId) -> Result<(), InodeId> {
        self.value.compare_exchange(0, val.0, Ordering::AcqRel, Ordering::Acquire)
            .map(|_| ())
            .map_err(|v| InodeId(v))
    }
}

/// Per-inode overlay state. Tracks which layers contribute to a merged
/// view of this inode.
///
/// An `OverlayInode` is created on first lookup and cached in the VFS
/// inode cache. It is the filesystem-private data attached to the VFS
/// inode via `InodeId`.
pub struct OverlayInode {
    /// Inode in the upper layer. `Some` if the entry exists in upper
    /// (either originally or after copy-up). `None` if the entry exists
    /// only in lower layers.
    ///
    /// Protected by `copy_up_lock`: transitions from `None` to `Some`
    /// exactly once during copy-up. Once set, never changes back.
    /// Reads after copy-up are lock-free (Acquire load on the Option
    /// discriminant).
    pub upper: AtomicOption<InodeId>,

    /// Inode in the topmost lower layer that contains this entry.
    /// `None` if the entry exists only in upper (newly created file).
    pub lower: Option<LowerInodeRef>,

    /// 1 if this inode is a metacopy-only upper entry (metadata
    /// copied, data still in lower layer). Cleared to 0 after full
    /// data copy-up completes. Uses AtomicU8 (not AtomicBool) to avoid
    /// the bool validity invariant — Tier 1 intra-domain memory
    /// corruption from a co-domain module could write a non-0/1 value,
    /// which would be undefined behavior for AtomicBool.
    /// 0 = no metacopy, 1 = metacopy.
    pub metacopy: AtomicU8,

    /// True if this is an opaque directory. An opaque directory hides
    /// all entries from lower layers — readdir and lookup do not
    /// descend into lower layers below this point.
    pub opaque: bool,

    /// Redirect path for directory renames. When a merged directory is
    /// renamed in the upper layer, this field stores the original lower
    /// path so that lookups can find the renamed directory's lower
    /// contents. `None` for non-redirected entries.
    pub redirect: Option<Box<OsStr>>,

    /// Lock serializing copy-up operations on this inode. Only one
    /// thread may copy-up a given inode at a time. Other threads
    /// attempting to modify the same lower-layer file block on this
    /// lock until copy-up completes, then proceed against the upper copy.
    ///
    /// This is a `Mutex`, not an `RwLock`, because copy-up is an
    /// exclusive state transition (None -> Some). Read paths check
    /// `upper` with an Acquire load and only take the lock if they
    /// need to trigger copy-up.
    pub copy_up_lock: Mutex<()>,

    /// Overlay inode type. Needed because the overlay may present a
    /// different view than the underlying filesystem (e.g., a whiteout
    /// character device appears as "entry does not exist").
    pub inode_type: OverlayInodeType,
}

/// Reference to a lower-layer inode.
pub struct LowerInodeRef {
    /// Inode ID on the lower layer's filesystem.
    pub inode: InodeId,
    /// Which lower layer this inode resides on (index into
    /// `OverlaySuperBlock::lower_layers`).
    pub layer_index: u16,
}

/// Overlay inode type classification.
#[derive(Clone, Copy, PartialEq, Eq)]
pub enum OverlayInodeType {
    /// Regular file (may be metacopy).
    Regular,
    /// Directory (may be merged or opaque).
    Directory,
    /// Symbolic link.
    Symlink,
    /// Character device, block device, FIFO, or socket.
    Special,
    /// Whiteout entry (exists in upper layer to mark deletion of a
    /// lower-layer entry). Not visible to userspace — lookups return
    /// ENOENT. Internally represented as either a character device
    /// with major:minor 0:0 or a zero-size file with the
    /// `trusted.overlay.whiteout` xattr.
    Whiteout,
}

14.8.3 Overlay Dentry Operations

overlayfs requires custom DentryOps to handle the dynamic nature of the merged filesystem view. Copy-up changes which layer serves a file, so cached dentries must be revalidated.

/// Zero-sized carrier for overlayfs's `DentryOps` vtable implementation. Holds
/// no state; all overlay context is reached through the `InodeId`/`Dentry`
/// arguments and `overlay_super_block()`.
pub struct OverlayDentryOps;

/// overlayfs dentry operations.
impl DentryOps for OverlayDentryOps {
    /// Revalidate a cached overlay dentry.
    ///
    /// Returns `false` (forcing re-lookup) in these cases:
    /// 1. The overlay inode has been copied up since the dentry was cached
    ///    (detected by checking if `OverlayInode::upper` transitioned from
    ///    None to Some since the last lookup).
    /// 2. The underlying filesystem's dentry has been invalidated (delegates
    ///    to the underlying filesystem's `d_revalidate` if it implements one,
    ///    e.g., for NFS lower layers).
    /// 3. A whiteout has been created or removed in the upper layer for this
    ///    name (detected by checking upper-layer lookup result against cached
    ///    overlay state).
    ///
    /// Returns `true` (dentry is still valid) in all other cases.
    fn d_revalidate(&self, parent: InodeId, name: &OsStr) -> Result<bool>;

    /// Overlay dentries use the default VFS hash (SipHash-1-3).
    fn d_hash(&self, _name: &OsStr) -> Option<u64> {
        None
    }

    /// Overlay dentries are always eligible for LRU caching.
    fn d_delete(&self, _inode: InodeId, _name: &OsStr) -> bool {
        true
    }

    /// On dentry release, drop the overlay inode's references to
    /// underlying filesystem inodes.
    fn d_release(&self, inode: InodeId, name: &OsStr);
}

Dentry cache interaction: A copy-up performs NO dentry-cache invalidation. The overlay dcache entry (Section 14.1) binds the name to the same OverlayInode that copy-up updates in place, so after copy-up nothing the cached dentry reaches is stale. The post-copy-up sequence:

1. Copy-up completes (new file exists in upper layer).
2. OverlayInode::upper is set via an atomic Release store.
3. No dcache action is required: the cached dentry references the same OverlayInode, whose upper now loads Some on every access.
4. Any dentry cached against pre-copy-up state is additionally caught by OverlayDentryOps::d_revalidate case 1 above, which forces re-lookup.

Negative dentry handling: Negative dentries (cached ENOENT results) in the overlay dentry cache are invalidated when: - A new file is created in the upper layer (the negative dentry for that name must be purged). - A whiteout is removed (the previously-hidden lower-layer entry becomes visible again).

14.8.4 Lookup Algorithm

overlayfs lookup implements the layer search order:

OverlayInodeOps::lookup(parent: InodeId, name: &OsStr) -> Result<InodeId>:

  let overlay_parent = get_overlay_inode(parent)

  // Step 1: Search upper layer (if writable overlay).
  if let Some(upper_dir) = overlay_parent.upper {
      match underlying_lookup(upper_dir, name) {
          Ok(upper_inode) => {
              // Check if this is a whiteout.
              if is_whiteout(upper_inode) {
                  // Entry was deleted. Do NOT search lower layers.
                  // Cache a negative dentry.
                  return Err(ENOENT)
              }
              // Check if this is an opaque directory.
              let opaque = is_opaque_dir(upper_inode)
              // Found in upper. If directory and not opaque, may need
              // to merge with lower layers.
              if is_directory(upper_inode) && !opaque {
                  // Merged directory: upper exists, also search lower
                  // for the merge view.
                  let lower = find_in_lower_layers(overlay_parent, name)
                  return create_overlay_inode(Some(upper_inode), lower, ...)
              }
              // Non-directory or opaque directory: upper is authoritative.
              return create_overlay_inode(Some(upper_inode), None, ...)
          }
          Err(ENOENT) => {
              // Not in upper, fall through to lower layers.
          }
          Err(e) => return Err(e),  // Propagate I/O errors.
      }
  }

  // Step 2: Search lower layers (topmost first).
  // If parent directory has a redirect, follow it.
  for (layer_idx, lower_layer) in lower_layers_for(overlay_parent) {
      match underlying_lookup(lower_dir_at(lower_layer, overlay_parent), name) {
          Ok(lower_inode) => {
              if is_whiteout(lower_inode) {
                  // Whiteout in this lower layer. Stop searching.
                  return Err(ENOENT)
              }
              return create_overlay_inode(None, Some(LowerInodeRef {
                  inode: lower_inode,
                  layer_index: layer_idx,
              }), ...)
          }
          Err(ENOENT) => continue,  // Try next lower layer.
          Err(e) => return Err(e),
      }
  }

  // Not found in any layer.
  Err(ENOENT)

Whiteout detection: An upper-layer entry is a whiteout if either: - It is a character device with major:minor 0:0 (traditional format), OR - It is a zero-size regular file with the trusted.overlay.whiteout (or user.overlay.whiteout in userxattr mode) xattr set.

Both formats are supported for compatibility with existing container images. UmkaOS creates whiteouts using the xattr format by default (avoids requiring mknod capability for character device creation in unprivileged containers).

Opaque directory detection: A directory is opaque if it has the xattr trusted.overlay.opaque (or user.overlay.opaque) set to "y". An opaque directory hides all entries from lower layers — lookups do not descend past it. This is used when an entire directory is deleted and recreated in the upper layer.

14.8.5 Copy-Up Protocol

Copy-up is the central operation of overlayfs. When a lower-layer file must be modified, its contents (and/or metadata) are first copied to the upper layer. The copy-up must be atomic from the perspective of concurrent readers: at no point should a reader see a partially-copied file.

Full copy-up algorithm (for regular files when metacopy is disabled, or on first write to a metacopy-only file):

copy_up(overlay_inode: &OverlayInode) -> Result<InodeId>:

  // Fast path: already copied up.
  if let Some(upper) = overlay_inode.upper.load(Acquire) {
      if overlay_inode.metacopy.load(Acquire) == 0 {
          return Ok(upper)  // Fully copied up already.
      }
      // Metacopy exists but needs full data copy. Fall through.
  }

  // Slow path: take copy-up lock.
  let _guard = overlay_inode.copy_up_lock.lock()

  // Double-check after acquiring lock (another thread may have completed
  // copy-up while we waited).
  if let Some(upper) = overlay_inode.upper.load(Acquire) {
      if overlay_inode.metacopy.load(Acquire) == 0 {
          return Ok(upper)
      }
  }

  let lower = overlay_inode.lower.as_ref().expect("copy-up requires lower");
  let sb = overlay_super_block()
  // Copy-up only runs on writable overlays, where the work directory always
  // exists (validated at mount). Unwrap the Option<InodeId> explicitly.
  let work_dir = sb.work_dir.expect("writable overlay has workdir");

  // Step 1: Ensure parent directory exists in upper layer.
  // Recursively copy-up parent directories if needed.
  let upper_parent = ensure_upper_parent(overlay_inode)

  // Step 2: Create temporary file in workdir (same filesystem as upper).
  // The workdir is on the same device as upperdir, enabling atomic rename.
  let tmp_name = generate_temp_name()  // e.g., "#overlay.XXXXXXXX"
  let tmp_inode = underlying_create(work_dir, tmp_name, lower_mode)

  // Step 3: Copy metadata from lower to tmp.
  let lower_attr = underlying_getattr(lower.inode)

  // CVE-2023-0386 mitigation: Verify that the source file's UID/GID are valid
  // in the overlay mount's user namespace. A setuid file in the lower layer
  // whose UID has no mapping in the overlay's userns must NOT be copied up
  // with elevated privileges. Reject with EOVERFLOW if unmappable.
  // The translation must be STRICT (`uid_from_global`/`gid_from_global`,
  // [Section 17.1](17-containers.md#namespace-architecture)): a lenient translation substitutes the
  // overflow id instead of failing, so it can never report "unmappable" and
  // would silently disable this guard. `lower_attr.uid`/`.gid` are in the
  // kernel-internal global representation, which is what these take.
  if overlay_mnt_userns.uid_from_global(lower_attr.uid).is_none()
      || overlay_mnt_userns.gid_from_global(lower_attr.gid).is_none() {
      underlying_unlink(work_dir, tmp_name)  // clean up temp file
      return Err(EOVERFLOW)
  }

  underlying_setattr(tmp_inode, &lower_attr)  // owner, mode, timestamps

  // Step 4: Copy xattrs from lower to tmp.
  // Filter out overlay-private xattrs (trusted.overlay.*).
  copy_xattrs_filtered(lower.inode, tmp_inode, sb.xattr_prefix)

  // Step 5: Copy file data (skip if metacopy mode and this is a
  // metadata-only copy-up triggered by chmod/chown/utimes).
  if !metacopy_only {
      copy_file_data_chunked(lower.inode, tmp_inode, &overlay_inode.copy_up_lock)
      // Uses chunked I/O with periodic lock release. See "Chunked Copy-Up
      // and Cgroup I/O Throttling" below for the algorithm.
  } else {
      // Set metacopy xattr on the tmp file. This marks it as containing
      // metadata only — data will be copied on first write.
      underlying_setxattr(tmp_inode,
          concat_static(sb.xattr_prefix, "metacopy"), b"", XattrFlags::empty())

      // If the lower file is itself a metacopy (nested overlay), follow
      // the redirect chain to find the actual data source.
      if let Some(origin) = get_metacopy_origin(lower.inode) {
          underlying_setxattr(tmp_inode,
              concat_static(sb.xattr_prefix, "origin"), &encode_fh(origin), XattrFlags::empty())
      }
  }

  // Step 6: Set security context on tmp file.
  // Copy security.* xattrs that the security framework requires.

  // Step 7: Atomic rename from workdir to upperdir.
  // This is the commit point. Before this rename, the copy-up is invisible
  // to other processes. After this rename, the upper-layer file is live.
  underlying_rename(work_dir, tmp_name, upper_parent, target_name,
                    RenameFlags::RENAME_NOREPLACE)

  // Step 8: Update overlay inode state.
  let upper_inode = underlying_lookup(upper_parent, target_name)
  // set_once(): CAS from None to Some. The copy_up_lock guarantees single
  // writer, so set_once always succeeds (debug_assert to catch invariant violations).
  overlay_inode.upper.set_once(upper_inode)
      .expect("copy_up_lock guarantees single writer");
  if metacopy_only {
      overlay_inode.metacopy.store(1, Release)  // 1 = metacopy (AtomicU8)
  }

  // No dcache step follows: the cached overlay dentry names this same
  // `OverlayInode`, whose `upper` now loads `Some`. See "Dentry cache
  // interaction" above.

  Ok(upper_inode)

Atomicity guarantee: The rename in Step 7 is the single atomic commit point. If the system crashes before Step 7, the temporary file in workdir is orphaned and cleaned up on next mount (overlayfs scans workdir for stale temporaries during mount() and removes them). If the system crashes after Step 7, the upper-layer file is complete and consistent.

Error recovery (runtime failures): Each step that can fail must clean up all prior steps before returning an error to the caller. The copy-up state machine tracks progress through four states:

/// Copy-up state machine. Tracks the current phase of a copy-up operation
/// for error recovery. Stored on the stack (not persistent — crash recovery
/// uses workdir scan, not state machine replay).
#[derive(Clone, Copy, PartialEq, Eq, Debug)]
pub enum CopyUpState {
    /// Step 2: Creating the temporary file in workdir.
    /// Cleanup on failure: none (nothing created yet).
    Creating,

    /// Steps 3-6: Copying metadata, xattrs, data, and security context
    /// to the temporary file.
    /// Cleanup on failure: unlink temporary file from workdir.
    Copying,

    /// Step 7: Atomic rename from workdir to upperdir.
    /// Cleanup on failure: unlink temporary file from workdir.
    Renaming,

    /// Step 8: Rename succeeded. Updating overlay inode state.
    /// No cleanup needed — the upper file is committed and will be
    /// found on retry if post-rename steps fail.
    Complete,
}

Error recovery protocol (driven by CopyUpState):

CopyUpState when error occurs	Cleanup action	Returned error	Lower file status
Creating (Step 2 fails)	None (nothing was created)	EIO / ENOSPC	Unchanged
Copying (Steps 3-6 fail)	underlying_unlink(workdir, tmp_name) — remove partial temp file	EIO / ENOSPC	Unchanged
Renaming (Step 7 fails)	underlying_unlink(workdir, tmp_name) — remove complete-but-uncommitted temp file	EIO	Unchanged
Complete (Step 8 fails)	None — rename already committed; upper file is live and will be discovered on retry	EIO (rare)	Now shadowed by upper

The error recovery path:

copy_up_with_recovery(overlay_inode) -> Result<InodeId>:
  let mut state = CopyUpState::Creating;
  // A writable overlay always has a workdir (enforced at mount) — unwrap the
  // Option once so the underlying_* calls receive the InodeId they expect.
  let work_dir = sb.work_dir.expect("writable overlay has workdir");

  let result = (|| -> Result<InodeId> {
      // Step 2: Create temp file.
      let tmp = underlying_create(work_dir, tmp_name, mode)?;
      state = CopyUpState::Copying;

      // Steps 3-6: Copy metadata, xattrs, data, security context.
      copy_metadata(lower, tmp)?;
      copy_xattrs_filtered(lower, tmp, prefix)?;
      if !metacopy_only {
          match copy_file_data_chunked(lower, tmp, &copy_up_lock) {
              Ok(()) => {}
              Err(EALREADY) => {
                  // Another thread completed copy-up while we released
                  // the lock between chunks. Clean up our temp file and
                  // return the already-committed upper inode.
                  let _ = underlying_unlink(work_dir, tmp_name);
                  return Ok(overlay_inode.upper.load(Acquire).unwrap())
              }
              Err(e) => return Err(e),
          }
      }
      copy_security_context(lower, tmp)?;
      state = CopyUpState::Renaming;

      // Step 7: Atomic rename.
      underlying_rename(work_dir, tmp_name, upper_parent, name,
                        RENAME_NOREPLACE)?;
      state = CopyUpState::Complete;

      // Step 8: Update overlay state. No dcache step follows.
      overlay_inode.upper.set_once(upper)
          .expect("copy_up_lock guarantees single writer");
      Ok(upper)
  })();

  if let Err(e) = &result {
      match state {
          CopyUpState::Creating => {
              // Nothing to clean up.
          }
          CopyUpState::Copying | CopyUpState::Renaming => {
              // Best-effort cleanup: remove the temporary file.
              if let Err(unlink_err) = underlying_unlink(work_dir, tmp_name) {
                  // Unlink failed — orphan left in workdir.
                  // mount() workdir scan will remove it.
                  log_warn!("copy-up cleanup failed: {:?}", unlink_err);
              }
          }
          CopyUpState::Complete => {
              // Rename committed. Upper file is live. No cleanup.
          }
      }
  }
  result

Key invariant: The original lower-layer file is never modified or removed during copy-up. If any step fails before CopyUpState::Complete, the lower file remains intact and serves subsequent reads. The temporary file in workdir is either successfully cleaned up or left as an orphan for the next mount scan.

If cleanup of the temporary file itself fails (i.e., underlying_unlink() returns an error during recovery), the orphaned temporary is left in workdir and will be removed by the next mount() scan. The original copy-up failure is still returned to the caller as an error. The orphaned file does not affect correctness because the rename (Step 7) did not complete.

Parent directory copy-up: Directories are copied up recursively. When copying up /a/b/c/file.txt, if /a/b/c/ does not exist in upper, overlayfs creates /a/, then /a/b/, then /a/b/c/ in upper (each with appropriate metadata and the trusted.overlay.origin xattr pointing to the lower original). Only then does the file copy-up proceed. Each directory copy-up is itself atomic (created in workdir, renamed to upper).

Hard link handling on copy-up: If a lower-layer file has multiple hard links (nlink > 1), all names referencing the same lower inode must resolve to the same upper inode after copy-up. The overlay maintains an index directory (inside workdir) that maps lower file handles to upper inodes. On copy-up, the overlay checks the index first: - If an index entry exists, the file was already copied up via another name. Create a hard link in upper rather than copying data again. - If no index entry exists, perform a full copy-up and record the mapping.

This index is also used for NFS export (mapping file handles across copy-up).

14.8.5.1 Chunked Copy-Up and Cgroup I/O Throttling

Copy-up data transfer (Step 5) can be arbitrarily large — container images routinely contain multi-gigabyte database files, ML model weights, or log archives. A naive single-pass copy_file_data() holds copy_up_lock for the entire transfer duration. Under cgroup io.max throttling (Section 17.2), this duration is further extended because each write is subject to the cgroup's byte-rate and IOPS limits. The result: all other threads attempting any metadata or write operation on the same file block on copy_up_lock for the full throttled transfer time — potentially minutes for a large file under tight io.max limits.

Solution: Chunked copy-up with periodic lock release. The data copy is split into fixed-size chunks. Between chunks, the copy_up_lock is released, allowing blocked threads to observe the in-progress copy-up and either wait briefly or proceed (if the copy-up completed during their wait). The temporary file in workdir is not visible to other overlay operations until the final atomic rename (Step 7), so releasing the lock during data copy does not expose partial state.

/// Chunk size for copy-up data transfer. 2 MiB balances:
/// - Throughput: large enough to amortize splice/sendfile setup overhead.
/// - Latency: small enough that lock-hold time per chunk is bounded (~2ms
///   at 1 GB/s disk throughput, longer under io.max throttling).
/// - Memory: the chunk is transferred in-place (splice) or via a bounded
///   kernel buffer, not allocated as a contiguous 2 MiB region.
const COPY_UP_CHUNK_SIZE: u64 = 2 * 1024 * 1024;  // 2 MiB

copy_file_data_chunked(
    lower: InodeId,
    tmp: InodeId,
    lock: &Mutex<()>,
) -> Result<()>:
  let file_size = underlying_getattr(lower).size
  let mut offset: u64 = 0

  while offset < file_size {
      let chunk_len = min(COPY_UP_CHUNK_SIZE, file_size - offset)

      // Copy one chunk. Uses splice/sendfile for zero-copy where the
      // underlying filesystem supports it; falls back to read+write.
      // The write side is subject to the calling task's cgroup io.max
      // throttling — cgroup_io_throttle() is called inside the block
      // layer's submit() path, which may sleep if the cgroup's
      // byte-rate or IOPS budget is exhausted.
      copy_file_range(lower, tmp, offset, chunk_len)?

      offset += chunk_len

      if offset < file_size {
          // Release the copy-up lock between chunks. This allows other
          // threads blocked on copy_up_lock to wake and re-check state.
          // They will find upper still None (the rename has not happened)
          // and re-acquire the lock. The first thread to re-acquire
          // continues the copy from where it left off.
          //
          // SAFETY of releasing mid-copy: the temporary file is in workdir,
          // invisible to overlay lookups. No concurrent thread can observe
          // partial data. The only visible state change is the lock
          // becoming briefly available, which lets waiters check for
          // completion and yields the CPU to higher-priority tasks.
          drop(lock.unlock())

          // Yield point: allow the scheduler to run higher-priority tasks
          // (especially relevant when this copy-up is in a low-priority
          // cgroup). Also allows signal delivery (SIGKILL check).
          if signal_pending(current_task()) {
              // Copy-up interrupted by fatal signal. The temporary file
              // will be cleaned up by the error recovery path (CopyUpState::Copying).
              return Err(EINTR)
          }

          // Re-acquire the lock before the next chunk.
          lock.lock()

          // Double-check: another thread may have completed the copy-up
          // while we released the lock (race between two writers on the
          // same metacopy file). If so, abandon our temp file — the
          // error recovery path will unlink it.
          if overlay_inode.upper.load(Acquire).is_some()
              && overlay_inode.metacopy.load(Acquire) == 0 {
              return Err(EALREADY)  // caller detects this and returns Ok
          }
      }
  }

  Ok(())

Interaction with io.max throttling: Each chunk's write passes through the block layer's submit() path. If the calling task's cgroup has io.max limits, the throttle check (cgroup_io_throttle() in Section 17.2) sleeps until the cgroup's token bucket replenishes. This sleep occurs while holding the copy-up lock for the current chunk only — at most COPY_UP_CHUNK_SIZE / wbps seconds per chunk hold. Between chunks, the lock is released, so other threads are unblocked.

Worst-case lock-hold time per chunk: COPY_UP_CHUNK_SIZE / min(disk_throughput, io.max.wbps). At the minimum practical io.max setting of 1 MB/s, a 2 MiB chunk holds the lock for ~2 seconds. This is acceptable because: 1. Threads waiting on copy_up_lock are already blocked on a slow-path mutation. 2. The 2-second hold is bounded and predictable, unlike the unbounded hold of a single-pass copy of a multi-gigabyte file. 3. Under normal (unthrottled) I/O, the hold time per chunk is ~2ms at 1 GB/s.

EALREADY sentinel: When copy_file_data_chunked returns Err(EALREADY), the caller (copy_up_with_recovery) recognizes this as a benign race — another thread completed the copy-up. The caller cleans up its temporary file and returns the already-committed upper inode. This is handled in the CopyUpState::Copying error recovery branch (temp file unlink).

Signal handling: The signal_pending() check between chunks allows SIGKILL to abort a long-running copy-up promptly (within one chunk transfer time) instead of only after the entire file is copied. The error recovery path cleans up the partial temporary file.

14.8.6 Metacopy Mode

Metacopy is the performance-critical optimization for container startup. Without metacopy, any metadata operation (chmod, chown, utimes) on a lower-layer file triggers a full data copy. With metacopy enabled, only metadata is copied, and data copy is deferred until the file is opened for writing.

Metacopy lifecycle:

State transitions for a file in metacopy mode:

  [Lower-only]
      │
      │ chmod/chown/utimes/setxattr
      ▼
  [Metacopy in upper]   ← metadata copied, data in lower
      │                    upper has trusted.overlay.metacopy xattr
      │ open(O_WRONLY/O_RDWR) or truncate
      ▼
  [Full copy-up]         ← data + metadata in upper
                           trusted.overlay.metacopy xattr removed

Realfile mechanism: An overlay open resolves the underlying real inode — upper if it exists, else lower — and RETURNS it as OpenOutcome::data_inode from OverlayFileOps::open (Section 14.1). The generic open path writes that binding into the description's data_inode before the descriptor is published, so every page-cache operation on the resulting open file addresses the real filesystem's AddressSpace directly. Overlayfs has no page cache of its own — it delegates entirely to the underlying filesystems.

The binding is per-open and immutable once published: an fd opened before a later copy-up keeps its resolved lower inode, which is contract-faithful because that lower data was identical at copy-up time, and a write-class open triggers copy-up FIRST (see "Write trigger" below) and therefore always binds upper. No live open file is ever re-pointed.

Read path for metacopy files: When a metacopy file is opened for reading (O_RDONLY), data is served from the lower layer. The OverlayFileOps::read() implementation checks overlay_inode.metacopy and dispatches to the lower-layer FileOps::read() with the lower inode. No data copy occurs.

Concurrent metacopy copy-up: If two tasks trigger copy-up for the same metacopy inode simultaneously, the first task to acquire the inode's copy_up_lock mutex (see OverlayInode::copy_up_lock) performs the copy-up. The second task waits on copy_up_lock and, after acquiring it, checks whether copy-up already completed (oi.upper is now Some). If so, it skips the copy-up and returns the upper inode as the OpenOutcome::data_inode of the description it is constructing — all of this happens before that description is published, so the second task rebinds nothing that anyone can observe.

Write trigger: When a metacopy file is opened for writing (O_WRONLY, O_RDWR) or truncated, the overlay triggers a full data copy-up before allowing the write:

/// Zero-sized carrier for overlayfs's `FileOps` vtable implementation. Holds no
/// state; the served layer is selected per call from the `OverlayInode`.
pub struct OverlayFileOps;

impl FileOps for OverlayFileOps {
    fn open(&self, inode: InodeId, flags: OpenFlags) -> Result<OpenOutcome> {
        let oi = get_overlay_inode(inode);
        let sb = overlay_super_block();

        // If opening for write and file is metacopy-only, trigger
        // full data copy-up before returning the fd.
        if flags.is_writable() && oi.metacopy.load(Acquire) != 0 {
            copy_up_data(oi)?;
            // copy_up_data() copies file data from lower to upper,
            // removes the metacopy xattr, and clears oi.metacopy.
        }

        // Resolve the real inode backing THIS open's data, and the layer it
        // lives on: upper if the copy-up above (or an earlier one) produced
        // it, else the topmost lower entry. Resolved once, here — the
        // binding is per-open and never revisited afterwards.
        let (real, layer) = match oi.upper.load(Acquire) {
            Some(upper) => (
                upper,
                // `oi.upper` is `Some` only on an overlay that HAS an upper
                // layer: a read-only overlay never produces one.
                sb.upper_layer.as_ref().expect("upper entry implies upper layer"),
            ),
            None => {
                // Read-only open on a lower-only file. No copy-up needed.
                let lower = oi.lower.as_ref().unwrap();
                (lower.inode, &sb.lower_layers[lower.layer_index as usize])
            }
        };

        // Delegate the open to the filesystem that owns `real`.
        let inner = underlying_open(real, flags)?;

        // Bind the data inode. A NESTED stacking layer that resolved its own
        // binding wins — propagate the DEEPEST one, since that is the inode
        // whose `AddressSpace` actually holds the data.
        let data_inode = match inner.data_inode {
            Some(deeper) => deeper,
            None => layer_inode(layer, real)?,
        };
        Ok(OpenOutcome { private: inner.private, data_inode: Some(data_inode) })
    }
}

14.8.6.1 Metacopy Trust Model and Security Constraints

The metacopy mechanism is only safe when the kernel can trust that trusted.overlay.metacopy (or user.overlay.metacopy in userxattr mode) was written by the overlay itself during a copy-up, not forged by a process with write access to the upper layer. If forged, an attacker could create a file whose upper stub has a redirect xattr pointing to an arbitrary path in a lower layer, then set the metacopy xattr to tell the kernel to serve lower-layer data through the stub — exposing files the attacker would not otherwise be able to read via the overlay's merged view.

Xattr namespace privilege boundary

The trusted. xattr namespace is the primary safeguard. The kernel checks CAP_SYS_ADMIN via has_cap() — which verifies the capability against the initial user namespace — not via has_ns_cap() (which would accept a user namespace root). This means:

A process that holds CAP_SYS_ADMIN only within a user namespace (i.e., container root mapped to an unprivileged host UID) cannot set or read trusted.* xattrs on the host filesystem. Only a process with CAP_SYS_ADMIN in the initial user namespace can write trusted.overlay.* xattrs.

This provides complete protection for overlayfs mounts created in the initial user namespace: container processes cannot forge trusted.overlay.metacopy or trusted.overlay.redirect xattrs because they lack the required capability on the host filesystem.

Privileged container caveat: A container that runs with CAP_SYS_ADMIN in the initial user namespace (not just the container's user namespace) CAN write trusted.* xattrs and could forge metacopy stubs. This is a known trust boundary: granting CAP_SYS_ADMIN in the initial namespace to a container is equivalent to granting root on the host. Operators who grant this should not rely on overlayfs metacopy for isolation.

User-namespace-influenced mounts: the attack surface

Since Linux 5.11, overlayfs can be mounted from within a user namespace (CAP_SYS_ADMIN in the user namespace that owns the mount namespace suffices to call mount("overlay", ...)). Such mounts are required to use userxattr mode (-o userxattr), which substitutes the user.overlay.* xattr namespace for trusted.overlay.*. Unlike trusted.*, the user.* namespace is writable by the file owner without any privilege — specifically, the unprivileged host UID that the container root maps to can set user.overlay.metacopy and user.overlay.redirect xattrs on files in the upper layer.

A user-namespace-influenced mount is defined as any overlayfs mount where either:

1. The overlayfs mount() call was made from within a user namespace (the calling process's user namespace is not the initial user namespace), or
2. The upper directory's owning user namespace differs from the initial user namespace (detected by comparing the user namespace of the mount namespace that created the upper directory's filesystem mount against the level-0 initial user namespace).

Enforcement: metacopy disabled for user-namespace-influenced mounts

UmkaOS enforces the following rule at mount time and at metacopy lookup time:

Mount-time enforcement: When OverlayFs::mount() is called from a process not in the initial user namespace, the metacopy and redirect_dir options are forced to off regardless of what the caller requested. The mount proceeds with these features disabled. The kernel logs:

overlayfs: metacopy and redirect_dir disabled for user-namespace mount (CVE mitigation, Section 14.4.6.1)

This matches Linux's behaviour (since kernel 5.11, user-namespace overlayfs mounts are restricted to userxattr mode and metacopy is not permitted unless the caller has CAP_SYS_ADMIN in the initial user namespace).

The OverlaySuperBlock records whether the mount is user-namespace-influenced:

pub struct OverlaySuperBlock {
    // ... existing fields ...

    /// True if this overlay was mounted from within a user namespace (the
    /// mounting process's user namespace is not the initial user namespace)
    /// or if the upper layer's filesystem mount is owned by a non-initial
    /// user namespace. When true, metacopy and redirect_dir are disabled
    /// regardless of mount options, and userxattr mode is mandatory.
    ///
    /// Set once at mount time; immutable thereafter.
    pub userns_influenced: bool,
}

Lookup-time enforcement: Even if metacopy is enabled in the mount options, the metacopy lookup path checks userns_influenced before reading or acting on any metacopy xattr:

/// Attempt to read a metacopy stub from the given upper-layer dentry.
/// Returns `None` (treat as a regular upper file) if:
///   - The mount is user-namespace-influenced, or
///   - No metacopy xattr is present, or
///   - The xattr value fails validation.
fn ovl_lookup_metacopy(dentry: &Dentry, sb: &OverlaySuperBlock) -> Option<OverlayMetacopy> {
    // Never trust metacopy xattrs from user-namespace-influenced mounts.
    // The xattr namespace used by such mounts (user.overlay.*) is writable
    // by the file owner without privilege, so any metacopy xattr present
    // must be treated as potentially forged.
    if sb.userns_influenced {
        return None;
    }

    // Read the metacopy xattr from the upper-layer file.
    let xattr_name = concat_static(sb.xattr_prefix, "metacopy");
    let xattr = dentry.get_xattr(xattr_name)?;

    // Validate xattr value. The Linux-compatible format is either empty
    // (legacy, no digest) or a 4+N byte structure: 4-byte header followed
    // by an optional fs-verity SHA-256 digest (32 bytes). Reject anything
    // that does not match either form.
    validate_metacopy_xattr(xattr)
}

The lookup-time check is defence-in-depth: the mount-time enforcement already prevents metacopy=on from reaching OverlaySuperBlock::config on user-namespace mounts, so ovl_lookup_metacopy would not be called. The redundant check in ovl_lookup_metacopy protects against future code paths that might bypass the mount-time gate.

Userxattr mode and data-only layers

When userxattr=on is set (required for user-namespace mounts), user.overlay.* xattrs are used throughout. The user.overlay.redirect xattr controls directory rename semantics and, in data-only layer configurations, points metacopy stubs to their data sources. Because user.* xattrs are writable by the file owner, and because data-only layer configurations allow a metacopy file in one lower layer to redirect to a file in a data-only lower layer via user.overlay.redirect:

• redirect_dir=on is disallowed for user-namespace-influenced mounts (forced to off at mount time).
• Data-only lower layers are disallowed for user-namespace-influenced mounts: OverlayFs::mount() returns EPERM if any lower layer path is specified with the :: data-only separator syntax when userns_influenced is true.

These restrictions prevent the user.overlay.redirect xattr from being used to point a metacopy stub in one layer at a file in another layer that the container would not otherwise be able to access.

Summary of security invariants

Condition	trusted.overlay.* metacopy	user.overlay.* metacopy
Initial user namespace mount, metacopy=on	Trusted (forging requires host CAP_SYS_ADMIN)	N/A (userxattr not used in privileged mounts by default)
User-namespace mount	N/A (trusted.* inaccessible from user NS)	Disabled (forced off at mount time; ovl_lookup_metacopy returns None)
User-namespace mount, userxattr=on, data-only layers	N/A	Rejected at mount time (EPERM)

14.8.7 Directory Operations

Readdir merge: Reading a merged directory (one that exists in both upper and lower layers) requires combining entries from all layers, excluding whiteouts and applying opaque directory semantics.

OverlayFileOps::readdir(inode, private, offset, emit) -> Result<()>:

  let oi = get_overlay_inode(inode)

  // Phase 1: Collect entries from upper layer.
  let mut seen: HashSet<OsString> = HashSet::new()
  if let Some(upper) = oi.upper.load(Acquire) {
      underlying_readdir(upper, |entry_inode, entry_off, ftype, name| {
          // Skip whiteout entries — they indicate deleted lower entries.
          if is_whiteout_entry(entry_inode) {
              seen.insert(name.to_owned())  // Track for lower suppression.
              return true  // Continue iteration.
          }
          seen.insert(name.to_owned())
          emit(overlay_inode_for(entry_inode), entry_off, ftype, name)
      })
  }

  // Phase 2: If directory is opaque, stop here. Lower entries are hidden.
  if oi.opaque {
      return Ok(())
  }

  // Phase 3: Collect entries from lower layers, skipping duplicates.
  for lower_ref in lower_dirs_for(oi) {
      underlying_readdir(lower_ref.inode, |entry_inode, entry_off, ftype, name| {
          // Skip entries already seen in upper or higher lower layers.
          if seen.contains(name) {
              return true
          }
          // Skip whiteout entries from lower layers too.
          if is_whiteout_entry(entry_inode) {
              seen.insert(name.to_owned())
              return true
          }
          seen.insert(name.to_owned())
          emit(overlay_inode_for(entry_inode), entry_off, ftype, name)
      })
  }

  Ok(())

Deduplication uses byte-exact filename comparison. Case-insensitive upper/lower layer combinations may produce duplicate entries with different casing. This matches Linux overlayfs behavior.

Readdir caching: The merged directory listing is cached in the overlay file's private state (returned by open()) for the lifetime of the open directory file descriptor. This matches Linux's behavior: the merge is computed once per opendir() and subsequent readdir() calls return entries from the cache. The cache is invalidated on rewinddir() (seek to offset 0).

Performance note on seen HashSet: The HashSet<OsString> in the pseudocode above is allocated once per opendir() call (during the initial merge), not once per readdir() call. The cache stores the deduplicated entry list; subsequent readdir() calls walk the already-merged cache without re-allocating or re-hashing. For large directories (>10,000 entries), the initial opendir() merge is O(N) with one allocation per distinct entry name (stored in the HashSet during merge, then released when the merge completes and entries are stored in a flat Vec in the file private state). The hot path — repeated readdir() calls iterating through the cached Vec — is O(entries) with zero heap allocations. Bound: The HashSet is bounded by the sum of directory entries across all layers for this single directory (typically <10,000 in container images; capped by the filesystem's max directory entries). This is a warm-path allocation (once per opendir(), bounded by directory size) and is acceptable per the collection usage policy (Section 3.13).

Directory rename (redirect_dir=on): When a merged directory is renamed, overlayfs cannot rename the lower-layer directory (it is read-only). Instead:

1. Create the new directory name in the upper layer.
2. Set the trusted.overlay.redirect xattr on the new upper directory, containing the absolute path (from the overlay root) of the original lower directory. Maximum redirect path: 256 bytes. Encoding: raw bytes (the underlying filesystem's filename encoding — typically UTF-8). No escaping; path components are separated by /. Paths exceeding 256 bytes cause the rename to fall back to full copy-up of the directory tree (no redirect xattr is set; the renamed directory becomes an opaque copy). This 256-byte limit is an UmkaOS implementation choice (Linux has no specific limit). The fallback to full directory copy-up preserves correctness.
3. Lookups for the renamed directory follow the redirect: when searching lower layers, use the redirect path instead of the current name.
4. Create a whiteout at the old name to hide the lower-layer original.

Opaque directory creation (rmdir + mkdir of same name):

1. Create whiteout or opaque directory in upper layer.
2. Set trusted.overlay.opaque xattr to "y" on the new upper directory.
3. All lower-layer entries under this path are hidden.

14.8.8 Whiteout and Deletion

When a file or directory is deleted from a merged view, overlayfs must hide the lower-layer entry without modifying the lower layer:

File deletion (unlink on a merged file): 1. If the file exists in upper: remove the upper entry via underlying_unlink(). 2. If the file exists in any lower layer: create a whiteout in the upper layer at the same path. 3. The cached name is dropped by the VFS unlink dispatch — the flow that already holds the parent dentry from path resolution — calling dentry_invalidate_child() (Section 14.1). The overlay's own InodeId-world code never resolves a dentry to do this.

Directory deletion (rmdir on a merged directory): 1. Verify the merged view of the directory is empty (no entries from any layer that are not whiteouts). Return ENOTEMPTY if non-empty. 2. If the directory exists in upper: remove it. 3. If the directory exists in lower: create an opaque whiteout in upper.

Whiteout creation:

/// Create a whiteout entry in the upper layer.
///
/// UmkaOS uses the xattr-based whiteout format by default: a zero-size
/// regular file with the overlay whiteout xattr set. This avoids
/// requiring mknod(2) capability (character device 0:0 creation
/// requires CAP_MKNOD in the filesystem's user namespace).
///
/// For compatibility, the character-device whiteout format is also
/// recognized on read (lookup).
fn create_whiteout(upper_parent: InodeId, name: &OsStr) -> Result<()> {
    let sb = overlay_super_block();

    // Create zero-size regular file. The creation mode is a `umode_t`-style
    // value (file-type bits | permission bits): a regular file with no
    // permission bits set.
    let whiteout = underlying_create(upper_parent, name,
        0o100_000 /* S_IFREG */ | 0o000)?;

    // Set the whiteout xattr.
    underlying_setxattr(whiteout,
        concat_static(sb.xattr_prefix, "whiteout"), b"y", XattrFlags::CREATE)?;

    Ok(())
}

RENAME_WHITEOUT integration: The VFS rename() with RENAME_WHITEOUT flag (already supported in InodeOps::rename(), Section 14.1) atomically renames a file and creates a whiteout at the old name. overlayfs uses this during copy-up of directory entries: when a file is copied from lower to upper, the old lower path is hidden by a whiteout created atomically with the rename.

14.8.9 Volatile Mode

Volatile mode disables all durability guarantees for the upper layer. This is a deliberate trade-off for ephemeral container workloads.

Behavior: - fsync(), fdatasync(), and sync_fs() on overlay files are no-ops (return success without calling the underlying filesystem's sync). - On mount with volatile=true, create the sentinel directory $workdir/work/incompat/volatile/. - On unmount, remove the sentinel directory (clean shutdown). - On next mount, if the sentinel exists, return EINVAL with a diagnostic message: the previous volatile session was not cleanly unmounted, and the upper/work directories may be inconsistent. The operator must delete upper and work directories and recreate them. - After any writeback error on the upper filesystem, subsequent fsync() calls on overlay files return EIO persistently (matching Linux's error stickiness behavior from Section 15.1).

Container runtime usage: Docker enables volatile mode for containers started with --storage-opt overlay2.volatile=true. This is common for CI/CD runners, build containers, and test environments where container state is discarded after each run.

14.8.10 Extended Attribute Handling

overlayfs must handle xattrs carefully because it uses private xattrs for internal bookkeeping (whiteouts, metacopy, redirects, opaque markers) and must pass through user-visible xattrs correctly.

Xattr namespace partitioning:

Namespace	Behavior
trusted.overlay.* (or user.overlay.* in userxattr mode)	Internal: overlay-private. Not visible to userspace via listxattr()/getxattr(). Used for whiteout, opaque, metacopy, redirect, origin markers.
security.*	Pass-through with copy-up: Copied from lower to upper during copy-up. setxattr() triggers copy-up. Includes security.selinux, security.capability (file caps), security.ima.
system.posix_acl_access, system.posix_acl_default	Pass-through with copy-up: POSIX ACLs are copied during copy-up. setfacl triggers copy-up.
user.* (excluding user.overlay.* in userxattr mode)	Pass-through with copy-up: User-defined xattrs. Copied during copy-up.
trusted.* (excluding trusted.overlay.*)	Pass-through with copy-up: Only accessible to CAP_SYS_ADMIN processes. Copied during copy-up.

getxattr/setxattr dispatch:

OverlayInodeOps::getxattr(inode, name, buf) -> Result<usize>:
  // Block access to overlay-private xattrs.
  if name.starts_with(overlay_xattr_prefix()) {
      return Err(ENODATA)
  }
  // Serve from upper if available, otherwise from lower.
  let target = upper_or_lower(inode)
  underlying_getxattr(target, name, buf)

OverlayInodeOps::setxattr(inode, name, value, flags) -> Result<()>:
  // Block writes to overlay-private xattrs.
  if name.starts_with(overlay_xattr_prefix()) {
      return Err(EPERM)
  }
  // setxattr triggers copy-up (xattr must be set on upper).
  let upper = copy_up(inode)?
  underlying_setxattr(upper, name, value, flags)

OverlayInodeOps::listxattr(inode, buf) -> Result<usize>:
  // List xattrs from upper (if exists) or lower.
  // Filter out overlay-private xattrs from the result.
  let target = upper_or_lower(inode)
  let raw = underlying_listxattr(target, buf)?
  filter_out_overlay_xattrs(buf, raw)

Nested overlayfs: When overlayfs is mounted on top of another overlayfs (nested container images, uncommon but valid), the inner overlay's xattrs must not collide with the outer overlay's. Linux handles this via "xattr escaping": the inner overlay stores its xattrs under trusted.overlay.overlay.* instead of trusted.overlay.*. UmkaOS implements the same escaping mechanism. This is transparent to the filesystem — the inner overlay simply uses a longer prefix.

14.8.11 statfs Behavior

OverlayFs::statfs() returns statistics from the upper layer's filesystem (if present). For read-only overlays (no upper), statistics from the topmost lower layer are returned. This matches Linux behavior and ensures that df on a container's root filesystem shows the available space on the writable layer.

14.8.12 Inode Number Composition (xino)

To guarantee unique inode numbers across the merged view, overlayfs composes inode numbers from the underlying filesystem's inode number and the layer index:

composed_ino = (layer_index << xino_bits) | underlying_ino

Where xino_bits is the number of bits available for the underlying inode (typically 32 for ext4 with default inode sizes). This ensures that stat() returns unique inode numbers for files from different layers that happen to share the same underlying inode number (common when layers are on the same filesystem).

When xino=off or when underlying inode numbers exceed the available bit width, overlayfs falls back to using the underlying inode numbers directly. In this mode, st_dev differs between upper and lower files (the VFS assigns a unique device number per overlay mount), but st_ino may collide across layers. Applications that rely on (st_dev, st_ino) pairs for file identity (e.g., tar, rsync, find -inum) may exhibit incorrect behavior. xino=auto avoids this by enabling composition only when it is safe.

14.8.13 Mount and Unmount Flow

Mount:

OverlayFs::mount(source, flags, data) -> Result<SuperBlock>:

  1. Parse mount options from `data` into `OverlayMountOptions`.

  2. Determine user-namespace influence (security policy, Section 14.4.6.1):
     userns_influenced = (current_task().cred.user_ns.level != 0)

     If userns_influenced:
       a. Force options.metacopy = false.
          Force options.redirect_dir = RedirectDirMode::Off.
          Log: "overlayfs: metacopy and redirect_dir disabled for
                user-namespace mount (Section 14.4.6.1)"
       b. Require options.userxattr == true. If not set, return EPERM.
          (User-namespace mounts cannot use trusted.overlay.* xattrs.)
       c. If any lower_dir entry uses the data-only '::' separator syntax:
          return EPERM. (Data-only layers with userxattr are disallowed
          because user.overlay.redirect is owner-writable.)

  3. Resolve each lower_dir path to an InodeId via VFS path lookup.
     Verify each is a directory. Hold references for mount lifetime.

  4. If upper_dir is set:
     a. Resolve upper_dir to InodeId. Verify it is a writable directory.
     b. Resolve work_dir to InodeId. Verify same superblock as upper_dir.
     c. Check work_dir is empty.
     d. Create `$workdir/work/` subdirectory if it does not exist.
     e. If volatile mode:
        - Check for `$workdir/work/incompat/volatile/` sentinel.
          If exists: return EINVAL ("previous volatile session unclean").
        - Create the sentinel directory.
     f. If nfs_export: create `$workdir/index/` subdirectory.
     g. Clean stale temporary files from workdir (names starting with
        `#overlay.`). These are remnants of interrupted copy-ups.

  5. Verify upper filesystem supports required operations:
     - xattr support (getxattr/setxattr succeed with overlay prefix).
     - rename with RENAME_WHITEOUT (test with a dummy file in workdir).

  6. Construct `OverlaySuperBlock` with userns_influenced as determined
     in step 2, and `SuperBlock`.

  7. Register overlay dentry ops with the VFS.

  8. Emit mount options for /proc/mounts via show_options().

Unmount:

OverlayFs::unmount(sb) -> Result<()>:

  1. If volatile mode: remove sentinel directory
     `$workdir/work/incompat/volatile/`.

  2. Flush and release upper layer (must happen FIRST):
     a. Sync all dirty pages and metadata in the upper filesystem's
        writeback queue. This ensures that any copy-up data, whiteouts,
        and metadata changes written to the upper layer are on stable
        storage before the upper SuperBlock reference is released.
        Flushes `upper_sb`'s dirty pages and metadata with a full barrier.
     b. Release the upper directory InodeId reference. This decrements
        the upper SuperBlock's mount reference count.
     c. Release the workdir InodeId reference (same SuperBlock as upper).

     The upper layer MUST be flushed and released before the lower layers
     because:
     - Dirty data in the upper layer may reference inodes from lower
       layers (metacopy files whose data still resides on a lower layer).
       If a lower SuperBlock were dropped first, its block device could
       be detached, making the lower data unreachable and causing I/O
       errors during upper flush.
     - The upper filesystem's journal commit may reference lower-layer
       block addresses (in filesystems like ext4 where the journal
       records physical block numbers). Releasing the lower device
       before journal commit would corrupt the journal.

  3. Release lower layer references (in reverse stacking order,
     topmost first):
     a. For each lower layer (from layer N down to layer 1):
        - Release the lower directory InodeId reference.
        - Decrement the lower SuperBlock's mount reference count.
     b. Reverse order ensures that if multiple lower layers share a
        SuperBlock (uncommon but valid), the last reference is released
        on the final iteration, not mid-traversal.
     c. Lower layers are read-only — no flush is needed. Their data
        is immutable for the lifetime of the overlay mount.

  4. Drop the OverlaySuperBlock (overlay's own VFS superblock metadata).
     At this point, all underlying filesystem references have been
     released. If this was the last mount referencing an underlying
     filesystem, that filesystem's `FileSystemOps::unmount()` is triggered, which
     flushes its own metadata and releases the block device.

Race with concurrent unmount of underlying filesystems: The VFS mount reference counting prevents an underlying filesystem from being unmounted while the overlay holds references to its inodes. An umount of the lower or upper filesystem while the overlay is mounted returns EBUSY (the overlay's InodeId references pin the underlying SuperBlock). This is identical to Linux behavior.

MountNamespace teardown cleanup order: During MountNamespace teardown, overlayfs cleanup follows the same ordering as explicit unmount: (1) flush pending copy-ups, (2) remove workdir temporary files, (3) unmount upper layer, (4) unmount lower layers. Workdir cleanup MUST precede upper unmount — the workdir is on the upper filesystem. If the workdir is cleaned after upper unmount, the workdir files become inaccessible and leak storage until the next fsck on the underlying filesystem.

14.8.14 Performance Characteristics

Operation	Overhead vs. direct filesystem access	Notes
Path lookup (cached)	+1 hash lookup per component	Overlay dentry points to underlying dentry
Read (lower-only file)	~0%	Direct delegation to lower filesystem
Read (upper file)	~0%	Direct delegation to upper filesystem
Read (metacopy file)	~0%	Reads from lower, same as lower-only
Write (upper file)	~0%	Direct delegation to upper filesystem
Write (first write, copy-up)	O(file_size) one-time	Sequential read+write of file data
Write (metacopy first write)	O(file_size) one-time	Deferred from container startup
chmod/chown (metacopy)	O(1) ~10μs	Metadata-only copy-up (no data copy)
chmod/chown (no metacopy)	O(file_size)	Full copy-up triggered
readdir (merged)	O(entries × layers)	Hash-based dedup over all layers
stat (cached)	~0%	Overlay inode cached in VFS

Container startup optimization: With metacopy enabled, pulling and starting a container image avoids copying any file data during the initial setup phase (only metadata operations occur: chmod, chown, symlink creation for the container's init process). Data is copied lazily on first write. For typical container images (200-500 MB of layers), this reduces container start time from seconds to tens of milliseconds for the filesystem setup phase.

14.8.15 dm-verity Integration for Container Image Layers

Read-only lower layers in a container overlay can be protected by dm-verity (Section 9.3). The container runtime mounts each image layer's block device with dm-verity verification, then stacks them as overlayfs lower layers:

Container image mount sequence:
  1. Pull image layers: layer1.img, layer2.img, ..., layerN.img
  2. For each layer:
     a. Set up dm-verity on the layer's block device (Merkle tree
        verification, Section 9.2.6)
     b. Mount the verified block device read-only (ext4/XFS)
  3. Mount overlayfs:
     mount -t overlay overlay \
       -o lowerdir=/mnt/layerN:...:/mnt/layer1,upperdir=...,workdir=...
       /container/rootfs

This provides block-level integrity verification for all read-only container layers. The writable upper layer is covered by IMA (Section 9.5) for runtime integrity measurement of modified files. Together, dm-verity (lower layers) + IMA (upper layer) provide complete integrity coverage for container filesystems.

The optional verity=require mount option (Section 14.8) provides an additional layer of verification at the overlayfs level using fs-verity digests, independent of dm-verity block device verification.

14.8.16 Internal Helper Reference

The pseudocode above calls a small set of overlayfs-internal helpers. They are not part of any KABI or userspace interface; they are defined here so the implementation is unambiguous.

/// A `'static` byte string, used for compile-time-composed xattr names.
pub type StaticStr = &'static [u8];

/// Parsed `trusted.overlay.metacopy` (or `user.overlay.metacopy` in userxattr
/// mode) stub. Present on an upper-layer inode whose data still lives in the
/// lower layer — only metadata was copied up. Internal parsed form, not an
/// on-disk/wire struct.
pub struct OverlayMetacopy {
    /// Metacopy format version from the 4-byte xattr header: `0` = legacy
    /// empty-value form (no digest), `1` = header followed by an fs-verity
    /// digest.
    pub version: u8,
    /// fs-verity SHA-256 digest of the authoritative lower data, present only
    /// when `version >= 1`. `None` for the legacy empty form.
    pub digest: Option<[u8; 32]>,
}

/// Fetch the `OverlayInode` backing a merged-view inode number. Infallible for
/// a live overlay inode (the caller already holds a reference to it).
fn get_overlay_inode(inode: InodeId) -> &'static OverlayInode { /* ... */ }

/// Return the overlay superblock for the current mount context.
fn overlay_super_block() -> &'static OverlaySuperBlock { /* ... */ }

/// Copy a metacopy file's data from its lower layer up to the existing upper
/// inode, then remove the `metacopy` xattr and clear `OverlayInode::metacopy`.
/// Runs at most once per inode (first write/truncate of a metacopy file).
fn copy_up_data(oi: &OverlayInode) -> Result<(), Errno> { /* ... */ }

/// Open `inode` on whichever underlying filesystem currently backs it,
/// returning that filesystem's `OpenOutcome`. Delegates to that
/// filesystem's `FileOps::open` ([Section 14.1](#virtual-filesystem-layer)).
///
/// An ordinary layer returns `data_inode: None`; a layer that is ITSELF a
/// stacking filesystem returns its own resolved binding, which the overlay
/// propagates unchanged rather than re-resolving — the deepest binding is
/// the one whose `AddressSpace` holds the data.
fn underlying_open(inode: InodeId, flags: OpenFlags) -> Result<OpenOutcome> { /* ... */ }

/// Resolve `ino` on `layer` to the live `Arc<Inode>` whose embedded
/// `AddressSpace` backs its data, for use as an `OpenOutcome::data_inode`
/// binding. Probes that layer superblock's `inode_cache` — every live inode
/// is a member of its superblock's cache for the whole of its life
/// ([Section 14.1](#virtual-filesystem-layer--inode-cache-icache), Invariants) — and
/// instantiates through the layer filesystem's inode-get path on a miss.
fn layer_inode(layer: &OverlayLayer, ino: InodeId) -> Result<Arc<Inode>, Errno> { /* ... */ }

/// Compile-time-compose an xattr name from the mount's `xattr_prefix` and a
/// static suffix (e.g. `xattr_prefix + "metacopy"`).
fn concat_static(prefix: StaticStr, suffix: &'static str) -> StaticStr { /* ... */ }

/// Validate a raw `metacopy` xattr value and parse it into an `OverlayMetacopy`.
/// Returns `None` (treat as a regular upper file) if the value matches neither
/// the legacy empty form nor the 4+N byte header-plus-digest form.
fn validate_metacopy_xattr(xattr: &[u8]) -> Option<OverlayMetacopy> { /* ... */ }

/// Create a new entry `name` under upper-layer directory `parent` with mode
/// `mode`, returning the new inode. `mode` is a `umode_t`-style value (Linux
/// `S_IFMT` file-type bits OR-ed with permission bits, identical to
/// `InodeAttr::mode`), NOT the open-time `FileMode` (`FMODE_*`) bitflags.
/// Delegates to the upper filesystem's `InodeOps::create`/`mknod`.
fn underlying_create(parent: InodeId, name: &OsStr, mode: u32)
    -> Result<InodeId, Errno> { /* ... */ }

/// Set xattr `name` = `value` (with `flags`) on an upper-layer inode. Delegates
/// to the upper filesystem's `InodeOps::setxattr`.
fn underlying_setxattr(target: InodeId, name: StaticStr, value: &[u8],
    flags: XattrFlags) -> Result<(), Errno> { /* ... */ }

14.8.17 Linux Compatibility

overlayfs is compatible with Linux's overlayfs at the mount interface and xattr format level:

• Upper and lower directories created by Linux overlayfs are mountable by UmkaOS and vice versa. The xattr format (trusted.overlay.* names and values) is identical.
• Mount option syntax matches Linux exactly (-o lowerdir=...,upperdir=..., workdir=...).
• Whiteout formats (both character device 0:0 and xattr-based) are recognized.
• Metacopy xattr format is compatible: layers created with metacopy=on on Linux work on UmkaOS.
• redirect_dir xattr format and path encoding match Linux.
• /proc/mounts output format matches Linux for container introspection tools.
• /sys/module/overlay/parameters/* is not emulated (UmkaOS does not use kernel modules); per-mount options in the mount command are the sole configuration mechanism.

Docker/containerd/Podman compatibility: These runtimes interact with overlayfs exclusively through the mount(2) syscall and standard file operations. They do not use any overlayfs-specific ioctls or sysfs interfaces. UmkaOS's implementation of mount("overlay", ...) with the standard option string is sufficient for full compatibility. The overlay2 storage driver in Docker and the overlayfs snapshotter in containerd are fully supported.

---

14.9 binfmt_misc — Arbitrary Binary Format Registration

binfmt_misc is a VFS-level mechanism that allows userspace to register handlers for arbitrary binary formats, identified by magic bytes or file extension. When the kernel's exec path attempts to start a file and neither the native ELF handler nor the #! script handler matches, the kernel delegates to a registered binfmt_misc interpreter. The registered interpreter binary is invoked with the original file path as an additional argument.

Critical use cases:

• Multi-architecture containers: qemu-aarch64-static is registered as the interpreter for AArch64 ELF binaries, identified by the AArch64 ELF magic header. This allows running unmodified ARM64 Docker images on an x86-64 host without hardware virtualisation.
• Java: .jar files executed as if they were executables via a registration that maps the .jar extension to /usr/bin/java -jar.
• .NET: PE32+ executables identified by the MZ magic bytes are mapped to dotnet exec.
• Wine: 16-bit and 32-bit Windows PE files mapped to wine.

14.9.1 Data Structures

/// A single registered binfmt_misc entry.
/// Kernel-internal, not KABI or wire format. `Option<[u8; N]>` fields use
/// the discriminant to distinguish "magic match" from "extension match"
/// without requiring sentinel values in the array.
pub struct BinfmtMiscEntry {
    /// Registration name. Shown as the filename under the binfmt_misc mount.
    /// Alphanumeric, hyphen, and underscore only. NUL-terminated.
    pub name:         [u8; 64],
    /// Matching strategy: magic bytes or file extension.
    pub match_type:   BinfmtMatch,
    /// Magic bytes to compare against file content (BinfmtMatch::Magic only).
    /// Maximum 128 bytes. Length of `magic` and `mask` must be equal.
    /// Comparison is byte-by-byte (no endianness interpretation) — each
    /// byte in the file at `magic_offset + i` is ANDed with `mask[i]` and
    /// compared to `magic[i]`. Multi-byte values embedded in magic patterns
    /// must be specified in the byte order they appear in the file.
    pub magic:        Option<[u8; 128]>,
    /// Length of the valid portion of `magic` and `mask` arrays.
    pub magic_len:    u8,
    /// Bitmask applied to each file byte before comparison with `magic`.
    /// A mask byte of `0xff` means "match exactly"; `0x00` means "ignore".
    pub mask:         Option<[u8; 128]>,
    /// Byte offset within the file at which `magic` is compared.
    pub magic_offset: u16,
    /// File extension string (BinfmtMatch::Extension only).
    /// Case-sensitive. Does not include the leading `.`. NUL-terminated.
    /// 32 bytes: 31 chars + NUL. Covers all reasonable extensions.
    pub extension:    Option<[u8; 32]>,
    /// Absolute path to the interpreter binary.
    pub interpreter:  [u8; PATH_MAX],
    /// Behavioural flags.
    pub flags:        BinfmtFlags,
    /// Whether this entry participates in exec matching.
    pub enabled:      AtomicBool,
}

/// How the entry identifies matching binaries.
pub enum BinfmtMatch {
    /// Match by magic bytes at a fixed offset within the file.
    Magic,
    /// Match by the file extension of the executed path.
    Extension,
}

bitflags! {
    /// Behavioural flags for a binfmt_misc entry.
    pub struct BinfmtFlags: u32 {
        /// Pass the original filename as argv[0] to the interpreter instead
        /// of substituting the interpreter path.
        const PRESERVE_ARGV0 = 0x01;
        /// Open the binary file and pass it to the interpreter as an open fd,
        /// delivered as the auxv entry `AT_EXECFD(2)` (NOT as a
        /// `/proc/self/fd/N` argv path). Required when the binary is not
        /// world-readable and the interpreter runs without elevated privilege.
        const OPEN_BINARY    = 0x02;
        /// Compute the new process credentials from the ORIGINAL executed
        /// binary (its setuid/setgid bits and `security.capability` xattr)
        /// rather than from the interpreter. IMPLIES `OPEN_BINARY` (the
        /// original binary is retained open as the credential source). Without
        /// this flag (default) credentials come from the INTERPRETER binary and
        /// ITS setuid IS honored. Matches the Linux `binfmt_misc` `C` flag:
        /// `fs/binfmt_misc.c` sets `bprm->execfd_creds` (and forces `O`), and
        /// Linux `fs/exec.c bprm_creds_from_file()` then reads `bprm->executable`
        /// (the original binary) instead of the final interpreter `bprm->file`.
        const CREDENTIALS    = 0x04;
        /// Fix binary: the interpreter is not itself subject to further
        /// binfmt_misc or personality transformation. Prevents recursion.
        const FIX_BINARY     = 0x08;
    }
}

/// Maximum registered binfmt_misc entries. Real systems have fewer than 64;
/// this bound makes the table fixed-size and avoids heap allocation on the
/// exec hot path.
pub const MAX_BINFMT_MISC: usize = 64;

The global entry table is an RcuCell<ArrayVec<Arc<BinfmtMiscEntry>, MAX_BINFMT_MISC>>. The exec path reads the table under an RCU read guard (lock-free) and performs a bounded scan (at most MAX_BINFMT_MISC entries). Registration, enable/disable, and removal are cold-path operations: the writer clones the current ArrayVec, applies the modification, and publishes the new version via RcuCell::update() (RCU grace period). The list is short in practice (fewer than 64 entries on any real system), so O(N) scan cost is negligible relative to exec overhead.

14.9.2 Registration Interface

The binfmt_misc filesystem is mounted at /proc/sys/fs/binfmt_misc (also accessible at /sys/kernel/umka/binfmt_misc/ via the umkafs namespace — see Section 20.5). It exposes:

Path	Type	Description
register	write-only file	Register a new entry
status	read/write file	1 = all entries active; 0 = all disabled globally
<name>/enabled	read/write file	1 enable, 0 disable, -1 remove this entry
<name>	read-only file	Shows entry details (flags, interpreter, magic/extension)

Writing to register or any <name>/enabled file requires Capability::SysAdmin in the caller's capability set.

Registration format (written as a single line to register):

:name:type:offset:magic:mask:interpreter:flags

Fields are separated by the same delimiter character as the leading :. Any printable non-alphanumeric character may be used as the delimiter (allowing paths that contain colons).

Field	Description
name	Identifier: alphanumeric, -, _. Maximum 63 characters.
type	M for magic-byte match; E for extension match.
offset	Decimal byte offset for magic comparison (type M). 0 for most formats.
magic	Hex-escaped bytes for type M (e.g., \x7fELF). Extension string for type E.
mask	Hex-escaped bitmask for type M; same length as magic. Empty for type E.
interpreter	Absolute path to the interpreter binary. Must exist at registration time.
flags	Subset of POCF: P = PRESERVE_ARGV0, O = OPEN_BINARY, C = CREDENTIALS, F = FIX_BINARY. (Linux binfmt_misc defines exactly these four; there is no S flag.)

Example — registering QEMU user-mode for AArch64 ELF binaries on an x86-64 host:

:qemu-aarch64:M:0:\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\xb7\x00::qemu-aarch64-static:OC

• Type M, offset 0: compare 20 magic bytes starting at file byte 0.
• No mask: all bytes compared exactly (\xff mask is implied).
• O (OPEN_BINARY): interpreter receives the pre-opened binary as an fd via the AT_EXECFD(2) auxv entry (argv is unchanged), for cross-uid access.
• C (CREDENTIALS): credentials computed from the ORIGINAL binary instead of the interpreter (implies O).

Parsing algorithm:

parse_registration(line: &[u8]) -> Result<BinfmtMiscEntry>:
  1. delimiter = line[0]
  2. Split line on delimiter into fields: [name, type, offset, magic_or_ext,
     mask, interpreter, flags_str].
  3. Validate name: alphanumeric + '-' + '_', length 1–63.
  4. Parse type: 'M' → BinfmtMatch::Magic, 'E' → BinfmtMatch::Extension.
  5. For type M:
     a. Parse offset as decimal u16.
     b. Decode hex-escaped bytes into magic array (max 128 bytes).
     c. If mask non-empty: decode hex-escaped bytes; must equal magic.len().
     d. If mask empty: fill mask with 0xff bytes (exact match).
  6. For type E:
     a. Validate extension: printable ASCII, no '/', no '.', max 31 chars.
     b. Store extension without leading '.'.
  7. Validate interpreter: starts with '/', exists in VFS (path lookup),
     is a regular file with execute permission for at least one uid.
  8. Parse flags_str: accept 'P', 'O', 'C', 'F' in any order; reject any other
     character (EINVAL).
  9. Construct BinfmtMiscEntry with enabled = AtomicBool::new(true).
  10. Clone current ArrayVec from RcuCell; reject if name already exists.
  11. Push Arc<BinfmtMiscEntry> to cloned table; RCU-publish via RcuCell::update().

14.9.3 Exec Path Integration

During the exec path (Section 8.1), after the ELF handler and the #! script handler both decline the binary (return ENOEXEC), the kernel calls binfmt_misc_load_binary(bprm, file, argv, envp), which rewrites bprm and returns BinfmtResult::Rewrite so exec_binprm() re-probes the same BinPrm.

Matching algorithm:

binfmt_misc_load_binary(bprm, file, argv, envp) -> Result<BinfmtResult>:  // ENOEXEC on no match
  1. Acquire RCU read guard on global entry table (lock-free).
  2. If global status is disabled: return ENOEXEC.
  3. Read a probe buffer of min(128 + max_magic_offset, 256) bytes from
     offset 0 of `file`. This single read covers all registered magic ranges.
  4. For each entry in table order (bounded by MAX_BINFMT_MISC = 64):
     a. If !entry.enabled.load(Relaxed): skip.
     b. If entry.match_type == Magic:
        i.  end = entry.magic_offset as usize + entry.magic_len as usize.
        ii. If end > probe_buffer.len(): skip (file too short).
        iii.For each byte i in 0..magic_len:
              file_byte = probe[magic_offset + i] & mask[i]
              if file_byte != magic[i] & mask[i]: break → no match
        iv. If all bytes matched: entry is selected.
     c. If entry.match_type == Extension:
        i.  Take `bprm.interp` — the executed path string, NOT argv[0].
            argv[0] is user-controlled and can differ from the exec path, so
            keying selection on it is observably divergent (a caller could
            select an arbitrary interpreter). Find the LAST '.' in
            `bprm.interp` (Linux `fs/binfmt_misc.c` `search_binfmt_handler`:
            `char *p = strrchr(bprm->interp, '.')`).
        ii. If a '.' is present and the substring after it equals `extension`
            byte-for-byte (case-sensitive — Linux `strcmp(e->magic, p + 1)`,
            i.e. only the single trailing component after the last dot, never
            a longer multi-component suffix): entry is selected. No '.' in the
            path → no match.
  5. If no entry matched: drop RCU guard; return ENOEXEC.
  6. Clone the matched entry (Arc clone, no copy of byte arrays).
  7. Drop RCU read guard.
  8. Rewrite argv in place via the splice primitives
     ([Section 8.1](08-process.md#process-and-task-management--argument-copying-and-stack-setup-copystrings-setupargpages)).
     First, if `bprm.path_inaccessible` (the synthesized `/dev/fd/<fd>` name
     derives from an `O_CLOEXEC` fd — `BinPrm.filename`, [Section 8.3](08-process.md#elf-loader)),
     reject with `ENOENT` BEFORE splicing: the handler binary could not re-open
     the path (Linux `fs/binfmt_misc.c`, the same gate as `fs/binfmt_script.c`).
     Otherwise:
     a. Unless PRESERVE_ARGV0 is set, `bprm_drop_arg0()` removes the original
        argv[0] (PRESERVE_ARGV0 keeps it as the second argument).
     b. `bprm_push_kernel_str(original_path)` prepends the original binary
        file path — ALWAYS, regardless of OPEN_BINARY. OPEN_BINARY does NOT
        alter argv. Instead, when OPEN_BINARY is set, `Arc::clone(&bprm.file)`
        (still the original binary at this point, before step 10 replaces it)
        is stashed into `bprm.execfd_file`; the interpreter then receives the
        pre-opened binary as the auxv entry `AT_EXECFD(2)` — a real installed
        fd, not a `/proc/self/fd/<N>` argv token. Exec step 6 installs the fd
        into the new process's fd table and pushes the auxv entry
        ([Section 8.1](08-process.md#process-and-task-management--program-execution-exec),
        [Section 8.3](08-process.md#elf-loader)). This matches how real interpreters (e.g.
        `qemu-user`) consume the pre-opened binary.
     c. `bprm_push_kernel_str(interpreter_path)` prepends the interpreter last,
        so it lands at `bprm.p` as the new argv[0].
  9. Credential source (corrects the Linux `C`-flag contract). By DEFAULT the
     post-rewrite credential transformation computes the new credentials from
     the INTERPRETER file (the rewritten `bprm.file`) — the interpreter's
     setuid/setgid/fscaps ARE honored (Linux `bprm_creds_from_file()` against
     the final `bprm->file`, `fs/exec.c`). If CREDENTIALS is set, retain the
     ORIGINAL binary (still `bprm.file` at this point, before step 10 replaces
     it) as the credential source —
     `bprm.cred_source = Some(Arc::clone(&bprm.file))` — so the transformation
     reads the ORIGINAL binary's setuid/setgid/fscaps instead (Linux `C`:
     `bprm->execfd_creds` / `bprm->executable`, `fs/binfmt_misc.c` +
     `fs/exec.c`). CREDENTIALS IMPLIES OPEN_BINARY (forced when the registration
     is parsed, `fs/binfmt_misc.c`), so step 8b already stashed the original
     into `bprm.execfd_file`. The exec loop consumes `bprm.cred_source` at its
     credential-recompute seam ([Section 8.3](08-process.md#elf-loader--script-handler)).
  10. Replace `bprm.file` with the interpreter's `OpenFile` — when FIX_BINARY is
      set, this is the file opened at `register` time in the registrar's mount
      namespace (immune to later path replacement); otherwise open the
      interpreter path now — and return `BinfmtResult::Rewrite`. `exec_binprm()`
      loops with the SAME BinPrm ([Section 8.3](08-process.md#elf-loader)): there is NO recursive
      exec syscall recursion. Re-matching is bounded ONLY by the loop depth counter
      (<= 5, then -ELOOP); no per-exec skip flag exists (Linux contract:
      Linux `fs/binfmt_misc.c load_misc_binary` mutates the carrier and returns;
      recursion control is the `exec_binprm` depth limit).

The loop's next iteration processes the interpreter through the normal ELF handler. QEMU user-mode binaries are statically linked ELF executables, so one further iteration terminates the chain.

14.9.4 The binfmt_misc Filesystem

binfmt_misc_fs is a minimal VFS filesystem type (FsType::BinfmtMisc) with the following FsOps implementation:

impl FsOps for BinfmtMiscFs {
    fn mount(&self, flags: MountFlags, _data: &[u8]) -> Result<Arc<SuperBlock>>;
    fn statfs(&self, sb: &SuperBlock) -> Result<StatFs>;
}

impl InodeOps for BinfmtMiscDir {
    fn lookup(&self, name: &OsStr) -> Result<Arc<Dentry>>;
    fn iterate_dir(&self, ctx: &mut DirContext) -> Result<()>;
}

impl FileOps for BinfmtMiscRegister {
    fn write(&self, buf: &[u8], _offset: u64) -> Result<usize>; // parse_registration
}

impl FileOps for BinfmtMiscStatus {
    fn read(&self, buf: &mut [u8], _offset: u64) -> Result<usize>; // "enabled\n" or "disabled\n"
    fn write(&self, buf: &[u8], _offset: u64) -> Result<usize>;    // "1" / "0"
}

impl FileOps for BinfmtMiscEntryFile {
    fn read(&self, buf: &mut [u8], _offset: u64) -> Result<usize>; // entry details
    fn write(&self, buf: &[u8], _offset: u64) -> Result<usize>;    // "1" / "0" / "-1"
}

The filesystem has no on-disk backing store. All state lives in the in-kernel RcuCell<ArrayVec<Arc<BinfmtMiscEntry>, MAX_BINFMT_MISC>>. Directory inodes are synthesised dynamically: lookup reads the entry table under an RCU guard, scans for a matching name, and returns a synthetic inode. iterate_dir emits register, status, and all current entry names.

RCU synchronization on handler removal: When an entry is removed (write -1 to the entry file), the removal sequence is: 1. Acquire the global binfmt_misc_lock (spinlock, serializes writers). 2. Create a new ArrayVec with the entry removed. 3. Publish via RcuCell::update() (rcu_assign_pointer semantics). 4. Release binfmt_misc_lock. 5. Call synchronize_rcu() to wait for all readers to complete. 6. Drop the old ArrayVec (releases the Arc<BinfmtMiscEntry>). Step 5 is critical: without it, a concurrent execve() holding an RCU read lock could still be matching against the removed entry. The synchronize_rcu() ensures that by the time the Arc is dropped (and potentially the interpreter binary's file reference released), all in-flight entry-table searches have either completed or moved past the entry table read. For FIX_BINARY entries (where the interpreter file is pinned at registration time), the pinned file reference is released only after the RCU grace period completes.

Multiple mounts of the binfmt_misc filesystem share the same global entry table (identical to Linux semantics). Unmounting does not clear registrations; entries persist until explicitly removed via echo -1 > /proc/sys/fs/binfmt_misc/<name>/enabled or until the kernel reboots.

Mount point: The standard location is /proc/sys/fs/binfmt_misc, mounted by systemd-binfmt.service at early boot before loading entries from /etc/binfmt.d/*.conf and /usr/lib/binfmt.d/*.conf.

14.9.5 Persistence and systemd Integration

The kernel holds registrations only in memory. Registrations are lost on reboot. The systemd-binfmt.service unit re-registers all entries at each boot by reading configuration files with the format:

# /etc/binfmt.d/qemu-aarch64.conf
:qemu-aarch64:M:0:\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\xb7\x00::qemu-aarch64-static:OC

Each non-comment, non-empty line is written verbatim to /proc/sys/fs/binfmt_misc/register. Drop-in files in /usr/lib/binfmt.d/ are processed first, then /etc/binfmt.d/ (higher priority). Conflicting entries with the same name are rejected by the kernel (duplicate-name check in parse_registration).

14.9.6 Security Model

• Privilege: Writing to register or any enabled file requires Capability::SysAdmin. Unprivileged processes cannot add or modify entries.
• Interpreter credentials: By default (no CREDENTIALS flag), the new process's credentials are computed from the INTERPRETER binary — its setuid/setgid bits and security.capability xattr ARE honored (Linux Linux bprm_creds_from_file() against the final bprm->file, fs/exec.c). Registering a binfmt_misc entry requires Capability::SysAdmin, so the interpreter is an administrator-vetted, trusted binary; honoring its setuid is the Linux-compatible default. A setuid SCRIPT's own bit is still never honored — the script is not the file whose credentials are read.
• CREDENTIALS flag: Switches the credential source to the ORIGINAL executed binary (its setuid/setgid bits and security.capability xattr) instead of the interpreter; implies OPEN_BINARY. Matches the Linux C flag (bprm->execfd_creds → bprm->executable, fs/binfmt_misc.c + fs/exec.c).
• OPEN_BINARY flag: The kernel keeps the already-opened binary (bprm.execfd_file), so the interpreter receives an already-open fd. This allows the interpreter to read the file even when the binary is not world-readable (e.g., chmod 700 user-owned binaries run through QEMU on a shared host). The fd is delivered as the auxv entry AT_EXECFD(2) — a real installed fd in the new process's fd table (exec step 6, Section 8.1; auxv push in Section 8.3) — NOT as a /proc/self/fd/N argv path. argv is left untouched by OPEN_BINARY; interpreters that need the fd (e.g. qemu-user) read it from the auxiliary vector.
• Recursion guard: The FIX_BINARY flag pins the interpreter file opened at register time (immune to later path replacement), and interpreter recursion is bounded solely by the exec_binprm loop depth counter (<= 5, then -ELOOP) — there is no per-exec skip flag. This prevents pathological interpreter chains where an interpreter is itself a binfmt_misc-dispatched binary.

---

14.10 autofs — Kernel Automount Trigger

autofs is the kernel side of the automount subsystem. Its role is narrow: detect access to a path that has not yet been mounted, suspend the filesystem lookup, notify a userspace daemon, and resume the lookup after the daemon has performed the mount. The kernel does not decide what to mount or where it comes from — that is entirely the daemon's responsibility.

Used extensively by systemd through .automount units: lazy NFS home directories (/home/$user), removable media (/media/disk), and network shares that should only connect on demand.

14.10.1 Architecture

autofs registers a VFS filesystem type (FsType::Autofs). An autofs filesystem instance covers a single mount point. Inside that mount point, the kernel may see directory entries that are not yet backed by a real mount. When path resolution (Section 14.1) traverses one of these directories whose dentry requests automount resolution, it invokes the dentry's automount operation.

The two fundamental mount modes are:

Mode	Description
indirect	autofs mount covers a directory; lookups of subdirectories trigger mounts. /nfs is autofs; accessing /nfs/fileserver triggers a mount of fileserver:/export onto /nfs/fileserver.
direct	The autofs mount point IS the trigger. Accessing the exact path (e.g., /mnt/backup) triggers the mount.

14.10.2 Data Structures

/// State for one autofs filesystem instance (one mount point).
pub struct AutofsMount {
    /// Pipe to the automount daemon. Kernel writes AutofsPacket messages here.
    pub pipe:             Arc<Pipe>,
    /// Protocol version negotiated with the daemon (UmkaOS implements v5).
    /// The daemon declares its version via `AUTOFS_IOC_PROTOVER` ioctl on
    /// the autofs mount fd. If the daemon's version is < 5, the kernel
    /// responds with v4 compatibility packets (no UID/GID/PID fields).
    /// If the daemon's version is > 5, the kernel uses v5 (the kernel
    /// never speaks a protocol newer than it implements). Version mismatch
    /// logging: "autofs: daemon v{N}, kernel v5 — using v{min(N,5)}".
    pub proto_version:    u32,
    /// Whether the daemon has declared itself gone (catatonic state).
    pub catatonic:        AtomicBool,
    /// Idle timeout in seconds after which expire packets are sent.
    pub timeout_secs:     AtomicU32,
    /// All outstanding lookup requests waiting for daemon response.
    /// Keyed by token (u64). XArray provides O(1) lookup with internal
    /// xa_lock for write serialization, replacing the external Mutex.
    pub pending:          XArray<Arc<AutofsPendingRequest>>,
    /// Monotonically increasing token counter. Internal counter is u64
    /// (exhaustion-proof: at 100 tokens/sec, wraps in 5.8 billion years).
    /// The Linux ABI wire protocol (`AutofsPacketMissing::wait_queue_token`)
    /// carries the low 32 bits only. The XArray is keyed by the wire token
    /// (u32, zero-extended to u64 for XArray indexing). Only the low 32 bits
    /// of the counter are used as XArray keys and wire tokens. Lookup on daemon
    /// response is O(1) via `pending.get(wire_token as u64)`. Collision is
    /// impossible: at 100 tokens/sec,
    /// the u32 space covers 49 days of tokens, but pending requests time out
    /// within `timeout_secs` (typically 30-300 seconds).
    pub next_token:       AtomicU64,
    /// Mount type: indirect or direct.
    pub mount_type:       AutofsMountType,
}

pub enum AutofsMountType {
    Indirect,
    Direct,
    Offset, // Internal: used for sub-mounts within a multi-mount map.
}

/// One outstanding automount request.
pub struct AutofsPendingRequest {
    /// Token echoed back in the daemon's IOC_READY / IOC_FAIL ioctl.
    pub token:   u32,
    /// Path component that triggered the lookup (indirect) or full path (direct).
    pub name:    KernelString,
    /// Sleeping callers blocked on this mount.
    pub waitq:   WaitQueue,
    /// Result set by the daemon: Ok(()) on success, Err(errno) on failure.
    pub result:  OnceLock<Result<()>>,
}

/// Packet written to the daemon pipe for a missing mount (protocol v5).
/// Layout matches Linux `struct autofs_v5_packet`. 304 bytes on all
/// UmkaOS-supported architectures: the `ino: u64` field has 8-byte alignment
/// on all targets (ARMv7 AAPCS, PPC32 System V ABI, and all 64-bit ABIs),
/// so trailing padding is always 4 bytes (300 named → 304 aligned).
/// The daemon reads `mem::size_of::<AutofsPacketMissing>()` bytes from the pipe.
#[repr(C)]
pub struct AutofsPacketMissing {
    pub hdr:              AutofsPacketHdr,       // offset  0, size  8
    /// Token for AUTOFS_IOC_READY / AUTOFS_IOC_FAIL.
    pub wait_queue_token: u32,                   // offset  8, size  4
    /// Device number of the autofs mount.
    pub dev:              u32,                    // offset 12, size  4
    /// Inode number of the trigger dentry.
    pub ino:              u64,                    // offset 16, size  8
    /// UID of the process that triggered the lookup.
    pub uid:              u32,                    // offset 24, size  4
    /// GID of the process that triggered the lookup.
    pub gid:              u32,                    // offset 28, size  4
    /// PID (thread group leader) of the process that triggered the lookup.
    /// Autofs wire protocol uses __u32 (not pid_t).
    pub pid:              u32,                    // offset 32, size  4
    /// TGID of the triggering process. Autofs wire protocol uses __u32.
    pub tgid:             u32,                    // offset 36, size  4
    /// Length of `name` (not including NUL). Autofs wire: __u32.
    pub len:              u32,                    // offset 40, size  4
    /// Name of the missing directory component (NUL-terminated).
    pub name:             [u8; NAME_MAX + 1],     // offset 44, size 256
    // Named fields: 8+4+4+8+4+4+4+4+4+256 = 300 bytes.
    // u64 alignment on all UmkaOS targets → 4 bytes trailing padding → 304.
}
// 304 on all UmkaOS-supported architectures: u64 has 8-byte alignment on
// ARMv7 (AAPCS), PPC32 (System V ABI), and all 64-bit ABIs. The 32-bit
// value of 300 would only apply on i386 (4-byte u64 alignment), which
// UmkaOS does not support.
const_assert!(size_of::<AutofsPacketMissing>() == 304);

/// Packet written to the daemon pipe requesting expiry of an idle mount.
/// Layout matches Linux `struct autofs_v5_packet_expire` (304 bytes on all
/// UmkaOS-supported architectures — identical to AutofsPacketMissing). In
/// Linux, `autofs_packet_expire_direct_t` is a typedef alias for `autofs_v5_packet`.
#[repr(C)]
pub struct AutofsPacketExpire {
    pub hdr:              AutofsPacketHdr,
    pub wait_queue_token: u32,
    pub dev:              u32,
    pub ino:              u64,
    pub uid:              u32,
    pub gid:              u32,
    pub pid:              u32,
    pub tgid:             u32,
    pub len:              u32, // Autofs wire: __u32.
    pub name:             [u8; NAME_MAX + 1],
}
// Same reasoning as AutofsPacketMissing: 304 on all UmkaOS targets.
const_assert!(size_of::<AutofsPacketExpire>() == 304);

/// Common packet header.
/// Field types match Linux's `struct autofs_packet_hdr` exactly:
/// both `proto_version` and `type` are `int` (i32) in the Linux C struct.
#[repr(C)]
pub struct AutofsPacketHdr {
    pub proto_version: i32,
    pub packet_type:   i32,
}
const_assert!(size_of::<AutofsPacketHdr>() == 8);

/// Autofs packet type constants. Values match Linux `auto_fs.h`.
/// The header's `packet_type` field is `i32` (not enum repr) for C ABI
/// compatibility. These constants are used for matching:
///
/// | Value | Name | Protocol | Description |
/// |-------|------|----------|-------------|
/// | 0 | Missing | v1 | Legacy missing (v1/v2 only) |
/// | 1 | Expire | v1 | Legacy expire (v1/v2 only) |
/// | 2 | ExpireMulti | v4 | Multi-mount expire |
/// | 3 | MissingIndirect | v5 | Indirect mount trigger |
/// | 4 | ExpireIndirect | v5 | Indirect mount expiry |
/// | 5 | MissingDirect | v5 | Direct mount trigger |
/// | 6 | ExpireDirect | v5 | Direct mount expiry |
///
/// Types 3-6 are required for v5 protocol. systemd dispatches on these values.
/// UmkaOS uses types 3-6 for v5 operation (types 0-1 only for v4 compat).
#[repr(i32)]
pub enum AutofsPacketType {
    Missing         = 0,
    Expire          = 1,
    ExpireMulti     = 2,
    MissingIndirect = 3,
    ExpireIndirect  = 4,
    MissingDirect   = 5,
    ExpireDirect    = 6,
}

14.10.3 Packetized Pipe Protocol

The autofs kernel-to-daemon communication channel is a packetized pipe: each write() from the kernel writes exactly one complete packet (mem::size_of::<AutofsPacketMissing>() bytes — 304 on all UmkaOS-supported architectures), and each read() from the daemon must read exactly that many bytes to consume one packet. The pipe is opened with O_DIRECT semantics (Linux pipe O_DIRECT flag, since Linux 3.4) to ensure atomic packet-sized writes — a partial write never occurs as long as the packet size (304 bytes) is less than PIPE_BUF (4096 bytes, POSIX-guaranteed atomicity threshold). See Section 14.17 for the UmkaOS pipe implementation.

If the pipe buffer is full (all slots occupied), the kernel's write() returns -EAGAIN (the pipe fd is set to non-blocking mode by the daemon at setup). The autofs trigger path converts this to -ENOMEM and returns to the caller — the daemon is overloaded and cannot accept new mount requests.

14.10.4 Automount Protocol

Trigger sequence (the fast path through VFS path resolution):

autofs_d_automount(at: &MountDentry) -> Result<Option<Arc<Mount>>>:
  Implements DentryOps::d_automount ([Section 14.1](#virtual-filesystem-layer)).
  `at` is the walk position; the steps below read the triggering dentry as
  `dentry = &at.dentry`.
  Precondition: called from REF-walk (never RCU-walk; see
  [Section 14.1](#virtual-filesystem-layer--path-resolution)).

  1. Obtain the AutofsMount for this dentry's superblock.
  2. If catatonic: return Err(ENOENT) immediately.
  3. Check if `dentry` is already a mount point (DCACHE_MOUNTED set):
     return Ok(None) — another thread raced and completed the mount.
  4. Allocate token = next_token.fetch_add(1, Relaxed).
  5. Construct AutofsPacketMissing with v5 packet type:
     - indirect mode: `packet_type = AutofsPacketType::MissingIndirect`
     - direct mode: `packet_type = AutofsPacketType::MissingDirect`
     Set `{ hdr: { proto_version: 5, packet_type }, token, name = dentry.name or full path }`.
  6. Insert Arc<AutofsPendingRequest> into pending table under token.
  7. Write packet to pipe (non-blocking; if pipe is full, return ENOMEM —
     the daemon is overloaded).
  8. Sleep on pending.waitq with timeout = timeout_secs seconds.
  9. On wake:
     a. Remove request from pending table.
     b. If result is Ok(()):
        - Verify dentry is now a mount point (DCACHE_MOUNTED).
        - Return Ok(None) (path resolution will cross the newly attached mount).
     c. If result is Err(e): return Err(e).
  10. On timeout:
     a. Remove request from pending table.
     b. Return Err(ETIMEDOUT).

Daemon response (via ioctl on the autofs pipe fd or mount point fd):

AUTOFS_IOC_READY(token: u32):
  1. Acquire pending lock; look up token.
  2. If not found: return ENXIO (stale token; request already timed out).
  3. Set request.result = Ok(()).
  4. Wake all waiters on request.waitq.
  5. Remove from pending table.

AUTOFS_IOC_FAIL(token: u32):
  1. Acquire pending lock; look up token.
  2. If not found: return ENXIO.
  3. Set request.result = Err(ENOENT).
  4. Wake all waiters.
  5. Remove from pending table.

Multiple callers may race to access the same missing path simultaneously. All of them find the same AutofsPendingRequest in the pending table (inserted by the first caller) and sleep on the same waitq. When the daemon responds, all waiters wake together.

14.10.5 Control Interface

All autofs control operations are performed via ioctl(2) on the file descriptor of the autofs pipe (passed to the kernel at mount time via the fd=N mount option) or on a file descriptor opened on the autofs mount point itself.

ioctl	Direction	Description
AUTOFS_IOC_READY	daemon→kernel	Mount succeeded for token.
AUTOFS_IOC_FAIL	daemon→kernel	Mount failed for token.
AUTOFS_IOC_CATATONIC	daemon→kernel	Daemon is exiting; all future lookups fail with ENOENT.
AUTOFS_IOC_PROTOVER	kernel→daemon	Returns protocol version (5 for UmkaOS).
AUTOFS_IOC_SETTIMEOUT	daemon→kernel	Sets idle expiry timeout in seconds.
AUTOFS_IOC_EXPIRE	kernel→daemon	Requests daemon to expire (unmount) one idle subtree.
AUTOFS_IOC_EXPIRE_MULTI	kernel→daemon	Requests daemon to expire up to N idle subtrees.
AUTOFS_IOC_EXPIRE_INDIRECT	kernel→daemon	Like EXPIRE but limited to indirect-mode subtrees.
AUTOFS_IOC_EXPIRE_DIRECT	kernel→daemon	Like EXPIRE but limited to direct-mode mount points.
AUTOFS_IOC_PROTOSUBVER	kernel→daemon	Returns protocol sub-version (UmkaOS: 6, matching Linux 5.4+).
AUTOFS_IOC_ASKUMOUNT	daemon→kernel	Query whether the autofs mount point can be unmounted.

14.10.6 Expiry

After an autofs-triggered mount has been idle for timeout_secs seconds, the kernel initiates expiry. Expiry is cooperative: the kernel asks the daemon to consider unmounting; the daemon decides whether conditions are met (no processes have open files under the mount, no active chdir into it) and issues umount(2) if appropriate.

autofs_expire_run(mount: &AutofsMount):
  Executed from a kernel timer callback at intervals of timeout_secs / 4.

  1. Walk all mounts that are children of this autofs mount point.
  2. For each child mount M:
     a. Compute idle_time = now - M.last_access_time.
     b. If idle_time < timeout_secs: skip.
     c. If any process has an open fd into M's subtree (check mount's
        open-file reference count): skip.
     d. Allocate token = next_token.fetch_add(1, Relaxed).
     e. Write AutofsPacketExpire { hdr: { proto_version: 5,
        packet_type: ExpireIndirect (indirect) or ExpireDirect (direct) },
        token, name = M.mountpoint_name } to pipe.
     f. Insert AutofsPendingRequest into pending table.
     g. Daemon calls AUTOFS_IOC_READY(token) after umount(2) succeeds, or
        AUTOFS_IOC_FAIL(token) if the mount is still busy.
  3. The timer reschedules itself unless the mount is in catatonic state.

The expiry path does not sleep in the kernel; it is fire-and-forget from the kernel's perspective. The daemon drives the actual unmount.

14.10.7 VFS Integration

autofs inserts itself into VFS path walking through the dentry automount operation hook, which is called by follow_automount() inside the path resolution loop (Section 14.1):

follow_automount(at: &MountDentry, nd) -> Result<()>:
  1. If nd.flags includes LookupFlags::NO_AUTOMOUNT: return without
     triggering (the caller continues the walk on the bare dentry).
  2. If at.dentry's inode does not carry InodeFlags::AUTOMOUNT: there is no
     trigger at this position — return without calling the hook. This flag
     is the sole gate; the hook is never reached on an ordinary dentry.
  3. If the walk is still in RCU mode, downgrade it to REF-walk before the
     call (see "RCU-walk downgrade" below) — the hook may sleep.
  4. Call at.dentry.ops.d_automount(at) → new_mnt (may be None).
  5. If new_mnt is Some(mnt): call attach_mount(mnt, at)
     ([Section 14.6](#mount-tree-data-structures-and-operations--attachmount-attach-a-constructed-mount-into-the-tree)).
     Policy flags such as MNT_SHRINKABLE are already set on mnt by the
     producing filesystem; attach_mount never sets them.
  6. Continue path walk over the now-mounted subtree.

RCU-walk downgrade: the automount operation cannot sleep, and sleeping is required to wait for the daemon response. Therefore, if the path walk is in RCU mode (the optimistic lockless fast path), it is downgraded to REF-walk before the dentry automount operation is called. The downgrade is performed by fs/namei.c try_to_unlazy() in Linux, which acquires references to the path components traversed so far. Once in REF-walk, the kernel can sleep safely in autofs_d_automount.

LookupFlags::NO_AUTOMOUNT: Certain operations (stat, openat with O_NOFOLLOW | O_PATH, utimensat with AT_SYMLINK_NOFOLLOW) set this flag to avoid triggering automounts on stat-only access. This matches Linux semantics.

14.10.8 Mount Options

autofs is mounted by the daemon at startup with options passed via the data argument to mount(2):

Option	Description
fd=N	File descriptor of the daemon-side pipe end. Required.
uid=N	UID of the daemon process. Used for permission checks on expire.
gid=N	GID of the daemon process.
minproto=N	Minimum acceptable protocol version (daemon's minimum).
maxproto=N	Maximum acceptable protocol version (daemon's maximum).
indirect	Mount in indirect mode (default).
direct	Mount in direct mode.
offset	Mount in offset mode (internal; used by the daemon for sub-mounts).

UmkaOS implements autofs protocol version 5, sub-version 6 (AUTOFS_PROTO_SUBVERSION = 6), matching the version supported by Linux kernel 5.4+ and systemd's automount daemon v252+. The protocol version is negotiated at mount time: the kernel picks min(maxproto, UMKA_PROTO_VERSION) and returns it via AUTOFS_IOC_PROTOVER.

14.10.9 systemd Integration

A systemd .automount unit creates an autofs mount point at the path specified by Where=, paired with a .mount unit of the same name. systemd acts as the automount daemon:

1. At unit activation, systemd calls mount("autofs", Where, "autofs", 0, "fd=N,...").
2. When AutofsPacketMissing arrives on the pipe, systemd activates the corresponding .mount unit (which runs mount(2) for the real filesystem).
3. On success, systemd calls AUTOFS_IOC_READY(token); on failure, AUTOFS_IOC_FAIL(token).
4. TimeoutIdleSec= in the .automount unit maps directly to AUTOFS_IOC_SETTIMEOUT.
5. After the idle timeout, systemd receives AutofsPacketExpire and issues umount(2) if the mount is not busy, then calls AUTOFS_IOC_READY(token).

Example unit (/etc/systemd/system/home.automount):

[Unit]
Description=Automount /home via NFS

[Automount]
Where=/home
TimeoutIdleSec=300

[Install]
WantedBy=multi-user.target

Paired with /etc/systemd/system/home.mount which specifies the NFS source and options. systemd creates the autofs mount point when the .automount unit starts and tears it down when the unit stops.

14.10.10 Linux Compatibility

UmkaOS's autofs implementation is wire-compatible with Linux autofs4:

• Protocol version 5, sub-version 6 — matches Linux kernel 5.4+.
• All ioctl numbers are identical to Linux (AUTOFS_IOC_* from <linux/auto_fs.h>).
• AutofsPacketMissing and AutofsPacketExpire structs are #[repr(C)] and match the Linux kernel ABI exactly.
• Mount option string format (fd=N,uid=N,...) matches Linux.
• systemd's automount daemon, autofs(5) userspace tools, and mount.autofs all operate without modification against UmkaOS's autofs implementation.

---

14.11 FUSE — Filesystem in Userspace

FUSE allows user-space processes to implement complete filesystems. A FUSE filesystem daemon opens /dev/fuse (character device, major 10, minor 229), mounts via FUSE_SUPER_MAGIC, and serves kernel VFS calls by reading and writing structured FUSE messages over the device fd. Any FUSE protocol-compliant daemon runs without modification on UmkaOS.

14.11.1 Architecture

User Process (e.g., sshfs, rclone, glusterfs-fuse)
       │  write(fuse_fd, fuse_out_header + reply)
       │  read(fuse_fd, fuse_in_header + args)
       ▼
  /dev/fuse  (character device, major 10 minor 229)
       │
  ┌────┴────────────────────────────────────────┐
  │  FuseConn: pending request queue            │
  │  FuseInode: nodeid → dentry mapping         │
  └────┬────────────────────────────────────────┘
       │  VFS callbacks → fuse_request dispatch
       ▼
  UmkaOS VFS layer (lookup, read, write, open, ...)
       │
  POSIX application

The FUSE connection object (FuseConn) is the central coordination point. It maintains two queues: pending (requests waiting for the daemon to pick up) and processing (requests sent to the daemon, awaiting reply). Each VFS thread that triggers a FUSE operation enqueues a request and blocks until the daemon writes the corresponding reply.

14.11.2 Core Data Structures

/// Maximum pending FUSE requests per connection. Prevents unbounded kernel
/// memory growth from a slow or misbehaving FUSE daemon.
const FUSE_MAX_PENDING: usize = 4096;

/// One FUSE connection — shared between all fds opened on this mount.
pub struct FuseConn {
    /// Pending requests waiting for the daemon to read.
    /// Lock-free bounded MPMC ring (defined in Section 3.1.11). VFS
    /// operations push requests (producer side); the FUSE daemon reads
    /// from the ring via `/dev/fuse` (consumer side). `try_push()`
    /// returns `Err(Full)` for backpressure — foreground callers block
    /// on `waitq` until the daemon drains entries; background callers
    /// receive `EAGAIN`.
    /// Per-request overhead: ~60-80 cycles for Arc refcount operations
    /// (4 atomic ops across pending ring and processing XArray). Acceptable:
    /// each FUSE request involves a user-kernel round-trip (~2-10 us),
    /// making the ~30-40 ns refcount overhead <2%.
    pub pending:      BoundedMpmcRing<Arc<FuseRequest>, FUSE_MAX_PENDING>,
    /// Number of currently outstanding background (async) requests.
    /// Incremented when a background request is submitted; decremented on reply.
    pub num_background: AtomicU32,
    /// Maximum background requests before blocking new submissions.
    /// Default: 12 (matching Linux `FUSE_DEFAULT_MAX_BACKGROUND`).
    /// Negotiated via FUSE_INIT: the daemon may set `max_background` in
    /// `FuseInitOut` to override the default.
    pub max_background: u32,
    /// When `num_background >= congestion_threshold`, the VFS marks the
    /// backing device as congested, causing writeback to throttle.
    /// Default: 9 (matching Linux `FUSE_DEFAULT_CONGESTION_THRESHOLD`,
    /// which is `max_background * 3 / 4`).
    ///
    /// **Units note**: Both `max_background` and `congestion_threshold` are
    /// measured in **request count** (not pages or bytes). Each background
    /// FUSE request may transfer a variable number of pages (e.g., a single
    /// WRITE request carries up to `max_write` bytes, default 128 KiB = 32
    /// pages). The request-count limit provides coarse backpressure; memory
    /// consumption is bounded by `max_background * max_write`.
    pub congestion_threshold: u32,
    /// Wait queue for tasks blocked due to backpressure (background request
    /// count exceeding `max_background`).
    pub bg_waitq:     WaitQueue,
    /// Requests sent to the daemon, awaiting reply. Keyed by monotonic
    /// request ID (u64). XArray provides O(1) lookup with native RCU reads
    /// and internal xa_lock for write serialization, eliminating the need
    /// for an external Mutex on the lookup structure.
    pub processing:   XArray<Arc<FuseRequest>>,
    /// Wait queue: daemon blocked in read() waiting for new requests.
    pub waitq:        WaitQueue,
    /// Connection options negotiated via FUSE_INIT.
    pub opts:         FuseConnOpts,
    /// Next unique request ID (monotonically increasing).
    pub next_unique:  AtomicU64,
    /// True after the daemon has exchanged FUSE_INIT.
    /// Intra-domain (FuseConn lives entirely within umka-vfs Tier 1).
    /// AtomicBool validity maintained by Rust type safety.
    pub initialized:  AtomicBool,
    /// True when the connection is shutting down. Intra-domain.
    pub destroyed:    AtomicBool,
    /// Maximum write size negotiated (from FUSE_INIT reply).
    pub max_write:    u32,
    /// Maximum read size.
    pub max_read:     u32,
}

/// A single FUSE request/reply pair.
pub struct FuseRequest {
    /// Monotonic ID — matches `FuseInHeader.unique` and `FuseOutHeader.unique`.
    pub unique:  u64,
    pub opcode:  FuseOpcode,
    /// Serialized FUSE input args (everything after the `FuseInHeader`).
    /// **Collection policy exception**: Vec<u8> on a warm/hot path. FUSE input
    /// args are variable-length (path names up to PATH_MAX, write data up to
    /// max_write). A fixed-size buffer would waste memory for small ops or
    /// truncate large ones. Allocation is bounded by max_write (negotiated
    /// at FUSE_INIT, typically 128 KiB) and occurs once per FUSE operation.
    pub in_args: Vec<u8>,
    pub reply:   Mutex<FuseReply>,
    /// Woken when `reply` transitions to `Done`.
    pub waker:   WaitEntry,
}

/// State of a request's reply.
pub enum FuseReply {
    /// Not yet answered by the daemon.
    Pending,
    /// Reply bytes, or a negative errno on error.
    /// Collection policy exception: Vec<u8> on warm/hot path. FUSE replies
    /// are variable-length (stat: ~100 bytes, read data: up to max_read,
    /// readdir: variable). Allocation bounded by max_read (negotiated at
    /// FUSE_INIT, typically 128 KiB). The FUSE userspace round-trip (~2-10 us)
    /// dominates; Vec allocation (~50-100 ns) is <5% overhead.
    Done(Result<Vec<u8>, i32>),
}

/// FUSE connection options negotiated during FUSE_INIT.
pub struct FuseConnOpts {
    pub max_write:           u32,
    pub max_read:            u32,
    pub max_pages:           u16,
    /// Capabilities declared by the daemon (server side).
    pub capable:             FuseInitFlags,
    /// Capabilities the kernel requests (client side).
    pub want:                FuseInitFlags,
    /// Timestamp granularity in nanoseconds (0 = 1 ns, i.e., full precision).
    pub time_gran:           u32,
    pub writeback_cache:     bool,
    pub parallel_dirops:     bool,
    pub async_dio:           bool,
    pub posix_acl:           bool,
    pub default_permissions: bool,
    pub allow_other:         bool,
}

FuseConn is reference-counted via Arc and held by: - The superblock of the mounted filesystem. - Every open file descriptor on /dev/fuse belonging to that mount.

When the last daemon fd is closed, FuseConn.destroyed is set and all further VFS operations return EIO. The mount point must then be explicitly unmounted with fusermount -u or umount.

14.11.2.1 Request Backpressure

FUSE distinguishes foreground requests (synchronous VFS operations: lookup, open, read, write) from background requests (async writeback, readahead, background FUSE_NOTIFY replies). Backpressure is applied to background requests to prevent a slow daemon from causing unbounded kernel memory growth:

fuse_submit_background(conn, request):
  loop:
    n = conn.num_background.load(Acquire)
    if n < conn.max_background:
      if conn.num_background.compare_exchange(n, n + 1, AcqRel, Acquire).is_ok():
        break
    else:
      // Block until the daemon processes a reply and decrements num_background.
      // Non-blocking callers (e.g., writeback from kthread) get EAGAIN instead.
      if request.is_nonblocking():
        return Err(EAGAIN)
      conn.bg_waitq.wait_until(|| conn.num_background.load(Acquire) < conn.max_background)

  // Congestion marking: when background requests exceed the threshold,
  // inform the VFS writeback layer so it throttles dirty page generation.
  if conn.num_background.load(Acquire) >= conn.congestion_threshold:
    set_bdi_congested(conn.backing_dev_info)

  conn.pending.try_push(request)  // lock-free; returns Err(Full) if ring is full
  conn.waitq.wake_one()  // wake daemon blocked in read(/dev/fuse)

fuse_complete_background(conn):
  prev = conn.num_background.fetch_sub(1, AcqRel)
  if prev <= conn.congestion_threshold:
    clear_bdi_congested(conn.backing_dev_info)
  if prev <= conn.max_background:
    conn.bg_waitq.wake_one()

Foreground requests are not subject to max_background — they always enter the pending ring (bounded by FUSE_MAX_PENDING = 4096). If try_push() returns Err(Full), the foreground caller blocks on conn.waitq until the daemon drains entries. This matches Linux semantics where synchronous FUSE operations never return EAGAIN (except with O_NONBLOCK on the file, which is handled at the VFS layer above FUSE).

14.11.3 Wire Protocol

All FUSE communication is framed with fixed headers. The kernel writes a request header followed by opcode-specific arguments; the daemon writes a reply header followed by opcode-specific data.

/// Fixed header preceding every FUSE request (kernel → daemon).
#[repr(C)]
pub struct FuseInHeader {
    /// Total request length (this header + opcode args).
    pub len:     u32,
    /// Opcode (FuseOpcode value).
    pub opcode:  u32,
    /// Unique request ID; matched by the reply.
    pub unique:  u64,
    /// Target inode number (0 for FUSE_INIT / FUSE_STATFS).
    pub nodeid:  u64,
    /// Effective UID of the calling process.
    pub uid:     u32,
    /// Effective GID of the calling process.
    pub gid:     u32,
    /// PID of the calling process.
    pub pid:     u32,
    /// Length of extended request data appended after the standard opcode
    /// arguments (protocol 7.36+). Zero when no extensions are present.
    /// Used by FUSE_SECURITY_CTX, FUSE_CREATE_SUPP_GROUP.
    pub total_extlen: u16,
    pub padding: u16,
}
const_assert!(size_of::<FuseInHeader>() == 40);

/// Fixed header preceding every FUSE reply (daemon → kernel).
#[repr(C)]
pub struct FuseOutHeader {
    /// Total reply length (this header + reply data).
    pub len:    u32,
    /// 0 on success; negative errno on error (e.g., -ENOENT = -2).
    pub error:  i32,
    /// Matches the `unique` field from the corresponding `FuseInHeader`.
    pub unique: u64,
}
const_assert!(size_of::<FuseOutHeader>() == 16);

Requests and replies are variable-length. The daemon must read exactly FuseInHeader.len bytes per request and must write exactly FuseOutHeader.len bytes per reply. A short read or write is a protocol error and terminates the connection.

FUSE_FORGET and FUSE_BATCH_FORGET are the only opcodes that carry no reply; the daemon must not write a reply for them.

14.11.4 FUSE Opcodes

The direction column records who initiates the message: K→D = kernel to daemon (a VFS call from a user process), D→K = daemon to kernel (a notify or retrieve reply with no corresponding VFS initiator).

Opcode	Value	Direction	Description
FUSE_LOOKUP	1	K→D	Lookup a name within a directory
FUSE_FORGET	2	K→D	Decrement inode reference count (no reply)
FUSE_GETATTR	3	K→D	Fetch inode attributes
FUSE_SETATTR	4	K→D	Modify inode attributes
FUSE_READLINK	5	K→D	Read the target of a symbolic link
FUSE_SYMLINK	6	K→D	Create a symbolic link
(reserved)	7	—	Reserved (unused in FUSE protocol; sequence intentionally skips from 6 to 8)
FUSE_MKNOD	8	K→D	Create a special or regular file
FUSE_MKDIR	9	K→D	Create a directory
FUSE_UNLINK	10	K→D	Remove a file
FUSE_RMDIR	11	K→D	Remove a directory
FUSE_RENAME	12	K→D	Rename a file (v1; same mount)
FUSE_LINK	13	K→D	Create a hard link
FUSE_OPEN	14	K→D	Open a file
FUSE_READ	15	K→D	Read file data
FUSE_WRITE	16	K→D	Write file data
FUSE_STATFS	17	K→D	Query filesystem statistics
FUSE_RELEASE	18	K→D	Close file (last close releases the handle)
(reserved)	19	—	Unassigned in FUSE protocol (intentionally skipped)
FUSE_FSYNC	20	K→D	Sync file data to stable storage
FUSE_SETXATTR	21	K→D	Set an extended attribute
FUSE_GETXATTR	22	K→D	Get an extended attribute value
FUSE_LISTXATTR	23	K→D	List all extended attribute names
FUSE_REMOVEXATTR	24	K→D	Remove an extended attribute
FUSE_FLUSH	25	K→D	Flush on close (sent before FUSE_RELEASE)
FUSE_INIT	26	K→D	Initialize connection (first message exchanged)
FUSE_OPENDIR	27	K→D	Open a directory
FUSE_READDIR	28	K→D	Read directory entries
FUSE_RELEASEDIR	29	K→D	Close a directory
FUSE_FSYNCDIR	30	K→D	Sync directory metadata to stable storage
FUSE_GETLK	31	K→D	Test a POSIX byte-range lock
FUSE_SETLK	32	K→D	Acquire or release a POSIX lock (non-blocking)
FUSE_SETLKW	33	K→D	Acquire a POSIX lock (blocking)
FUSE_ACCESS	34	K→D	Check access (used only when default_permissions is false)
FUSE_CREATE	35	K→D	Atomically create and open a file
FUSE_INTERRUPT	36	K→D	Cancel a pending request
FUSE_BMAP	37	K→D	Map logical file block to device block
FUSE_DESTROY	38	K→D	Tear down the connection
FUSE_IOCTL	39	K→D	Forward an ioctl to the userspace filesystem
FUSE_POLL	40	K→D	Poll a file for readiness events
FUSE_NOTIFY_REPLY	41	D→K	Deliver data in response to FUSE_NOTIFY_RETRIEVE
FUSE_BATCH_FORGET	42	K→D	Drop references for multiple inodes at once
FUSE_FALLOCATE	43	K→D	Pre-allocate or de-allocate file space
FUSE_READDIRPLUS	44	K→D	Read directory entries together with their attributes
FUSE_RENAME2	45	K→D	Rename with RENAME_EXCHANGE or RENAME_NOREPLACE
FUSE_LSEEK	46	K→D	Seek with SEEK_DATA or SEEK_HOLE
FUSE_COPY_FILE_RANGE	47	K→D	Server-side copy (copy_file_range)
FUSE_SETUPMAPPING	48	K→D	Set up a DAX direct memory mapping
FUSE_REMOVEMAPPING	49	K→D	Remove a DAX mapping
FUSE_SYNCFS	50	K→D	Sync the entire filesystem
FUSE_TMPFILE	51	K→D	Create an unnamed temporary file (O_TMPFILE)
FUSE_STATX	52	K→D	Extended stat (statx(2))
FUSE_COPY_FILE_RANGE_64	53	K→D	Server-side copy (64-bit variant, returns bytes_copied via fuse_copy_file_range_out). Added in FUSE protocol 7.45.

Notify messages (daemon → kernel, unsolicited; no reply is sent by the kernel except for FUSE_NOTIFY_RETRIEVE which expects FUSE_NOTIFY_REPLY):

Notify code	Value	Description
FUSE_NOTIFY_POLL	1	Wake all pollers on the specified file handle
FUSE_NOTIFY_INVAL_INODE	2	Invalidate cached attributes and, optionally, a byte range of page cache
FUSE_NOTIFY_INVAL_ENTRY	3	Invalidate a specific dentry in a parent directory
FUSE_NOTIFY_STORE	4	Pre-populate a byte range of the page cache
FUSE_NOTIFY_RETRIEVE	5	Request the kernel to send page-cache contents back to the daemon
FUSE_NOTIFY_DELETE	6	Remove a dentry without a round-trip FUSE_LOOKUP failure
FUSE_NOTIFY_RESEND	7	Daemon notification that a previously interrupted request should be resent. Paired with HAS_RESEND capability flag (bit 39). Protocol 7.41+, Linux 6.12+
FUSE_NOTIFY_INC_EPOCH	8	Increment the kernel-side epoch counter for cache invalidation coordination
FUSE_NOTIFY_PRUNE	9	Request the kernel to prune (evict) dentries from a directory

14.11.5 FUSE_INIT Handshake

FUSE_INIT is always the first message exchanged. The kernel sends FuseInitIn and the daemon replies with FuseInitOut. The two sides negotiate protocol version and capability flags; the connection uses the minimum agreed minor version.

/// FUSE_INIT request body (kernel → daemon).
#[repr(C)]
pub struct FuseInitIn {
    /// FUSE major protocol version (kernel sends 7).
    pub major:         u32,
    /// FUSE minor protocol version (kernel sends 45 for Linux 6.14+ equivalent).
    pub minor:         u32,
    pub max_readahead: u32,
    /// Capability bitmask the kernel supports (low 32 bits of FuseInitFlags).
    /// Wire format: flags = FuseInitFlags bits 0-31 (low 32 bits).
    pub flags:         u32,
    /// Extended capability flags (protocol minor ≥ 36, FUSE_INIT_EXT must be set in flags).
    /// Wire format: flags2 = FuseInitFlags bits 32-63 shifted down 32 bits.
    /// This matches the FUSE protocol extension for large flag sets (kernel 5.13+).
    pub flags2:        u32,
    pub unused:        [u32; 11],
}
// Layout: 5 × u32 + 11 × u32 = 16 × 4 = 64 bytes.
const_assert!(size_of::<FuseInitIn>() == 64);

/// FUSE_INIT reply body (daemon → kernel).
#[repr(C)]
pub struct FuseInitOut {
    pub major:               u32,
    pub minor:               u32,
    pub max_readahead:       u32,
    /// Capabilities the daemon acknowledges and enables (low 32 bits of FuseInitFlags).
    /// Wire format: flags = FuseInitFlags bits 0-31 (low 32 bits).
    pub flags:               u32,
    /// Maximum number of outstanding background requests.
    pub max_background:      u16,
    /// Congestion threshold: kernel slows down at this many background requests.
    pub congestion_threshold: u16,
    /// Maximum bytes per WRITE request.
    pub max_write:           u32,
    /// Timestamp granularity in nanoseconds (0 = 1 ns, i.e., full precision).
    pub time_gran:           u32,
    /// Maximum scatter-gather page count per request.
    pub max_pages:           u16,
    /// Alignment required for DAX mappings.
    pub map_alignment:       u16,
    /// Extended flags (protocol minor ≥ 36, requires FUSE_INIT_EXT set in flags).
    /// Wire format: flags2 = FuseInitFlags bits 32-63 shifted down 32 bits.
    /// This matches the FUSE protocol extension for large flag sets (kernel 5.13+).
    pub flags2:              u32,
    pub max_stack_depth:     u32,
    /// Negotiated request timeout in seconds. Valid when `FUSE_REQUEST_TIMEOUT`
    /// (bit 42) is set in the negotiated flags. 0 = no timeout. Matches Linux
    /// `include/uapi/linux/fuse.h` field `request_timeout`.
    pub request_timeout:     u16,
    pub unused:              [u16; 11],
}
// Layout: 4+4+4+4+2+2+4+4+2+2+4+4+2+22 = 64 bytes.
// (8×u32 = 32) + (4×u16 = 8) + (11×u16 = 22) + (request_timeout u16 = 2) = 64.
const_assert!(size_of::<FuseInitOut>() == 64);

bitflags! {
    /// Capability flags exchanged during FUSE_INIT.
    pub struct FuseInitFlags: u64 {
        /// Daemon supports asynchronous read requests.
        const ASYNC_READ          = 1 << 0;
        /// Daemon handles POSIX advisory byte-range locks.
        const POSIX_LOCKS         = 1 << 1;
        /// Daemon uses file handles returned in open replies.
        const FILE_OPS            = 1 << 2;
        /// Daemon handles O_TRUNC atomically in open.
        const ATOMIC_O_TRUNC      = 1 << 3;
        /// Filesystem supports NFS export (node IDs are stable across reboots).
        const EXPORT_SUPPORT      = 1 << 4;
        /// Daemon supports writes larger than 4 KiB.
        const BIG_WRITES          = 1 << 5;
        /// Kernel should not apply the process umask to create operations.
        const DONT_MASK           = 1 << 6;
        /// Daemon supports splice(2)-based writes.
        const SPLICE_WRITE        = 1 << 7;
        /// Daemon supports splice(2)-based moves.
        const SPLICE_MOVE         = 1 << 8;
        /// Daemon supports splice(2)-based reads.
        const SPLICE_READ         = 1 << 9;
        /// Daemon handles BSD flock() locking.
        const FLOCK_LOCKS         = 1 << 10;
        /// Daemon supports ioctl on directories.
        const HAS_IOCTL_DIR       = 1 << 11;
        /// Kernel auto-invalidates cached data on attribute changes.
        const AUTO_INVAL_DATA     = 1 << 12;
        /// Kernel uses FUSE_READDIRPLUS instead of FUSE_READDIR.
        const DO_READDIRPLUS      = 1 << 13;
        /// Kernel switches adaptively between READDIRPLUS and READDIR.
        const READDIRPLUS_AUTO    = 1 << 14;
        /// Daemon supports asynchronous direct I/O.
        const ASYNC_DIO           = 1 << 15;
        /// Daemon supports writeback caching (batched dirty page writeback).
        const WRITEBACK_CACHE     = 1 << 16;
        /// Daemon does not need FUSE_OPEN (open is a no-op).
        const NO_OPEN_SUPPORT     = 1 << 17;
        /// Parallel directory operations are safe (no serialization needed).
        const PARALLEL_DIROPS     = 1 << 18;
        /// Kernel clears setuid/setgid bits on write (v1).
        const HANDLE_KILLPRIV     = 1 << 19;
        /// Daemon supports POSIX ACLs.
        const POSIX_ACL           = 1 << 20;
        /// Daemon sets error on abort rather than returning EIO.
        const ABORT_ERROR         = 1 << 21;
        /// `max_pages` field in FuseInitOut is valid.
        const MAX_PAGES           = 1 << 22;
        /// Daemon caches symlink targets.
        const CACHE_SYMLINKS      = 1 << 23;
        /// Daemon does not need FUSE_OPENDIR.
        const NO_OPENDIR_SUPPORT  = 1 << 24;
        /// Daemon explicitly invalidates data (FUSE_NOTIFY_INVAL_INODE).
        const EXPLICIT_INVAL_DATA = 1 << 25;
        /// `map_alignment` field in FuseInitOut is valid.
        const MAP_ALIGNMENT       = 1 << 26;
        /// Daemon is aware of submount semantics.
        const SUBMOUNTS           = 1 << 27;
        /// Kernel clears setuid/setgid bits on write (v2, extended semantics).
        const HANDLE_KILLPRIV_V2  = 1 << 28;
        /// Extended setxattr arguments (flags field present).
        const SETXATTR_EXT        = 1 << 29;
        /// `flags2` fields in FuseInitIn/Out are valid.
        const INIT_EXT            = 1 << 30;
        const INIT_RESERVED       = 1 << 31;

        // --- flags2 bits (require INIT_EXT set in flags) ---
        // Wire format: flags2 = FuseInitFlags bits 32-63 shifted down 32 bits.
        // All bit positions match Linux `include/uapi/linux/fuse.h` (torvalds/linux master).

        /// Security context support (protocol 7.36+). Linux 6.0+.
        /// Daemon can receive security context (e.g., SELinux label) with
        /// create/mkdir/mknod requests via extended headers (total_extlen).
        const SECURITY_CTX           = 1 << 32;
        /// Per-inode DAX hint (protocol 7.36+). Linux 6.0+.
        /// Daemon can set per-inode DAX mode via FUSE_ATTR_DAX.
        const HAS_INODE_DAX          = 1 << 33;
        /// Supplementary group support in create (protocol 7.38+). Linux 6.6+.
        const CREATE_SUPP_GROUP      = 1 << 34;
        /// Expire-only entry invalidation (protocol 7.38+). Linux 6.6+.
        const HAS_EXPIRE_ONLY        = 1 << 35;
        /// Allow mmap on direct-I/O files (protocol 7.39+). Linux 6.6+.
        const DIRECT_IO_ALLOW_MMAP   = 1 << 36;
        /// I/O passthrough to backing file (protocol 7.40+). Linux 6.9+.
        const PASSTHROUGH            = 1 << 37;
        /// Opt out of NFS export support (protocol 7.40+). Linux 6.9+.
        const NO_EXPORT_SUPPORT      = 1 << 38;
        /// Daemon supports request resend on interrupted operations.
        /// When set, the kernel may resend a FUSE request that was interrupted
        /// (e.g., by a signal) if the daemon has not yet replied. The daemon must
        /// handle duplicate `unique` IDs idempotently (protocol 7.41+). Linux 6.12+.
        const HAS_RESEND             = 1 << 39;
        /// ID-mapped FUSE mounts (protocol 7.42+). Linux 6.12+.
        const ALLOW_IDMAP            = 1 << 40;
        /// io_uring-based FUSE request transport (protocol 7.43+). Linux 6.14+.
        /// When negotiated, requests are submitted and completed via io_uring
        /// SQEs/CQEs instead of read()/write() on `/dev/fuse`, eliminating
        /// two syscalls per FUSE operation.
        const OVER_IO_URING          = 1 << 41;
        /// Per-request timeout (protocol 7.45+). Linux 6.14+.
        const REQUEST_TIMEOUT        = 1 << 42;
    }
}

If the daemon returns a minor version lower than what the kernel sent, the kernel downconverts: fields that did not exist in the older protocol minor are ignored. If the daemon sends a major version other than 7, the kernel closes the connection.

14.11.6 VFS Integration

FUSE registers filesystem type "fuse" with superblock magic FUSE_SUPER_MAGIC = 0x65735546. Mounting proceeds as follows:

mount(2) path

1. User invokes mount -t fuse -o fd=N,... or uses the fusermount3 helper.
2. The kernel parses the fd=N mount option and resolves the fd to an open /dev/fuse file.
3. A FuseConn is allocated and attached to the fd and the new superblock.
4. The kernel sends FUSE_INIT and waits for the daemon's reply; on success, FuseConn.initialized is set and the mount completes.

VFS → FUSE dispatch

For every VFS operation on a FUSE mount (lookup, read, write, getattr, etc.) the kernel:

1. Allocates a FuseRequest with a fresh unique ID.
2. Serializes the opcode-specific arguments into in_args.
3. Appends the request to FuseConn.pending and wakes the daemon's wait queue.
4. Blocks on FuseRequest.waker until the daemon writes a reply.
5. Deserializes the reply from FuseRequest.reply and returns to the VFS caller.

The daemon loop is simply:

loop {
    bytes = read(fuse_fd, buf)          // blocks until a request is pending
    handle_opcode(parse(buf))
    write(fuse_fd, reply_bytes)         // unblocks the kernel thread
}

Interrupt handling

If the calling thread receives a fatal signal while waiting for a FUSE reply, the kernel enqueues a FUSE_INTERRUPT message targeting the original request's unique ID. It then waits a short grace period (default 20 milliseconds). If the daemon does not abort the request and send a reply within that window, the kernel forcibly removes the request from FuseConn.processing and returns EINTR to the caller. The daemon is expected to ignore any subsequent reply it sends for the interrupted unique.

Writeback cache (WRITEBACK_CACHE flag)

When this capability is negotiated, dirty pages accumulate in the kernel page cache and are written to the daemon in larger batches via FUSE_WRITE. Without it, every write(2) to a FUSE file generates an immediate, synchronous FUSE_WRITE to the daemon, serializing all write traffic. Most performance- sensitive FUSE filesystems negotiate WRITEBACK_CACHE.

Connection death

When the last daemon fd is closed (daemon exits, crashes, or explicitly calls FUSE_DESTROY):

1. FuseConn.destroyed is set atomically.
2. All requests in FuseConn.processing are completed with error ENOTCONN.
3. All requests in FuseConn.pending are discarded.
4. Subsequent VFS operations on the mount return EIO.
5. The mount point persists in the namespace; an explicit umount or fusermount -u is required to remove it.

14.11.7 Security Model

Mount-owner restriction (default)

Unless the allow_other mount option is passed, only the UID that opened /dev/fuse and performed the mount may access the filesystem. All other UIDs receive EACCES from the UmkaOS VFS layer before the request reaches the daemon, regardless of the file mode bits the daemon returns.

allow_other option

Permits any UID to access the filesystem subject to normal Unix permission checks. Because allow_other exposes the daemon process to arbitrary user requests, it requires either: - The SysAdmin capability in the mount namespace, or - The /proc/sys/fs/fuse/user_allow_other sysctl set to 1 (off by default).

default_permissions option

When set, the kernel enforces standard Unix permission checks (owner, group, other; st_mode, st_uid, st_gid) against the attributes the daemon returns in FUSE_GETATTR. The kernel never sends FUSE_ACCESS in this mode. Without default_permissions, the daemon is responsible for its own access control and receives FUSE_ACCESS for every access check.

Privilege requirement for mounting

Unprivileged FUSE mounts (without SysAdmin) are permitted only through fusermount3, which is installed setuid-root and validates that the user owns the target mountpoint. Direct mount(2) requires SysAdmin in the current user namespace.

14.11.8 io_uring FUSE

UmkaOS supports the io_uring-based FUSE I/O path (OVER_IO_URING feature, equivalent to Linux 6.14+). The daemon opts in by negotiating the OVER_IO_URING capability during FUSE_INIT and then submitting SQEs of type IORING_OP_URING_CMD to the /dev/fuse fd rather than using blocking read/write.

Benefits over the classic blocking I/O path:

• Asynchronous request handling — the daemon can have many requests in flight simultaneously without blocking threads.
• Reduced syscall overhead — requests are batched via io_uring_submit; one syscall drains or fills multiple queue slots.
• CPU affinity — the daemon can pin io_uring workers to specific CPUs, reducing cross-socket latency for NUMA-aware FUSE filesystems.

The FUSE daemon registers a fixed buffer pool at startup. The kernel delivers requests into pre-registered buffers, and the daemon submits replies via the same ring. The wire format (FuseInHeader, FuseOutHeader, opcode bodies) is unchanged; only the transport mechanism differs.

Capability requirement: OVER_IO_URING negotiation requires the daemon process to hold CAP_IPC_LOCK (needed for the io_uring fixed buffer registration, which pins user pages via IORING_REGISTER_BUFFERS). If the daemon lacks CAP_IPC_LOCK, the OVER_IO_URING capability is silently cleared from the FUSE_INIT response and the connection falls back to the classic blocking I/O path.

14.11.9 Linux Compatibility

• /dev/fuse device node (major 10, minor 229): identical to Linux.
• FUSE protocol version 7.45 (Linux 6.14+ equivalent) is the maximum negotiated kernel version. Daemons advertising higher minors receive 7.45 in the reply.
• libfuse3 (3.x series): works without modification.
• fusermount3 and the fuse.ko-equivalent path: built into the UmkaOS VFS layer; no kernel module is required.
• All widely deployed FUSE filesystems run without modification: sshfs, rclone mount, glusterfs-fuse, ceph-fuse, bindfs, s3fs-fuse, encfs, gvfs, ntfs-3g.
• DAX (FUSE_SETUPMAPPING / FUSE_REMOVEMAPPING) is supported on persistent memory-backed FUSE mounts, providing zero-copy access to file data.

---

14.11.10 VFS Service Provider

Provider model: VFS service is always a host-native provider (the host's kernel manages the filesystem). Device-native Tier M providers do not apply here — storage devices provide BLOCK_STORAGE, not FILESYSTEM. The VFS service provider runs on the host that mounts the filesystem and serves it to remote peers. Sharing model: multiple remote peers mount the same export simultaneously (close-to-open consistency, DLM-coordinated locking).

A host can provide mounted filesystems as cluster services via the peer protocol. Remote peers mount the export as a local filesystem and perform file operations transparently — the VFS dispatches operations to the remote host, which executes them against its local filesystem.

This is the VFS instantiation of the capability service provider model described in Section 5.7. In a uniform UmkaOS cluster, VFS service provider provides file sharing without requiring NFS, nfsd, portmapper, idmapd, or any external daemon. The cluster infrastructure (peer protocol, DLM, PeerRegistry, heartbeat) provides everything needed.

// umka-vfs/src/service_provider.rs

/// Opaque handle to a local mount point being exported as a cluster service.
/// Holds a strong reference to the underlying `Mount`
/// ([Section 14.6](#mount-tree-data-structures-and-operations)) so the mount cannot be
/// torn down while it is being served; dropping the handle releases the
/// export's reference. Kernel-internal, not an ABI type.
pub struct MountHandle(pub Arc<Mount>);

/// Cluster-unique identifier for an exported filesystem (an "export").
/// Assigned when a `VfsServiceProvider` is created and used as the DLM lock
/// namespace for every file lock on the export (see `VfsLockResource`). u64
/// (50-year rule: monotonically allocated per cluster, never recycled while
/// an export is live). Kernel-internal newtype.
#[derive(Clone, Copy, PartialEq, Eq, Hash)]
pub struct ExportId(pub u64);

/// Provides a local mount point as a cluster service to remote peers.
pub struct VfsServiceProvider {
    /// The local mount point being served (e.g., "/data").
    mount: MountHandle,
    /// Unique service identifier. Used as the DLM lock namespace for all
    /// file locks on this service (Section 14.7.10.3).
    service_id: ExportId,
    /// Transport endpoint for receiving remote VFS operations.
    endpoint: PeerEndpoint,
    /// Lease duration for metadata caching (default: 30 seconds).
    /// Remote peers cache metadata (stat, readdir) for this duration.
    /// On expiry, they must re-validate with the server.
    lease_duration_ms: u32,
    /// Maximum concurrent remote operations.
    max_inflight: u32,
    /// Connected clients, tracked for lease invalidation and recovery.
    /// Keyed by PeerId (u64). XArray provides O(1) lookup with native
    /// RCU-protected reads (no read-side locking) and ordered iteration.
    clients: XArray<ServiceClientState>,
}

/// Per-client state on the server side. Tracks leases and open files
/// for recovery after client disconnect/reconnect.
pub struct ServiceClientState {
    /// Peer ID of the connected client.
    peer_id: PeerId,
    /// PeerRegistry generation at last sync. Used to detect stale clients.
    last_registry_gen: u64,
    /// Active inode leases held by this client. Keyed by InodeId (u64);
    /// value is `()` (presence-only tracking). XArray per collection policy
    /// (integer-keyed mapping, warm path — lease grant/revoke).
    leases: XArray<()>,
    /// Open file handles (for recovery after server reboot). Keyed by
    /// FileHandle (u64). XArray per collection policy (integer-keyed mapping).
    /// Maximum 4096 open files per client (enforced at Open time; server
    /// returns -EMFILE if exceeded).
    open_files: XArray<OpenFileRecord>,
}

/// Recovery metadata for one open file on a remote client.
/// Used during server reboot recovery (grace period) to validate
/// client reclaim requests. Stored in `ServiceClientState::open_files`,
/// keyed by `FileHandle` (u64). The server populates this on every
/// successful `Open` and removes it on `Release`.
pub struct OpenFileRecord {
    /// Server-assigned file handle.
    handle: FileHandle,
    /// Inode this file handle refers to.
    inode_id: InodeId,
    /// Open flags (O_RDONLY, O_WRONLY, O_RDWR, etc.).
    flags: u32,
    /// Client's UID at open time (for permission re-verification on reclaim).
    uid: u32,
    /// Client's GID at open time.
    gid: u32,
}

/// VFS operation forwarded from a remote peer. Modeled after FUSE opcodes
/// but uses native UmkaOS VFS types, not FUSE wire format.
///
/// Every mutating operation carries the caller's `uid` and `gid` for
/// permission checking on the server (Section 14.7.10.2).
/// Fixed-size filename for wire protocol. NUL-padded, max 255 bytes.
#[repr(C)]
pub struct FileName {
    /// Actual name length in bytes (excluding NUL).
    pub len: u8,
    /// NUL-padded name bytes (only first `len` bytes are significant).
    pub bytes: [u8; 255],
}
// Layout: 1 + 255 = 256 bytes.
const_assert!(size_of::<FileName>() == 256);

/// Fixed-size xattr name for wire protocol (same layout as FileName).
pub type XattrName = FileName;

/// Xattr value descriptor. Values <= 224 bytes are inlined; larger values
/// use bulk transfer via a shared memory region.
#[repr(C)]
pub struct XattrValue {
    /// Actual value length in bytes.
    pub len: u32,
    /// Inline data (valid for first `min(len, 224)` bytes).
    pub inline_data: [u8; 224],
    /// Explicit padding after inline_data (offset 228) to align bulk_offset (u64, align 8).
    /// 228 % 8 = 4, need 4 bytes. CLAUDE.md rule 11.
    pub _pad: [u8; 4],
    /// Non-zero if value was transferred via bulk region (offset into
    /// the shared data region). Zero if fully inlined.
    pub bulk_offset: u64,
}
// Layout: len(4) + inline_data(224) + _pad(4) + bulk_offset(8) = 240 bytes.
// All padding explicit.
const_assert!(size_of::<XattrValue>() == 240);

/// Wire protocol discriminant. Append-only: new operations are added at
/// the end with incrementing values. Do not reorder or remove variants.
#[repr(C, u16)]
pub enum VfsServiceOp {
    Lookup { parent: InodeId, name: FileName, uid: u32, gid: u32 },
    Getattr { inode: InodeId },
    /// `attrs` is a `SetAttrMask` bitmask specifying which attributes to set.
    Setattr { inode: InodeId, attrs: SetAttrMask, uid: u32, gid: u32 },
    Open { inode: InodeId, flags: u32, uid: u32, gid: u32 },
    Read { handle: FileHandle, offset: u64, len: u32 },
    Write { handle: FileHandle, offset: u64, data_region_offset: u64, data_len: u32 },
    Release { handle: FileHandle },
    /// `offset` is an opaque server-assigned cookie (NOT a byte offset or
    /// entry index). Value 0 starts from the beginning of the directory.
    /// Each Readdir response includes the cookie for the next batch.
    /// This matches the NFS cookie model and avoids issues with
    /// concurrent directory mutations invalidating positional offsets.
    Readdir { inode: InodeId, offset: u64, uid: u32, gid: u32 },
    Create { parent: InodeId, name: FileName, mode: u32, flags: u32, uid: u32, gid: u32 },
    Unlink { parent: InodeId, name: FileName, uid: u32, gid: u32 },
    Mkdir { parent: InodeId, name: FileName, mode: u32, uid: u32, gid: u32 },
    Rmdir { parent: InodeId, name: FileName, uid: u32, gid: u32 },
    Rename { src_parent: InodeId, src_name: FileName,
             dst_parent: InodeId, dst_name: FileName, uid: u32, gid: u32 },
    Fsync { handle: FileHandle, datasync: u8 }, // 0 = fsync, 1 = fdatasync. u8 for wire safety.
    Statfs,
    /// File locking operations. Lock state is managed by the DLM
    /// (Section 14.7.10.3); these ops coordinate with the server's
    /// local filesystem lock state.
    Lock { handle: FileHandle, lock_type: LockType, start: u64, len: u64, uid: u32 },
    Unlock { handle: FileHandle, start: u64, len: u64 },
    /// Create a symbolic link. `target` is the symlink destination path.
    Symlink { parent: InodeId, name: FileName, target: FileName, uid: u32, gid: u32 },
    /// Read the target of a symbolic link.
    Readlink { inode: InodeId },
    /// Create a hard link. `inode` is the existing file; `new_parent`/`new_name`
    /// specify the new directory entry pointing to it.
    Link { inode: InodeId, new_parent: InodeId, new_name: FileName, uid: u32, gid: u32 },
    /// Get an extended attribute value.
    Getxattr { inode: InodeId, name: XattrName, uid: u32, gid: u32 },
    /// Set an extended attribute. `flags` follows Linux semantics:
    /// `XATTR_CREATE` (1) = fail if exists, `XATTR_REPLACE` (2) = fail if absent.
    Setxattr { inode: InodeId, name: XattrName, value: XattrValue, flags: u32, uid: u32, gid: u32 },
    /// List all extended attribute names on an inode.
    Listxattr { inode: InodeId, uid: u32, gid: u32 },
    /// Remove an extended attribute.
    Removexattr { inode: InodeId, name: XattrName, uid: u32, gid: u32 },
}
// VfsServiceOp is #[repr(C, u16)]: overall alignment = 8 (from u64 fields).
// Discriminant u16 at offset 0 (2 bytes), 6 bytes padding to offset 8 for
// first field alignment.
//
// Largest variant: Rename { src_parent: InodeId(8), src_name: FileName(256),
//   dst_parent: InodeId(8), dst_name: FileName(256), uid: u32(4), gid: u32(4) }
// Layout: offset 8..16(InodeId) + 16..272(FileName) + 272..280(InodeId)
//   + 280..536(FileName) + 536..540(u32) + 540..544(u32) = 544 bytes total.
//
// Runner-up: Symlink { parent(8), name(256), target(256), uid(4), gid(4) }
//   = 8+256+256+4+4 + 6(discrim pad) = 536 bytes.
// Setxattr { inode(8), name(256), value(240), flags(4), uid(4), gid(4) }
//   = 8+256+240+4+4+4 + 6(discrim pad) + 4(trailing align) = 528 bytes.
//
// EVERY variant on the wire takes 544 bytes, even a Getattr (16 bytes of
// actual data). Acceptable for a KABI ring (not a network wire protocol);
// consider a header+opcode+payload redesign if ring bandwidth is a concern.
const_assert!(size_of::<VfsServiceOp>() == 544);

/// Xattr encoding rules:
///
/// - `XattrName`: up to 255 bytes (`XATTR_NAME_MAX`, matching Linux). Sent
///   inline in the `ServiceMessage` payload.
/// - `XattrValue`: up to 65536 bytes (`XATTR_SIZE_MAX`, matching Linux).
///   Values <= 224 bytes are sent inline in the `ServiceMessage`. Values
///   > 224 bytes use bulk transfer via the peer transport: the client
///   writes the value into a bounce buffer and sends `Setxattr` with
///   `data_region_offset` pointing to the buffer; the server fetches the
///   value via remote fetch through the peer transport.
/// - `Listxattr` returns a null-separated list of attribute names. If the
///   total list exceeds 224 bytes, it is transferred via bulk push from
///   the server to the client's bounce buffer.
///
/// The 224-byte inline threshold is chosen to fit within a single
/// `ServiceMessage` payload (256 bytes minus header overhead), avoiding
/// a separate bulk transfer for small xattr values (the common case for
/// security labels, POSIX ACLs, and user attributes).

/// SetAttrMask bitmask specifying which attributes to set. Matches Linux
/// `ATTR_*` values for compatibility with `fuse_setattr_in`.
bitflags! {
    pub struct SetAttrMask: u32 {
        /// Set file mode (permissions).
        const MODE      = 1 << 0;
        /// Set owner UID.
        const UID       = 1 << 1;
        /// Set owner GID.
        const GID       = 1 << 2;
        /// Set file size (truncate).
        const SIZE      = 1 << 3;
        /// Set access time to a specific value.
        const ATIME     = 1 << 4;
        /// Set modification time to a specific value.
        const MTIME     = 1 << 5;
        /// Set change time (server updates ctime automatically on any change;
        /// this flag is for explicit ctime override when restoring backups).
        const CTIME     = 1 << 6;
        /// Set access time to current server time.
        const ATIME_NOW = 1 << 7;
        /// Set modification time to current server time.
        const MTIME_NOW = 1 << 8;
    }
}

/// Server-assigned file handle. Opaque to the client. Maps to an open
/// file descriptor on the server's VFS. Valid for the lifetime of the
/// client connection (or until Release).
pub type FileHandle = u64;

InodeId scope: InodeId values are scoped to a single VFS service export (one server mount point). They are NOT globally unique across the cluster. The combination (server_peer_id, service_id, inode_id) is globally unique. Clients must not compare InodeId values across different peerfs mounts.

Wire protocol: operation forwarding over the peer protocol. Data transfers (Read, Write) use remote write/read via the peer transport for zero-copy. Metadata and control operations use ring pair send/recv.

Server side: the VFS service provider receives operations, dispatches them to the local VFS layer (which invokes the local filesystem — ext4, XFS, etc.), and returns results. The server is entirely in-kernel — no userspace daemon involved (unlike NFS's nfsd or FUSE daemons). Permission checks use the caller's UID/GID against the local filesystem's ownership and mode bits.

Client side: remote peers mount the export using a dedicated filesystem type (mount -t peerfs host_peer_id:/data /mnt/remote). The client filesystem translates local VFS operations into VfsServiceOp messages and sends them to the server peer.

14.11.10.1 Scope and Relationship to NFS

VFS service provider is designed for uniform UmkaOS clusters — all nodes run UmkaOS, share a consistent UID/GID namespace, and trust each other at the kernel level (mutual peer authentication via the capability system). In this environment, it replaces NFS entirely: no RPC/XDR, no portmapper, no separate daemon, no exports file. File sharing is a native cluster capability.

VFS service provider does not cover the full NFS feature set:

Feature	VFS Export	NFS v4.2
Wire protocol	Native peer protocol (RDMA)	RPC/XDR (+ optional RDMA)
Authentication	Peer capability + UID pass-through	Kerberos/GSSAPI, AUTH_SYS
UID mapping	None (consistent namespace assumed)	idmapd, Kerberos principal
File locking	DLM (Section 15.15)	Built-in NLM/v4 locks
Consistency	Close-to-open + leases	Close-to-open + delegations
Server recovery	DLM lock recovery + heartbeat	Grace period + reclaim
Parallel data	Not supported	pNFS
Referrals	Not supported	v4.1 referrals
ACL model	POSIX ACLs (from underlying FS)	NFSv4 ACLs
Configuration	Zero (auto-discovered via PeerRegistry)	exports file, mount options
Daemons required	None	nfsd, mountd, idmapd, gssproxy

For mixed environments (Linux clients, Windows clients, NAS appliances, or clusters where per-user Kerberos authentication is required), NFS (Section 15.14) remains the right choice.

14.11.10.2 Identity and Permission Model

VFS service provider uses UID/GID pass-through: the client sends the calling process's UID and GID with every operation that requires permission checking. The server applies standard POSIX permission checks (owner, group, other, POSIX ACLs) against the local filesystem using the received UID/GID.

Assumptions (non-negotiable for VFS service provider):
1. All peers in the cluster share a consistent UID/GID namespace.
   (Same /etc/passwd, LDAP, or equivalent directory service on all nodes.)
2. The client peer is authenticated via the capability system (Section 9.1).
   UID/GID are trusted because the client kernel is trusted.
3. Root squash: optional, configurable per-export. When enabled, uid 0 from
   remote peers is mapped to nobody (65534). Default: enabled.

This is intentionally simple. UmkaOS clusters are managed as a single system with a single identity domain. Cross-domain authentication (Kerberos, GSSAPI) is the job of NFS, not the VFS service provider.

14.11.10.3 File Locking via DLM

File locks on exported filesystems are managed by the cluster's DLM (Section 15.15). The DLM already provides distributed lock coordination, deadlock detection (Section 15.15), and lock recovery on node failure.

/// DLM lock resource name for a file lock on an exported filesystem.
/// Composed from export ID and inode number — globally unique within
/// the cluster.
pub struct VfsLockResource {
    pub service_id: ExportId,
    pub inode_id: InodeId,
}

/// Lock type for VFS service provider locks. Maps directly to POSIX lock types.
#[repr(u8)]
pub enum LockType {
    /// flock() shared lock or fcntl() F_RDLCK.
    Shared = 0,
    /// flock() exclusive lock or fcntl() F_WRLCK.
    Exclusive = 1,
}

Lock flow (process on Host B locks file on Host A's export):

1. Process calls flock(fd, LOCK_EX) on a file mounted via peerfs.
2. Client VFS sends Lock { handle, Exclusive, 0, WHOLE_FILE, uid } to the server peer.
3. Server acquires a DLM lock on VfsLockResource { service_id, inode_id } in exclusive mode. If the DLM lock is already held by another peer, the request blocks (or returns EWOULDBLOCK for LOCK_NB).
4. After DLM grant, server also acquires the local filesystem lock (so local processes and remote clients see consistent lock state).
5. Server responds with success. Client flock() returns.

fcntl() byte-range locks: supported. The DLM lock resource includes the byte range: VfsLockResource { service_id, inode_id, start, len }. DLM handles range overlap and splitting.

Deadlock detection: the DLM's WaitForGraph (Section 15.15) detects cross-node deadlocks. If a deadlock cycle is found, one holder receives EDEADLK.

Lock recovery on client failure: when a client peer is declared Dead (heartbeat timeout, Section 5.8), the DLM releases all locks held by that peer. Server-side open file state for the dead client is cleaned up.

14.11.10.4 Consistency Model: Close-to-Open

VFS service provider uses close-to-open consistency, the same model as NFS. This is simple, well-understood, and sufficient for the vast majority of workloads.

Rules:
1. CLOSE flushes: when a client closes a file (Release op), all dirty data
   and metadata are flushed to the server before Release returns. The server
   commits to the underlying filesystem (fsync if O_SYNC, writeback otherwise).

2. OPEN validates: when a client opens a file (Open op), the client
   invalidates all cached attributes for that inode and fetches fresh
   metadata from the server. If the file was modified by another client
   since the last open, the new data is visible.

3. Between open and close: the client may cache read data and buffer
   writes. Concurrent access to the same file from multiple clients
   without locking has undefined ordering (same as NFS). Use flock()
   or fcntl() for coordination.

4. Lease-assisted invalidation: the server sends lease invalidation to
   clients holding metadata leases when an inode is modified. This
   provides best-effort visibility between open/close cycles —
   not guaranteed, but usually works within 1-2 lease durations.

fsync() semantics: Fsync operation is forwarded to the server, which calls fsync() on the underlying filesystem. Returns only after data is persistent on the server's storage.

14.11.10.5 Metadata Caching and Leases

The client caches stat and readdir results for the lease duration (default: 30 seconds, configurable per-export). Leases are per-inode, granted implicitly on Lookup and Getattr responses.

Lease invalidation: when the server modifies an inode (local write, unlink, rename, chmod, etc.), it sends invalidation messages to all clients holding leases for that inode. Clients receiving invalidation drop their cached attributes; the next access re-fetches from the server.

Unreachable client: if a client is unreachable during invalidation, the lease expires naturally after the lease duration. The server does NOT block on client acknowledgment — invalidation is best-effort, and close-to-open consistency provides the correctness backstop.

Read caching: file data may be cached on the client for the lease duration. On lease invalidation, cached data for the invalidated inode is also dropped. This provides reasonable read performance for read-heavy workloads without complex delegation machinery.

Write buffering: dirty writes are buffered on the client and flushed on close(), fsync(), or when the write buffer exceeds a threshold (default: 1 MB per file). The server may send an early flush request if another client opens the same file (to make close-to-open consistency work without waiting for the first client's close).

14.11.10.5.1 Lease Invalidation Wire Protocol

The server sends a ServiceMessage with opcode LEASE_INVALIDATE (0x0100) containing a batch of InodeId values to invalidate (up to 32 per message, batched to reduce round trips).

/// Lease invalidation message payload. Sent from server to client when
/// inodes held in the client's lease set are modified on the server.
#[repr(C)]
pub struct LeaseInvalidation {
    /// Number of inodes in this batch (1-32).
    count: u16,
    _pad: [u8; 6],
    /// Array of inode IDs to invalidate. Only the first `count` entries
    /// are valid; remaining entries are undefined.
    inodes: [InodeId; 32],
}
// Layout: 2 + 6(pad) + 32×8 = 264 bytes.
const_assert!(size_of::<LeaseInvalidation>() == 264);

Delivery: fire-and-forget (no ACK required). The server does NOT block on delivery — invalidation is best-effort. If the ring pair send fails (client unreachable), the message is discarded. Close-to-open consistency (Section 14.11) provides the correctness backstop: any client that opens a file will always see fresh data regardless of whether invalidation was delivered.

Server-side invalidation trigger: any local operation that modifies an inode (write, truncate, chmod, chown, rename, unlink, link, setxattr) scans the clients XArray for leases containing that inode_id and batches invalidation messages. The scan uses RCU-protected reads on the per-client leases XArray — no locking required on the read side. Batching collects up to 32 inodes per message before sending, with a 1 ms flush timer to bound latency when fewer than 32 inodes are pending.

14.11.10.6 Server Reboot Recovery

When the exporting host reboots, connected clients detect the reboot via the heartbeat protocol (generation change in PeerRegistry, Section 5.2).

Recovery protocol:

1. Server reboots, re-joins the cluster with an incremented generation number. Re-exports its filesystems.
2. Server enters a grace period (default: 45 seconds). During the grace period, only lock reclaim operations are accepted — no new opens or mutations. This prevents new clients from acquiring locks that conflict with locks held by recovering clients.
3. Clients detect the generation change, reconnect to the server, and reclaim their open file state:
4. Re-send Open for each file that was open before the reboot.
5. Re-acquire DLM locks (DLM recovery protocol handles this automatically — Section 15.15).
6. After the grace period, normal operations resume. Clients that did not reclaim within the grace period have their open files and locks invalidated.

Grace period operation filtering: during the grace period, the server classifies each incoming VfsServiceOp:

RECLAIM operations (allowed during grace):
  - Open with RECLAIM flag set (client re-opening a file it had open
    before reboot). Server validates against OpenFileRecord from the
    client's pre-reboot state (persisted in the recovery log or re-sent
    by the client in the reclaim Open message).
  - Lock with RECLAIM flag (DLM lock reclaim — handled by DLM subsystem,
    see [Section 15.15](15-storage.md#distributed-lock-manager)).

NON-RECLAIM operations (rejected during grace with -EAGAIN):
  - Any Open without RECLAIM flag.
  - Create, Mkdir, Unlink, Rmdir, Rename, Setattr, Setxattr.
  - Write (new writes, not reclaim of buffered data).
  - Readdir, Getattr, Lookup (metadata reads also deferred — server
    state may be inconsistent during recovery).

RECLAIM flag encoding:
  const OPEN_RECLAIM: u32 = 1 << 31;
  Set in the VfsServiceOp::Open.flags field. The server masks this bit
  before passing flags to the local filesystem open.

After the grace period expires (default 45 seconds, configurable via
per-export grace_period_s mount option), the RECLAIM flag is ignored
and all operations proceed normally. Clients that did not reclaim within
the window have their handles invalidated — subsequent operations on
stale handles return -ESTALE.

Client-side handling: processes with open files on the rebooting server experience a brief stall (grace period duration) followed by normal operation. No EIO unless the server remains unreachable beyond the heartbeat dead threshold.

14.11.10.7 Capability Gating and Discovery

Remote filesystem access requires CAP_FS_REMOTE (Section 9.1). The capability is checked per-connection (at mount time), not per-operation.

Discovery: hosts exporting filesystems advertise FILESYSTEM in their PeerRegistry capabilities (Section 5.2). Remote peers discover available exports by querying PeerRegistry::peers_with_cap(FILESYSTEM), then requesting an export list from the serving peer. No /etc/exports file, no showmount — discovery is automatic via the cluster membership protocol.

14.11.10.8 PeerFS Client Filesystem Implementation

PeerFS is the client-side kernel filesystem that mounts a remote VFS service export as a local filesystem. It translates VFS operations into VfsServiceOp messages, manages a local inode cache, integrates with the page cache for data caching, and handles server reconnection transparently.

Tier assignment: Tier 1 (hardware memory domain isolated). PeerFS runs in the VFS isolation domain alongside other filesystem drivers. Network I/O is delegated to the peer protocol transport in Core.

Phase: 2 (core cluster filesystem, required for multi-node operation).

14.11.10.8.1 PeerFs Struct

/// Client-side filesystem for mounting remote VFS service exports.
///
/// Registered with VFS as filesystem type `"peerfs"`. Each mount creates
/// one `PeerFs` instance attached to the superblock via `s_fs_info`.
///
/// **Superblock magic**: `PEERFS_SUPER_MAGIC = 0x50454552` (`"PEER"` in ASCII).
///
/// **Lifecycle**: Created during `fill_super`. Destroyed on unmount after
/// all open files are released and dirty data flushed to the server.
pub struct PeerFs {
    /// Peer ID of the serving host.
    pub server_peer_id: PeerId,

    /// Export path on the server (e.g., "/data").
    pub export_path: ArrayString<256>,

    /// Service connection established via ServiceBind
    /// ([Section 5.7](05-distributed.md#network-portable-capabilities--capability-service-providers)).
    /// Provides the peer queue pair and control channel to the server.
    pub conn: PeerFsConn,

    /// Local cache of remote inodes. Keyed by server-assigned `InodeId` (u64).
    /// XArray provides O(1) lookup with RCU-protected reads on the hot path.
    pub inode_cache: XArray<Arc<PeerFsInode>>,

    /// Number of cached inodes. Used for LRU eviction decisions.
    pub nr_cached_inodes: AtomicU64,

    /// Maximum cached inodes before LRU eviction begins (default: 65536).
    pub max_cached_inodes: u64,

    /// LRU list for inode cache eviction. Head = most recently used.
    /// Protected by a dedicated spinlock (not the inode cache XArray lock)
    /// to avoid contention between lookups (read-side RCU on XArray) and
    /// LRU reordering. Eviction walks from tail (least recently used).
    pub inode_lru: SpinLock<IntrusiveList<PeerFsInode>>,

    /// Lease duration in milliseconds. Cached attributes and data are valid
    /// for this duration after the server grants the lease. Default: 30000 (30s).
    pub lease_duration_ms: u32,

    /// Write buffer flush threshold per file in bytes. When buffered dirty
    /// data for a single file exceeds this, writeback is triggered without
    /// waiting for close. Default: 1 MiB (1_048_576).
    pub writeback_threshold: u32,

    /// Maximum concurrent in-flight operations to the server. Backpressure:
    /// new operations block when this limit is reached. Default: 256.
    pub max_inflight: u32,

    /// Current in-flight operation count.
    pub inflight: AtomicU32,

    /// Wait queue for tasks blocked on inflight limit.
    pub inflight_waitq: WaitQueue,

    /// Retry timeout for server-unreachable in milliseconds. Operations
    /// block for this duration before returning -EIO. Default: 60000 (60s).
    pub retry_timeout_ms: u32,

    /// Server generation (from PeerRegistry). Used to detect server reboots.
    pub server_generation: AtomicU64,

    /// True when the server is in grace period (lock reclaim only).
    pub in_grace_period: AtomicBool,

    /// Root squash: when true, uid 0 from this client is mapped to
    /// nobody (65534) by the server. Default: true.
    pub root_squash: bool,

    /// Read-only mount. When true, all mutating operations return -EROFS.
    pub read_only: bool,
}

/// Connection state to the remote VFS service provider.
pub struct PeerFsConn {
    /// Peer protocol endpoint for control messages (ServiceBind channel).
    pub endpoint: PeerEndpoint,

    /// Data region registered with the peer transport at ServiceBind time.
    /// Covers the client's bounce buffer pool for zero-copy bulk transfers.
    pub data_region: ServiceDataRegion,

    /// Bounce buffer pool for bulk data transfers. Pre-allocated at mount
    /// time. Size: `max_inflight * 128 KiB` (covers max concurrent I/O).
    /// Slab-backed, no hot-path allocation.
    pub bounce_pool: SlabPool<BounceBuffer>,

    /// Connection state.
    pub state: AtomicU8, // PeerFsConnState discriminant

    /// Sequence number for request/response matching.
    pub next_seq: AtomicU64,
}

/// Pre-allocated bounce buffer for bulk data transfers. Fixed-size,
/// slab-allocated from `PeerFsConn::bounce_pool`. One buffer per in-flight
/// I/O operation. Never heap-allocated on the hot path — the pool is
/// sized at mount time to `max_inflight` entries. Bounce buffer pool uses
/// vmalloc-backed allocation (not buddy allocator) to avoid high-order
/// page allocation failures. Each buffer is page-aligned within the
/// vmalloc region.
#[repr(C, align(4096))]
pub struct BounceBuffer {
    /// Data area. Size: 128 KiB (covers the maximum single read/write
    /// transfer size). Page-aligned for transport registration requirements.
    pub data: [u8; 131072],
    /// Transport-local key for this buffer (from the registered data region).
    pub local_key: u32,
    /// Explicit padding after local_key (u32, offset 131076) to align
    /// region_offset (u64, align 8). 131076 % 8 = 4, need 4 bytes.
    /// CLAUDE.md rule 11.
    pub _pad: [u8; 4],
    /// Offset of this buffer within the registered data region. Used to
    /// compute the remote-accessible address for bulk transfers.
    pub region_offset: u64,
}
// Layout: data(131072) + local_key(4) + _pad(4) + region_offset(8) = 131088 bytes.
// align(4096) pads to 135168. All padding explicit.
const_assert!(size_of::<BounceBuffer>() == 135168);

/// Connection state machine.
#[repr(u8)]
pub enum PeerFsConnState {
    /// ServiceBind in progress.
    Connecting = 0,
    /// Normal operation.
    Connected = 1,
    /// Server unreachable, retrying.
    Reconnecting = 2,
    /// Server rebooted, in grace period (reclaim only).
    GracePeriod = 3,
    /// Unmounting, draining in-flight operations.
    Draining = 4,
    /// Terminal: connection dead, all ops return -EIO.
    Dead = 5,
}

14.11.10.8.2 FileSystemOps Implementation

PeerFS implements the FileSystemOps trait (Section 14.1) to register as filesystem type "peerfs". It does not require a block device (FS_REQUIRES_DEV is not set).

mount / fill_super:

peerfs_mount(source: &str, flags: MountFlags, data: &[u8]) -> Result<SuperBlock>:
  1. Parse `source` as "<peer_id>:<export_path>".
     - peer_id: decimal u64 or hostname resolved via PeerRegistry.
     - export_path: absolute path on the server.
     If parse fails: return -EINVAL.

  2. Parse mount options from `data`:
     - lease_duration=N (seconds, default 30, range 1-3600)
     - writeback_threshold=N (bytes, default 1048576, range 4096-67108864)
     - max_inflight=N (default 256, range 16-4096)
     - retry_timeout=N (seconds, default 60, range 5-600)
     - ro (read-only)
     - norootsquash (disable root squash)

  3. Resolve server_peer_id via PeerRegistry
     ([Section 5.2](05-distributed.md#cluster-topology-model--peer-registry)).
     Check FILESYSTEM flag in server's PeerCapFlags. If absent: return -ENOENT.

  4. Check CAP_FS_REMOTE capability on calling task
     ([Section 9.1](09-security.md#capability-based-foundation)). If absent: return -EPERM.

  5. ServiceBind to the server's VFS service
     ([Section 5.7](05-distributed.md#network-portable-capabilities--capability-service-providers)).
     Payload includes export_path and client's lease_duration preference.
     Server may adjust lease_duration downward. On failure: return -ECONNREFUSED.

  6. Register data region with peer transport for bounce buffer pool.

  7. Allocate PeerFs struct, populate all fields.

  8. Send VfsServiceOp::Lookup { parent: ROOT_INODE, name: "." } to
     fetch root inode attributes from server.

  9. Allocate SuperBlock:
     - s_type = "peerfs"
     - s_blocksize = server-reported block size (from Statfs)
     - s_maxbytes = i64::MAX
     - s_flags = flags | MS_NOSUID | MS_NODEV
     - s_fs_info = PeerFs pointer
     - s_root = dentry for root inode
     - magic = PEERFS_SUPER_MAGIC (0x50454552)
     - s_bdev = None (no local block device)
     - s_bdi = None (no local backing device; writeback managed by peerfs)

  10. Register heartbeat callback for server_peer_id to detect reboots.
      Return SuperBlock.

statfs: Forwards VfsServiceOp::Statfs to the server. Returns the server's StatFs values directly (total/free/available blocks and inodes). Cached for lease_duration_ms to avoid round-trips on repeated df calls.

sync_fs: Flushes all dirty pages for all open files on this mount to the server, then sends VfsServiceOp::Fsync for each dirty inode. Blocks until all server acknowledgments arrive.

unmount: Flushes dirty data (sync_fs), releases all DLM locks, sends VfsServiceOp::Release for all open files, tears down the data region, and disconnects from the server.

show_options: Emits peer=<peer_id>,export=<path>,lease=<N> for /proc/mounts.

14.11.10.8.3 InodeOps and FileOps

All VFS inode and file operations are translated to VfsServiceOp messages and forwarded to the server. The mapping is direct:

VFS Operation	VfsServiceOp	Notes
lookup	Lookup	Populates local PeerFsInode cache on hit
getattr	Getattr	Served from cache if lease valid
setattr	Setattr	Invalidates cached attrs on success
create	Create	Returns new inode + open handle
mkdir	Mkdir
unlink	Unlink	Invalidates parent dir cache
rmdir	Rmdir	Invalidates parent dir cache
rename	Rename	Invalidates both parent dir caches
readdir	Readdir	Populates inode cache for returned entries
open	Open	Invalidates cached attrs (close-to-open)
read	Read (remote fetch)	Zero-copy from server's page cache
write	Buffered, Write (remote push)	Flushed on close/fsync/threshold
release	Release	Flush dirty pages first
fsync	Fsync	Flush dirty pages, then forward
symlink	Symlink	Creates symlink; returns new inode
readlink	Readlink	Returns symlink target path
link	Link	Creates hard link; invalidates parent dir cache
getxattr	Getxattr	Phase 2 — required for POSIX ACLs
setxattr	Setxattr	Phase 2 — required for POSIX ACLs
listxattr	Listxattr	Phase 2
removexattr	Removexattr	Phase 2

Read path (hot):

peerfs_read(file, buf, offset, len):
  inode = file.inode
  pi = inode.i_private as &PeerFsInode

  // Check page cache first (lease must be valid).
  if pi.lease_valid():
    pages = page_cache_lookup(inode, offset, len)
    if pages.is_complete():
      copy_to_user(buf, pages)
      return len

  // Cache miss or lease expired: fetch from server via peer transport.
  bounce = conn.bounce_pool.alloc()  // pre-allocated, no heap alloc
  send VfsServiceOp::Read { handle, offset, len } to server
  // Server responds with region offset for the data.
  // Client fetches data from remote region into bounce buffer.
  transport_fetch(bounce, server_region_offset, len)
  insert_into_page_cache(inode, offset, bounce.data, len)
  copy_to_user(buf, bounce.data, len)
  conn.bounce_pool.free(bounce)
  return len

Write path (warm — buffered, flushed asynchronously):

peerfs_write(file, buf, offset, len):
  if self.read_only: return -EROFS
  pi = file.inode.i_private as &PeerFsInode

  // Buffer in page cache. Mark pages dirty.
  copy_from_user_to_page_cache(file.inode, offset, buf, len)
  // Relaxed ordering: dirty_bytes is a best-effort counter for writeback
  // triggering, not a synchronization mechanism. Races between concurrent
  // writers may cause slight over- or under-counting, but writeback
  // correctness does not depend on exact counts — the page cache dirty
  // flags are the source of truth. Relaxed avoids unnecessary fence cost
  // on the write hot path (~2-5ns saved per write on weakly-ordered archs).
  pi.dirty_bytes.fetch_add(len, Relaxed)

  // Trigger writeback if threshold exceeded.
  if pi.dirty_bytes.load(Relaxed) >= self.writeback_threshold:
    peerfs_writeback(file.inode)

  return len

peerfs_writeback(inode):
  // Collect dirty pages, push to server via peer transport.
  for each dirty page range (offset, data, len):
    bounce = conn.bounce_pool.alloc()
    copy_pages_to_bounce(bounce, data, len)
    send VfsServiceOp::Write { handle, offset, data_region_offset: bounce.region_offset, data_len: len } to server
    // Server fetches data from client's bounce buffer via remote read.
    wait_for_server_ack()
    conn.bounce_pool.free(bounce)
    clear_page_dirty(inode, offset, len)
  pi.dirty_bytes.store(0, Relaxed)

Release (close) path: enforces close-to-open consistency.

peerfs_release(file):
  // Flush all dirty pages to server before releasing handle.
  peerfs_writeback(file.inode)
  send VfsServiceOp::Release { handle }
  wait_for_server_ack()
  // Do NOT invalidate cached attrs here — other opens may hold leases.

Open path: enforces close-to-open consistency (revalidation side).

peerfs_open(inode, flags):
  pi = inode.i_private as &PeerFsInode

  // Close-to-open: invalidate cached attrs and page cache on open.
  pi.invalidate_attrs()
  invalidate_inode_pages(inode)  // drop cached read data

  send VfsServiceOp::Open { inode: pi.remote_inode_id, flags, uid, gid }
  handle = wait_for_reply()
  // Store server-assigned handle in file->private_data.
  file.private_data = handle
  // Refresh cached attrs from Open reply.
  pi.update_attrs(reply.attrs, current_time_ms() + self.lease_duration_ms)

14.11.10.8.4 PeerFsInode: Remote Inode Cache

/// Local representation of a remote inode. Attached to the VFS `Inode`
/// via `i_private`. Caches attributes received from the server to avoid
/// round-trips on `stat()` / `getattr()` within the lease window.
pub struct PeerFsInode {
    /// Server-assigned inode ID. Matches `InodeId` on the server's filesystem.
    pub remote_inode_id: InodeId,

    /// Cached attributes (mode, uid, gid, size, mtime, ctime, nlink).
    pub attrs: SpinLock<PeerFsInodeAttrs>,

    /// Absolute time (monotonic ms) when the cached attrs expire.
    /// Operations after this time must re-fetch from the server.
    pub lease_expiry_ms: AtomicU64,

    /// Server generation at the time this inode was cached. If the server
    /// reboots (generation changes), all cached inodes are stale.
    pub server_generation: u64,

    /// Dirty bytes buffered in the page cache for this inode. Used to
    /// trigger writeback when exceeding `PeerFs::writeback_threshold`.
    pub dirty_bytes: AtomicU64,

    /// Server-assigned file handle for the current open. Zero if not open.
    /// Multiple opens share the same PeerFsInode but get distinct handles
    /// (handles are stored in `OpenFile::private_data`, not here).
    /// This field tracks the *last known* handle for recovery purposes.
    pub last_handle: AtomicU64,

    /// LRU linkage for inode cache eviction.
    pub lru_node: IntrusiveListNode,
}

/// Cached attributes for a remote inode.
pub struct PeerFsInodeAttrs {
    pub mode: u32,
    pub uid: u32,
    pub gid: u32,
    pub size: u64,
    pub nlink: u32,
    pub mtime_sec: u64,
    pub mtime_nsec: u32,
    pub ctime_sec: u64,
    pub ctime_nsec: u32,
    pub blocks: u64,
    pub blksize: u32,
}

impl PeerFsInode {
    /// Returns true if cached attributes are still within the lease window.
    pub fn lease_valid(&self) -> bool {
        current_monotonic_ms() < self.lease_expiry_ms.load(Acquire)
    }

    /// Invalidate cached attributes. Next getattr will fetch from server.
    pub fn invalidate_attrs(&self) {
        self.lease_expiry_ms.store(0, Release);
    }

    /// Update cached attributes from a server response.
    pub fn update_attrs(&self, attrs: PeerFsInodeAttrs, new_expiry_ms: u64) {
        *self.attrs.lock() = attrs;
        self.lease_expiry_ms.store(new_expiry_ms, Release);
    }
}

Cache eviction: triggered when nr_cached_inodes exceeds max_cached_inodes (default 65536). Eviction runs in a background kthread (peerfs_evictor), not on the hot lookup path.

peerfs_evict_inodes(pfs: &PeerFs):
  1. Acquire inode_lru spinlock.
  2. Walk the LRU list from tail (least recently used).
  3. For each candidate inode:
     a. Skip if VFS reference count > 0 (inode has active opens).
     b. Skip if dirty_bytes > 0 (must writeback first — schedule
        async writeback via peerfs_writeback() and re-check on next
        eviction pass).
     c. Remove from LRU list and inode_cache XArray.
     d. Drop the Arc<PeerFsInode> (deallocates if refcount reaches 0).
     e. Decrement nr_cached_inodes.
  4. Stop when nr_cached_inodes <= max_cached_inodes * 7/8 (hysteresis
     to avoid thrashing — evict 12.5% below threshold before stopping).
  5. Release inode_lru spinlock.

Memory budget: 65536 inodes x ~128 bytes per PeerFsInode = 8 MiB. Configurable via mount option max_inodes=N.

Lease-driven invalidation: The server sends lease invalidation messages when a remote inode is modified by another client. On receipt:

on_lease_invalidation(inode_id: InodeId):
  pi = inode_cache.load(inode_id)
  if pi.is_some():
    pi.invalidate_attrs()
    invalidate_inode_pages(pi.vfs_inode)  // drop cached read data

This provides best-effort visibility between open/close cycles. The close-to-open protocol is the correctness backstop — invalidation is an optimization that improves freshness for long-lived opens.

Revalidation: When a VFS operation accesses an inode with an expired lease, the client sends VfsServiceOp::Getattr and updates the cache. This is transparent to the caller.

14.11.10.8.5 Page Cache Integration

PeerFS uses the standard VFS page cache (Section 14.1) for read and write caching. The caching policy is deliberately simple — no delegations, no complex cache consistency state machines.

Design principles (lessons from NFS client bugs): - No "delegation" concept. Leases are the only cache validity mechanism. - Page cache validity is tied to the inode's lease. When the lease expires or is invalidated, all cached pages for that inode are dropped. - No speculative cache retention after lease loss. This wastes some bandwidth on re-reads but eliminates stale data bugs. - Dirty pages are ALWAYS flushed before releasing a file handle. No deferred or lazy flush that could lose data.

Read caching: - On read(), check page cache first. If the page is present AND the inode's lease is valid, serve from cache (zero network I/O). - If the page is absent or the lease is expired, fetch from the server via remote fetch (peer transport) and insert into the page cache. - Readahead: the VFS readahead infrastructure (Section 14.1) drives prefetch. PeerFS implements AddressSpaceOps::readahead() to batch multiple pages into a single remote fetch.

Write caching: - Writes go to the page cache. Pages are marked dirty. - Dirty pages are flushed to the server on: 1. close() (mandatory — close-to-open consistency). 2. fsync() (explicit sync request). 3. Per-file dirty bytes exceeding writeback_threshold. 4. Server sends early-flush request (another client opened the file). 5. Memory pressure (VFS shrinker callback).

mmap support: - mmap() is supported. Page faults trigger the read path (fetch from server if not cached). Dirty mmap pages are tracked via the page cache dirty mechanism and flushed on msync(), munmap(), or close().

No direct-I/O operation: PeerFS does not support O_DIRECT. All I/O goes through the page cache. This simplifies the implementation and avoids the NFS O_DIRECT coherency bugs. Applications requiring direct server access should use the DLM-based locking path.

Readdir caching: - Directory entries are cached as a serialized readdir buffer in the page cache, using the same page cache machinery as regular file data. Pages are keyed by (directory inode, byte offset into the readdir stream). - Cached readdir data is valid for lease_duration_ms. After expiry, the next readdir() re-fetches from the server. - Any directory mutation (Create, Unlink, Mkdir, Rmdir, Rename affecting the directory as source or destination) invalidates the directory's cached readdir pages immediately. - No negative dentry caching. Caching "file not found" results is error-prone with multiple clients modifying the same directory concurrently. A lookup miss always goes to the server.

14.11.10.8.6 Mount Syntax

mount -t peerfs <peer_id>:<export_path> <mountpoint> [-o <options>]

Option	Type	Default	Range	Description
lease_duration	seconds	30	1-3600	Metadata/data cache validity period
writeback_threshold	bytes	1048576	4096-64M	Per-file dirty data flush threshold
max_inflight	count	256	16-4096	Max concurrent server operations
retry_timeout	seconds	60	5-600	Timeout before -EIO on server unreachable
ro	flag	off		Read-only mount
norootsquash	flag	off		Disable root squash (uid 0 not mapped)
max_inodes	count	65536	1024-1048576	Maximum cached inodes before LRU eviction

Examples:

mount -t peerfs 42:/data /mnt/remote
mount -t peerfs 42:/home /mnt/home -o lease_duration=60,ro
mount -t peerfs 7:/scratch /mnt/scratch -o writeback_threshold=4194304,max_inflight=512

Peer ID can also be a hostname if DNS/mDNS resolution is available in the cluster. The PeerRegistry resolves hostnames to PeerId values.

14.11.10.8.7 Error Handling

Condition	Behavior
Server unreachable	Operations block for retry_timeout (default 60s), then return -EIO. Connection transitions to Reconnecting.
Server rebooted	Detected via PeerRegistry generation change. Connection transitions to GracePeriod. Client reclaims open files and DLM locks. Normal operations resume after grace period. See Section 14.11.
Stale file handle	Server returns -ESTALE. Client re-lookups the inode from parent directory: walk up to nearest cached valid parent, re-lookup each path component. If re-lookup succeeds, retry the operation with the new handle. If the file was deleted, return -ESTALE to the application.
Server returns error	Passed through to the application as-is (e.g., -EACCES, -ENOSPC, -ENOENT).
Transport failure	Connection transitions to Reconnecting. Re-register data region after reconnect. In-flight operations receive -EIO.
Client-side memory pressure	VFS shrinker evicts clean cached inodes (LRU). Dirty inodes are written back first.
Inflight limit reached	New operations block on inflight_waitq until an in-flight operation completes.

Stale inode recovery detail:

peerfs_handle_estale(inode, parent_path_components):
  // Walk up the cached path to find the nearest valid ancestor.
  for component in parent_path_components.reverse():
    ancestor = lookup_cached(component)
    if ancestor.is_valid():
      // Re-lookup from this ancestor downward.
      for child in path_from(ancestor, inode):
        result = send VfsServiceOp::Lookup { parent: ancestor.id, name: child }
        if result.is_err():
          return result  // file was deleted or renamed
        ancestor = result.inode
      // Update local inode cache with fresh server state.
      update_inode_cache(inode, ancestor)
      return Ok(())
  return Err(ESTALE)  // entire path is stale

14.11.10.8.8 Advantages Over NFS

Aspect	PeerFS	NFS v4.2
Wire overhead	Native structs over peer transport — no RPC/XDR marshaling	Sun RPC + XDR encoding/decoding on every operation
Daemons	Zero (pure in-kernel)	nfsd, mountd, idmapd, gssproxy, rpc.statd, rpcbind
Configuration	mount -t peerfs peer:/path /mnt	/etc/exports, /etc/fstab, Kerberos keytabs, idmapd.conf
Discovery	Automatic via PeerRegistry (Section 5.2)	Manual: must know server IP and export path
Locking	DLM (Section 15.15) — already running for DFS	NLM (v3) or built-in (v4) — separate protocol, separate recovery
Data transfer	Zero-copy via peer transport (bounce buffer pool, no kernel copy on large I/O; ~3-5μs on RDMA, ~50-200μs on TCP)	TCP or RPC-over-RDMA (still has XDR framing overhead)
Caching model	Leases only — simple, predictable, few bugs	Delegations — complex state machine, known source of client bugs
Recovery	DLM lock recovery + PeerRegistry heartbeat	Grace period + reclaim + edge cases around delegation return

14.11.10.8.9 Intentional Non-Goals

The following NFS features are not implemented in PeerFS. Each omission is deliberate, not a gap:

• pNFS (parallel NFS): UmkaOS clusters use the block service provider (Section 15.14) for parallel storage access. PeerFS serves the simpler "one server, one export" use case.
• Referrals: PeerRegistry discovery handles service location. No need for filesystem-level redirect.
• NFSv4 ACLs: PeerFS uses POSIX ACLs from the underlying filesystem. The server enforces them using the passed UID/GID.
• Client-side deduplication: The server's local filesystem handles this.
• Kerberos/GSSAPI: PeerFS assumes a uniform trust domain (all peers mutually authenticated via Section 9.1). Cross-domain authentication requires NFS.
• O_DIRECT: All I/O goes through the page cache for simplicity. Applications needing direct access use DLM locks and RDMA directly.

---

14.12 configfs — Kernel Object Configuration Filesystem

configfs is a RAM-resident pseudo-filesystem (similar to sysfs) that allows user-space to create, configure, and destroy kernel objects by manipulating directories and files under a single mount point. The key distinction from sysfs is direction of control: sysfs exports kernel-managed objects to user-space, while configfs gives user-space the power to instantiate new kernel objects via mkdir.

configfs is used by: - LIO iSCSI / NVMe-oF target (/sys/kernel/config/target/, /sys/kernel/config/nvmet/) — see Section 11 for the block-layer and NVMe-oF protocol details. - USB gadget framework (/sys/kernel/config/usb_gadget/) - 9pnet and netconsole subsystems

14.12.1 Architecture

                 User Space
         mkdir / rmdir / cat / echo
                     │
              /sys/kernel/config/
                     │  (VFS operations)
        ┌────────────┴────────────────────────┐
        │          configfs VFS layer          │
        │  ConfigfsSubsystem → ConfigGroup     │
        │  ConfigItem → ConfigAttribute        │
        └────────────┬────────────────────────┘
                     │  callbacks
              Kernel subsystem
         (LIO, nvmet, USB gadget, ...)

User-space operates exclusively with POSIX filesystem primitives. No ioctl or dedicated syscall is needed. The kernel subsystem registers callback functions that the configfs VFS layer invokes in response to standard filesystem operations.

14.12.2 Data Structures

/// A configfs subsystem, registered by a kernel module at init time.
pub struct ConfigfsSubsystem {
    /// Directory name created under /sys/kernel/config/.
    pub name: &'static str,
    /// Root group of this subsystem.
    pub root: Arc<ConfigGroup>,
}

/// A configfs group — a directory that may contain items, subgroups, and
/// attributes. Groups may also carry a set of default child groups that are
/// created automatically when the group itself is created.
pub struct ConfigGroup {
    pub item:           ConfigItem,
    /// Active children (items and subgroups) keyed by name.
    /// Bounded by the subsystem's make_item/make_group callbacks (return
    /// ENOSPC when subsystem-specific limits are exceeded). All configfs
    /// mutation operations require CAP_SYS_ADMIN.
    ///
    /// **Lock ordering**: Parent group's `children` lock MUST be acquired
    /// before any child group's `children` lock (top-down ordering). This
    /// is deadlock-free by the tree structure: no cycle can form because
    /// locks are always acquired in root-to-leaf order. Concurrent mkdir
    /// operations on sibling groups do not conflict (different RwLock
    /// instances). Lock level: LOCK_LEVEL_CONFIGFS_CHILDREN (within the
    /// VFS lock ordering hierarchy, after VFS inode lock, before
    /// attribute file I/O locks).
    pub children:       RwLock<BTreeMap<String, ConfigChild>>,
    /// Type descriptor controlling allowed operations on this group.
    pub item_type:      Arc<ConfigItemType>,
    /// Subgroups automatically created alongside this group (not user-removable).
    /// Bounded: typically 1-4 default groups per subsystem (e.g., target_core_mod
    /// creates "alua" and "statistics"). Cold path (group creation).
    /// Enforced: registration fails with ENOSPC if len >= CONFIGFS_MAX_DEFAULT_GROUPS.
    pub default_groups: Vec<Arc<ConfigGroup>>,
}

const CONFIGFS_MAX_DEFAULT_GROUPS: usize = 16;

/// Discriminated union of group children.
pub enum ConfigChild {
    Item(Arc<ConfigItem>),
    Group(Arc<ConfigGroup>),
}

/// A configfs item — the leaf directory representing one kernel object.
pub struct ConfigItem {
    /// Item name within its parent group. Set at creation time (mkdir);
    /// immutable thereafter — no lock required. Inline storage avoids
    /// heap allocation for typical names (container IDs, device names).
    pub name:      ArrayString<256>,
    /// Reference count; item is dropped when it reaches zero.
    /// AtomicU64 for consistent width across 32-bit and 64-bit platforms
    /// (per project policy — avoids usize width variation).
    pub kref:      AtomicU64,
    pub parent:    Weak<ConfigGroup>,
    pub item_type: Arc<ConfigItemType>,
}

/// Type descriptor: defines the callbacks and attributes for an item or group.
pub struct ConfigItemType {
    pub name: &'static str,
    /// Called when the item's reference count drops to zero.
    pub release:    fn(&ConfigItem),
    /// Attribute files exposed in every instance of this item type.
    pub attrs:      &'static [&'static dyn ConfigAttribute],
    /// Returns additional child groups (used for complex multi-level objects).
    pub groups:     Option<fn(&ConfigItem) -> Vec<Arc<ConfigGroup>>>,
    /// Create a new leaf item inside this group (triggered by mkdir).
    pub make_item:  Option<fn(group: &ConfigGroup, name: &str)
                              -> Result<Arc<ConfigItem>, KernelError>>,
    /// Create a new subgroup inside this group (triggered by mkdir).
    pub make_group: Option<fn(group: &ConfigGroup, name: &str)
                               -> Result<Arc<ConfigGroup>, KernelError>>,
    /// Notify the subsystem before an item is removed (triggered by rmdir).
    pub drop_item:  Option<fn(group: &ConfigGroup, item: &ConfigItem)>,
}

/// A single configfs attribute — a regular file in the item directory.
pub trait ConfigAttribute: Send + Sync {
    /// File name within the item directory.
    fn name(&self) -> &str;
    /// Unix permission bits (typically 0644 for read-write, 0444 for read-only).
    fn mode(&self) -> u32;
    /// Populate `buf` with a text representation of the attribute value.
    /// Returns the number of bytes written.
    fn show(&self, item: &ConfigItem, buf: &mut [u8]) -> Result<usize, KernelError>;
    /// Parse `buf` and apply the new attribute value.
    /// Returns the number of bytes consumed.
    fn store(&self, item: &ConfigItem, buf: &[u8]) -> Result<usize, KernelError>;
}

Lifetimes and reference counting mirror those of the objects the subsystem manages. A ConfigItem is kept alive as long as the directory exists in the configfs namespace. Removal (rmdir) calls drop_item, decrements the kref, and invokes release when the count reaches zero.

14.12.3 Mount Point and Directory Layout

configfs is mounted at boot by configfs_init() and exposed at /sys/kernel/config. User-space may also mount it manually:

mount -t configfs configfs /sys/kernel/config

Illustrative layout showing the NVMe-oF and iSCSI target subsystems (see Section 11 for full protocol details):

/sys/kernel/config/
├── target/                              ← LIO iSCSI / generic target subsystem
│   ├── core/
│   │   └── iblock_0/                   ← mkdir: create iblock backstore group
│   │       └── lio_disk0/              ← mkdir: create a new block device object
│   │           ├── dev                 ← echo /dev/sda > dev
│   │           ├── udev_path           ← echo /dev/sda > udev_path
│   │           └── enable              ← echo 1 > enable
│   └── iscsi/
│       └── iqn.2024-01.com.example:storage/   ← mkdir: create iSCSI target IQN
│           └── tpgt_1/                         ← mkdir: create target portal group
│               ├── enable
│               ├── lun/
│               │   └── lun_0 → ../../core/iblock_0/lio_disk0   ← symlink
│               ├── acls/
│               │   └── iqn.2024-01.com.client:host1/
│               │       ├── auth/
│               │       └── mapped_lun0/
│               └── fabric_statistics/
├── nvmet/                               ← NVMe-oF target subsystem
│   ├── subsystems/
│   │   └── nqn.2024-01.com.example:nvme-ssd/  ← mkdir: create NVMe subsystem NQN
│   │       ├── attr_allow_any_host
│   │       └── namespaces/
│   │           └── 1/                          ← mkdir: create namespace ID 1
│   │               ├── device_path             ← echo /dev/nvme0n1 > device_path
│   │               └── enable                  ← echo 1 > enable
│   └── ports/
│       └── 1/                                  ← mkdir: create NVMe-oF port
│           ├── addr_trtype                     ← echo tcp > addr_trtype
│           ├── addr_traddr                     ← echo 192.0.2.1 > addr_traddr
│           ├── addr_trsvcid                    ← echo 4420 > addr_trsvcid
│           └── subsystems/
│               └── nqn.2024-01.com.example:nvme-ssd  ← symlink
└── usb_gadget/                          ← USB gadget framework
    └── g1/                             ← mkdir: create a gadget instance
        ├── idVendor
        ├── idProduct
        └── functions/
            └── mass_storage.0/
                └── lun.0/
                    └── file            ← echo /dev/sdb > file

The directory hierarchy encodes object relationships. Symlinks express associations between independently-created objects (e.g., linking a LUN to its backing store, or attaching a subsystem to a port).

14.12.4 VFS Operations

configfs maps the five fundamental filesystem operations onto subsystem callbacks:

mkdir(path) The parent directory's ConfigItemType is consulted. If make_group is defined, a new ConfigGroup is allocated and returned as a subdirectory dentry. If make_item is defined, a new ConfigItem is allocated and returned. Only one of the two may be non-null for a given group type; attempting mkdir on a group that defines neither returns EPERM. Default child groups are created automatically alongside any new group.

rmdir(path) The directory must be empty (no user-created children; default children are exempt from this check and are removed automatically). drop_item is invoked on the parent's ConfigItemType, then the item's kref is decremented. If the kref reaches zero, release is called. Attempting to remove a non-empty directory returns ENOTEMPTY.

open(attr_path) / read(attr_fd) The fd is associated with the specific ConfigAttribute. read(2) invokes ConfigAttribute::show(), which populates the kernel buffer with a text representation. The output is always \n-terminated for shell compatibility.

open(attr_path) / write(attr_fd) write(2) invokes ConfigAttribute::store() with the user-supplied buffer. The subsystem parses and validates the value; on error it returns a negative errno. Writes larger than PAGE_SIZE (4 KiB) are rejected with EINVAL to prevent unbounded allocations.

symlink(src, dst) Used to express dependencies between items: for example, associating a LUN directory with a backstore object, or adding a subsystem to a port's subscriber list. configfs validates that both the source and destination are within the same configfs mount before creating the link. The subsystem's ConfigItemType may reject symlinks by returning EPERM from an optional allow_link callback.

readdir Returns all children of a group: items, subgroups, attribute files, and symlinks. Attribute names are synthesized from ConfigItemType.attrs; no inode backing store is needed.

14.12.5 Linux Compatibility

• /sys/kernel/config/ mount point and directory layout: byte-for-byte identical to Linux configfs (kernel 5.0+).
• The ConfigAttribute read/write text format (newline-terminated strings, echo value > file idiom) matches Linux.
• LIO iSCSI target tools (targetcli, targetcli-fb, rtslib-fb) work without modification.
• NVMe-oF target tools (nvmetcli) work without modification; see Section 11 for NVMe-oF transport configuration details.
• USB gadget framework (configfs-gadget, libusbgx) works without modification.
• Symlink semantics (cross-item dependencies) are identical to Linux: both source and destination must reside within the same configfs mount.

---

14.13 File Notification System

UmkaOS implements inotify and fanotify with full Linux syscall and wire-format compatibility. Internal delivery uses typed structured channels rather than raw fd-write protocols; the external syscall interfaces are byte-for-byte identical to Linux.

Two interfaces are provided:

• inotify: informational events (IN_CREATE, IN_MODIFY, etc.), delivered asynchronously via a file descriptor readable with read(2).
• fanotify: superset of inotify, plus permission events (FAN_OPEN_PERM, FAN_ACCESS_PERM, FAN_OPEN_EXEC_PERM) that block the originating syscall until userspace responds with allow or deny. Used by malware scanners, file integrity monitors, and backup software.

Both are implemented in umka-vfs. Event delivery hooks are called from within the VFS operation dispatch paths — after permission checks pass, before returning to userspace.

14.13.1 inotify

14.13.1.1 In-Kernel Objects

/// inotify watch descriptor — the `wd` value returned to userspace by
/// `inotify_add_watch(2)` and echoed in `struct inotify_event::wd`. A 1-based
/// positive index allocated per `InotifyInstance` (see `next_wd` below). The
/// external ABI value is a signed 32-bit `wd`; internally it is carried as a
/// `u32` XArray key. Kernel-internal newtype, not itself an ABI struct.
#[derive(Clone, Copy, PartialEq, Eq, Hash)]
pub struct WatchDescriptor(pub u32);

bitflags! {
    /// inotify event mask — mirrors the Linux `IN_*` ABI bit-for-bit
    /// (`include/uapi/linux/inotify.h`, torvalds/linux master). Passed to
    /// `inotify_add_watch(2)` and reported in `struct inotify_event::mask`.
    /// Values are the external userspace ABI and MUST NOT change.
    pub struct InotifyMask: u32 {
        // --- events userspace can watch for ---
        /// `IN_ACCESS` — file was accessed (read).
        const ACCESS        = 0x0000_0001;
        /// `IN_MODIFY` — file was modified (write).
        const MODIFY        = 0x0000_0002;
        /// `IN_ATTRIB` — metadata changed (chmod, chown, timestamps, link count).
        const ATTRIB        = 0x0000_0004;
        /// `IN_CLOSE_WRITE` — writable file was closed.
        const CLOSE_WRITE   = 0x0000_0008;
        /// `IN_CLOSE_NOWRITE` — unwritable file was closed.
        const CLOSE_NOWRITE = 0x0000_0010;
        /// `IN_OPEN` — file was opened.
        const OPEN          = 0x0000_0020;
        /// `IN_MOVED_FROM` — file moved out of watched directory.
        const MOVED_FROM    = 0x0000_0040;
        /// `IN_MOVED_TO` — file moved into watched directory.
        const MOVED_TO      = 0x0000_0080;
        /// `IN_CREATE` — file/subdirectory created in watched directory.
        const CREATE        = 0x0000_0100;
        /// `IN_DELETE` — file/subdirectory deleted from watched directory.
        const DELETE        = 0x0000_0200;
        /// `IN_DELETE_SELF` — watched file/directory itself was deleted.
        const DELETE_SELF   = 0x0000_0400;
        /// `IN_MOVE_SELF` — watched file/directory itself was moved.
        const MOVE_SELF     = 0x0000_0800;
        // --- events sent unconditionally to any watch ---
        /// `IN_UNMOUNT` — backing filesystem was unmounted.
        const UNMOUNT       = 0x0000_2000;
        /// `IN_Q_OVERFLOW` — event queue overflowed (wd == -1).
        const Q_OVERFLOW    = 0x0000_4000;
        /// `IN_IGNORED` — watch was removed (explicitly or by deletion/unmount).
        const IGNORED       = 0x0000_8000;
        // --- control / special flags ---
        /// `IN_ONLYDIR` — fail `add_watch` unless the target is a directory.
        const ONLYDIR       = 0x0100_0000;
        /// `IN_DONT_FOLLOW` — do not dereference a symlink target.
        const DONT_FOLLOW   = 0x0200_0000;
        /// `IN_EXCL_UNLINK` — stop reporting events for unlinked children.
        const EXCL_UNLINK   = 0x0400_0000;
        /// `IN_MASK_CREATE` — fail if a watch already exists (no mask merge).
        const MASK_CREATE   = 0x1000_0000;
        /// `IN_MASK_ADD` — OR the mask into an existing watch instead of replacing.
        const MASK_ADD      = 0x2000_0000;
        /// `IN_ISDIR` — reported event refers to a directory.
        const ISDIR         = 0x4000_0000;
        /// `IN_ONESHOT` — deliver a single event then auto-remove the watch.
        const ONESHOT       = 0x8000_0000;
    }
}

/// `IN_CLOSE` helper: any close event (`CLOSE_WRITE | CLOSE_NOWRITE`).
pub const IN_CLOSE: InotifyMask = InotifyMask::CLOSE_WRITE.union(InotifyMask::CLOSE_NOWRITE);
/// `IN_MOVE` helper: any move event (`MOVED_FROM | MOVED_TO`).
pub const IN_MOVE: InotifyMask = InotifyMask::MOVED_FROM.union(InotifyMask::MOVED_TO);

/// Per-inotify-instance state. Created by inotify_init() / inotify_init1().
/// Exposed to userspace as a file descriptor (the fd is backed by a synthetic
/// inode in the anonymous inode filesystem; read(2) on it drains the event queue).
pub struct InotifyInstance {
    /// Watch descriptors: maps wd → InotifyWatch.
    /// WatchDescriptor is the Linux `wd` (i32 cast to u32 index). XArray
    /// provides O(1) lookup with native RCU-protected reads (no read-side
    /// locking) and internal xa_lock for write serialization, replacing the
    /// external RwLock. Watch addition/removal is infrequent (warm path).
    pub watches: XArray<Arc<InotifyWatch>>,

    /// Monotonically increasing allocator for watch descriptors.
    /// WDs are 1-based positive integers per inotify_add_watch(2) contract.
    ///
    /// **Longevity**: i32 allows ~2.1 billion watch additions per inotify fd.
    /// At 1000 add/remove cycles per second, wraps in ~24.8 days. In practice,
    /// applications rarely exceed thousands of watches. If a long-lived daemon
    /// exhausts the space, the application must close and re-open the inotify fd
    /// (matches Linux behavior — Linux also does not recycle WDs).
    /// WD recycling (reusing released WDs via a free list) would extend
    /// the practical lifetime but is deferred: Linux does not recycle WDs,
    /// and changing this would be a behavioral difference.
    pub next_wd: AtomicI32,

    /// Per-instance event queue. Pre-allocated at `inotify_init()` time with
    /// capacity `max_queued_events` (sysctl, default 16384). Fixed capacity avoids
    /// heap allocation under spinlock. Overflow policy: **newest events are dropped**
    /// when the queue is full (the newest event that would be enqueued is discarded,
    /// not an existing queued event). This matches Linux inotify behavior: the
    /// `overflow` flag is set and a synthetic `IN_Q_OVERFLOW` event is prepended
    /// to the next `read(2)` response.
    pub event_queue: SpinLock<BoundedRing<InotifyEventBuf>>,

    /// Set when the event queue overflowed since the last read(2). A synthetic
    /// `IN_Q_OVERFLOW` event is prepended to the next read response and this flag
    /// is cleared. Separate from the queue to avoid occupying a queue slot.
    pub overflow: AtomicBool,

    /// Wait queue for poll()/select()/epoll() on this instance.
    pub wait_queue: WaitQueueHead,

    /// Flags from inotify_init1() (IN_CLOEXEC, IN_NONBLOCK).
    pub flags: u32,
}

/// One inotify watch: a single inode being monitored for specific events.
pub struct InotifyWatch {
    /// Watch descriptor (the value returned to userspace by inotify_add_watch).
    pub wd: WatchDescriptor,

    /// The inode being watched. Holds an Arc reference to prevent premature eviction
    /// while the watch is active.
    pub inode: Arc<Inode>,

    /// Bitmask of watched events (IN_CREATE | IN_MODIFY | IN_CLOSE_WRITE | etc.).
    pub mask: u32,

    /// Back-reference to the owning InotifyInstance (weak to avoid cycles).
    pub instance: Weak<InotifyInstance>,
}

/// Event delivered to userspace via read(2) on the inotify fd.
/// Matches the Linux inotify_event ABI exactly.
#[repr(C)]
pub struct InotifyEvent {
    /// Watch descriptor that fired.
    pub wd: i32,
    /// Event type (IN_CREATE, IN_MODIFY, IN_DELETE, etc.).
    pub mask: u32,
    /// Links related IN_MOVED_FROM and IN_MOVED_TO events (same cookie = same rename).
    pub cookie: u32,
    /// Length of the name[] field in bytes, including the null terminator and any
    /// trailing padding bytes. 0 if no filename is associated with this event
    /// (e.g., IN_ATTRIB on a non-directory inode).
    pub len: u32,
    // Followed immediately by name[len]: null-terminated filename, valid only for
    // events on directory inodes. Padded to a 4-byte boundary.
}
const_assert!(size_of::<InotifyEvent>() == 16);

/// Internal buffer holding a complete inotify event + filename bytes.
/// Uses a fixed-size array instead of `Vec<u8>` to avoid heap allocation
/// inside the spinlock-protected event queue. NAME_MAX is 255; with a null
/// terminator the maximum name is 256 bytes. Padded to 4-byte alignment,
/// the worst case is 256 bytes (255 + 1 null, already 4-byte aligned).
pub struct InotifyEventBuf {
    pub header: InotifyEvent,
    /// The filename, null-terminated and padded to a 4-byte boundary.
    /// Only the first `header.len` bytes are valid; the rest is unused.
    /// Fixed capacity avoids heap allocation under the event_queue SpinLock.
    ///
    /// **Memory budget**: 272 * max_queued_events bytes per instance
    /// (default ~4.25 MiB). With max_user_instances=128, worst-case per-user
    /// kernel memory: ~544 MiB. This is bounded by per-user rlimits and
    /// is comparable to Linux's inotify memory consumption for the same
    /// event depth. Phase 3 optimization: variable-length event entries
    /// reduce typical memory by 10-20x.
    pub name: [u8; 256],
}

14.13.1.2 VFS Integration Hooks

inotify events are generated from dentry/inode operation call sites within the VFS dispatch layer. The fast path check costs a single pointer load:

VFS operation	Event(s) generated
create, mkdir, mknod, symlink	IN_CREATE on parent dir inode
unlink, rmdir	IN_DELETE on parent dir; IN_DELETE_SELF on the target inode
rename (source side)	IN_MOVED_FROM on old parent + cookie
rename (destination side)	IN_MOVED_TO on new parent + same cookie
open	IN_OPEN on the inode
read, readdir	IN_ACCESS on the inode
write, truncate, fallocate	IN_MODIFY on the inode
setattr (chmod/chown/utimes)	IN_ATTRIB on the inode
close (file was written)	IN_CLOSE_WRITE on the inode
close (read-only open)	IN_CLOSE_NOWRITE on the inode
inotify watch removed (inode evicted or inotify_rm_watch)	IN_IGNORED on the watch descriptor

Each Inode carries an inotify_watches field:

/// Maximum number of inotify watches that can be attached to a single inode.
/// Bounded to avoid unbounded heap allocation on the per-inode watch list,
/// which is scanned under a SpinLock on every VFS event delivery. 128 is
/// generous — even heavily-monitored inodes rarely exceed a handful of
/// watchers (one per inotify instance). The system-wide per-user limit
/// (`max_user_watches` sysctl, default 8192) bounds the total; this
/// per-inode cap prevents pathological concentration on a single inode.
const MAX_WATCHES_PER_INODE: usize = 128;

/// Per-inode inotify watch list. Null when no watches are active (the common case).
/// This field is checked on every relevant VFS operation; a null pointer load
/// has zero overhead (no branch misprediction for the vast majority of inodes).
///
/// Uses `OnceLock` for the `None` → `Some` transition: the first `inotify_add_watch`
/// calls `inotify_watches.get_or_init(|| SpinLock::new(ArrayVec::new()))`.
/// `OnceLock` provides internal synchronization for the initialization race —
/// if two threads add the first watch concurrently, only one performs the init.
/// Subsequent accesses are a simple pointer load (no locking overhead).
/// Reverting to "no watches" does NOT clear the OnceLock (the empty SpinLock
/// persists, consuming only the lock + ArrayVec header — ~24 bytes); this avoids
/// an ABA race on the pointer.
pub inotify_watches: OnceLock<SpinLock<ArrayVec<Arc<InotifyWatch>, MAX_WATCHES_PER_INODE>>>,

When the field is None (no watches active), the check is a single null pointer comparison — zero overhead on the fast path for the vast majority of inodes.

14.13.1.3 Event Delivery Algorithm

fsnotify_inode_event(inode, event_mask, name, cookie):
  watches_opt = inode.inotify_watches.get()  // single load, no locking
  if watches_opt is None: return             // fast path: no watches on this inode

  watches = watches_opt.lock()
  for watch in watches.iter():
    fired_mask = watch.mask & event_mask
    if fired_mask == 0: continue
    if let Some(instance) = watch.instance.upgrade():
      buf = InotifyEventBuf {
        header: InotifyEvent { wd: watch.wd, mask: fired_mask, cookie, len: name.len() + padding },
        name: name_bytes_padded_to_4_bytes,
      }
      queue = instance.event_queue.lock()
      if !queue.is_full():
        queue.push(buf)
      else:
        // Queue overflow: set the overflow flag so that the next read(2) prepends
        // a synthetic IN_Q_OVERFLOW event. The AtomicBool lives outside the spinlock;
        // store is done while still holding the lock to ensure the writer side sees
        // the flag before any reader drains the queue.
        instance.overflow.store(true, Ordering::Release)
      drop(queue)
      instance.wait_queue.wake_up_one()  // unblock read()/poll()

14.13.1.4 Syscall Implementations

inotify_add_watch(fd, path, mask) → wd: 1. Resolve path → inode using normal path resolution. 2. Look up fd → InotifyInstance. 3. Scan instance.watches for an existing watch on this inode: - If found: update watch.mask = mask (OR behavior if IN_MASK_ADD flag is set; replace otherwise). Return the existing wd. 4. Enforce max_user_watches: count total watches across all inotify instances for the calling user's real UID. If total >= sysctl.max_user_watches, return ENOSPC (matches Linux errno for this limit). The per-user watch count is tracked via an AtomicU32 in the per-user credential structure for O(1) checking. 5. Allocate a new WatchDescriptor from instance.next_wd.fetch_add(1). If the result is negative (wrapped past i32::MAX), return ENOSPC — the WD space is exhausted. The application must close and re-open the inotify fd. (Linux also wraps without checking; UmkaOS adds the guard for 50-year uptime correctness at negligible cost.) 6. Construct InotifyWatch { wd, inode: inode.clone(), mask, instance: Arc::downgrade(&instance) }. 7. Initialize inode.inotify_watches if it was None. 8. Insert the watch into both inode.inotify_watches and instance.watches. 9. Increment the per-user watch count. 10. Return wd.

inotify_rm_watch(fd, wd) → 0: 1. Look up fd → InotifyInstance. 2. Remove the watch from instance.watches by wd. Return EINVAL if not found. 3. Remove the corresponding entry from inode.inotify_watches. 4. If inode.inotify_watches is now empty, the OnceLock persists with an empty ArrayVec (not cleared — see OnceLock design note above). The ~24-byte overhead avoids ABA races on the pointer. 5. Deliver an IN_IGNORED event to the instance. 6. Drop the Arc<InotifyWatch>.

14.13.1.5 Mandatory Event Coalescing

Coalescing rule (mandatory): Before enqueuing a new event, the delivery path checks whether the tail of the instance's EventQueue is an identical event. If so, the new event is discarded (coalesced) rather than enqueued. Two events are identical if and only if:

fn events_are_identical(a: &InotifyEvent, b: &InotifyEvent) -> bool {
    a.wd     == b.wd     &&
    a.mask   == b.mask   &&
    a.cookie == b.cookie &&
    a.name   == b.name    // byte-for-byte name comparison
}

The check is against the tail only (O(1)), not the entire queue. Events are coalesced only when consecutive and identical — non-consecutive duplicates are not coalesced (ordering is preserved for different events between duplicates).

IN_MOVED_FROM / IN_MOVED_TO cookie pairing: Cookie values are assigned by a per-VFS-instance AtomicU32 cookie_counter. Consecutive rename operations get consecutive cookie values. Coalescing does NOT apply to cookie-bearing events (mask has IN_MOVED_FROM or IN_MOVED_TO set) — rename pairs must always be delivered in full.

IN_Q_OVERFLOW: When the fixed-capacity BoundedRing is full and a new event cannot be enqueued (even after attempting coalescing), the InotifyInstance.overflow AtomicBool is set to true. On the next read(2), the read path checks this flag first: if set, it clears the flag and prepends a synthetic IN_Q_OVERFLOW event (wd=-1, mask=IN_Q_OVERFLOW, cookie=0, name="") before draining normal events. This keeps the overflow sentinel out of the ring buffer itself, preserving all max_queued_events slots for real events. The queue is never silently dropped without this sentinel.

Performance: Under cargo build workloads (10k+ file writes), inotify watchers on the build directory receive IN_MODIFY storms. Coalescing reduces queue pressure by 10-100x for write-heavy workloads where the application re-reads the file on any change (editor reload, build system).

Linux compatibility: Linux inotify performs the same tail-coalescing. UmkaOS mandates it (Linux specifies it informally). The IN_Q_OVERFLOW sentinel behaviour is identical to Linux.

14.13.1.6 inotify Sysctls (/proc/sys/fs/inotify/)

Sysctl	Default	Enforced at	Description
max_user_instances	128	inotify_init() / inotify_init1()	Maximum inotify file descriptors per real UID. Returns EMFILE when exceeded.
max_user_watches	8192	inotify_add_watch()	Maximum watches across all inotify instances per real UID. Returns ENOSPC when exceeded.
max_queued_events	16384	Event enqueue (§14.9.1.3)	Maximum pending events per inotify instance before IN_Q_OVERFLOW. Set at inotify_init() time.

Note on max_user_watches default: Linux kernels 5.11+ dynamically increase this limit based on available memory (up to 1048576). UmkaOS uses the static default of 8192 (matching the historical Linux default) but allows runtime tuning via the sysctl. The event queue capacity per instance is max_queued_events (not the compile-time generic parameter — the BoundedRing is allocated with capacity max_queued_events at inotify_init() time).

Enforcement: - inotify_init() / inotify_init1(): check per-user instance count against max_user_instances. If exceeded, return EMFILE. - inotify_add_watch(): check per-user watch count against max_user_watches (step 4 in §14.9.1.4). If exceeded, return ENOSPC. - Event enqueue: when the per-instance event queue reaches max_queued_events, new events are dropped and the overflow flag is set (§14.9.1.5).

14.13.1.7 read(2) Serialization Protocol

Events are packed contiguously in the user buffer provided to read(2):

1. Each event consists of struct inotify_event (16 bytes: wd i32 + mask u32
2. cookie u32 + len u32) followed by len bytes of filename data.
3. The filename is null-terminated and padded with additional null bytes to align the total event size (sizeof(inotify_event) + len) to the next 4-byte boundary. The len field includes all null bytes (terminator + padding).
4. For events without a filename (e.g., IN_ATTRIB on a non-directory), len is 0 and no name bytes follow the header.
5. If the user buffer is smaller than sizeof(inotify_event) (16 bytes), read(2) returns EINVAL (matching Linux ≥ 2.6.21 behavior).
6. Partial events are never returned: if the next event in the queue does not fit in the remaining buffer space, read(2) stops and returns the number of bytes written so far. If no events have been written yet (first event does not fit), return EINVAL.
7. If the overflow flag is set, a synthetic IN_Q_OVERFLOW event (wd=-1, mask= IN_Q_OVERFLOW, cookie=0, len=0, total 16 bytes) is prepended before draining normal events. The flag is cleared after prepending.

14.13.2 fanotify

fanotify extends inotify with:

1. Filesystem-wide and mount-wide marks (not just per-inode): a single mark can cover an entire mount point or filesystem, eliminating the need to add per-inode watches for directories being monitored for new file creation.
2. Permission events (FAN_OPEN_PERM, FAN_ACCESS_PERM, FAN_OPEN_EXEC_PERM): the originating syscall blocks until the fanotify daemon responds with allow or deny, subject to a mandatory per-group timeout (default 5000ms) to prevent system-wide I/O stalls.

14.13.2.1 Data Structures

/// Per-fanotify-instance state. Created by fanotify_init().
pub struct FanotifyInstance {
    /// Mark tables: one XArray per mark type, keyed by the object's u64 ID.
    /// Three separate XArrays (matching the QuotaCache precedent in
    /// disk-quota-subsystem.md) instead of a single `BTreeMap<FanotifyMarkKey>`:
    /// (1) XArray provides O(1) lookup with RCU-compatible reads,
    /// (2) avoids BTreeMap with enum-wrapped integer keys (collection policy),
    /// (3) allows independent locking per mark type.
    /// Mark management (fanotify_mark() syscall) is warm-path. Event delivery
    /// traverses per-inode/mount/sb mark lists (attached when marks are added),
    /// not these central tables.
    pub inode_marks: XArray<Arc<FanotifyMark>>,
    pub mount_marks: XArray<Arc<FanotifyMark>>,
    pub sb_marks: XArray<Arc<FanotifyMark>>,

    /// Informational event queue (non-permission events).
    /// Uses a pre-allocated fixed-capacity ring buffer (`BoundedRing`) to avoid
    /// heap allocation under spinlock. Capacity is set at `fanotify_init()` time
    /// (default 16384, matching Linux `FANOTIFY_DEFAULT_MAX_EVENTS`).
    /// Events beyond capacity are dropped and a `FAN_Q_OVERFLOW` synthetic event
    /// is generated (matching Linux behavior).
    pub event_queue: SpinLock<BoundedRing<FanotifyEvent>>,

    /// Set when the event queue overflowed since the last read(2). A synthetic
    /// `FAN_Q_OVERFLOW` event is prepended to the next read response and this
    /// flag is cleared. Kept outside the ring to avoid occupying an event slot.
    pub overflow: AtomicBool,

    /// Pending permission requests: keyed by a unique request ID (u64)
    /// assigned at creation. Entries are removed when the daemon writes a
    /// response. XArray provides O(1) lookup with internal xa_lock for write
    /// serialization, replacing the external SpinLock.
    pub perm_queue: XArray<Arc<FanotifyPermRequest>>,

    /// Next permission request ID (monotonically increasing).
    pub next_perm_id: AtomicU64,

    /// Wait queue for poll()/select()/epoll() on this instance.
    pub wait_queue: WaitQueueHead,

    /// Notification class: determines permission event delivery order when multiple
    /// fanotify instances watch the same inode.
    /// FAN_CLASS_NOTIF=0x00000000: informational only.
    /// FAN_CLASS_CONTENT=0x00000004: content scanners (see file after open).
    /// FAN_CLASS_PRE_CONTENT=0x00000008: DLP / integrity monitors (see file before open).
    /// Higher class is notified first. Within the same class, order is unspecified.
    pub class: FanotifyClass,

    /// Flags from fanotify_init() (FAN_CLOEXEC, FAN_NONBLOCK, FAN_REPORT_FID, etc.).
    pub flags: u32,

    /// Maximum time to wait for a permission event response.
    /// Default: 5000ms. Configurable per group at fanotify_init() time via
    /// FANOTIFY_INIT_PERM_TIMEOUT_MS (UmkaOS extension, not in Linux).
    /// A value of 0 means: use the system default from
    /// /proc/sys/fs/fanotify/perm_timeout_ms.
    pub perm_timeout: Duration,

    /// Action taken when a permission request times out:
    /// - PermTimeoutAction::Deny: return EPERM to the originating syscall (safe default)
    /// - PermTimeoutAction::Allow: allow the operation (permissive mode for monitoring-only daemons)
    pub perm_timeout_action: PermTimeoutAction,
}

pub enum PermTimeoutAction {
    Deny,   // Return EPERM to originating syscall on timeout (default)
    Allow,  // Allow the operation on timeout (for monitoring daemons that tolerate loss)
}

/// Mark type discriminant for fanotify_mark() dispatch. Determines which
/// XArray (`inode_marks`, `mount_marks`, or `sb_marks`) to use for the
/// mark operation. The u64 ID is extracted from the discriminant and used
/// as the XArray key directly.
#[derive(Clone, Copy, PartialEq, Eq)]
pub enum FanotifyMarkKey {
    /// Inode mark: watches a specific file or directory.
    /// `inode_id` is the filesystem-wide inode number.
    Inode { inode_id: u64 },
    /// Mount mark: watches all files under a mount point.
    /// `mount_id` is the unique mount ID from the mount tree.
    Mount { mount_id: u64 },
    /// Filesystem mark: watches all files on a filesystem (superblock scope).
    /// `sb_id` is a unique identifier for the superblock (device number).
    Filesystem { sb_id: u64 },
}

/// A single fanotify mark: attaches event interest to an inode, mount, or superblock.
pub struct FanotifyMark {
    pub mark_type: FanotifyMarkType,  // FAN_MARK_INODE, FAN_MARK_MOUNT, FAN_MARK_FILESYSTEM
    /// Object identifier: inode_id (for inode marks), mount_id (for mount marks),
    /// or superblock pointer (for filesystem marks).
    pub object_id: u64,
    /// Event mask this mark is listening for.
    pub mask: u64,
    /// Ignore mask: events matching this mask are suppressed even if mask is set.
    pub ignored_mask: u64,
    pub instance: Weak<FanotifyInstance>,
}

/// A pending permission request: holds the event plus the response channel.
pub struct FanotifyPermRequest {
    /// The event as delivered to userspace via read(2) on the fanotify fd.
    pub event: FanotifyEvent,
    /// Unique request ID (matches the fd-based identification in the response).
    pub request_id: u64,
    /// Set to FAN_ALLOW or FAN_DENY by the daemon's write(2) response.
    /// Uses OnceLock<u32> (first-writer-wins via `.set()`): the first
    /// writer (daemon's write() or timeout handler) wins; the loser's
    /// `.set()` returns Err. This enforces the semantic at the type level.
    pub response: OnceLock<u32>,
    /// Wakes the blocked originating syscall when response becomes Some.
    pub waker: WaitQueueHead,
}

/// Event delivered to userspace via read(2) on the fanotify fd.
/// Matches Linux's fanotify_event_metadata ABI.
#[repr(C)]
pub struct FanotifyEvent {
    pub event_len: u32,    // Total length of this event record (including variable info records)
    pub vers: u8,          // FANOTIFY_METADATA_VERSION (always 3)
    pub reserved: u8,
    pub metadata_len: u16, // sizeof(FanotifyEvent)
    pub mask: u64,         // Event type bitmask
    pub fd: i32,           // Opened fd for the file (or -1 with FAN_REPORT_FID)
    pub pid: i32,          // PID of the process that triggered the event
}
// mask field matches Linux `__aligned_u64` — naturally 8-byte aligned at offset 8.
const_assert!(size_of::<FanotifyEvent>() == 24);

/// Variable-length event information record header.
/// Appended after `FanotifyEvent` when `FAN_REPORT_FID` is set in the
/// fanotify group flags. Multiple info records may follow a single event
/// (e.g., `FAN_REPORT_FID | FAN_REPORT_DFID_NAME` produces two records).
/// Linux ABI: `struct fanotify_event_info_header` (linux/fanotify.h).
#[repr(C)]
pub struct FanotifyEventInfoHeader {
    /// Info record type. Determines the layout of the data following
    /// this header. Values:
    /// - `FAN_EVENT_INFO_TYPE_FID` (1): file handle info.
    /// - `FAN_EVENT_INFO_TYPE_DFID_NAME` (2): directory + name.
    /// - `FAN_EVENT_INFO_TYPE_DFID` (3): directory file handle only.
    /// - `FAN_EVENT_INFO_TYPE_PIDFD` (4): pidfd info (Linux 5.15+).
    /// - `FAN_EVENT_INFO_TYPE_ERROR` (5): filesystem error info (Linux 5.16+).
    pub info_type: u8,
    /// Padding for alignment.
    pub pad: u8,
    /// Total length of this info record (header + payload), in bytes.
    /// Must be a multiple of 4 (aligned to u32 boundary).
    pub len: u16,
}

/// File identifier info record. Follows `FanotifyEventInfoHeader` when
/// `info_type == FAN_EVENT_INFO_TYPE_FID` (or DFID/DFID_NAME).
/// Linux ABI: `struct fanotify_event_info_fid` (linux/fanotify.h).
///
/// The file handle bytes follow immediately after this struct. The total
/// record size is `sizeof(FanotifyEventInfoHeader) + sizeof(FanotifyEventInfoFid)
/// + handle_bytes`, padded to a 4-byte boundary.
#[repr(C)]
pub struct FanotifyEventInfoFid {
    /// Header identifying this record type and total length.
    pub hdr: FanotifyEventInfoHeader,
    /// Filesystem identifier (same as `statfs.f_fsid`). Allows userspace
    /// to correlate the file handle with a specific mounted filesystem.
    pub fsid: FsId,
    /// Variable-length file handle. The first 4 bytes are the handle
    /// length (matching `struct file_handle.handle_bytes`), followed by
    /// the handle type (4 bytes) and the handle data. The total length
    /// including padding is recorded in `hdr.len`.
    // Followed by: struct file_handle { u32 handle_bytes; i32 handle_type; u8 f_handle[]; }
}

/// Filesystem identifier (matches `__kernel_fsid_t` / `statfs.f_fsid`).
#[repr(C)]
pub struct FsId {
    pub val: [i32; 2],
}
const_assert!(size_of::<FanotifyEventInfoHeader>() == 4);
const_assert!(size_of::<FsId>() == 8);
const_assert!(size_of::<FanotifyEventInfoFid>() == 12);

/// fanotify event info type constants. Match Linux `FAN_EVENT_INFO_TYPE_*`.
pub const FAN_EVENT_INFO_TYPE_FID: u8 = 1;
pub const FAN_EVENT_INFO_TYPE_DFID_NAME: u8 = 2;
pub const FAN_EVENT_INFO_TYPE_DFID: u8 = 3;
pub const FAN_EVENT_INFO_TYPE_PIDFD: u8 = 4;
pub const FAN_EVENT_INFO_TYPE_ERROR: u8 = 5;
/// Byte-range info for pre-content events (fanotify pre-content scanning).
/// Linux 6.12+.
pub const FAN_EVENT_INFO_TYPE_RANGE: u8 = 6;
/// Mount ID info for mount-aware fanotify. Linux 6.12+.
pub const FAN_EVENT_INFO_TYPE_MNT: u8 = 7;
// Types 8, 9 reserved by Linux.
/// Source directory+name for rename events. Linux 6.6+.
pub const FAN_EVENT_INFO_TYPE_OLD_DFID_NAME: u8 = 10;
// Type 11 reserved by Linux.
/// Destination directory+name for rename events. Linux 6.6+.
pub const FAN_EVENT_INFO_TYPE_NEW_DFID_NAME: u8 = 12;

pub enum FanotifyMarkType { Inode, Mount, Filesystem }

pub enum FanotifyClass {
    Notif = 0x0000_0000,      // FAN_CLASS_NOTIF
    Content = 0x0000_0004,    // FAN_CLASS_CONTENT
    PreContent = 0x0000_0008, // FAN_CLASS_PRE_CONTENT
}

14.13.2.2 Permission Event Flow

When a VFS operation triggers a permission-event mask bit (e.g., FAN_OPEN_PERM on open(2)):

fanotify_perm_event(inode, event_type, opener_pid):
  // Collect all matching fanotify instances in class order (PreContent first).
  matching = collect_matching_marks(inode, event_type)
  if matching is empty: return Ok(())  // fast path

  for instance in matching sorted by class descending:
    id = instance.next_perm_id.fetch_add(1)
    event_fd = open_file_for_fanotify(inode)  // opens fd for daemon to inspect
    event = FanotifyEvent { mask: event_type, fd: event_fd, pid: opener_pid, ... }
    req = Arc::new(FanotifyPermRequest { event, request_id: id, response: None, waker })

    instance.perm_queue.lock().insert(id, req.clone())
    queue = instance.event_queue.lock()
    if !queue.is_full():
      queue.push(event)
    else:
      // Queue overflow: drop event, set FAN_Q_OVERFLOW flag (matching Linux)
      instance.overflow.store(true, Ordering::Release)
    instance.wait_queue.wake_up_one()

    // Block with mandatory timeout — never block indefinitely
    match req.channel.wait_timeout(instance.perm_timeout):
      Ok(response):
        if response.allow: close(event_fd); continue  // allow: close fd, check next instance
        else: close(event_fd); return Err(EPERM)
      Err(Timeout):
        // Log timeout: fanotify daemon too slow
        log_warn!("fanotify: perm request timed out after {:?}, action={:?}",
                  instance.perm_timeout, instance.perm_timeout_action)
        // Increment per-group timeout counter (visible in /proc/PID/fdinfo/<fafd>)
        instance.timeout_count.fetch_add(1, Ordering::Relaxed)
        match instance.perm_timeout_action:
          PermTimeoutAction::Deny  → close(event_fd); return Err(EPERM)
          PermTimeoutAction::Allow → close(event_fd); continue  // allow on timeout

  return Ok(())  // all instances allowed

Timeout vs late response race: If the daemon responds after the timeout fires but before the requesting thread fully unblocks, the response is discarded. The req.response uses an OnceLock<u32> first-writer-wins pattern: the first writer (either the daemon's write() or the timeout handler) wins. The loser's write is a no-op. This prevents both double-free of the event fd and contradictory allow-then-deny sequences. The daemon's late response is logged at DEBUG level for diagnostic purposes.

Mandatory permission event timeout: Permission events (FAN_OPEN_PERM, FAN_ACCESS_PERM, FAN_OPEN_EXEC_PERM) have a mandatory response timeout to prevent system-wide I/O stalls.

System-wide timeout knob: /proc/sys/fs/fanotify/perm_timeout_ms (default: 5000). Can be set to 0 to disable timeout (not recommended; requires CAP_SYS_ADMIN).

Monitoring: /proc/sys/fs/fanotify/perm_timeout_count — system-wide count of permission request timeouts (monotonic counter, reset on boot). Per-group count in /proc/PID/fdinfo/<fafd> as perm_timeout_count: N.

Linux compatibility note: Linux fanotify has no timeout on permission events (daemon death causes permanent block — requires daemon restart or fanotify fd close). UmkaOS's timeout is an improvement over Linux; existing fanotify daemons work unchanged (they don't set FANOTIFY_INIT_PERM_TIMEOUT_MS, so they get the 5s default with Deny on timeout). Tools like systemd-oomd, CrowdStrike Falcon, and audit daemons that use fanotify will benefit automatically from the safety timeout.

Userspace daemon writes FAN_ALLOW / FAN_DENY:

write(fanotify_fd, &fanotify_response { fd: event_fd, response: FAN_ALLOW_or_DENY }):
  // Match the response to a pending request by event_fd.
  // Linux ABI compatibility: the userspace `fanotify_response` struct uses `fd` as
  // the matching key. Internally, the kernel maps fd → request using the per-group
  // fd-to-request XArray (O(1) lookup). An internal request_id is used only for
  // kernel-side tracking and logging; it is never exposed to userspace.
  req = find_perm_request_by_fd(instance.perm_queue, event_fd)
  if req is None: return Err(EINVAL)  // stale or already answered
  req.response.call_once(|| FAN_ALLOW_or_DENY)
  req.waker.wake_up_one()  // unblock the blocked syscall

UmkaOS improvement over Linux fanotify: Linux matches responses to pending permission requests by the fd number inside the fanotify_response struct, which becomes ambiguous if the daemon closes and reopens fds in the event window. UmkaOS uses a typed FanotifyPermRequest with a structured response channel keyed by a monotonically increasing request_id. The Arc<FanotifyPermRequest> lifetime guarantees the blocked syscall's stack is valid until the response arrives, eliminating the lifetime ambiguity in the fd-matching approach.

14.13.3 UmkaOS-Native File Watch Capabilities

UmkaOS provides a capability-based file watching API as a modern alternative to inotify. Unlike inotify (global watch descriptor namespace, process-scoped), FileWatchCap watches are:

• Capability-scoped: unforgeable, revocable, auditable
• Memory-bounded: each watch is a capability slot (no global state)
• Automatically revoked: when the capability is dropped or the process exits
• Ring-delivered: events go to a typed UmkaOS ring buffer, not a read() queue
• Composable: multiple watches can share one ring

inotify remains fully supported for Linux compatibility. FileWatchCap is the recommended API for new UmkaOS code.

/// A capability granting the holder the right to watch a specific inode for
/// specific events. Cannot be forged; issued by the kernel only.
/// Revocable via the standard capability revocation path (Section 9.1).
pub struct FileWatchCap {
    /// The inode to watch. Kernel-internal reference — not a path (immune to rename).
    inode: Arc<Inode>,
    /// Events to deliver (subset of InotifyMask).
    mask: InotifyMask,
    /// Watch children of this directory (if inode is a directory).
    watch_children: bool,
    /// Watch children recursively (deep watch — UmkaOS extension, not in inotify).
    watch_recursive: bool,
}

/// Kernel handle for an active watch registration, returned by
/// `inode_watch()`. Ownership is unique (neither `Copy` nor `Clone`).
/// `WatchHandle::drop()` unregisters the watch from its inode and releases the
/// event ring reference — so a watch lives exactly as long as the handle. When
/// a process exits, all its `WatchHandle`s are dropped automatically, requiring
/// no explicit teardown. Kernel-internal; not exposed across a KABI boundary.
pub struct WatchHandle {
    /// The inode the watch is registered against; the `Arc` keeps the inode
    /// resident for the lifetime of the watch.
    inode: Arc<Inode>,
    /// Stable per-registration identifier used to locate and remove this
    /// watch's entry on drop. u64 (50-year rule; monotonically allocated).
    watch_id: u64,
}

/// Error returned by watch registration and capability-open operations
/// (`inode_watch()`, `open_watch_cap()`).
#[derive(Clone, Copy, PartialEq, Eq, Debug)]
pub enum WatchError {
    /// The supplied capability does not grant watch rights on the target
    /// (maps to `EACCES`/`EPERM` at the syscall boundary).
    PermissionDenied,
    /// Per-user or per-instance watch limit reached
    /// (`fs.inotify.max_user_watches`; maps to `ENOSPC`).
    LimitExceeded,
    /// The target inode no longer exists or was unlinked (maps to `ENOENT`).
    NotFound,
    /// `IN_ONLYDIR` was requested but the target is not a directory
    /// (maps to `ENOTDIR`).
    NotDirectory,
    /// The event ring is full or has been torn down (maps to `EAGAIN`).
    RingUnavailable,
}

/// Subscribe to inode events via a capability.
/// Events are delivered to `ring` as typed `FileWatchEvent` structs.
///
/// Returns a `WatchHandle` — dropping the handle unregisters the watch.
pub fn inode_watch(
    cap: FileWatchCap,
    ring: Arc<EventRing<FileWatchEvent>>,
) -> Result<WatchHandle, WatchError>;

/// A single file watch event, delivered to the ring.
/// C-compatible layout: uses explicit length + fixed array instead of
/// `Option<ArrayString<255>>` (which has Rust-internal layout).
// kernel-internal, not KABI
#[repr(C)]
pub struct FileWatchEvent {
    pub event_type: FileWatchEventType, // enum (see below)
    pub cookie: u32,                    // for rename pairs (FROM/TO share cookie)
    pub inode_id: u64,                  // stable inode number
    pub name_len: u8,                   // 0 = no name; >0 = first `name_len` bytes valid
    pub name: [u8; 255],                // filename (for directory events), NUL-padded
    pub timestamp: MonotonicInstant,    // UmkaOS extension: not in inotify
}
// FileWatchEvent layout: event_type(u32=4) + cookie(u32=4) + inode_id(u64=8) +
// name_len(u8=1) + name([u8;255]=255) + timestamp(MonotonicInstant(u64)=8).
// After name_len+name: offset = 4+4+8+1+255 = 272. 272 % 8 = 0, no padding.
// Total: 272 + 8 = 280 bytes.
const_assert!(core::mem::size_of::<FileWatchEvent>() == 280);

/// Monotonic timestamp (nanoseconds since boot, from CLOCK_MONOTONIC).
/// Used for UmkaOS extensions where wall-clock time is not needed.
#[repr(transparent)]
#[derive(Clone, Copy, PartialEq, Eq, PartialOrd, Ord)]
pub struct MonotonicInstant(pub u64);

#[repr(u32)]
pub enum FileWatchEventType {
    Access,       // File was read
    Modify,       // File was written
    Attrib,       // Metadata changed (chmod, chown, timestamps)
    CloseWrite,   // File opened for writing was closed
    CloseNoWrite, // File opened read-only was closed
    Open,         // File was opened
    MovedFrom,    // File moved out (cookie matches MovedTo)
    MovedTo,      // File moved in (cookie matches MovedFrom)
    Create,       // File created in watched directory
    Delete,       // File deleted from watched directory
    DeleteSelf,   // Watched file itself was deleted
    MoveSelf,     // Watched file itself was moved
    Unmount,      // Filesystem containing watched file was unmounted
}

Deep watch (watch_recursive: true): watches a directory tree recursively. UmkaOS maintains a kernel-side tree of watch registrations, automatically adding watches for new subdirectories as they are created (IN_CREATE on a directory). inotify has no recursive watch; tools like inotifywait -r simulate it with userspace polling, which has TOCTOU races. UmkaOS's deep watch is race-free.

Deep watch resource limits: Recursive watches can consume significant kernel memory on deep directory trees (e.g., a deep watch on / with millions of directories). Limits are enforced per-user to prevent denial-of-service: - max_deep_watches_per_user: default 128 (sysctl fs.inotify.max_deep_watches). Exceeding this returns ENOSPC. - max_watch_entries_per_deep_watch: default 65536. If the directory tree contains more subdirectories than this, the kernel stops adding new watches beyond the limit and delivers IN_Q_OVERFLOW to signal the user that coverage is incomplete. - Memory accounting: each internal watch node costs ~128 bytes. A deep watch on a tree with 65536 directories consumes ~8 MB of kernel memory, charged to the user's RLIMIT_MEMLOCK limit. - CAP_SYS_ADMIN can override max_deep_watches_per_user up to the system-wide hard limit of 1024.

Obtaining a FileWatchCap: capability is issued via:

/// Open a FileWatchCap for a path (requires read permission on the path).
pub fn open_watch_cap(
    dirfd: DirFd,
    path: &Path,
    mask: InotifyMask,
    watch_children: bool,
    watch_recursive: bool,
) -> Result<FileWatchCap, WatchError>;

Revocation: WatchHandle::drop() unregisters the watch. When the process exits, all WatchHandles are dropped automatically — no cleanup required. Capability revocation (Section 9.1) also revokes all file watches derived from the revoked capability.

Linux compatibility: FileWatchCap is an UmkaOS-only API. inotify_init(), inotify_add_watch(), inotify_rm_watch() work identically to Linux. FileWatchCap is intended for new UmkaOS applications; existing Linux software uses inotify unchanged.

14.13.4 Cross-References

• Section 14.1 (VFS Traits): inotify/fanotify hooks are inserted at the VFS operation dispatch layer, after InodeOps/FileOps call sites complete successfully.
• Section 17.1 (Namespace Implementation): fanotify marks survive CLONE_NEWNS and remain attached to the underlying inode/mount, not to a specific mount namespace. Marks set in a parent namespace remain visible in child namespaces for the same underlying mount.
• Section 9.1 (Security): fanotify_init(FAN_CLASS_CONTENT) and fanotify_init(FAN_CLASS_PRE_CONTENT) require CAP_SYS_ADMIN. Informational fanotify (FAN_CLASS_NOTIF) requires no capability (Linux 5.13+, unprivileged fanotify); UmkaOS follows the same requirement for compatibility.

---

14.14 Local File Locking (flock / fcntl POSIX Locks / OFD Locks)

UmkaOS provides three advisory file locking interfaces, each with distinct semantics:

Interface	Granularity	Lock scope	Inherited on fork	Released on
flock(2)	Whole file	Per open-file-description	Yes (child shares the open file description, so the same lock is shared; either process can release it)	Last close of the description
fcntl F_SETLK	Byte-range (POSIX)	Per process (PID)	No	Process exit OR any close of the file
fcntl F_OFD_SETLK	Byte-range (OFD)	Per open-file-description	Yes	Last close of the description

All three are advisory: a process can read and write a file regardless of locks held by other processes. Locks only prevent other processes from acquiring conflicting locks. Mandatory locking (Linux MS_MANDLOCK) is deliberately not implemented — it was deprecated in Linux 5.15 and is incompatible with modern VFS semantics.

14.14.1 Data Structures

/// A single file lock entry. Stored in the per-inode `FileLockTree`.
pub struct FileLock {
    /// Lock type: read (shared) or write (exclusive).
    pub lock_type: FileLockType,

    /// Byte range: [start, end] inclusive. 0..=u64::MAX represents the whole file.
    /// For flock locks, start=0 and end=u64::MAX always.
    pub start: u64,
    pub end: u64,

    /// For POSIX locks: the PID of the owning process.
    /// All POSIX locks held by a process are released when it exits OR when
    /// any file descriptor for the file is closed (POSIX semantics).
    /// For OFD locks: None. The lock is owned by the open-file-description.
    /// For flock locks: None. The lock is owned by the open-file-description.
    pub owner_pid: Option<Pid>,

    /// The open-file-description that created this lock.
    /// Weak reference: if the description is dropped (last fd closed), the lock
    /// is released. For POSIX locks, `owner_pid` is the primary ownership token
    /// and `owner_fd` is advisory for conflict matching.
    pub owner_fd: Weak<OpenFile>,

    /// Wait queue: tasks blocked waiting for this lock to be released sleep here.
    pub wait_queue: WaitQueueHead,
}

pub enum FileLockType {
    /// Shared (read) lock. Multiple readers can hold simultaneously.
    Read,
    /// Exclusive (write) lock. No other lock may be held concurrently.
    Write,
}

/// Per-inode lock state. Present only on inodes that have had locks acquired;
/// None on inodes that have never been locked (zero overhead on the fast path).
pub struct InodeLocks {
    /// Augmented interval tree of active locks (POSIX, flock, and OFD locks).
    /// Sorted by `l_start`; each node carries `subtree_max: u64` = maximum
    /// `l_end` in its subtree. This enables O(log n) range overlap queries.
    /// See Section 14.10.3 for the full algorithm specification.
    pub locks: FileLockTree,
    /// Protects the lock tree. Operations must be atomic with respect to each other.
    pub lock: SpinLock<()>,
}

/// Augmented interval tree for file lock conflict detection.
/// Red-black tree sorted by `l_start`, augmented with `subtree_max` for
/// O(log n) range overlap queries.
pub struct FileLockTree {
    /// Root of the red-black tree. None when no locks are held.
    root: Option<Box<FileLockNode>>,
    /// Number of locks currently in the tree.
    count: usize,
}

/// FileLockNode allocation uses a dedicated slab cache (`file_lock_slab`)
/// with per-CPU magazines, matching Linux's `file_lock_cache`. This provides
/// bounded warm-path allocation without general-heap contention.
pub struct FileLockNode {
    pub lock: FileLock,
    /// Maximum `l_end` value in this node's subtree (including this node).
    /// Updated on every insert/delete along the path to the root.
    pub subtree_max: u64,
    pub left: Option<Box<FileLockNode>>,
    pub right: Option<Box<FileLockNode>>,
    pub color: RbColor,
}

pub enum RbColor { Red, Black }

14.14.2 Conflict Detection

Two locks conflict if: 1. At least one is a write lock (FileLockType::Write). 2. Their byte ranges overlap: !(lock_a.end < lock_b.start || lock_b.end < lock_a.start). 3. They have different owners: - For POSIX locks: different PIDs. - For OFD/flock locks: different Weak<OpenFile> pointers. - A POSIX lock can upgrade/replace an existing POSIX lock from the same PID without conflict.

14.14.3 Locking Algorithm

UmkaOS uses an augmented interval tree (red-black tree with subtree_max augmentation) for O(log n) file lock conflict detection. This is the correct data structure; there is no O(n) fallback. Linux used an O(n) linked-list scan for decades before adding interval trees in Linux 3.13; UmkaOS starts with the correct design.

FileLockTree structure: - Sorted by l_start (range start) - Each node carries subtree_max: u64 = maximum l_end in its subtree - This augmentation enables O(log n) range overlap queries

Conflict query for range [req_start, req_end): Walk the tree: at each node, if node.subtree_max < req_start, the entire subtree has no overlapping locks — prune. Otherwise check the node itself and recurse into both children. O(log n + k) where k = number of conflicts found.

Insert/delete: O(log n) standard red-black tree operations, plus O(log n) subtree_max recomputation on the path to root. During red-black tree rotations (left-rotate, right-rotate), subtree_max is recomputed for the two rotated nodes: node.subtree_max = max(node.lock.l_end, left_child_max(node), right_child_max(node)). This is the standard augmented red-black tree technique (CLRS §14.2).

fcntl_setlk(fd, lock_type, start, end, wait: bool) → Result:
  inode = fd.inode()
  ensure inode.locks is initialized

  inode.locks.lock.lock()

  loop:
    // O(log n + k) interval tree query for conflicting locks in [start, end).
    for existing in inode.locks.locks.query_conflicts(start, end, lock_type, &fd):
      if !wait:
        inode.locks.lock.unlock()
        return Err(EAGAIN)           // F_SETLK: fail immediately

      // F_SETLKW: deadlock detection before sleeping
      if would_deadlock(current_pid, existing.owner_pid):
        inode.locks.lock.unlock()
        return Err(EDEADLK)

      inode.locks.lock.unlock()
      existing.wait_queue.wait_event(|| !lock_conflicts_anymore(...))
      inode.locks.lock.lock()
      continue loop                  // re-check after wakeup (spurious wakeup safe)

    // No conflict: coalesce adjacent/overlapping locks of the same type and owner,
    // then insert the new lock. O((k+1) log n).
    coalesce_and_insert(inode, fd, lock_type, start, end)
    inode.locks.lock.unlock()
    return Ok(())

Lock Coalescing Algorithm (Greedy Interval Merge)

The following batch coalescing algorithm is used during lock migration and crash recovery (when multiple lock requests are replayed). The per-call coalescing path is coalesce_and_insert() below, which operates on the interval tree directly with no Vec allocation.

Input: a set of pending lock requests sorted by (offset, len). Output: a minimal set of merged lock requests covering the same byte ranges.

Data structure:

/// Lock mode of a byte-range or whole-file lock request. A read lock (shared)
/// is compatible with other shared locks on overlapping ranges; a write lock
/// (exclusive) conflicts with any other lock on an overlapping range. Maps to
/// `flock(2)` `LOCK_SH`/`LOCK_EX` and to the `l_type` `F_RDLCK`/`F_WRLCK`
/// values of `fcntl(2)` locks. (Unlock is not a lock *request* — it is a
/// separate release path — so it is not a variant here.)
#[derive(Clone, Copy, PartialEq, Eq)]
pub enum LockOp {
    /// Shared / read lock (`LOCK_SH`, `F_RDLCK`).
    Shared,
    /// Exclusive / write lock (`LOCK_EX`, `F_WRLCK`).
    Exclusive,
}

struct PendingLockRequest {
    offset: u64,
    len: u64,
    op: LockOp,  // Shared or Exclusive
}

Algorithm (O(n log n) for n requests): 1. Collect all pending requests into Vec<PendingLockRequest>. 2. Sort by offset (ascending), then by len (descending) as tiebreaker. 3. Sweep left to right: - Start with current = requests[0]. - For each subsequent request r: - If r.offset <= current.offset + current.len (overlapping or adjacent) AND r.op == current.op (same lock type): - current.len = max(current.offset + current.len, r.offset + r.len) - current.offset - Otherwise: emit current, set current = r. 4. Emit final current.

Rationale: coalescing reduces the number of kernel lock table entries for byte-range locking (POSIX fcntl(F_SETLK)), avoiding fragmentation in the per-file lock list.

coalesce_and_insert(new_lock) — called after conflict check passes:

1. Query the interval tree for all locks owned by new_lock.pid that are adjacent to or overlapping new_lock's range [l_start, l_end) (adjacent = existing.l_end == new_lock.l_start or vice versa)
2. Compute the union range: min(all.l_start) to max(all.l_end)
3. Remove all found locks from the interval tree (O(k log n))
4. Insert a single merged lock covering the union range (O(log n))

Complexity: O((k+1) log n) where k = number of locks merged. Coalescing reduces tree size over time for processes that acquire many adjacent byte-range locks (common in database file locking patterns).

14.14.4 Deadlock Detection

Wait-For Graph data structure:

/// Directed graph of lock-wait relationships between threads.
/// Edge (A → B) means thread A is currently blocked waiting for a byte-range
/// lock held by thread B. The graph is maintained in the lock manager and
/// updated on each lock acquisition attempt that would block.
///
/// Bounded at compile time; exceeding `LOCK_GRAPH_MAX_THREADS` causes
/// `detect_deadlock` to abort with `LockError::DeadlockDetectionOverflow`.
pub struct FlockWaitForGraph {
    /// Sparse adjacency list: (waiting_thread, [holder_threads]).
    /// Each entry represents one blocked thread and the set of threads
    /// it is waiting on (typically 1, but can be multiple for range locks).
    edges: ArrayVec<(ThreadId, ArrayVec<ThreadId, LOCK_GRAPH_MAX_HOLDERS>), LOCK_GRAPH_MAX_THREADS>,
}

// FlockWaitForGraph is ~20 KiB inline (256 * 80 bytes), which exceeds safe kernel
// stack depth (8-16 KiB). It MUST NOT be allocated on the kernel stack.
// Allocation strategy: per-CPU pre-allocated buffer. Only one deadlock
// detection runs per CPU at a time because InodeLocks.lock is a SpinLock,
// which disables preemption. No other thread on this CPU can enter a locking
// path while the SpinLock is held.
static LOCK_DEADLOCK_GRAPH: PerCpu<FlockWaitForGraph> = PerCpu::new(FlockWaitForGraph::new);

impl FlockWaitForGraph {
    pub const fn new() -> Self { Self { edges: ArrayVec::new() } }

    /// Record that `waiter` is blocked on a lock held by `holder`.
    pub fn add_edge(&mut self, waiter: ThreadId, holder: ThreadId) { /* ... */ }

    /// Remove all outgoing edges from `waiter` (called on lock release or
    /// wakeup so stale edges don't pollute subsequent detections).
    pub fn remove_waiter(&mut self, waiter: ThreadId) { /* ... */ }

    /// Return an iterator over all threads that currently hold locks that
    /// `waiter` is blocked on. O(n) scan over the edge list.
    pub fn holders_of(&self, waiter: ThreadId) -> impl Iterator<Item = ThreadId> + '_ {
        self.edges.iter()
            .find(|(tid, _)| *tid == waiter)
            .map(|(_, holders)| holders.iter().copied())
            .into_iter()
            .flatten()
    }
}

/// Maximum number of threads tracked simultaneously in the wait-for graph.
/// This is the per-inode concurrent contention limit, not a system-wide
/// thread limit. 256 concurrent threads blocked on the same inode's lock
/// tree is well beyond any realistic workload. Overflow is treated
/// conservatively (EDEADLK) — correctness is preserved at the cost of
/// a false deadlock report.
pub const LOCK_GRAPH_MAX_THREADS: usize = 256;
/// Maximum number of holders per waiting thread (range locks can be split).
pub const LOCK_GRAPH_MAX_HOLDERS: usize = 8;

Deadlock Detection: Wait-For Graph DFS (3-Color)

Each lock holder is a node; each blocked waiter is a directed edge (waiter → holder).

Node state per thread: - WHITE: not yet visited in current DFS - GRAY: currently in the DFS recursion stack (potential cycle node) - BLACK: fully explored, no cycle reachable from here

Constants:

const VFS_LOCK_MAX_DEPTH: usize = 64;  // Max wait-chain depth before abort

Algorithm (invoked before blocking on a contested lock):

fn detect_deadlock(start: ThreadId, graph: &FlockWaitForGraph) -> bool:
  // Stack-allocated: FlockWaitForGraph limits to VFS_LOCK_MAX_DEPTH threads,
  // so linear scan on ≤64 entries is faster than HashMap heap allocation.
  color = ArrayVec<(ThreadId, Color), VFS_LOCK_MAX_DEPTH>::new()
  return dfs(start, &mut color, graph, depth=0)

fn dfs(node: ThreadId, color: &mut ArrayVec, graph: &FlockWaitForGraph, depth: usize) -> bool:
  if depth > VFS_LOCK_MAX_DEPTH:
    return true   // treat as deadlock (conservative)
  color[node] = GRAY
  for each holder in graph.holders_of(node):
    match color.get(holder):
      GRAY  => return true   // back-edge: cycle detected
      BLACK => continue      // already explored, safe
      WHITE | None:
        color[holder] = WHITE
        if dfs(holder, color, graph, depth+1): return true
  color[node] = BLACK
  return false

On true return: the blocking call returns Err(LockError::Deadlock) / EDEADLK. The caller must release all currently held locks and retry with a backoff.

The graph is constructed on-demand per lock request and is not persisted. Returning true on depth overflow is safe: it causes the lock request to fail with EDEADLK, which is better than silently allowing a potential deadlock. The depth limit prevents deadlock detection from becoming a denial-of-service vector in pathological chains.

14.14.5 Lock Release on File Description Close

When an OpenFile's reference count drops to zero (the last file descriptor pointing to it is closed):

• OFD locks: all locks where owner_fd matches this description are removed.
• flock locks: the flock lock associated with this description (if any) is removed.
• POSIX locks: all locks where owner_pid == current_process.pid are removed. This is the POSIX-mandated behavior: closing any file descriptor for a file releases all POSIX locks the process holds on that file, regardless of which fd was used to acquire them.

After removing locks, wake all tasks in the wait_queue of each removed lock so they can retry acquisition.

14.14.6 memfd Sealing (F_ADD_SEALS / F_GET_SEALS)

memfd_create(2) returns an anonymous file (backed by tmpfs, with no pathname). Seals are write-once restrictions placed on the file's mutation capabilities:

/// Seal flags for memfd files. Once set, seals cannot be removed.
/// SEAL_SEAL prevents any further seals from being added.
bitflags! {
    pub struct SealFlags: u32 {
        /// Prevent any further seals from being added.
        const SEAL_SEAL         = 0x0001;
        /// Prevent the file from shrinking (ftruncate to a smaller size returns EPERM).
        const SEAL_SHRINK       = 0x0002;
        /// Prevent the file from growing (writes past EOF, ftruncate to larger size return EPERM).
        const SEAL_GROW         = 0x0004;
        /// Prevent all writes: write(2) returns EPERM, mmap(PROT_WRITE) returns EPERM.
        const SEAL_WRITE        = 0x0008;
        /// Prevent future mmap(PROT_WRITE) but allow existing writable mappings to remain.
        const SEAL_FUTURE_WRITE = 0x0010;
    }
}

fcntl(fd, F_ADD_SEALS, seals): add the specified seals atomically via a compare_exchange on the inode's AtomicU32 seal field. Fails with EPERM if SEAL_SEAL is already set. Fails with EBUSY if SEAL_WRITE is being added while a writable mmap exists on the file.

fcntl(fd, F_GET_SEALS): return the current seal set (atomic load, lock-free).

Seal enforcement in VFS paths: - write(2) and pwrite64(2): check SEAL_WRITE. - ftruncate(2) to smaller size: check SEAL_SHRINK. - ftruncate(2) to larger size: check SEAL_GROW. - mmap(PROT_WRITE): check SEAL_WRITE | SEAL_FUTURE_WRITE.

UmkaOS improvement: seals are stored as an AtomicU32 in the memfd's inode — seal reads are lock-free (a single atomic load), which is important because the seal check appears on every write(2) and mmap(2) call for sealed fds.

14.14.7 Cross-References

• Section 15.15 (Distributed Lock Manager): the DLM provides cluster-wide advisory locks that extend the local flock/POSIX lock semantics across nodes. Local file locks (this section) are node-local only.
• Section 14.1 (VFS Architecture): FileOps::release() is the call site where OFD and flock locks are released when the last fd to a file description is closed.
• Section 17.1 (Containers): POSIX lock ownership is per-PID-namespace-PID. Within a container's PID namespace, lock ownership semantics are unchanged.

14.14.8 Lock Semantics Mode (POSIX Default / OFD Opt-in)

UmkaOS keeps POSIX semantics as the default for F_SETLK to preserve full Linux binary compatibility. Applications and deployments that want the correct OFD semantics as default can opt in at three levels, with the highest-priority source winning:

Priority order (highest first): 1. Per-call explicit constant 2. Per-process prctl 3. Per-user-namespace sysctl 4. System global default: POSIX

---

14.14.8.1.1 Per-call explicit (always available, no mode setting needed)

F_OFD_SETLK    // Always OFD semantics (Linux 3.15+, UmkaOS supported)
F_OFD_SETLKW   // Always OFD semantics, blocking
F_SETLK_POSIX  // UmkaOS extension: always POSIX semantics, explicit
F_SETLKW_POSIX // UmkaOS extension: always POSIX semantics, blocking

F_SETLK_POSIX exists so code inside an OFD-default process can still request POSIX semantics for specific locks (e.g., a bundled library that requires process-death lock release for crash detection).

---

14.14.8.1.2 Per-process opt-in

prctl(PR_SET_LOCK_SEMANTICS, LOCK_SEM_OFD)    // F_SETLK means OFD for this process
prctl(PR_SET_LOCK_SEMANTICS, LOCK_SEM_POSIX)  // Explicit POSIX (escape hatch)
prctl(PR_GET_LOCK_SEMANTICS, 0, 0, 0, 0)      // Query current mode

pub const LOCK_SEM_POSIX: u64 = 0;  // default
pub const LOCK_SEM_OFD:   u64 = 1;

Stored in Task.lock_semantics: LockSemanticsMode (per-thread but inherited from the process — all threads in a process share the same mode via Process.lock_semantics).

Inheritance rules: - fork(): child inherits parent's lock_semantics - exec(): inherited (sticky) — a container runtime sets it once; all descendant processes inherit - exec() of setuid/setgid binary: reset to the user-namespace sysctl default (security: a privilege-elevating binary must not blindly inherit)

---

14.14.8.1.3 Per-user-namespace sysctl

/proc/sys/fs/file_lock_default

Values: posix (default) | ofd

This sysctl is per-user-namespace, not global. Each container has its own user namespace and therefore its own file_lock_default. The container runtime sets it at container creation:

# Inside an UmkaOS-native container's user namespace:
echo ofd > /proc/sys/fs/file_lock_default

/// Per-user-namespace lock semantics default.
/// Stored in UserNamespace.file_lock_default.
pub enum LockSemanticsMode {
    Posix = 0,  // F_SETLK uses POSIX semantics (default)
    Ofd   = 1,  // F_SETLK uses OFD semantics
    Unset = 2,  // Not explicitly configured; falls through to namespace/global default
}

Requires CAP_SYS_ADMIN in the target user namespace to change. Affects new processes only — running processes keep their current mode.

---

14.14.8.1.4 Deployment model

Scenario	Recommended config
Host with legacy software	sysctl = posix (default), no change needed
UmkaOS-native container	runtime sets sysctl = ofd in container's user namespace
Mixed container (some legacy binaries)	sysctl = posix, UmkaOS-native apps use prctl
Wine / NFS lockd / old SQLite	prctl(LOCK_SEM_POSIX) in launch wrapper

---

14.14.8.1.5 Internal resolution

/// Kernel-internal decode of the `fcntl(2)` lock command. Derived at the
/// syscall boundary from the userspace `F_*` command values
/// (`F_GETLK`/`F_SETLK`/`F_SETLKW` and `F_OFD_GETLK`/`F_OFD_SETLK`/`F_OFD_SETLKW`,
/// per `fcntl(2)`), then split so the resolver can distinguish commands whose
/// lock model is fixed (`*Ofd*`, and the `*Posix` forms produced when a caller
/// or namespace policy pins classic POSIX semantics) from the ambiguous
/// `SetLk`/`SetLkW`, whose model (POSIX vs OFD vs per-process policy) is
/// resolved at call time by `effective_lock_semantics()`. This is the internal
/// discriminant, not the external ABI value.
#[derive(Clone, Copy, PartialEq, Eq)]
pub enum FcntlCmd {
    /// `F_GETLK` (resolved semantics) — test whether a lock could be placed.
    GetLk,
    /// `F_SETLK` with unresolved model — non-blocking set; resolved at call time.
    SetLk,
    /// `F_SETLKW` with unresolved model — blocking set; resolved at call time.
    SetLkW,
    /// `F_SETLK` pinned to classic POSIX (per-process) semantics.
    SetLkPosix,
    /// `F_SETLKW` pinned to classic POSIX (per-process) semantics.
    SetLkWPosix,
    /// `F_OFD_SETLK` — open-file-description lock, non-blocking.
    OfdSetLk,
    /// `F_OFD_SETLKW` — open-file-description lock, blocking.
    OfdSetLkW,
}

fn effective_lock_semantics(
    task: &Task,
    cmd: FcntlCmd,
) -> LockSemanticsMode {
    match cmd {
        FcntlCmd::OfdSetLk | FcntlCmd::OfdSetLkW     => LockSemanticsMode::Ofd,
        FcntlCmd::SetLkPosix | FcntlCmd::SetLkWPosix  => LockSemanticsMode::Posix,
        FcntlCmd::SetLk | FcntlCmd::SetLkW => {
            // Resolve: per-process > per-namespace sysctl > global POSIX
            if task.process.lock_semantics != LockSemanticsMode::Unset {
                task.process.lock_semantics
            } else {
                task.user_namespace.file_lock_default
            }
        }
        _ => LockSemanticsMode::Posix,
    }
}

Linux compatibility: existing binaries calling F_SETLK on a system where no mode is set get identical POSIX behaviour to Linux. F_OFD_SETLK was added in Linux 3.15 and is already supported. F_SETLK_POSIX and PR_SET_LOCK_SEMANTICS are UmkaOS extensions with no Linux equivalent.

---

14.15 Disk Quota Subsystem (quotactl)

Disk quotas enforce per-user, per-group, and per-project limits on filesystem space and inode usage. Required for multi-tenant storage environments and Linux compatibility.

14.15.1 Data Structures

/// Internal kernel quota accounting structure. NOT the UAPI struct — see
/// `IfDqblk` below for the Linux-compatible quotactl(2) wire format.
/// This struct extends the UAPI layout with `bgrace` and `igrace` fields
/// for in-kernel grace period tracking (not exposed to userspace directly).
pub struct DiskQuota {
    /// Hard block limit (bytes). 0 = no limit. Writes that would exceed this
    /// are rejected with EDQUOT immediately, regardless of grace period.
    pub bhardlimit: u64,

    /// Soft block limit (bytes). Exceeding this triggers a grace period timer.
    /// Once the grace period expires, further writes are rejected with EDQUOT.
    pub bsoftlimit: u64,

    /// Current block usage (bytes). Updated on every successful write and truncate.
    pub bcurrent: u64,

    /// Hard inode limit. 0 = no limit. File creation that would exceed this
    /// is rejected with EDQUOT.
    pub ihardlimit: u64,

    /// Soft inode limit. Exceeding this triggers an inode grace period.
    pub isoftlimit: u64,

    /// Current inode count (files + directories + symlinks owned by this subject).
    pub icurrent: u64,

    /// Quota grace period expiry deadline for blocks: the absolute timestamp
    /// (seconds since epoch) at which the block soft limit grace period expires.
    /// Set to `now + bgrace` when the soft block limit is first exceeded.
    /// 0 if the soft limit has not been exceeded.
    /// After this deadline, writes that would keep usage above `bsoftlimit`
    /// are rejected with EDQUOT (same enforcement as the hard limit).
    /// Matches Linux `dqb_btime` semantics ("time limit for excessive disk use").
    /// Type is u64, matching the Linux UAPI `struct if_dqblk.dqb_btime` (`__u64`).
    pub btime: u64,

    /// Quota grace period expiry deadline for inodes: the absolute timestamp
    /// (seconds since epoch) at which the inode soft limit grace period expires.
    /// Semantics mirror `btime` but for inode counts instead of block usage.
    /// 0 if the soft inode limit has not been exceeded.
    /// Type is u64, matching the Linux UAPI `struct if_dqblk.dqb_itime` (`__u64`).
    pub itime: u64,

    /// Grace period for the block soft limit, in seconds. Default: 7 days (604800).
    pub bgrace: u32,

    /// Grace period for the inode soft limit, in seconds. Default: 7 days (604800).
    pub igrace: u32,
}

/// Linux UAPI quota structure for quotactl(2). Matches `struct if_dqblk`
/// from `<linux/quota.h>` exactly — this is the struct that userspace tools
/// (quota, repquota, edquota) read and write via Q_GETQUOTA / Q_SETQUOTA.
///
/// Field order and sizes must match the Linux definition exactly:
///   __u64 dqb_bhardlimit, dqb_bsoftlimit, dqb_curspace,
///   __u64 dqb_ihardlimit, dqb_isoftlimit, dqb_curinodes,
///   __u64 dqb_btime, dqb_itime,
///   __u32 dqb_valid
#[repr(C)]
pub struct IfDqblk {
    pub dqb_bhardlimit: u64,
    pub dqb_bsoftlimit: u64,
    pub dqb_curspace:    u64,
    pub dqb_ihardlimit:  u64,
    pub dqb_isoftlimit:  u64,
    pub dqb_curinodes:   u64,
    /// Grace period expiry deadline for blocks (seconds since epoch).
    /// 0 if the soft block limit has not been exceeded.
    pub dqb_btime:       u64,
    /// Grace period expiry deadline for inodes (seconds since epoch).
    pub dqb_itime:       u64,
    /// Bitmask of QIF_* flags indicating which fields are valid.
    /// QIF_BLIMITS=1, QIF_SPACE=2, QIF_ILIMITS=4, QIF_INODES=8,
    /// QIF_BTIME=16, QIF_ITIME=32, QIF_ALL=0x3F.
    pub dqb_valid:       u32,
    // repr(C) adds 4 bytes implicit trailing padding for u64 alignment,
    // matching Linux's `struct if_dqblk` exactly (9 fields, 72 bytes).
    // No explicit `_pad` field — Linux has 9 fields, not 10. The implicit
    // padding is zero-initialized by the kernel before copy_to_user().
}
// Layout: 8×u64 + u32 + 4(implicit pad) = 64 + 4 + 4 = 72 bytes.
const_assert!(size_of::<IfDqblk>() == 72);

/// Conversion between internal `DiskQuota` and UAPI `IfDqblk`:
/// - Q_GETQUOTA: kernel reads `DiskQuota` from cache, converts to `IfDqblk`,
///   copies to userspace. `dqb_valid` is set to `QIF_ALL` (all fields valid).
/// - Q_SETQUOTA: kernel copies `IfDqblk` from userspace, updates only the
///   `DiskQuota` fields indicated by `dqb_valid` in the cache.

/// Quota subject type.
pub enum QuotaType {
    User    = 0,  // USRQUOTA
    Group   = 1,  // GRPQUOTA
    Project = 2,  // PRJQUOTA
}

/// Quota operations implemented by filesystems that support quotas.
/// Optional — filesystems without quota support omit this and quotactl(2) returns ENOSYS.
pub trait QuotaOps: Send + Sync {
    /// Enable quota enforcement for the given type, reading limits from `quota_file`.
    fn quota_on(&self, quota_type: QuotaType, quota_file: &str) -> Result<(), VfsError>;

    /// Disable quota enforcement for the given type.
    fn quota_off(&self, quota_type: QuotaType) -> Result<(), VfsError>;

    /// Read the quota entry for subject `id` (UID, GID, or project ID).
    fn get_quota(&self, quota_type: QuotaType, id: u32) -> Result<DiskQuota, VfsError>;

    /// Set limits and accounting for subject `id`. Requires CAP_SYS_ADMIN.
    fn set_quota(&self, quota_type: QuotaType, id: u32, quota: &DiskQuota) -> Result<(), VfsError>;

    /// Read global quota state (grace periods, flags) for the given type.
    fn get_info(&self, quota_type: QuotaType) -> Result<QuotaInfo, VfsError>;

    /// Set global quota state (grace periods). Requires CAP_SYS_ADMIN.
    fn set_info(&self, quota_type: QuotaType, info: &QuotaInfo) -> Result<(), VfsError>;

    /// Flush in-memory quota accounting to the quota database file.
    fn sync_quota(&self, quota_type: QuotaType) -> Result<(), VfsError>;
}

/// Global quota state (grace periods and enabled flags) for a single quota type.
pub struct QuotaInfo {
    /// Block grace period in seconds.
    pub bgrace: u32,
    /// Inode grace period in seconds.
    pub igrace: u32,
    /// Quota flags (QIF_FLAGS: quota enabled, quota accounting-only, etc.).
    pub flags: u32,
}

14.15.2 quotactl(2) Dispatch

The quotactl(2) syscall encodes both the quota command and the quota type in a single 32-bit cmd argument: bits [31:8] are the command (Q_QUOTAON=0x800002, Q_QUOTAOFF=0x800003, Q_GETQUOTA=0x800007, Q_SETQUOTA=0x800008, Q_GETINFO=0x800005, Q_SETINFO=0x800006, Q_SYNC=0x800001) and bits [7:0] are the quota type (USRQUOTA=0, GRPQUOTA=1, PRJQUOTA=2). This matches the Linux QCMD(cmd, type) = (cmd << 8) | type macro. Subcmd values range up to 0x800009 (Q_GETNEXTQUOTA).

quotactl(cmd, dev, id, addr):
  qt_cmd  = cmd >> 8
  qt_type = QuotaType::from(cmd & 0xff)  // USRQUOTA/GRPQUOTA/PRJQUOTA

  sb = resolve_superblock_from_device_path(dev)
  if sb.quota_ops is None: return Err(ENOSYS)

  // Capability check for mutating operations
  if qt_cmd in [Q_QUOTAON, Q_QUOTAOFF, Q_SETQUOTA, Q_SETINFO]:
    check_capability(CAP_SYS_ADMIN)?

  match qt_cmd:
    Q_QUOTAON   → sb.quota_ops.quota_on(qt_type, addr_as_path)
    Q_QUOTAOFF  → sb.quota_ops.quota_off(qt_type)
    Q_GETQUOTA  → quota = sb.quota_ops.get_quota(qt_type, id)?; uapi = quota.to_if_dqblk(); copy_to_user(addr, uapi)
    Q_SETQUOTA  → uapi = copy_from_user::<IfDqblk>(addr)?; quota = DiskQuota::from_if_dqblk(&uapi); sb.quota_ops.set_quota(qt_type, id, &quota)
    Q_GETINFO   → info = sb.quota_ops.get_info(qt_type)?; copy_to_user(addr, info)
    Q_SETINFO   → info = copy_from_user(addr)?; sb.quota_ops.set_info(qt_type, &info)
    Q_SYNC      → sb.quota_ops.sync_quota(qt_type)
    _           → return Err(EINVAL)

14.15.3 VFS Enforcement Hooks

On every write(2), fallocate, create, mkdir, mknod, and symlink call, the VFS checks quotas for all three subject types:

vfs_quota_check_blocks(inode, bytes_requested) → Result:
  creds = current_task().creds
  for qt in [QuotaType::User, QuotaType::Group, QuotaType::Project]:
    id = match qt:
      User    → creds.fsuid
      Group   → creds.fsgid
      Project → inode.project_id  // already namespace-translated (canonical init-absolute ID); translated at FS_IOC_FSSETXATTR via map_projid_to_init() — no per-check translation here
    quota = inode.sb.quota_ops.get_quota(qt, id)?  // from in-memory quota cache
    new_usage = quota.bcurrent + bytes_requested
    if new_usage > quota.bhardlimit && quota.bhardlimit != 0:
      return Err(EDQUOT)  // hard limit exceeded: reject immediately
    if new_usage > quota.bsoftlimit && quota.bsoftlimit != 0:
      now = current_time_secs()
      // The get_quota() → check btime → update_quota_cache() sequence
      // must be serialized per (qt, id) to prevent a TOCTOU race: two
      // concurrent writers could both read btime == 0 and both set btime,
      // with the second overwriting the first. Serialization is provided
      // by the per-quota-entry SpinLock in the quota cache (acquired by
      // get_quota() and held until update_quota_cache() completes).
      if quota.btime == 0:
        quota.btime = now + quota.bgrace as u64  // start grace period timer
        update_quota_cache(qt, id, &quota)
      elif now > quota.btime:
        return Err(EDQUOT)  // grace period expired: reject
      // else: within grace period, allow the write
  return Ok(())

vfs_quota_check_inodes(inode, count) → Result:
  // Identical structure to vfs_quota_check_blocks but uses icurrent/isoftlimit/ihardlimit.

14.15.4 In-Memory Quota Cache

Quota accounting state is kept in a per-filesystem in-memory cache to avoid hitting the quota database file on every write. The cache structure mirrors DiskQuota with an additional dirty: bool field. Cache entries are written back to the quota file asynchronously via sync_quota(), which is called:

• Periodically by the writeback daemon (default interval: 30 seconds).
• On quotactl(Q_SYNC).
• On filesystem unmount.
• On sync(2) / syncfs(2) when the filesystem's quota is dirty.

The cache uses three per-filesystem XArrays — one per QuotaType — keyed by subject ID (u32 UID, GID, or project ID). Quota checks on the write(2) hot path use RCU read guards (lock-free, no contention between concurrent writers). Updates (usage accounting, limit changes via quotactl(Q_SETQUOTA)) acquire the XArray's internal lock on the affected entry only.

/// Per-filesystem in-memory quota cache.
///
/// Three XArrays partition by quota type so that user, group, and project
/// lookups are fully independent (no false contention). XArray provides
/// O(1) lookup by integer key with native RCU read support.
pub struct QuotaCache {
    /// User quota cache, keyed by UID.
    pub user: XArray<QuotaCacheEntry>,
    /// Group quota cache, keyed by GID.
    pub group: XArray<QuotaCacheEntry>,
    /// Project quota cache, keyed by project ID.
    pub project: XArray<QuotaCacheEntry>,
}

pub struct QuotaCacheEntry {
    pub quota: DiskQuota,
    /// True if this entry has been modified since the last writeback.
    pub dirty: bool,
}

impl QuotaCache {
    /// Look up a quota entry. RCU read — no lock, no allocation.
    /// Called on every write(2), fallocate, create, mkdir — hot path.
    pub fn get(&self, qt: QuotaType, id: u32) -> Option<RcuRef<QuotaCacheEntry>> {
        self.array_for(qt).get_rcu(id as u64)
    }

    /// Insert or update a quota entry. Acquires the XArray's internal lock
    /// for the affected slot only — does not block concurrent reads.
    pub fn set(&self, qt: QuotaType, id: u32, entry: QuotaCacheEntry) {
        self.array_for(qt).store(id as u64, entry);
    }

    fn array_for(&self, qt: QuotaType) -> &XArray<QuotaCacheEntry> {
        match qt {
            QuotaType::User    => &self.user,
            QuotaType::Group   => &self.group,
            QuotaType::Project => &self.project,
        }
    }
}

This replaces the previous RwLock<HashMap<(QuotaType, u32), DiskQuota>> design, which had three problems: (1) HashMap with integer keys violates collection policy (§3.1.13); (2) the global RwLock serialises all quota checks across all subjects; (3) the composite (QuotaType, u32) key prevents independent access by quota type. The XArray design gives O(1) lookup, lock-free RCU reads, and natural partitioning.

14.15.5 Linux Compatibility

• quotactl(2) with all seven commands (Q_QUOTAON, Q_QUOTAOFF, Q_GETQUOTA, Q_SETQUOTA, Q_GETINFO, Q_SETINFO, Q_SYNC) is fully implemented.
• The UAPI IfDqblk structure matches the Linux struct if_dqblk layout exactly (9 fields: dqb_bhardlimit through dqb_valid). The internal DiskQuota struct extends this with bgrace/igrace fields for in-kernel grace period tracking.
• quota tools (quota, quotacheck, repquota, edquota) work without modification.
• ext4, XFS, and tmpfs quota implementations are in scope for the initial release.
• Project quotas (PRJQUOTA) are supported; project IDs are stored in the inode's project_id field as the namespace-translated id — the canonical (init-namespace-absolute) project id, the analogue of a Linux kprojid_t. Translation from the caller's user-namespace project id to this stored (init-namespace-absolute) id happens ONCE, at the FS_IOC_FSSETXATTR write boundary, via UserNamespace::map_projid_to_init() — the ns-relative→global parent-relative-to-init direction, since project_id holds the canonical init-absolute project ID (Section 17.1). UmkaOS stores each user namespace's projid_map in its PARENT's terms (unlike Linux, which pre-composes the outer column to init-absolute at map-write time), so map_projid_to_init() is a per-hop WALK up the user-ns chain to init — at nesting depth ≥ 2 a single parent hop would yield an intermediate-namespace value and silently alias quotas across namespaces. The projid_map is unmapped (translation returns None) until /proc/[pid]/projid_map is written, and any unmapped hop makes the ioctl fail EINVAL (Linux invalid-project-ID parity with Linux). Because the stored project_id is already translated, the quota-check path (vfs_quota_check_blocks, above) keys the quota cache on inode.project_id DIRECTLY and performs no per-check translation — exactly as Linux keys project quota by kprojid. This is the consumer the UserNamespace.projid_map field doc refers to; the only map_projid_to_init() call site is the FS_IOC_FSSETXATTR ioctl handler.

14.15.6 Cross-References

• Section 14.1 (VFS Architecture): quota checks are inserted into the VFS dispatch layer at write, create, mkdir, mknod, and fallocate call sites.
• Section 17.1 (Containers): cgroup v2 io.max and memory.max provide resource controls complementary to quota; quota enforces per-UID/GID storage limits while cgroups enforce per-container I/O and memory limits.
• Section 15.1 (Storage): ext4, XFS, and btrfs filesystem drivers implement QuotaOps as part of their SuperBlock initialization.

---

14.16 Extended Attributes (xattr)

Extended attributes are name-value pairs associated with inodes, providing metadata beyond the standard POSIX file attributes (owner, group, mode, timestamps). They are the storage mechanism for POSIX ACLs (Section 9.2), SELinux labels, IMA hashes (Section 9.5), overlayfs whiteouts (Section 14.8), file capabilities (Section 9.9), and user-defined metadata.

UmkaOS implements the complete Linux xattr ABI: identical syscall numbers, identical namespace rules, identical size limits, and identical wire format for POSIX ACLs stored in system.posix_acl_access / system.posix_acl_default.

14.16.1 Syscall Interface

Twelve syscalls implement four operations across three path resolution variants:

Operation	Path-based (follows symlinks)	Link-based (no follow)	FD-based
Get	getxattr(path, name, value, size)	lgetxattr(path, name, value, size)	fgetxattr(fd, name, value, size)
Set	setxattr(path, name, value, size, flags)	lsetxattr(path, name, value, size, flags)	fsetxattr(fd, name, value, size, flags)
List	listxattr(path, list, size)	llistxattr(path, list, size)	flistxattr(fd, list, size)
Remove	removexattr(path, name)	lremovexattr(path, name)	fremovexattr(fd, name)

Return values: getxattr returns the number of bytes written to value (or the required buffer size if size == 0). listxattr returns the total length of the null-separated name list (or required size if size == 0). setxattr and removexattr return 0 on success.

Error codes: ENODATA (attribute not found), EEXIST (CREATE flag, attribute already exists), ERANGE (buffer too small), EPERM (namespace permission denied), ENOTSUP (filesystem does not support xattrs or namespace not valid for this inode type).

The l-variants operate on the symlink inode itself rather than following the symlink target. The f-variants use an open file descriptor, bypassing path resolution entirely.

14.16.2 XattrFlags

bitflags! {
    /// Flags for setxattr / lsetxattr / fsetxattr. Matches Linux XATTR_CREATE
    /// and XATTR_REPLACE from <linux/xattr.h>.
// kernel-internal, not KABI
    #[repr(C)]
    pub struct XattrFlags: u32 {
        /// Fail with EEXIST if the attribute already exists.
        const CREATE  = 0x1;
        /// Fail with ENODATA if the attribute does not exist.
        const REPLACE = 0x2;
        // 0 (no flags) = create or replace unconditionally.
    }
}

Setting both CREATE | REPLACE simultaneously is invalid and returns EINVAL.

14.16.3 Namespace Prefixes

Extended attribute names are partitioned into four namespaces by their prefix string. Each namespace has independent permission semantics:

14.16.3.1 user.*

User-defined attributes. No capability required.

Operation	Requirement
Get	Read permission on the file
Set	Write permission on the file

Inode type restriction: user.* xattrs are permitted only on regular files and directories. Attempts to set user.* on symlinks, device nodes, pipes, or sockets return EPERM. Rationale: symlinks must be transparent (a symlink's xattrs should not be confused with those of its target); device node xattrs would create ambiguity between the device file and the underlying device.

14.16.3.2 trusted.*

Trusted attributes for kernel subsystems and privileged daemons.

Operation	Requirement
Get	CAP_SYS_ADMIN
Set	CAP_SYS_ADMIN

Stored on disk and persistent across reboots. Examples:

• trusted.overlay.opaque — overlayfs opaque directory marker (Section 14.8)
• trusted.overlay.redirect — overlayfs rename redirect

14.16.3.3 security.*

Security labels written by LSMs and integrity subsystems.

Operation	Requirement
Set	Delegated to LSM hooks. Default (commoncap): CAP_SYS_ADMIN for generic security.* attributes; security.capability requires CAP_SETFCAP (checked while converting namespaced file capabilities). SELinux/AppArmor may impose additional type enforcement rules via lsm_call_inode_security(Setxattr, ...).
Get	Varies by LSM; SELinux allows read by any process with appropriate type enforcement

Examples:

• security.selinux — SELinux security context (Section 9.8)
• security.ima — IMA measurement hash (Section 9.5)
• security.capability — file capabilities (VFS_CAP_REVISION_3) (Section 9.9)
• security.evm — EVM HMAC over protected xattrs (Section 9.5)

14.16.3.4 system.*

System attributes for kernel-managed metadata. Two attributes are defined:

• system.posix_acl_access — POSIX access ACL (Section 9.2)
• system.posix_acl_default — POSIX default ACL (directories only)

Permission model: read follows normal file permission checks; set requires write permission plus ownership (uid == i_uid) or CAP_FOWNER.

14.16.4 Size Limits

/// Maximum length of an extended attribute name, including the namespace prefix
/// (e.g., "user." is 5 bytes of the 255-byte budget). Matches Linux XATTR_NAME_MAX.
pub const XATTR_NAME_MAX: usize = 255;

/// Maximum size of an extended attribute value in bytes (64 KiB).
/// Matches Linux XATTR_SIZE_MAX.
pub const XATTR_SIZE_MAX: usize = 65536;

/// Maximum total size of a listxattr() output buffer in bytes (64 KiB).
/// Matches Linux XATTR_LIST_MAX.
pub const XATTR_LIST_MAX: usize = 65536;

These are hard limits enforced by the VFS layer before dispatching to filesystem code. Individual filesystems may impose smaller limits (e.g., ext4 inline xattr space is limited by the inode size minus i_extra_isize).

14.16.5 VFS Dispatch Pipeline

All xattr syscalls route through InodeOps methods defined in Section 14.1. The VFS layer performs namespace permission checks and LSM hooks before dispatching to the filesystem:

1. Parse namespace prefix — extract "user.", "trusted.", "security.", or "system." from the attribute name. Unknown prefixes return EOPNOTSUPP.
2. Validate name length — reject if name.len() > XATTR_NAME_MAX.
3. Validate value size — reject if value.len() > XATTR_SIZE_MAX (set operations).
4. Check namespace permissions — verify the caller holds the required capability for the namespace (see tables above). Check inode type restriction for user.*.
5. Call LSM hooks — lsm_call_inode_security(Setxattr | Getxattr | Removexattr | Listxattr, ...) (Section 9.8). LSMs may deny the operation (e.g., SELinux type enforcement) or intercept security.* writes.
6. Dispatch to filesystem — call the InodeOps::getxattr / setxattr / listxattr / removexattr method on the filesystem driver.
7. EVM re-computation (set/remove of security.* xattrs only) — after the filesystem write succeeds, trigger EVM HMAC re-computation (Section 9.5).

14.16.6 Per-Filesystem Storage

Each filesystem implements xattr storage according to its on-disk format. The VFS layer is agnostic to the storage mechanism; it delegates entirely to InodeOps.

Filesystem	Storage mechanism	Inline capacity	Overflow strategy
ext4	Inode body (after i_extra_isize) or external xattr block	~100 bytes (256-byte inode default)	Separate 4 KiB block, shared across inodes via block refcount
XFS	Inode attribute fork (shortform, leaf, or B-tree)	~256 bytes (shortform)	B-tree of 4 KiB attr leaf blocks
Btrfs	Xattr items in the filesystem B-tree (same tree as data extent refs)	~3900 bytes (single leaf item)	Additional B-tree items (no single-xattr limit, tree grows)
tmpfs	XArray per-inode, keyed by FNV-1a hash of xattr name	Memory-only, no disk limit	Bounded by tmpfs size limit and system memory
ZFS	System Attributes (SA) in dnode bonus buffer or ZAP objects	~48 KiB (bonus buffer)	Fat ZAP (on-disk hash table)

tmpfs xattr storage: tmpfs has no backing disk, so xattrs are stored in memory. Each inode with xattrs carries an XArray<XattrEntry> keyed by fnv1a(name) as u64 with open-addressing collision resolution (same triangular probing scheme as Section 14.18). On collision, the probe sequence h, h+1, h+3, h+6, … is followed, comparing XattrEntry.name at each occupied slot. This gives O(1) lookup for the common case (no collisions) with bounded worst-case O(k) where k is the number of collisions for a given hash.

/// tmpfs xattr entry. Stored in the per-inode XArray.
pub struct TmpfsXattrEntry {
    /// Full attribute name including namespace prefix (e.g., "user.mime_type").
    /// Heap-allocated because xattr names are variable-length.
    pub name: Box<[u8]>,
    /// Attribute value. Heap-allocated, up to XATTR_SIZE_MAX bytes.
    pub value: Box<[u8]>,
}

14.16.7 POSIX ACL Wire Format

The POSIX draft ACL (Section 9.2) is stored on disk as the value of system.posix_acl_access (access ACL) and system.posix_acl_default (default ACL, directories only). The wire format is identical to Linux <linux/posix_acl_xattr.h>:

/// POSIX ACL xattr header. Appears once at the start of the xattr value.
/// All fields are little-endian on disk (Le32/Le16 wrappers enforce
/// explicit conversion at read/write boundaries).
/// **Canonical definition** of the POSIX ACL xattr wire structs (this section
/// owns the on-disk format). Field names match Linux
/// `<linux/posix_acl_xattr.h>` exactly (`a_version`, `e_tag`, `e_perm`, `e_id`).
#[repr(C, packed)]
pub struct PosixAclXattrHeader {
    /// ACL format version. Must be POSIX_ACL_XATTR_VERSION (0x0002).
    pub a_version: Le32,
}
// Packed layout: 4 bytes.
const_assert!(size_of::<PosixAclXattrHeader>() == 4);

/// POSIX_ACL_XATTR_VERSION — the only version defined by the POSIX draft standard.
pub const POSIX_ACL_XATTR_VERSION: u32 = 0x0002;

/// A single ACL entry. Follows the header; repeated N times.
/// All fields are little-endian on disk.
#[repr(C, packed)]
pub struct PosixAclXattrEntry {
    /// ACL entry tag identifying the entry type.
    pub e_tag: Le16,
    /// Permission bits: ACL_READ (0x04) | ACL_WRITE (0x02) | ACL_EXECUTE (0x01).
    pub e_perm: Le16,
    /// Qualifier: uid for ACL_USER, gid for ACL_GROUP.
    /// ACL_UNDEFINED_ID (0xFFFFFFFF) for USER_OBJ, GROUP_OBJ, MASK, OTHER.
    pub e_id: Le32,
}
// Packed layout: 2 + 2 + 4 = 8 bytes.
const_assert!(size_of::<PosixAclXattrEntry>() == 8);

/// ACL entry tag values. Values match Linux `include/uapi/linux/posix_acl.h`.
/// NOTE: Although the values are powers of two, these are enum
/// discriminants, NOT combinable flags — each `e_tag` must be exactly one
/// of these values.
pub const ACL_USER_OBJ:  u16 = 0x01; // File owner
pub const ACL_USER:      u16 = 0x02; // Named user (uid in qualifier)
pub const ACL_GROUP_OBJ: u16 = 0x04; // File owning group
pub const ACL_GROUP:     u16 = 0x08; // Named group (gid in qualifier)
pub const ACL_MASK:      u16 = 0x10; // Upper bound on USER/GROUP/GROUP_OBJ permissions
pub const ACL_OTHER:     u16 = 0x20; // Everyone else

/// `e_perm` permission bits (same values as the low 3 bits of `st_mode`).
/// Values match Linux `include/uapi/linux/posix_acl.h`.
pub const ACL_READ:    u16 = 0x04;
pub const ACL_WRITE:   u16 = 0x02;
pub const ACL_EXECUTE: u16 = 0x01;

/// Sentinel value for entries that do not reference a specific uid/gid
/// (`USER_OBJ`, `GROUP_OBJ`, `MASK`, `OTHER`). Linux defines it as `(-1)`;
/// `0xFFFF_FFFF` is the same value as an unsigned 32-bit qualifier.
pub const ACL_UNDEFINED_ID: u32 = 0xFFFF_FFFF;

Wire layout: 4-byte header followed by N 8-byte entries. Total size = 4 + 8 * N bytes.

Minimum ACL: 3 entries (USER_OBJ, GROUP_OBJ, OTHER) = 28 bytes. This is the "minimal ACL" equivalent to standard POSIX mode bits.

Extended ACL: When named users or groups are present, a MASK entry is mandatory. The mask defines the maximum permissions for ACL_USER, ACL_GROUP, and ACL_GROUP_OBJ entries (the "effective permissions" are entry.e_perm & mask.e_perm).

14.16.8 chmod / ACL Mask Interaction

When chmod() is called on a file that has a POSIX access ACL, the ACL must be updated to reflect the new mode bits. The POSIX draft standard defines this mapping:

Mode bits	ACL entry updated
Owner bits (mode >> 6) & 0o7	ACL_USER_OBJ.e_perm
Group bits (mode >> 3) & 0o7	ACL_MASK.e_perm (NOT ACL_GROUP_OBJ)
Other bits (mode) & 0o7	ACL_OTHER.e_perm

The group bits of the file mode always correspond to ACL_MASK, not ACL_GROUP_OBJ. This is a common source of confusion but is required by POSIX.1e: the mask entry is the upper bound on group-class permissions, and ls -l displays the mask as the group permission bits.

Conversely, when an ACL is set via setxattr("system.posix_acl_access", ...), the file's mode bits are updated to match: owner bits from ACL_USER_OBJ.e_perm, group bits from ACL_MASK.e_perm, other bits from ACL_OTHER.e_perm.

14.16.9 Default ACL Inheritance

When creating a new inode in a directory that has system.posix_acl_default set:

New file creation: 1. The directory's default ACL becomes the new file's access ACL. 2. The ACL_MASK entry (if present) is ANDed with the umask-adjusted creation mode to produce the file's effective permissions. 3. The file's mode bits are set from the resulting ACL (owner from USER_OBJ, group from MASK, other from OTHER). 4. The new file does NOT receive a default ACL (only directories inherit defaults).

New directory creation: 1. Same as file creation for the access ACL. 2. Additionally, the parent's default ACL is copied as the new directory's own default ACL, ensuring recursive inheritance for all future children.

No default ACL: If the parent directory has no system.posix_acl_default xattr, standard umask-based permission inheritance applies and no ACL is created on the new inode.

14.16.10 EVM Integration

EVM (Extended Verification Module) protects security-critical xattrs against offline tampering (Section 9.5).

Protected xattr set: security.selinux, security.ima, security.capability, and any other security.* xattr registered with EVM at boot.

Flow on security.* xattr modification: 1. VFS calls InodeOps::setxattr() to persist the new value. 2. On success, VFS acquires the per-inode evm_lock (spinlock). 3. VFS concatenates the inode number and all protected xattr values in a deterministic order. 4. VFS computes HMAC-SHA3-256 over the concatenation using the boot-derived EVM key. 5. VFS writes the resulting HMAC as the value of security.evm. 6. VFS releases evm_lock.

Lock ordering: The evm_lock is acquired AFTER the inode's i_rwsem (which protects xattr storage). The ordering is: i_rwsem (exclusive, acquired by setxattr() VFS path) → evm_lock (spinlock, acquired in step 2). Reversing this order would deadlock: evm_lock protects only the HMAC recomputation (steps 3-5), not the underlying xattr storage write (step 1). Concurrent getxattr("security.evm") reads do NOT acquire evm_lock — they read the stored HMAC value directly. This is safe because setxattr holds i_rwsem exclusive, which prevents concurrent setxattr (but not concurrent getxattr, which takes i_rwsem shared). A getxattr concurrent with step 5 may see either the old or new HMAC — both are valid (the old HMAC matches the old xattr set; the new HMAC matches the new xattr set).

Appraisal on file open: When a file is opened, EVM re-computes the HMAC and compares it against the stored security.evm value. Mismatch returns EINTEGRITY (or EPERM when evm_mode is set to enforce).

14.16.11 Performance Budget

Operation	Path class	Typical cost	Notes
getxattr (inline, ext4)	Warm	~500 ns	Inode already in page cache; inline scan of i_extra_isize region
getxattr (external block, ext4)	Cold	~5 us	Requires reading the shared xattr block from disk
setxattr (inline, ext4)	Warm	~1 us	Journal transaction for inode update
listxattr	Cold	~2 us	Iterates all xattr entries in inode + overflow
LSM hook overhead per xattr op	Hot	~20 ns	Static dispatch through LSM hook array (Section 9.8)
EVM HMAC re-computation	Warm	~3 us	HMAC-SHA3-256 over protected xattr set; dominated by hash computation
tmpfs getxattr (XArray lookup)	Warm	~100 ns	In-memory XArray traversal, no disk I/O

Hot-path note: xattr operations are not on the per-packet or per-syscall hot path. The most frequent xattr consumer is the LSM label check performed when a file opens, which caches the resolved label in the inode's LSM blob and do not re-read the xattr on every access. The performance budget above reflects the actual xattr syscall cost, not the cached LSM check cost (which is ~5 ns via the blob pointer).

14.17 Pipes and FIFOs

Pipes (pipe(2), pipe2(2)) and named FIFOs (mkfifo(2)) are anonymous unidirectional byte streams. They are the oldest and most widely used IPC primitive in UNIX.

14.17.1 Pipe Data Buffer

The pipe data buffer uses the page-array model defined in Section 17.3. Each pipe holds an array of page references (PipePage), supporting partial-page writes for small messages and zero-copy page gifting for vmsplice(SPLICE_F_GIFT). The PipeBuffer struct provides:

• Default 65536 bytes (16 pages), matching Linux default
• Lock-free single-writer fast path with active_writer CAS
• Mutex-protected multi-writer slow path for POSIX atomicity
• Inline storage for the common case (16 pages), heap fallback for fcntl(F_SETPIPE_SZ) beyond 64 KB
• Seqlock-based resize safety for concurrent fcntl(F_SETPIPE_SZ)

Key PipeBuffer fields (summary; full definition in Section 17.3): - pages: ArrayVec<PipePage, 16> — page ring (inline for default 64 KiB) - r_idx: u32, w_idx: u32 — read/write indices into the page ring - capacity: u32 — current capacity in bytes - active_writer: AtomicU64 — CAS-based single-writer detection

See Section 17.3 for the complete struct definition, write/read algorithms, memory ordering rationale, and resize protocol.

14.17.2 Capacity and fcntl(F_SETPIPE_SZ)

Default pipe capacity: 65536 bytes (matches Linux default).

fcntl(F_SETPIPE_SZ, size) resizes the pipe: - Rounds up to the next power of 2 (minimum 4096 bytes) - Maximum: /proc/sys/fs/pipe-max-size (default 1MB, same as Linux) - Requires CAP_SYS_RESOURCE to exceed /proc/sys/fs/pipe-max-size - Data currently in the pipe is preserved (pages migrated to new array) - If the new size is smaller than current content: EBUSY

fcntl(F_GETPIPE_SZ) returns the current capacity.

14.17.3 MPSC Pipes (Multiple Writers)

When more than one process/thread writes to the same pipe (e.g., shell { cmd1; cmd2; } | cmd3), the single-writer fast path cannot be used. UmkaOS detects multiple writers via PipeBuffer.writer_count: AtomicU32:

• writer_count == 1: lock-free single-writer fast path (CAS on active_writer)
• writer_count > 1: writer acquires PipeBuffer.ring_lock: Mutex<()> before writing

Writes <= PIPE_BUF (4096 bytes) are always atomic (no interleaving with other writers) -- same guarantee as POSIX.

14.17.4 O_DIRECT Pipe Mode

pipe2(O_DIRECT): each write() is a discrete message; read() returns exactly one message. Implemented by prepending a 4-byte length header before each message in the page array:

/// O_DIRECT pipe message header (4 bytes, native endian).
/// Followed immediately by `len` bytes of payload.
/// Alignment: none required (page data is byte-addressable).
/// packed is defensive: ensures no trailing padding if fields are added later.
#[repr(C, packed)]
pub struct PipeMessageHdr {
    pub len: u32,
}
const_assert!(size_of::<PipeMessageHdr>() == 4);

Maximum message size: PIPE_BUF (4096 bytes) for atomic writes.

14.17.5 Named FIFOs (mkfifo)

Named FIFOs use the same PipeBuffer struct, but with a VFS inode for pathname lookup:

• mkfifo(path, mode): creates a VFS inode of type InodeKind::Fifo
• open(path, O_RDONLY): blocks until a writer opens (unless O_NONBLOCK)
• open(path, O_WRONLY): blocks until a reader opens (unless O_NONBLOCK)
• Once both ends are open: identical semantics to anonymous pipe

A FIFO is a VFS node (VfsNode) that, when opened, creates a reference to an existing PipeBuffer or creates a new one. Multiple readers and writers can open a FIFO; the reader_count and writer_count fields track opens/closes. Writers use the multi-writer slow path when concurrent writes are detected. When the last reader and last writer close, the buffer is freed.

14.17.6 Splice and Zero-Copy

UmkaOS's page-array pipe model (PipeBuffer) uses the same fundamental design as Linux's struct pipe_buffer array: each pipe page is a reference to a physical page with offset and length. This enables true zero-copy splice operations via page-reference transfer.

Pipe-to-pipe splice (splice(pipe_fd_in, pipe_fd_out)): Transfers page references from the source pipe to the destination pipe. The source pipe's PipePage entries are moved (not copied) to the destination, incrementing the underlying page refcount. No data copy occurs -- only metadata (page pointer, offset, length) is transferred. This matches Linux's pipe-to-pipe splice semantics exactly.

File-to-pipe splice (splice(file_fd, pipe_fd)): The filesystem's FileOps::splice_read populates pipe pages directly from page cache pages, transferring page references (incrementing refcount) without copying data. The pipe's PipePage entries point directly into the page cache.

Pipe-to-socket splice (splice(pipe_fd, socket_fd)): The network stack receives page references from the pipe and uses scatter-gather DMA to transmit directly from the pipe's pages (zero kernel-side copy). Each PipePage maps to a scatter-gather entry for the NIC.

Pipe-to-file splice (splice(pipe_fd, file_fd)): The filesystem's FileOps::splice_write transfers page references from the pipe into the page cache (for filesystems that support it) or copies data from pipe pages to page cache pages.

vmsplice zero-copy (vmsplice(pipe_fd, iov, SPLICE_F_GIFT)): When SPLICE_F_GIFT is set, the user pages described by the iovec are unmapped from the sender's address space and gifted to the pipe as PipePage entries with is_gifted == true. The reader can then access the data without any copy. Without SPLICE_F_GIFT, data is copied from user pages into pipe pages.

14.17.7 Linux Compatibility

• Default capacity 65536 bytes: identical to Linux
• F_SETPIPE_SZ / F_GETPIPE_SZ: identical semantics
• PIPE_BUF = 4096 bytes: POSIX required, identical
• O_DIRECT pipe mode: identical to Linux 3.4+
• pipe2(O_CLOEXEC | O_NONBLOCK | O_DIRECT): all flags supported
• Splice semantics: identical to Linux (page-reference transfer)
• /proc/sys/fs/pipe-max-size: identical default (1MB), same permission model
• Signal on broken pipe: SIGPIPE + EPIPE on write to pipe with no readers
• select()/poll()/epoll(): EPOLLIN when data available, EPOLLOUT when space available, EPOLLHUP on last writer close

---

14.18 Pseudo-Filesystems

Pseudo-filesystems are RAM-resident virtual filesystems that expose kernel state to userspace. Unlike disk-backed filesystems, they have no persistent storage — all content is generated dynamically by the kernel on read and consumed on write. UmkaOS provides a common registration framework and six standard pseudo-filesystems required for Linux compatibility.

procfs and sysfs are specified in Section 14.19. tmpfs and devtmpfs are built into the VFS core (Section 14.1). cgroupfs is specified in Section 17.2, and configfs in Section 14.12. This section covers the remaining pseudo-filesystems needed for complete Linux workload support: debugfs, tracefs, hugetlbfs, bpffs, securityfs, and efivarfs.

14.18.1 Common Registration Framework

All pseudo-filesystems share a uniform registration path through the VFS layer (Section 14.1). Each pseudo-fs defines a static PseudoFsType descriptor and submits it to the VFS filesystem registry during kernel init (Section 2.3).

/// Registration descriptor for a pseudo-filesystem type.
///
/// Each pseudo-fs defines exactly one static instance. The VFS layer stores
/// registered types in an XArray keyed by a monotonic u64 registration
/// sequence number. Name-based lookup (mount -t <name>) walks the XArray
/// linearly — acceptable because the total number of filesystem types is
/// small (<50) and registration/mount are cold-path operations.
pub struct PseudoFsType {
    /// Filesystem type name as it appears in mount(2) and /proc/filesystems.
    /// e.g., "debugfs", "tracefs", "hugetlbfs", "bpf", "securityfs", "efivarfs".
    pub name: &'static str,

    /// Filesystem flags. Pseudo-filesystems typically set NODEV | NOEXEC | NOSUID
    /// to prevent device node creation, executable mapping, and setuid escalation.
    pub fs_flags: FsFlags,

    /// Filesystem magic number returned by statfs(2). Each pseudo-fs has a
    /// unique magic defined by the Linux UAPI (include/uapi/linux/magic.h).
    pub magic: u32,

    /// Populate the superblock and create the root inode. Called once per mount.
    /// The implementation creates the root directory inode with appropriate mode
    /// and ownership, then populates any initial directory structure.
    pub populate_super: fn(&mut SuperBlock) -> Result<(), VfsError>,

    /// Accepted mount options. The VFS parses `-o key=value` pairs from mount(2)
    /// and validates them against this table before calling `populate_super`.
    pub mount_opts: &'static [MountOptDesc],
}

bitflags! {
    /// Filesystem-level flags applied at mount time.
    pub struct FsFlags: u32 {
        /// No device special files may be created or accessed on this filesystem.
        const NODEV  = 1 << 0;
        /// No files on this filesystem may be executed (mmap PROT_EXEC denied).
        const NOEXEC = 1 << 1;
        /// Setuid and setgid bits are ignored for all files on this filesystem.
        const NOSUID = 1 << 2;
        /// This filesystem may be mounted inside a non-initial user namespace.
        /// Only set for filesystems that are safe for unprivileged mounting
        /// (e.g., tmpfs, procfs subset). None of the six pseudo-filesystems
        /// in this section set this flag — they all require initial namespace.
        const USERNS_MOUNT = 1 << 3;
    }
}

/// Descriptor for a single mount option accepted by a pseudo-filesystem.
pub struct MountOptDesc {
    /// Option name (e.g., "pagesize", "mode", "uid").
    pub name: &'static str,
    /// Type of the option value.
    pub kind: MountOptKind,
}

/// Value type for a mount option.
pub enum MountOptKind {
    /// Boolean flag (present = true, absent = false). Example: "noexec".
    Flag,
    /// Unsigned 64-bit integer. Example: "size=1073741824".
    U64,
    /// Unsigned 32-bit integer. Example: "uid=1000", "mode=0700".
    U32,
    /// String value. Example: (currently unused by pseudo-fs, reserved).
    Str,
}

A duplicate filesystem name is rejected with EBUSY. Built-in pseudo-filesystems remain registered for the kernel's lifetime; unregistration is unsupported.

14.18.1.1 PseudoInode and File Operations

All pseudo-filesystems share a simplified inode representation for RAM-backed directory trees. Unlike disk-backed inodes (Section 14.1), pseudo-inodes carry no block mappings, no page cache association, and no filesystem-specific opaque data.

/// Simplified inode for RAM-backed pseudo-filesystems.
///
/// Pseudo-inodes are allocated on first access and freed when the
/// dentry is removed. They are never written to disk.
pub struct PseudoInode {
    /// Inode number. Allocated from a per-superblock `AtomicU64`
    /// counter starting at 2 (inode 1 is reserved for the root).
    /// u64 counters never wrap within the kernel's operational
    /// lifetime (50+ years at billions of allocations per second).
    pub ino: u64,
    /// POSIX file mode (type + permission bits).
    pub mode: u32,
    /// Owner UID.
    pub uid: Uid,
    /// Owner GID.
    pub gid: Gid,
    /// Access time.
    pub atime: Timespec,
    /// Modification time.
    pub mtime: Timespec,
    /// Status change time.
    pub ctime: Timespec,
    /// Inode content, determined by the file type.
    pub data: PseudoInodeData,
}

/// Content discriminant for pseudo-inodes.
pub enum PseudoInodeData {
    /// Directory: children stored in `HashedXArray<DirEntry>`, a wrapper
    /// that encapsulates FNV-1a hashing + triangular probing + tombstone
    /// protocol on top of XArray. This is necessary because XArray's built-in
    /// `xa_store(key, value)` overwrites existing entries at the same key —
    /// using raw `xa_store(fnv1a(name), entry)` would silently lose entries
    /// on hash collision.
    ///
    /// `HashedXArray<V>` provides:
    ///   - `insert(name: &[u8], value: V) -> Result<(), Exists>`
    ///   - `lookup(name: &[u8]) -> Option<&V>`
    ///   - `remove(name: &[u8]) -> Option<V>`
    ///
    /// Implementation: on insert, hash `h = fnv1a(name) as u64`. If the slot
    /// at `h` is occupied by a different name, probe `h+1, h+3, h+6, ...`
    /// (triangular probing: offset k = k*(k+1)/2). Lookup uses the same
    /// probe sequence, comparing `DirEntry.name` at each occupied slot.
    /// Remove stores a tombstone sentinel (a DirEntry with empty name) so
    /// the probe chain is not broken. Periodic compaction removes tombstones
    /// when the tombstone ratio exceeds 25%.
    ///
    /// Common-case (no collision): single XArray lookup, O(1). With 64-bit
    /// FNV-1a, collisions are negligible for directories under ~2^32 entries.
    Directory(HashedXArray<DirEntry>),
    /// Regular file: read/write behavior defined by the `PseudoFileOps`
    /// implementation provided at file creation time.
    RegularFile(&'static dyn PseudoFileOps),
    /// Symbolic link: target path stored inline.
    Symlink(Box<[u8]>),
}

/// Directory entry within a pseudo-filesystem directory.
pub struct DirEntry {
    /// Entry name (variable-length, heap-allocated).
    pub name: Box<[u8]>,
    /// Inode number of the target.
    pub ino: u64,
    /// File type for getdents64 optimization. Values match Linux UAPI
    /// `include/uapi/linux/dirent.h`:
    ///   DT_UNKNOWN = 0, DT_FIFO = 1, DT_CHR = 2, DT_DIR = 4,
    ///   DT_BLK = 6, DT_REG = 8, DT_LNK = 10, DT_SOCK = 12.
    pub d_type: u8,
}

/// Collision-safe name-to-value map layered on `XArray`. `XArray`'s `xa_store`
/// overwrites on key equality, so hashing names directly into it would silently
/// drop entries whose 64-bit hashes collide. `HashedXArray` resolves collisions
/// explicitly: FNV-1a hash + open-addressed triangular probing + tombstones.
///
/// Contract:
///   - `insert(name, value)`: place `value` at the first free or tombstone slot
///     on the probe chain for `name`; `EEXIST` if a live entry already matches.
///   - `lookup(name)`: walk the probe chain, comparing stored names, until a
///     match, an empty slot (miss), or the chain end.
///   - `remove(name)`: replace the matching slot with a tombstone (keeping the
///     probe chain intact) and return the value.
///
/// On insert, hash `h = fnv1a(name)`; if slot `h` holds a different name, probe
/// `h + k*(k+1)/2` for `k = 1, 2, ...` (triangular). Compaction removes
/// tombstones when their ratio exceeds 25%. Common case (no collision): one
/// `XArray` access, O(1). `&self` mutation is sound: the backing `XArray`
/// provides its own interior locking.
pub struct HashedXArray<V> {
    /// Backing store: probe-slot index (masked to capacity) to an occupied
    /// slot. Absent keys are empty slots. Integer-keyed `XArray` per the
    /// collection policy (RCU-friendly reads).
    slots: XArray<HashedSlot<V>>,
    /// Number of live (non-tombstone) entries. u64 (50-year rule).
    live: u64,
    /// Number of tombstone slots awaiting compaction. u64.
    tombstones: u64,
}

/// One occupied slot of a `HashedXArray`. A tombstone is a slot whose `value`
/// is `None`, which keeps a probe chain unbroken after a removal.
struct HashedSlot<V> {
    /// Entry name (the probe key).
    name: Box<[u8]>,
    /// Stored value; `None` in a tombstone slot.
    value: Option<V>,
}

impl<V> HashedXArray<V> {
    /// Insert `value` under `name`. `Err(Errno::EEXIST)` if a live entry with
    /// the same name already exists.
    pub fn insert(&self, name: &[u8], value: V) -> Result<(), Errno> { /* ... */ }
    /// Look up the value for `name`, or `None` if absent.
    pub fn lookup(&self, name: &[u8]) -> Option<&V> { /* ... */ }
    /// Remove and return the value for `name` (leaving a tombstone), or `None`.
    pub fn remove(&self, name: &[u8]) -> Option<V> { /* ... */ }
}

/// Callback trait for pseudo-filesystem regular files.
///
/// Each file in a pseudo-filesystem implements this trait to define its
/// read/write behavior. Implementations are typically stateless — they
/// read from or write to kernel data structures referenced through the
/// inode's subsystem-specific context.
pub trait PseudoFileOps: Send + Sync {
    /// Read data from this pseudo-file into `buf` starting at `offset`.
    /// Returns the number of bytes written to `buf`.
    fn read(
        &self,
        inode: &PseudoInode,
        buf: &mut [u8],
        offset: u64,
    ) -> Result<usize, Errno>;

    /// Write data from `buf` to this pseudo-file at `offset`.
    /// Returns the number of bytes consumed from `buf`.
    fn write(
        &self,
        inode: &PseudoInode,
        buf: &[u8],
        offset: u64,
    ) -> Result<usize, Errno>;

    /// Called when the file is opened. Optional initialization.
    /// Default: no-op (returns Ok).
    fn open(&self, _inode: &PseudoInode) -> Result<(), Errno> {
        Ok(())
    }

    /// Called when the last fd referencing this file is closed.
    /// Default: no-op.
    fn release(&self, _inode: &PseudoInode) {}
}

14.18.1.2 Helper Functions

Convenience functions for creating and removing entries in pseudo-filesystem directories. Used internally by debugfs, tracefs, securityfs, and bpffs.

/// Create a regular file as a child of `parent`.
///
/// Allocates a new `PseudoInode` with `PseudoInodeData::RegularFile(ops)`,
/// inserts a `DirEntry` into the parent's `XArray`, and creates a VFS
/// dentry linking the two.
///
/// # Errors
/// - `EEXIST`: a child with the same name already exists.
/// - `ENOMEM`: inode or dentry allocation failed.
pub fn pseudo_create_file(
    parent: &PseudoInode,
    name: &[u8],
    mode: u16,
    ops: &'static dyn PseudoFileOps,
) -> Result<Arc<PseudoInode>, VfsError>;

/// Create a subdirectory as a child of `parent`.
///
/// Allocates a new `PseudoInode` with `PseudoInodeData::Directory(HashedXArray::new())`.
/// The new directory starts empty.
pub fn pseudo_create_dir(
    parent: &PseudoInode,
    name: &[u8],
    mode: u16,
) -> Result<Arc<PseudoInode>, VfsError>;

/// Remove a child entry from `parent` by name.
///
/// Looks up the child in the parent's directory XArray. If the target is a
/// directory, it must be empty (returns `ENOTEMPTY` otherwise).
///
/// **Tombstone protocol**: Because the directory uses open-addressing
/// collision resolution, naive deletion (clearing a slot to empty) would
/// break probe chains for entries inserted after the deleted entry via
/// collision. On removal, the slot is set to a tombstone sentinel
/// (`DirEntry::TOMBSTONE`) that:
/// - Is treated as "occupied" during probing (probe chains remain intact)
/// - Is skipped during name matching (not returned by lookup)
/// - Is overwritten by new inserts (reclaimed lazily)
///
/// When the directory's tombstone count exceeds 25% of occupied slots,
/// the directory is compacted: a new `HashedXArray` is allocated, all
/// live entries are inserted into it, the directory's `HashedXArray`
/// pointer is atomically swapped via `rcu_assign_pointer`, and the old
/// `HashedXArray` is freed after an RCU grace period. Concurrent RCU
/// readers always see a consistent directory (either old or new, never
/// partially rebuilt). Cold path, under the parent inode's directory lock.
pub fn pseudo_remove(
    parent: &PseudoInode,
    name: &[u8],
) -> Result<(), VfsError>;

14.18.2 debugfs — Kernel Debug Filesystem

Mount point: /sys/kernel/debug (mount -t debugfs debugfs /sys/kernel/debug) Magic: 0x64626720 (DEBUGFS_MAGIC) Flags: NODEV | NOEXEC | NOSUID

debugfs exposes kernel-internal debugging data. It carries no stable ABI guarantee: files may appear, disappear, or change format between kernel versions. Userspace tools must handle missing files gracefully.

Access control: mounted with mode=0700 by default, restricting access to CAP_SYS_ADMIN (Section 9.9). Distributions may remount with mode=0755 for read-only debug access, but this is a policy decision outside kernel scope.

14.18.2.1 debugfs Registration

static DEBUGFS_TYPE: PseudoFsType = PseudoFsType {
    name: "debugfs",
    fs_flags: FsFlags::NODEV | FsFlags::NOEXEC | FsFlags::NOSUID,
    magic: 0x64626720,
    populate_super: debugfs_populate_super,
    mount_opts: &[
        MountOptDesc { name: "uid",  kind: MountOptKind::U32 },
        MountOptDesc { name: "gid",  kind: MountOptKind::U32 },
        MountOptDesc { name: "mode", kind: MountOptKind::U32 },
    ],
};

14.18.2.2 debugfs Kernel API

Kernel subsystems create debugfs entries during their initialization. All functions are no-ops (returning a dummy handle) if debugfs is not mounted, ensuring subsystem init never fails due to debugfs unavailability.

/// Handle to a debugfs directory. Opaque to callers.
/// Internally holds the dentry reference for the directory inode.
pub struct DebugfsDir {
    dentry: Arc<Dentry>,
}

/// Handle to a debugfs file or value entry. Opaque to callers.
pub struct DebugfsEntry {
    dentry: Arc<Dentry>,
}

/// Create a directory under the debugfs root (or under `parent`).
/// Returns `DebugfsDir` used as the parent for subsequent entries.
pub fn debugfs_create_dir(
    name: &str,
    parent: Option<&DebugfsDir>,
) -> Result<DebugfsDir, VfsError>;

/// Create a file with custom read/write operations.
/// `mode` is the POSIX permission bits (e.g., 0o444 for read-only).
/// `fops` provides the read/write/open/release callbacks.
pub fn debugfs_create_file(
    name: &str,
    parent: &DebugfsDir,
    mode: u16,
    fops: &'static FileOps,
) -> Result<DebugfsEntry, VfsError>;

/// Create a file that reads/writes a single `AtomicU32` value.
/// Read returns the decimal ASCII representation; write parses decimal ASCII.
pub fn debugfs_create_u32(
    name: &str,
    parent: &DebugfsDir,
    mode: u16,
    value: &'static AtomicU32,
) -> Result<DebugfsEntry, VfsError>;

/// Create a file that reads/writes a single `AtomicU64` value.
pub fn debugfs_create_u64(
    name: &str,
    parent: &DebugfsDir,
    mode: u16,
    value: &'static AtomicU64,
) -> Result<DebugfsEntry, VfsError>;

/// Create a file that reads/writes a single `AtomicBool` value.
/// Read returns "Y\n" or "N\n"; write accepts "1"/"Y"/"y" or "0"/"N"/"n".
pub fn debugfs_create_bool(
    name: &str,
    parent: &DebugfsDir,
    mode: u16,
    value: &'static AtomicBool,
) -> Result<DebugfsEntry, VfsError>;

/// Remove a single debugfs entry (file or empty directory).
pub fn debugfs_remove(entry: DebugfsEntry);

/// Remove a directory and all entries beneath it recursively.
/// Safe to call from module teardown — removes all entries created
/// by the subsystem in one call.
pub fn debugfs_remove_recursive(dir: DebugfsDir);

14.18.2.3 Lockdown Integration

When kernel lockdown (Section 9.3) is active, debugfs access is restricted based on the lockdown level:

Lockdown Level	debugfs Behavior
none	Full read/write access (subject to mount permissions)
integrity	Read-only: writes to all debugfs files return EPERM
confidentiality	Fully disabled: mount returns EPERM; all reads/writes return EPERM

The debugfs=off boot parameter disables debugfs entirely (equivalent to confidentiality lockdown for debugfs). When disabled, debugfs_create_* functions return dummy handles and all file operations are no-ops, ensuring subsystem initialization never fails due to debugfs unavailability.

14.18.2.4 Standard debugfs Directories

Created at boot by their respective subsystems:

Directory	Subsystem	Content
/sys/kernel/debug/block/	Block layer (Section 15.2)	Per-device I/O stats, request queue state
/sys/kernel/debug/dma_buf/	DMA subsystem (Section 4.14)	DMA-buf allocation tracking
/sys/kernel/debug/clk/	Clock framework (Section 2.24)	Clock tree rates, enable counts
/sys/kernel/debug/regulator/	Regulator framework (Section 13.27)	Voltage/current state per regulator
/sys/kernel/debug/ieee80211/	WiFi subsystem (Section 13.15)	Per-PHY/per-STA debug counters
/sys/kernel/debug/bluetooth/	Bluetooth (Section 13.14)	HCI trace data

14.18.3 tracefs — Tracing Filesystem

Mount point: /sys/kernel/tracing (historically /sys/kernel/debug/tracing; UmkaOS creates a compatibility symlink at the legacy path) Magic: 0x74726163 (TRACEFS_MAGIC) Flags: NODEV | NOEXEC | NOSUID

tracefs exposes the tracepoint event catalog and ftrace ring buffers to userspace tracing tools (perf, bpftrace, trace-cmd). It is the primary interface for Section 20.2.

Access control: CAP_SYS_ADMIN or CAP_PERFMON for most operations. Reading available_events and event format files requires only read permission on the tracefs mount (allows unprivileged discovery of available tracepoints without enabling them).

14.18.3.1 tracefs Registration

static TRACEFS_TYPE: PseudoFsType = PseudoFsType {
    name: "tracefs",
    fs_flags: FsFlags::NODEV | FsFlags::NOEXEC | FsFlags::NOSUID,
    magic: 0x74726163,
    populate_super: tracefs_populate_super,
    mount_opts: &[
        MountOptDesc { name: "uid",  kind: MountOptKind::U32 },
        MountOptDesc { name: "gid",  kind: MountOptKind::U32 },
        MountOptDesc { name: "mode", kind: MountOptKind::U32 },
    ],
};

14.18.3.2 tracefs Directory Structure

/sys/kernel/tracing/
    available_events           # one "subsystem:event" per line
    available_tracers          # "nop function function_graph"
    current_tracer             # write tracer name to activate
    trace                      # human-readable trace output (snapshot)
    trace_pipe                 # streaming trace output (blocks on read)
    tracing_on                 # "1"=enabled, "0"=disabled (write to toggle)
    buffer_size_kb             # per-CPU ring buffer size (write to resize)
    events/                    # per-subsystem event directories
        sched/                 # scheduler events
            sched_switch/
                enable         # "1"=trace, "0"=disable
                filter         # BPF-style filter expression
                format         # printf-style field description
                id             # tracepoint numeric ID (u32)
        syscalls/              # syscall enter/exit events
        net/                   # networking events
        block/                 # block I/O events
        irq/                   # interrupt events
    per_cpu/                   # per-CPU trace data
        cpu0/
            trace              # CPU-specific trace snapshot
            trace_pipe         # CPU-specific streaming trace
            stats              # entries, overrun, commit overrun, bytes
    instances/                 # named trace instances (independent buffers)

14.18.3.3 tracefs Ring Buffer

Each CPU has a dedicated ring buffer (per-CPU, no lock contention). The buffer size defaults to 1408 KB per CPU (matching Linux default) and is configurable via buffer_size_kb. Ring buffer allocation uses the page allocator (Section 4.2); each buffer is a set of linked pages, not a single contiguous allocation (avoids high-order allocation failures on fragmented systems).

/// Per-CPU trace ring buffer. One instance per CPU per trace instance.
pub struct TraceRingBuffer {
    /// Per-CPU buffer pages. Each entry is a page-sized ring segment.
    /// Pages are linked in a circular list for wraparound.
    pub pages: PerCpu<TraceBufferPages>,
    /// Buffer size in KB (per CPU). Default: 1408.
    pub size_kb: AtomicU32,
    /// Overrun counter: events dropped due to full buffer (per CPU, u64).
    pub overrun: PerCpu<AtomicU64>,
    /// Total events written (per CPU, u64).
    pub entries: PerCpu<AtomicU64>,
}

/// The page set backing one CPU's slice of a `TraceRingBuffer`. A ring of
/// page-sized segments written head-to-tail by the local CPU (lock-free,
/// single-producer) and drained by readers. Pages are pre-allocated at buffer
/// (re)size; the ring wraps at `page_count`.
pub struct TraceBufferPages {
    /// Pointers to the page-sized ring segments (one per slot), allocated at
    /// buffer (re)size and traversed circularly for wraparound.
    pages: Box<[NonNull<u8>]>,
    /// Number of segments in the ring. u64 (50-year rule).
    page_count: u64,
    /// Producer write cursor, encoded `(segment_index << 16) | intra_page_off`.
    /// u64 — monotonic; wraps far beyond the 50-year horizon at any real rate.
    head: AtomicU64,
    /// Consumer read cursor, same encoding. u64.
    tail: AtomicU64,
}

14.18.3.4 Tracepoint Integration

Each tracepoint registered via the DECLARE_TRACEPOINT! macro (Section 20.2) automatically gets an events/<subsystem>/<name>/ directory in tracefs. The id file provides a stable u32 identifier that perf_event_open() (Section 20.8) uses to attach to tracepoints programmatically.

register_tracepoint delegates node creation to the tracefs owner (the pseudo-filesystem layer) through tracefs_add_event:

/// Create the tracefs entry for a registered tracepoint under
/// `events/<category>/<name>/`, exposing the read-only `id` (decimal tracepoint
/// id) and `format` (argument schema) pseudo-files. The per-category directory
/// is created idempotently (shared by sibling tracepoints); the per-tracepoint
/// directory is created once. Internally uses `pseudo_create_dir` /
/// `pseudo_create_file`; the `id`/`format` files are backed by tracefs event
/// ops that render their content from the tracepoint registry on read (the
/// content is fixed at registration — `id` from the assigned number, `format`
/// from the `&'static [TracepointArg]` schema).
///
/// Called from `register_tracepoint` ([Section 20.2](20-observability.md#stable-tracepoint-abi)) at boot or
/// module load (process context only). Returns `VfsError` if tracefs is not yet
/// mounted; the caller treats that as non-fatal — the tracepoint stays
/// registered and callable, and its entry is recreated on the next tracefs
/// mount by replaying the tracepoint registry.
pub fn tracefs_add_event(tp: &StableTracepoint, id: u32) -> Result<(), VfsError>;

14.18.3.5 Trace Instances

Named trace instances (mkdir /sys/kernel/tracing/instances/mytracer) create independent ring buffers with their own events/, trace, trace_pipe, and per_cpu/ directories. This allows multiple concurrent tracing sessions (e.g., one for system-wide scheduler tracing, another for application-specific I/O tracing) without interference.

14.18.4 hugetlbfs — Huge Page Filesystem

Mount point: /dev/hugepages (or any user-chosen mount point) Magic: 0x958458f6 (HUGETLBFS_MAGIC) Flags: NODEV | NOSUID

hugetlbfs provides huge page-backed file mappings for applications requiring large contiguous physical pages: databases (Oracle, PostgreSQL shared buffers), DPDK, HPC/AI workloads, and GPU pinned memory.

14.18.4.1 hugetlbfs Registration

static HUGETLBFS_TYPE: PseudoFsType = PseudoFsType {
    name: "hugetlbfs",
    fs_flags: FsFlags::NODEV | FsFlags::NOSUID,
    magic: 0x958458f6,
    populate_super: hugetlbfs_populate_super,
    mount_opts: &[
        MountOptDesc { name: "pagesize", kind: MountOptKind::U64 },
        MountOptDesc { name: "size",     kind: MountOptKind::U64 },
        MountOptDesc { name: "min_size", kind: MountOptKind::U64 },
        MountOptDesc { name: "nr_inodes", kind: MountOptKind::U64 },
        MountOptDesc { name: "uid",  kind: MountOptKind::U32 },
        MountOptDesc { name: "gid",  kind: MountOptKind::U32 },
        MountOptDesc { name: "mode", kind: MountOptKind::U32 },
    ],
};

14.18.4.2 Mount Options

Option	Type	Default	Description
pagesize	bytes	architecture default	Huge page size. Platform-dependent (see table below).
size	bytes	all available	Maximum total size of files on this mount.
min_size	bytes	0	Guaranteed reservation: this many bytes of huge pages are reserved at mount time and cannot be stolen by other mounts.
nr_inodes	count	unlimited	Maximum number of inodes (files + directories).
uid	uid_t	0	UID of the root directory.
gid	gid_t	0	GID of the root directory.
mode	octal	01777	Permissions of the root directory.

14.18.4.3 Supported Huge Page Sizes

Architecture	Default	Available Sizes
x86-64	2 MiB	2 MiB (PMD), 1 GiB (PUD)
AArch64	2 MiB	64 KiB (cont PTE), 2 MiB (PMD), 32 MiB (cont PMD), 1 GiB (PUD)
ARMv7	2 MiB	2 MiB (section)
RISC-V 64	2 MiB	2 MiB (PMD), 1 GiB (PUD)
PPC32	4 MiB	4 MiB (depends on MMU variant)
PPC64LE	2 MiB	2 MiB, 1 GiB (radix); 16 MiB (HPT mode also supports 16 MiB)
s390x	1 MiB	1 MiB (segment table large page)
LoongArch64	2 MiB	2 MiB (PMD), 1 GiB (PUD)

Sizes are discovered at boot from the hardware page table capabilities and reported in /proc/meminfo (Hugepagesize) and /sys/kernel/mm/hugepages/.

14.18.4.4 File Operations

Syscall	Behavior
open() / creat()	Creates a file backed by huge pages. No physical pages allocated yet.
mmap()	Maps huge pages into the process address space. Each VMA page fault allocates a single huge page from the pool. MAP_POPULATE pre-faults all pages.
read() / write()	Returns EINVAL. hugetlbfs files are mmap-only.
unlink()	Removes the directory entry. Huge pages are returned to the pool when the last mapping is removed (reference-counted).
fallocate()	mode=0: pre-allocate huge pages without mapping. FALLOC_FL_PUNCH_HOLE: release allocated pages for the given range.
ftruncate()	Resize the file. Shrinking releases pages beyond the new size.

14.18.4.5 Huge Page Pool Management

The system-wide huge page pool is managed via:

• /proc/sys/vm/nr_hugepages — persistent huge pages (survive memory pressure)
• /proc/sys/vm/nr_overcommit_hugepages — surplus pages (reclaimed under pressure)
• Per-NUMA node: /sys/devices/system/node/node<N>/hugepages/hugepages-<size>kB/nr_hugepages

The hugetlbfs pool is independent of THP (Section 4.7): hugetlbfs uses an explicit reservation pool while THP uses buddy allocator promotion. They do not compete for the same pages.

14.18.4.6 memfd_create Integration

memfd_create() with MFD_HUGETLB (Section 4.15) creates an anonymous file descriptor backed by hugetlbfs. The optional MFD_HUGE_2MB / MFD_HUGE_1GB flags select the page size. This is the preferred mechanism for applications that need huge pages without a visible filesystem mount.

14.18.5 bpffs — BPF Filesystem

Mount point: /sys/fs/bpf (mount -t bpf bpffs /sys/fs/bpf) Magic: 0xcafe4a11 (BPF_FS_MAGIC) Flags: NODEV | NOEXEC | NOSUID

bpffs persists BPF objects (programs, maps, links) beyond the lifetime of the loading process. Required by Cilium (Kubernetes CNI), systemd, bpftool, and any infrastructure that loads BPF programs at boot and expects them to survive across process restarts.

14.18.5.1 bpffs Registration

static BPFFS_TYPE: PseudoFsType = PseudoFsType {
    name: "bpf",
    fs_flags: FsFlags::NODEV | FsFlags::NOEXEC | FsFlags::NOSUID,
    magic: 0xcafe4a11,
    populate_super: bpffs_populate_super,
    mount_opts: &[
        MountOptDesc { name: "mode",            kind: MountOptKind::U32 },
        MountOptDesc { name: "delegate_cmds",   kind: MountOptKind::U64 },
        MountOptDesc { name: "delegate_maps",   kind: MountOptKind::U64 },
        MountOptDesc { name: "delegate_progs",  kind: MountOptKind::U64 },
        MountOptDesc { name: "delegate_attachs", kind: MountOptKind::U64 },
    ],
};

14.18.5.2 BPF Object Pinning

/// Pin a BPF object (program, map, or link) to a path in bpffs.
/// The object's kernel reference count is incremented. The object
/// remains alive as long as at least one pin or fd references it.
///
/// Called via bpf(BPF_OBJ_PIN, { fd, pathname }).
///
/// # Errors
/// - `EEXIST`: path already exists.
/// - `EINVAL`: fd does not refer to a BPF object.
/// - `EACCES`: caller lacks CAP_BPF or write permission on parent directory.
/// - `ENOSPC`: bpffs inode limit reached (if configured).
pub fn bpf_obj_pin(fd: BpfFd, pathname: &Path) -> Result<(), SyscallError>;

/// Retrieve a previously pinned BPF object by path, returning a new fd.
/// The caller receives a new file descriptor referencing the pinned object.
///
/// Called via bpf(BPF_OBJ_GET, { pathname }).
///
/// # Errors
/// - `ENOENT`: path does not exist.
/// - `EACCES`: caller lacks CAP_BPF or read permission on the path.
pub fn bpf_obj_get(pathname: &Path) -> Result<BpfFd, SyscallError>;

14.18.5.3 Object Lifecycle

A BPF object is freed when all references are removed:

1. All userspace file descriptors closed.
2. All bpffs pins removed (via unlink()).
3. All kernel-internal references released (e.g., a BPF program attached to a network hook holds an internal reference; detaching releases it).

Only when the reference count reaches zero does the kernel free the BPF program bytecode and map memory.

14.18.5.4 Directory Structure

bpffs supports arbitrary directory hierarchies via mkdir(2). Conventions used by standard tools:

Path	Creator	Content
/sys/fs/bpf/tc/globals/	iproute2	Shared maps for TC BPF programs
/sys/fs/bpf/cilium/	Cilium	Datapath programs and maps
/sys/fs/bpf/xdp/	xdp-tools	XDP programs
/sys/fs/bpf/ip/	iproute2	BPF programs for ip rule

Standard VFS operations: mkdir(), rmdir(), unlink(), readdir() for namespace management. Only unlink() on a pinned object file removes the pin; rmdir() requires the directory to be empty.

14.18.5.5 BPF Token Delegation

BPF tokens (Linux 6.9+) allow unprivileged processes to perform specific BPF operations within the scope of a bpffs mount. A privileged process creates a token by calling bpf(BPF_TOKEN_CREATE) on a bpffs file descriptor; the token inherits delegation rights from the mount options.

/// Bitmask of permitted `bpf()` commands. One bit per `BpfCmd`
/// ([Section 19.2](19-sysapi.md#ebpf-subsystem)); command C is permitted iff `(bits >> C as u32) & 1 == 1`.
/// Parallels `BpfProgTypeMask`. Populated from the bpffs `delegate_cmds` mount option.
pub struct BpfCmdSet(pub u64);

/// Bitmask of permitted BPF map types. One bit per Linux `bpf_map_type`
/// discriminant (the C-level `enum bpf_map_type` value used by
/// `bpf(BPF_MAP_CREATE)`; see [Section 19.2](19-sysapi.md#ebpf-subsystem)); map type `M` is permitted
/// iff `(bits >> M) & 1 == 1`. Populated from the bpffs `delegate_maps` mount
/// option.
pub struct BpfMapTypeSet(pub u64);

/// Bitmask of permitted BPF attach types. One bit per Linux `bpf_attach_type`
/// discriminant (the C-level `enum bpf_attach_type` value used by
/// `bpf(BPF_PROG_ATTACH)`; see [Section 19.2](19-sysapi.md#ebpf-subsystem)); attach type `A` is
/// permitted iff `(bits >> A) & 1 == 1`. Populated from the bpffs
/// `delegate_attachs` mount option.
pub struct BpfAttachTypeSet(pub u64);

/// A BPF token granting scoped BPF permissions to an unprivileged process.
/// Created via bpf(BPF_TOKEN_CREATE, { bpffs_fd }).
///
/// The token inherits its allowed operations from the mount-time
/// delegation options of the bpffs instance.
pub struct BpfToken {
    /// BPF commands this token permits (e.g., BPF_PROG_LOAD, BPF_MAP_CREATE).
    pub allowed_cmds: BpfCmdSet,
    /// Map types this token permits creating.
    pub allowed_map_types: BpfMapTypeSet,
    /// Program types this token permits loading.
    pub allowed_prog_types: BpfProgTypeMask,
    /// Attach types this token permits.
    pub allowed_attach_types: BpfAttachTypeSet,
}

Mount-time delegation options control what a token created on this mount may grant:

Mount Option	Effect
delegate_cmds=0x1f	Bitmask of BPF commands the token may delegate
delegate_maps=0xff	Bitmask of map types the token may delegate
delegate_progs=0x3f	Bitmask of program types the token may delegate
delegate_attachs=0x7f	Bitmask of attach types the token may delegate

Without delegation mount options, BPF_TOKEN_CREATE returns ENOENT — the mount does not support token creation.

14.18.5.6 Access Control

Standard POSIX permissions on directories and files control visibility. Creating or retrieving pinned objects additionally requires CAP_BPF (Section 9.9). Programs operating within a BPF token scope (Section 19.2) may pin objects without CAP_BPF if the token grants the appropriate permissions.

14.18.6 securityfs — Security Module Filesystem

Mount point: /sys/kernel/security Magic: 0x73636673 (SECURITYFS_MAGIC) Flags: NODEV | NOEXEC | NOSUID

securityfs provides per-LSM configuration and status interfaces. The overall LSM framework and the content exposed under securityfs are specified in Section 9.8. This section formalizes the filesystem registration and the kernel API for creating securityfs entries.

14.18.6.1 securityfs Registration

static SECURITYFS_TYPE: PseudoFsType = PseudoFsType {
    name: "securityfs",
    fs_flags: FsFlags::NODEV | FsFlags::NOEXEC | FsFlags::NOSUID,
    magic: 0x73636673,
    populate_super: securityfs_populate_super,
    mount_opts: &[],
};

14.18.6.2 securityfs Kernel API

/// Handle to a securityfs directory. Opaque to callers.
pub struct SecurityfsDir {
    dentry: Arc<Dentry>,
}

/// Handle to a securityfs file. Opaque to callers.
pub struct SecurityfsEntry {
    dentry: Arc<Dentry>,
}

/// Create a directory under the securityfs root (or under `parent`).
/// Each LSM creates its top-level directory during LSM init.
pub fn securityfs_create_dir(
    name: &str,
    parent: Option<&SecurityfsDir>,
) -> Result<SecurityfsDir, VfsError>;

/// Create a file with custom read/write callbacks.
/// `mode` is POSIX permission bits. LSMs typically use 0o444 for
/// status files and 0o600 or 0o200 for policy write interfaces.
pub fn securityfs_create_file(
    name: &str,
    parent: &SecurityfsDir,
    mode: u16,
    fops: &'static FileOps,
) -> Result<SecurityfsEntry, VfsError>;

/// Remove a securityfs entry. Called during LSM teardown (if supported)
/// or during live kernel evolution ([Section 13.18](13-device-classes.md#live-kernel-evolution)).
pub fn securityfs_remove(entry: SecurityfsEntry);

14.18.6.3 Standard securityfs Layout

Path	LSM	Content
/sys/kernel/security/lsm	Core	Comma-separated list of active LSMs (read-only)
/sys/kernel/security/apparmor/	AppArmor	Profile management, policy load
/sys/kernel/security/selinux/	SELinux	Enforce mode, policy, booleans, AVC stats
/sys/kernel/security/ima/	IMA	Measurement log, policy (Section 9.5)
/sys/kernel/security/evm/	EVM	EVM mode, status
/sys/kernel/security/landlock/	Landlock	ABI version (Section 9.8)

Access control varies by LSM: reading status files is typically unrestricted, while policy writes require CAP_MAC_ADMIN (Section 9.9).

14.18.7 efivarfs — EFI Variable Filesystem

Mount point: /sys/firmware/efi/efivars Magic: 0xde5e81e4 (EFIVARFS_MAGIC) Flags: NODEV | NOEXEC | NOSUID

efivarfs exposes UEFI firmware variables to userspace for reading and writing. Required for boot manager configuration (efibootmgr), Secure Boot key management, and firmware diagnostics.

Availability: UEFI systems only. On non-UEFI platforms (most ARMv7, PPC32, PPC64LE), the filesystem is not registered and mount -t efivarfs returns ENODEV.

Architecture	EFI Support	efivarfs Available
x86-64	Yes (UEFI standard)	Yes
AArch64	Yes (UEFI standard)	Yes
ARMv7	Rare (U-Boot)	Only if EFI runtime services present
RISC-V 64	Emerging (UEFI spec)	When EFI runtime services present
PPC32	No (Open Firmware / DTB)	No
PPC64LE	No (OPAL / SLOF)	No
s390x	No (z/VM / LPAR IPL)	No
LoongArch64	Emerging (UEFI standard on Loongson 3A5000+)	When EFI runtime services present

14.18.7.1 efivarfs Registration

static EFIVARFS_TYPE: PseudoFsType = PseudoFsType {
    name: "efivarfs",
    fs_flags: FsFlags::NODEV | FsFlags::NOEXEC | FsFlags::NOSUID,
    magic: 0xde5e81e4,
    populate_super: efivarfs_populate_super,
    mount_opts: &[],
};

14.18.7.2 File Naming Convention

Each file in efivarfs represents one UEFI variable, named as {VariableName}-{VendorGUID} where the GUID is in standard xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx format:

• Boot0001-8be4df61-93ca-11d2-aa0d-00e098032b8c — Boot entry 1 (EFI Global Variable GUID)
• BootOrder-8be4df61-93ca-11d2-aa0d-00e098032b8c — Boot order sequence
• SecureBoot-8be4df61-93ca-11d2-aa0d-00e098032b8c — Secure Boot state
• dbx-d719b2cb-3d3a-4596-a3bc-dad00e67656f — Secure Boot forbidden signature DB

14.18.7.3 File Format

/// Wire format for efivarfs file content. The first 4 bytes are the EFI
/// variable attributes; the remainder is the variable value.
/// This matches the Linux efivarfs file format exactly.
///
/// Read: returns attributes (4 bytes LE) + value
/// Write: caller provides attributes (4 bytes LE) + new_value
// Userspace ABI (efivarfs read/write wire format). DST: no const_assert on
// EFI variable file format: first 4 bytes are the attributes (little-endian u32),
// remainder is the variable value. Total size is attributes(4) + value(N).
// kernel-internal, not KABI

/// Parse an efivarfs file buffer into (attributes, value_slice).
/// Returns `Err(EINVAL)` if the buffer is too short (< 4 bytes).
fn parse_efivar(buf: &[u8]) -> Result<(EfiVariableAttributes, &[u8]), Errno> {
    if buf.len() < 4 {
        return Err(EINVAL);
    }
    let attrs = u32::from_le_bytes([buf[0], buf[1], buf[2], buf[3]]);
    Ok((EfiVariableAttributes::from_bits_truncate(attrs), &buf[4..]))
}

/// Build an efivarfs file buffer from (attributes, value).
/// Writes attributes as little-endian u32 prefix followed by value bytes.
fn build_efivar(attrs: EfiVariableAttributes, value: &[u8], out: &mut [u8]) -> usize {
    let total = 4 + value.len();
    out[..4].copy_from_slice(&attrs.bits().to_le_bytes());
    out[4..total].copy_from_slice(value);
    total
}

bitflags! {
    /// EFI variable attributes. Matches the UEFI specification (Table 14).
    pub struct EfiVariableAttributes: u32 {
        /// Variable persists across resets.
        const NON_VOLATILE                       = 0x0000_0001;
        /// Variable accessible during boot services.
        const BOOTSERVICE_ACCESS                 = 0x0000_0002;
        /// Variable accessible at OS runtime.
        const RUNTIME_ACCESS                     = 0x0000_0004;
        /// Hardware error record (separate NVRAM region on some firmware).
        const HARDWARE_ERROR_RECORD              = 0x0000_0008;
        /// Only authenticated writes accepted (deprecated by UEFI 2.8+).
        const AUTHENTICATED_WRITE_ACCESS         = 0x0000_0010;
        /// Time-based authenticated variable (used by Secure Boot db/dbx).
        const TIME_BASED_AUTHENTICATED_WRITE_ACCESS = 0x0000_0020;
        /// Append-only writes: new data is appended, existing data unchanged.
        const APPEND_WRITE                       = 0x0000_0040;
    }
}

14.18.7.4 Operations

Syscall	Behavior
read()	Returns attributes (4 bytes LE) followed by the variable value. Calls EFI GetVariable() runtime service (Section 2.20).
write()	Caller provides attributes (4 bytes LE) + new value. Calls EFI SetVariable(). Returns EIO on firmware error, ENOSPC if NVRAM is full.
creat()	Creates a new EFI variable. File name must follow the {Name}-{GUID} convention. Calls EFI SetVariable() with the new name.
unlink()	Deletes the EFI variable by calling SetVariable() with DataSize=0. Returns EPERM if the variable is immutable.
readdir()	Enumerates all EFI variables via GetNextVariableName(). Results are cached in memory after first enumeration; invalidated on any write.

14.18.7.5 Immutable Variable Protection

Certain variables are critical to system boot and must not be accidentally deleted:

/// Variables marked immutable (FS_IMMUTABLE_FL) by the kernel.
/// Users cannot unlink or write to these without first clearing
/// the immutable flag (requires CAP_LINUX_IMMUTABLE).
const IMMUTABLE_VARS: &[&str] = &[
    "SecureBoot",
    "SetupMode",
    "PK",        // Platform Key
    "KEK",       // Key Exchange Key
    "AuditMode",
    "DeployedMode",
];

The kernel sets FS_IMMUTABLE_FL on these files at mount time. Modifying them requires chattr -i first (which requires CAP_LINUX_IMMUTABLE), providing a two-step safeguard against accidental firmware corruption.

14.18.7.6 NVRAM Wear Protection

EFI NVRAM has limited write endurance (typically 100K-1M cycles per flash block). The kernel rate-limits writes to prevent userspace from wearing out NVRAM:

/// NVRAM write rate limiter. Shared across all efivarfs writes.
/// Uses a token bucket algorithm: one token per write, refilled at
/// `REFILL_RATE` tokens per second, maximum burst of `BUCKET_SIZE`.
pub struct EfiNvramRateLimiter {
    /// Current token count. Bounded gauge: range [0, NVRAM_BUCKET_SIZE].
    /// AtomicU32 is sufficient: max value is 64 (NVRAM_BUCKET_SIZE),
    /// never incremented beyond the bucket ceiling by the refill logic.
    pub tokens: AtomicU32,
    /// Last refill timestamp (nanoseconds, monotonic clock).
    pub last_refill_ns: AtomicU64,
}

/// Maximum burst writes before throttling.
const NVRAM_BUCKET_SIZE: u32 = 64;

/// Sustained write rate: 1 write per 100ms (10 writes/sec).
/// At this rate, 100K-cycle NVRAM endures ~2,800 hours of
/// continuous maximum-rate writes. Practical workloads (boot
/// manager changes, key rotations) are many orders of magnitude
/// below this rate.
const NVRAM_REFILL_INTERVAL_NS: u64 = 100_000_000;

When the bucket is empty, write() returns EBUSY. The caller (typically efibootmgr or mokutil) retries after a short delay.

14.18.8 Boot Initialization Order

All pseudo-filesystems register during Phase 5 (VFS initialization) of the boot sequence (Section 2.3). The ordering reflects dependency constraints:

Order	Filesystem	Dependency	Registration Guard
1	debugfs	VFS core initialized	None
2	tracefs	debugfs mounted (for legacy symlink)	debugfs mount point exists
3	hugetlbfs	Physical memory allocator + huge page pool (Section 4.2)	Huge page pool initialized
4	bpffs	eBPF verifier initialized (Section 19.2)	BPF subsystem ready
5	securityfs	LSM framework initialized (Section 9.8)	At least one LSM registered
6	efivarfs	EFI runtime services available (Section 2.20)	efi.runtime_services != null (skipped on non-UEFI)

After registration, each filesystem is mounted at its standard mount point by the init process (PID 1). The kernel does not auto-mount pseudo-filesystems — mount commands come from userspace init (systemd .mount units or /etc/fstab). The exception is the root filesystem and devtmpfs, which are mounted by the kernel before init executes.

14.19 procfs and sysfs

procfs and sysfs are the two primary pseudo-filesystems through which the kernel exposes runtime state to userspace. procfs (/proc) is process-oriented: per-PID directories, global memory/CPU statistics, and writable sysctl tunables. sysfs (/sys) is device-oriented: it mirrors the kernel's device model as a directory hierarchy with one-value-per-file attributes. Both are required for glibc, systemd, udev, ps, top, htop, lscpu, lsblk, and virtually every Linux system management tool.

Both filesystems build on the common pseudo-filesystem registration framework defined in Section 14.18.

---

14.19.1 procfs — Process Information Filesystem

Mount point: /proc (type proc) Magic: 0x9fa0 (PROC_SUPER_MAGIC) Flags: NODEV | NOEXEC | NOSUID | USERNS_MOUNT

procfs is the kernel's primary process-to-userspace information channel. It is mounted automatically during early init (before PID 1 executes) and is required for correct glibc operation (/proc/self/), systemd (cgroup discovery, mount enumeration, process introspection), and standard POSIX process tools.

14.19.1.1 procfs Registration

static PROCFS_TYPE: PseudoFsType = PseudoFsType {
    name: "proc",
    fs_flags: FsFlags::NODEV | FsFlags::NOEXEC | FsFlags::NOSUID
        | FsFlags::USERNS_MOUNT,
    magic: 0x9fa0,
    populate_super: proc_populate_super,
    mount_opts: &[
        MountOptDesc { name: "hidepid", kind: MountOptKind::U32 },
        MountOptDesc { name: "gid",     kind: MountOptKind::U32 },
        MountOptDesc { name: "subset",  kind: MountOptKind::Str },
    ],
};

Mount options:

Option	Values	Default	Description
hidepid	0, 1, 2, 4	0	Process visibility. 0 = world-readable. 1 = hide cmdline/status for other users' PIDs. 2 = invisible /proc/PID/ for non-owned processes. 4 (ptraceable) = /proc contains only the /proc/PID/ directories the caller is permitted to ptrace() (ptrace-based PID filtering); every non-ptraceable PID is invisible. No mode hides thread IDs.
gid	GID	0	GID whose members bypass the hidepid=1/hidepid=2 restrictions (see the permission model below), letting monitoring daemons (e.g., monit, Prometheus node_exporter) see all processes without running as root. The bypass does NOT apply at hidepid=4: there, visibility is decided solely by ptraceability, so group membership alone never reveals a PID the caller cannot ptrace().
subset	pid	none	Mount only per-PID entries (no global files). Used for container /proc mounts that only need process information.

14.19.1.2 ProcEntry Trait

Every file or directory in procfs is backed by an implementation of ProcEntry. Subsystems register entries during init; per-PID entries are instantiated lazily on first lookup.

/// Trait for procfs file content generation.
///
/// Implementations are typically zero-sized types that read kernel state
/// on demand. No per-file heap allocation occurs — the `ProcEntry` is a
/// `&'static dyn` reference.
pub trait ProcEntry: Send + Sync {
    /// Read content into `buf` starting at byte `offset`.
    /// Returns the number of bytes written to `buf`.
    ///
    /// For fixed-format files (e.g., `/proc/PID/stat`), the implementation
    /// generates the entire content into an internal buffer on the first
    /// read (offset=0) and serves subsequent reads from that snapshot.
    /// This ensures a consistent view even if the process state changes
    /// between read() calls.
    fn read(&self, ctx: &ProcReadCtx, buf: &mut [u8], offset: u64) -> Result<usize, Errno>;

    /// Write data from `buf`. Returns bytes consumed.
    /// Most procfs files are read-only and return `EACCES`.
    fn write(&self, ctx: &ProcWriteCtx, buf: &[u8]) -> Result<usize, Errno> {
        Err(Errno::EACCES)
    }

    /// Poll for readability/writability/events. Returns the events that are
    /// currently ready. Default: not pollable (empty events).
    ///
    /// This is required for `/proc/sys/` files: programs that `poll()` on
    /// sysctl files (e.g., `poll(/proc/sys/vm/overcommit_memory, POLLPRI)`)
    /// need notification when the value changes. The sysctl implementation
    /// calls `proc_sys_notify()` on write, which wakes pollers.
    fn poll(&self, _ctx: &ProcReadCtx, _events: PollEvents) -> PollEvents {
        PollEvents::empty()
    }
}

/// Notify pollers of a procfs sysctl entry that the value has changed.
/// Called by the sysctl write path after updating the kernel parameter.
/// Wakes all processes polling the file with `POLLPRI`.
fn proc_sys_notify(entry: &dyn ProcEntry) {
    // Implementation: the sysctl ProcEntry stores a WaitQueue. poll()
    // registers the caller on this WaitQueue. proc_sys_notify() does
    // wake_up_all(&entry.waitqueue, POLLPRI). Specific sysctl entries
    // that support notification override poll() to register on the waitqueue.
}

/// Context passed to ProcEntry::read().
pub struct ProcReadCtx<'a> {
    /// The PID this entry belongs to (None for global entries).
    pub pid: Option<Pid>,
    /// Credentials of the reading process (for permission checks).
    /// RCU-protected reference: credentials can change during a task's
    /// lifetime (setuid, capset), so a static reference would be unsound.
    pub cred: RcuRef<'a, Credentials>,
}

/// Context passed to ProcEntry::write().
pub struct ProcWriteCtx<'a> {
    /// The PID this entry belongs to (None for global entries).
    pub pid: Option<Pid>,
    /// Credentials of the writing process.
    /// RCU-protected reference — see `ProcReadCtx::cred` for rationale.
    pub cred: RcuRef<'a, Credentials>,
}

14.19.1.3 SeqFile Protocol

Large procfs files that enumerate variable-length records (e.g., /proc/net/tcp, /proc/mounts) use the SeqFile protocol to handle partial reads across multiple read() syscalls without missing or duplicating entries.

/// Sequential file generator. Each call to `show()` produces one logical
/// record. The SeqFile infrastructure handles buffering, partial reads,
/// and seek.
pub trait SeqFileOps: Send + Sync {
    /// Position the iterator at the beginning.
    /// `pos` is the logical position (0 on first read, or a saved position
    /// from a previous lseek). Returns an opaque iterator state, or None
    /// if `pos` is beyond the last record.
    fn start(&self, ctx: &ProcReadCtx, pos: u64) -> Option<SeqIterState>;

    /// Emit one record into `buf`. Returns bytes written.
    /// The SeqFile core calls `show()` repeatedly, appending output to an
    /// internal page-sized buffer. When the buffer is full, it is returned
    /// to the `read()` caller and the remaining records are served on the
    /// next `read()`.
    fn show(&self, state: &SeqIterState, buf: &mut [u8]) -> Result<usize, Errno>;

    /// Advance to the next record. Returns the updated iterator state,
    /// or None if iteration is complete.
    fn next(&self, state: SeqIterState) -> Option<SeqIterState>;

    /// Release any resources held by the iterator.
    /// Called when the file is closed or the read sequence is abandoned.
    fn stop(&self, state: SeqIterState);
}

/// Opaque iterator state for SeqFile traversal.
/// Subsystems typically store an index, a pointer to the current object,
/// and an RCU read-lock guard or reference count.
pub struct SeqIterState {
    /// Logical position counter (incremented by `next()`).
    /// Stored across read() calls for correct resume-after-partial-read.
    pub index: u64,
    /// Bytes emitted so far in the current read() call.
    pub count: usize,
    /// Subsystem-private state (cast from a concrete type).
    pub private: u64,
}

14.19.1.4 procfs Registration API

/// Register a global procfs entry (e.g., `/proc/meminfo`).
///
/// `name`: entry name (e.g., "meminfo").
/// `mode`: POSIX permission bits (e.g., 0o444 for read-only).
/// `parent`: parent directory (None = procfs root).
/// `ops`: implementation providing read/write behavior.
///
/// Returns a handle used for removal (live kernel evolution only;
/// built-in entries are never removed).
///
/// # Errors
/// - `EEXIST`: name already registered under parent.
/// - `ENOMEM`: allocation failure.
pub fn proc_create(
    name: &'static str,
    mode: u16,
    parent: Option<&ProcDir>,
    ops: &'static dyn ProcEntry,
) -> Result<ProcHandle, VfsError>;

/// Register a global procfs entry backed by SeqFile.
/// Convenience wrapper: creates a ProcEntry that delegates to SeqFileOps.
pub fn proc_create_seq(
    name: &'static str,
    mode: u16,
    parent: Option<&ProcDir>,
    ops: &'static dyn SeqFileOps,
) -> Result<ProcHandle, VfsError>;

/// Create a subdirectory under the procfs root (e.g., `/proc/net/`).
pub fn proc_mkdir(
    name: &'static str,
    parent: Option<&ProcDir>,
) -> Result<ProcDir, VfsError>;

/// Opaque handle to a registered procfs directory.
pub struct ProcDir {
    dentry: Arc<Dentry>,
}

/// Opaque handle to a registered procfs entry (file or directory).
/// Dropping this handle does NOT remove the entry — entries persist
/// for the kernel's lifetime. Explicit removal via `proc_remove()`
/// is for live kernel evolution only.
pub struct ProcHandle {
    dentry: Arc<Dentry>,
}

/// Remove a previously registered procfs entry.
pub fn proc_remove(handle: ProcHandle);

14.19.1.5 Per-PID Directory Lifecycle

A /proc/PID/ directory is created when a task is allocated (Section 8.1) and removed when the task is reaped (after wait() collects the zombie). The directory is lazily populated — subdirectory dentries are instantiated on first lookup, not at task creation time. This avoids per-fork allocation overhead for the ~20 entries per PID. The lookup callback (pid_dir_lookup()) resolves the PID to a Task, creates a dentry backed by a ProcInode, and populates on-demand. No dentries exist for PIDs that have not been accessed.

For thread-group leaders, /proc/PID/task/ contains subdirectories for each thread in the group. Each thread subdirectory mirrors the per-PID layout.

14.19.1.6 Mandatory /proc/PID/ Entries

These entries are required for glibc, systemd, ps, top, htop, and container runtimes. Format descriptions are normative — field order, separator characters, and field types must match Linux exactly for binary compatibility.

Entry	Mode	Format	Description
status	0o444	Key:\tValue\n lines	Human-readable status. Fields: Name, Umask, State, Tgid, Ngid, Pid, PPid, TracerPid, Uid (4 fields), Gid (4 fields), FDSize, Groups, NStgid, NSpid, NSpgid, NSsid, Kthread, VmPeak, VmSize, VmLck, VmPin, VmHWM, VmRSS, RssAnon, RssFile, RssShmem, VmData, VmStk, VmExe, VmLib, VmPTE, VmSwap, HugetlbPages, CoreDumping, THP_enabled, Threads, SigQ, SigPnd, ShdPnd, SigBlk, SigIgn, SigCgt, CapInh, CapPrm, CapEff, CapBnd, CapAmb, NoNewPrivs, Seccomp, Seccomp_filters, Speculation_Store_Bypass, SpeculationIndirectBranch, Cpus_allowed, Cpus_allowed_list, Mems_allowed, Mems_allowed_list, voluntary_ctxt_switches, nonvoluntary_ctxt_switches.
stat	0o444	Single line, space-separated	52 fields: pid (comm) state ppid pgrp session tty_nr tpgid flags minflt cminflt majflt cmajflt utime stime cutime cstime priority nice num_threads itrealvalue starttime vsize rss rsslim startcode endcode startstack kstkesp kstkeip signal blocked sigignore sigcatch wchan nswap cnswap exit_signal processor rt_priority policy delayacct_blkio_ticks guest_time cguest_time start_data end_data start_brk arg_start arg_end env_start env_end exit_code. Matches Linux fs/proc/array.c do_task_stat() — field 52 is exit_code. Note: core_dumping and thp_enabled are in /proc/PID/status (Key:Value format), not stat.
statm	0o444	Single line, 7 space-separated page counts	size resident shared text lib data dt
cmdline	0o444	NUL-separated argv bytes	Empty for zombie/kernel threads.
environ	0o400	NUL-separated envp bytes	Requires ptrace_access_permitted() or same UID. Returns empty for kernel threads.
maps	0o444	One line per VMA	start-end perms offset dev inode pathname. Hex addresses, rwxp perms.
smaps	0o444	Multi-line per VMA	Same header as maps, followed by Size, KernelPageSize, MMUPageSize, Rss, Pss, Pss_Dirty, Shared_Clean, Shared_Dirty, Private_Clean, Private_Dirty, Referenced, Anonymous, LazyFree, AnonHugePages, ShmemPmdMapped, FilePmdMapped, Shared_Hugetlb, Private_Hugetlb, Swap, SwapPss, Locked, THPeligible, VmFlags lines.
fd/	0o500	Directory of symlinks	Each entry is a decimal fd number; readlink returns the path of the open file.
fdinfo/	0o500	One file per fd	Lines: pos:, flags:, mnt_id:. Additional fields for epoll, eventfd, inotify, fanotify, timerfd, signalfd fds.
cgroup	0o444	hierarchy-ID:controller-list:cgroup-path per line	For cgroups v2: single line 0::/path.
mountinfo	0o444	SeqFile, one line per mount	Fields: mount_id, parent_id, major:minor, root, mount_point, mount_options, optional_fields, separator(-), fs_type, mount_source, super_options. See Section 14.6.
ns/	0o500	Directory of symlinks	Entries: cgroup, ipc, mnt, net, pid, pid_for_children, time, time_for_children, user, uts, ima. Readlink returns <type>:[<inode>]. The ima entry links to the task's IMA namespace; IMA measurement log output (e.g., /sys/kernel/security/ima/ascii_runtime_measurements) is scoped to the reading task's IMA namespace — a process in container namespace A sees only namespace A's measurement log entries. See Section 17.1, Section 9.5.
oom_score	0o444	Single integer (0-2000)	OOM badness score. See Section 4.5.
oom_score_adj	0o644	Single integer (-1000 to 1000)	OOM adjustment. -1000 = never kill. Writing requires CAP_SYS_RESOURCE for values < 0.
limits	0o444	Table format	Columns: Limit, Soft Limit, Hard Limit, Units. Rows: Max cpu time, Max file size, Max data size, Max stack size, Max core file size, Max resident set, Max processes, Max open files, Max locked memory, Max address space, Max file locks, Max pending signals, Max msgqueue size, Max nice priority, Max realtime priority, Max realtime timeout. See Section 8.8.
io	0o400	key: value lines	Fields: rchar, wchar, syscr, syscw, read_bytes, write_bytes, cancelled_write_bytes. Requires ptrace_access_permitted() or same UID.
task/	0o555	Directory of per-thread subdirectories	Each subdirectory is named by TID and contains the same entries as the parent /proc/PID/ (stat, status, maps, etc.) scoped to that thread.
exe	—	Symlink	Points to the executable file. If the binary is unlinked while the process is still running, the task keeps a live reference to its executable, so readlink SUCCEEDS and returns the original path with (deleted) appended by the path-formatting layer. readlink returns ENOENT only when the task has no executable at all — kernel threads, and tasks whose address space (mm) has been torn down. Consumers depend on the success-with-(deleted) behavior: lsof \| grep deleted, needrestart, security scanners, and ls -l /proc/PID/exe across package upgrades.
root	—	Symlink	Points to the process's root directory (as set by chroot()).
cwd	—	Symlink	Points to the process's current working directory.
comm	0o644	Single line, max 16 bytes	Executable name (truncated to TASK_COMM_LEN = 16). Writable: echo newname > /proc/PID/comm sets the task comm, but ONLY when the writer belongs to the same thread group as the target task; every other writer — including a CAP_SYS_PTRACE monitor — gets EINVAL.
wchan	0o444	Single symbol name	Wait channel: the kernel function the task is sleeping in, or "0" if running.
uid_map	0o644	%10u %10u %10u\n per mapping line (inner outer count)	UID mappings of the task's user namespace. Read shows the frozen map (empty until written). Write-once; parser, permission rules (CAP_SETUID-in-parent vs single-extent self-map), and errno surface: Section 17.1. Required by newuidmap/podman/rootless Docker.
gid_map	0o644	Same format as uid_map	GID mappings. Write-once; the unprivileged (no CAP_SETGID in parent) self-map write additionally requires setgroups = "deny" — see the write protocol.
projid_map	0o644	Same format as uid_map	Project-ID mappings (filesystem project quotas). Write-once; CAP_SYS_ADMIN-in-parent rule.
setgroups	0o644	allow\n or deny\n	setgroups(2) gate for the task's user namespace. Writable only while gid_map is unwritten. See the write protocol in Section 17.1.

Barrier discipline for ptrace_access_permitted()-gated entries: every /proc/PID entry whose access is gated by ptrace_access_permitted() — the private status/stat fields, maps/smaps, environ, io — performs the permission check AND the gated content emission inside ONE READ hold of the target's exec_cred_barrier (Section 3.4, entry 3b; the interruptible READ acquisition returns EINTR if interrupted). This keeps the sampled permission and the emitted values one consistent snapshot with respect to a concurrent execve() on the target, so a decision taken against the pre-exec credential can never gate freshly-committed post-exec content. The canonical three-argument gate is defined at Section 20.4.

14.19.1.7 Mandatory Global /proc Entries

These entries are required for system monitoring tools, container runtimes, and standard POSIX utilities.

Entry	Mode	Format	Primary Consumers
meminfo	0o444	Key: value kB lines	free, top, htop, systemd, Prometheus node_exporter. Fields: MemTotal, MemFree, MemAvailable, Buffers, Cached, SwapCached, Active, Inactive, Active(anon), Inactive(anon), Active(file), Inactive(file), Unevictable, Mlocked, SwapTotal, SwapFree, Zswap, Zswapped, Dirty, Writeback, AnonPages, Mapped, Shmem, KReclaimable, Slab, SReclaimable, SUnreclaim, KernelStack, PageTables, SecPageTables, NFS_Unstable, Bounce, WritebackTmp, CommitLimit, Committed_AS, VmallocTotal, VmallocUsed, VmallocChunk, Percpu, HardwareCorrupted, AnonHugePages, ShmemHugePages, ShmemPmdMapped, FileHugePages, FilePmdMapped, CmaTotal, CmaFree, HugePages_Total, HugePages_Free, HugePages_Rsvd, HugePages_Surp, Hugepagesize, Hugetlb, DirectMap4k, DirectMap2M, DirectMap1G.
cpuinfo	0o444	Per-arch key-value blocks	lscpu, nproc, /proc/cpuinfo parsers. Per-CPU block separated by blank line. Format is architecture-specific (x86: processor/vendor_id/model name/flags; ARM: processor/BogoMIPS/Features; RISC-V: hart/isa/mmu).
stat	0o444	Multi-line	top, mpstat, vmstat. Lines: cpu (aggregate), cpu0..cpuN (per-CPU: user nice system idle iowait irq softirq steal guest guest_nice), intr (per-IRQ counts), ctxt (context switches), btime (boot time epoch), processes (forks since boot), procs_running, procs_blocked, softirq (per-softirq counts).
loadavg	0o444	Single line	uptime, w, shell prompts. Format: 1min 5min 15min running/total last_pid.
uptime	0o444	Single line	uptime. Format: seconds_since_boot idle_seconds (both with centisecond precision).
version	0o444	Single line	uname, build identification. Format: UmkaOS version <version> (<build>) (<compiler>) #<build_number> <config> <date>.
filesystems	0o444	One per line	mount auto-detection. Format: [nodev]\tfstype. nodev prefix for pseudo-filesystems that have no backing device.
self	—	Symlink	Points to /proc/[getpid()]. Resolved per-access to the calling task's TGID. Required by glibc (/proc/self/exe, /proc/self/fd/).
thread-self	—	Symlink	Points to /proc/[getpid()]/task/[gettid()]. Resolved per-access to the calling thread's TID. Required for per-thread procfs access (e.g., /proc/thread-self/attr/current for SELinux).
mounts	—	Symlink	Points to self/mounts.
partitions	0o444	Table	lsblk, fdisk. Columns: major, minor, #blocks, name.
diskstats	0o444	One line per device	iostat, sar. 18 fields per line: major minor name reads_completed reads_merged sectors_read ms_reading writes_completed writes_merged sectors_written ms_writing ios_in_progress ms_io weighted_ms_io discards_completed discards_merged sectors_discarded ms_discarding flush_count ms_flushing.
net/	0o555	Directory	Network pseudo-files (per-net-namespace). Entries: dev (interface stats), tcp (TCP sockets), tcp6, udp, udp6, unix (UNIX sockets), route (IPv4 routing), ipv6_route, if_inet6, arp, snmp, snmp6, netstat, sockstat, sockstat6, raw, raw6, packet, protocols, wireless. Each file uses SeqFile.
sys/	0o555	Directory tree	Sysctl interface. Writable tunables organized as kernel/, vm/, fs/, net/, dev/. Each leaf file reads/writes one value. See Section 20.9 for the sysctl registration framework.
interrupts	0o444	Table	Per-CPU IRQ counts. Columns: IRQ number, per-CPU counts, IRQ chip name, hardware IRQ, action name.
softirqs	0o444	Table	Per-CPU softirq counts. One row per softirq type (HI, TIMER, NET_TX, NET_RX, BLOCK, IRQ_POLL, TASKLET, SCHED, HRTIMER, RCU).

14.19.1.8 procfs Namespace Awareness

Each PID namespace (Section 17.1) has its own view of /proc: processes in a child PID namespace see only PIDs visible within that namespace. When procfs is mounted inside a container, populate_super binds the superblock to the caller's PID namespace. /proc/1/ inside the container refers to the container's init process, not the host PID 1.

All pid_t-valued fields are translated against the procfs mount's PID namespace (matching Linux fs/proc/array.c do_task_stat()): pid/ppid via pid_nr_in(), and /proc/PID/stat fields 5-6 (pgrp, session) plus field 8 (tpgid) via pgid_nr_ns()/sid_nr_ns() (Section 8.7), reporting 0 when the group/session has no number at the mount's level. The /proc/PID/status per-level lists NStgid/NSpid/NSpgid/NSsid are produced by walking the captured chains directly — the task's Task.pid_links for NStgid/NSpid, the group's and session's pid_chain for NSpgid/NSsid — emitting one number per level from the mount's namespace down to the leaf. Field-to-struct source mapping: Section 8.8.

14.19.1.9 procfs Permission Model

hidepid	Effect
0	All /proc/PID/ directories visible to all users (Linux default)
1	/proc/PID/{cmdline,sched,status} restricted to owner; directory is visible
2	/proc/PID/ directory invisible to non-owners (opendir returns ENOENT)
4 (ptraceable)	/proc shows only the /proc/PID/ directories the caller is permitted to ptrace(); every non-ptraceable PID is invisible (absent from readdir, lookup returns ENOENT). NOT a superset of mode 2 — the gate is ptraceability, not ownership — and no mode hides thread IDs from /proc/PID/task/.

For hidepid=1 and hidepid=2, the gid mount option exempts a group from the restriction: a process whose supplementary groups include the configured GID sees all PIDs without CAP_SYS_PTRACE. At hidepid=4 this exemption does NOT apply — the ptraceability gate is evaluated before any group check — so a gid member still cannot see a PID it is not permitted to ptrace().

---

14.19.2 sysfs — Device Model Filesystem

Mount point: /sys (type sysfs) Magic: 0x62656572 (SYSFS_MAGIC) Flags: NODEV | NOEXEC | NOSUID

sysfs mirrors the kernel's device model hierarchy as a directory tree. Every registered bus, device, driver, and class gets a directory. Attributes are files that expose exactly one value each (the "sysfs one-value-per-file rule"). This filesystem is required for udev device discovery, systemd device management, and all /sys-reading tools (lspci, lsusb, lsblk, ip link, etc.).

14.19.2.1 sysfs Registration

static SYSFS_TYPE: PseudoFsType = PseudoFsType {
    name: "sysfs",
    fs_flags: FsFlags::NODEV | FsFlags::NOEXEC | FsFlags::NOSUID,
    magic: 0x62656572,
    populate_super: sysfs_populate_super,
    mount_opts: &[],
};

14.19.2.2 Kobject Model

Every kernel object that participates in sysfs inherits from Kobject. A kobject represents one directory in the sysfs tree. The parent-child relationship between kobjects defines the directory hierarchy.

/// Kernel object — the unit of representation in sysfs.
///
/// Every kobject corresponds to exactly one directory under `/sys`.
/// Kobjects form a tree: each kobject has at most one parent.
/// The root kobjects (no parent) appear directly under `/sys`.
/// **Cycle detection**: `sysfs_create_kobject()` walks the parent chain
/// (max depth 32) to verify the new parent is not a descendant of the
/// new kobject. If a cycle is detected, creation fails with `ELOOP` and
/// an FMA warning identifies the offending driver.
pub struct Kobject {
    /// Name of this object (= directory name in sysfs).
    pub name: Box<[u8]>,
    /// Parent kobject. None for top-level directories.
    pub parent: Option<Arc<Kobject>>,
    /// Attribute groups attached to this kobject.
    /// Each group's attributes appear as files in this directory.
    pub attr_groups: ArrayVec<&'static SysfsGroup, 8>,
    /// Kset membership (if any). A kset is a collection of related
    /// kobjects that share a uevent domain.
    pub kset: Option<Arc<Kset>>,
    /// Reference count. Kobject is freed when refcount reaches zero.
    pub refcount: AtomicU64,
    /// Uevent state: whether this kobject has been announced to userspace.
    pub uevent_sent: AtomicBool,
}

/// A kset groups related kobjects and provides the uevent emission
/// context. Bus, class, and device collections are ksets.
pub struct Kset {
    /// The kset's own kobject (its sysfs directory).
    pub kobj: Kobject,
    /// Uevent filter: if set, called before emitting uevents for
    /// member kobjects. Returns false to suppress the event.
    pub uevent_filter: Option<fn(&Kobject) -> bool>,
}

impl Kobject {
    /// Create a kobject with a fresh refcount and no attached groups.
    ///
    /// `name` becomes the sysfs directory name (copied into an owned buffer).
    /// `parent` places the directory under another kobject (`None` = sysfs root
    /// or the kset's directory). `kset` records membership in a collection whose
    /// class/bus attribute group supplies the standard files; per-kobject groups
    /// are attached afterwards with `sysfs_create_group` when a device needs its
    /// own attributes. Cold path (device registration): the name copy allocates.
    pub fn new(name: &str, parent: Option<Arc<Kobject>>, kset: Option<Arc<Kset>>) -> Self {
        Self {
            name: name.as_bytes().to_vec().into_boxed_slice(),
            parent,
            attr_groups: ArrayVec::new(),
            kset,
            refcount: AtomicU64::new(1),
            uevent_sent: AtomicBool::new(false),
        }
    }
}

14.19.2.3 SysfsAttribute Trait

/// Trait for sysfs file attributes.
///
/// Each attribute is a single file in a kobject's directory.
/// Implementations MUST follow the one-value-per-file rule:
/// `show()` returns exactly one scalar, string, or enumeration value.
/// `store()` parses exactly one value.
pub trait SysfsAttribute: Send + Sync {
    /// Attribute file name.
    fn name(&self) -> &'static str;

    /// POSIX permission bits (e.g., 0o444 read-only, 0o644 read-write).
    fn mode(&self) -> u16;

    /// Read the attribute value into `buf`. Returns bytes written.
    /// The output MUST end with a newline (`\n`) for shell compatibility.
    /// Maximum output: PAGE_SIZE - 1 bytes (4095 on all architectures).
    fn show(&self, kobj: &Kobject, buf: &mut [u8]) -> Result<usize, Errno>;

    /// Write a new value from `buf`. Returns bytes consumed.
    /// Returns `EACCES` for read-only attributes (mode without write bits).
    /// Returns `EINVAL` if the value cannot be parsed.
    fn store(&self, kobj: &Kobject, buf: &[u8]) -> Result<usize, Errno> {
        Err(Errno::EACCES)
    }
}

/// A named collection of attributes applied to a kobject together.
///
/// Groups provide atomic attachment: all attributes in a group are
/// created or destroyed as a unit. A kobject may have multiple groups;
/// each group can optionally create a subdirectory.
pub struct SysfsGroup {
    /// Subdirectory name. If Some, attributes appear in a named
    /// subdirectory of the kobject's directory. If None, attributes
    /// appear directly in the kobject's directory.
    pub name: Option<&'static str>,
    /// Attributes in this group.
    pub attrs: &'static [&'static dyn SysfsAttribute],
    /// Visibility filter. Called once per attribute at group creation
    /// time. Returns the effective mode (0 = skip this attribute).
    /// Allows conditional attributes based on hardware capabilities.
    pub is_visible: Option<fn(&Kobject, &dyn SysfsAttribute) -> u16>,
}

14.19.2.4 sysfs Kernel API

/// Create a kobject directory in sysfs and attach its attribute groups.
///
/// The directory is created under the parent's directory (or the sysfs
/// root if parent is None). All attribute groups are instantiated
/// atomically: if any attribute creation fails, the entire directory
/// is rolled back.
///
/// Typically reached through `DeviceRegistry::register()` or the bus/class
/// registration paths rather than directly by subsystem code.
pub fn sysfs_create_kobject(kobj: &Arc<Kobject>) -> Result<(), VfsError>;

/// Attach an additional attribute group to an existing kobject.
/// Used when subsystems add attributes after initial registration
/// (e.g., driver-specific attributes added at probe time).
pub fn sysfs_create_group(
    kobj: &Arc<Kobject>,
    group: &'static SysfsGroup,
) -> Result<(), VfsError>;

/// Remove an attribute group from a kobject.
pub fn sysfs_remove_group(
    kobj: &Arc<Kobject>,
    group: &'static SysfsGroup,
);

/// Create a symbolic link in sysfs. Used for cross-references:
/// e.g., `/sys/class/net/eth0/device` → `/sys/devices/pci0000:00/...`.
///
/// `kobj`: the directory where the symlink is created.
/// `target`: the kobject the symlink points to.
/// `name`: the symlink file name.
pub fn sysfs_create_link(
    kobj: &Arc<Kobject>,
    target: &Arc<Kobject>,
    name: &'static str,
) -> Result<(), VfsError>;

/// Remove a symbolic link.
pub fn sysfs_remove_link(kobj: &Arc<Kobject>, name: &'static str);

/// Notify userspace that an attribute value has changed.
/// Wakes any process blocked in `poll()` / `select()` on the
/// attribute file. Used by the thermal subsystem, battery driver,
/// and power management to notify udev/systemd of state changes.
pub fn sysfs_notify(kobj: &Arc<Kobject>, attr_name: &'static str);

/// Remove a kobject and all its attribute groups from sysfs.
/// Called during device removal or driver unbind.
pub fn sysfs_remove_kobject(kobj: &Arc<Kobject>);

14.19.2.5 Mandatory /sys Hierarchies

These top-level directories are required for udev, systemd, and standard Linux device management tools.

14.19.2.5.1 /sys/devices/ — Physical Device Tree

The canonical device hierarchy. Every device registered via Section 11.4 appears here, organized by physical topology: system/cpu/, pci0000:00/, platform/, virtual/. Each device directory contains:

• Standard attributes: uevent, power/ (runtime PM state), driver (symlink), subsystem (symlink to bus/class).
• Device-specific attributes added by the device driver at probe time.
• Child device directories for hierarchical devices (e.g., PCI bridge children).

14.19.2.5.2 /sys/bus/ — Bus Type Directories

One directory per registered bus type (pci, usb, platform, i2c, spi, etc.). Each bus directory contains:

Subdirectory	Content
devices/	Symlinks to /sys/devices/... for every device on this bus.
drivers/	One subdirectory per registered driver. Each driver directory contains bind (write device name to force-bind), unbind (write device name to force-unbind), and new_id (write vendor:device to add dynamic ID).
drivers_probe	Write a device name to trigger re-probe.
drivers_autoprobe	1 = auto-probe new devices (default), 0 = manual binding only.

14.19.2.5.3 /sys/class/ — Device Class Directories

One directory per device class. Each class directory contains symlinks to the device directories in /sys/devices/. Classes group devices by function rather than bus topology:

Class	Content	Primary Tool
net/	Network interfaces (eth0, wlan0, lo)	ip link, NetworkManager
block/	Block devices (sda, nvme0n1, dm-0)	lsblk
tty/	Terminal devices (ttyS0, ttyUSB0, pts/0)	stty, getty
input/	Input devices (event0, mice)	libinput, evtest
hwmon/	Hardware monitoring (temperatures, fans, voltages)	sensors, lm-sensors
thermal/	Thermal zones and cooling devices	thermald
power_supply/	Battery and AC adapter status	upower
backlight/	Display backlight	xrandr, brightnessctl
leds/	LED devices	ledctl
sound/	ALSA sound devices	aplay, alsamixer
drm/	DRM/KMS display devices	modetest, Xorg
misc/	Miscellaneous devices	Various

14.19.2.5.4 /sys/block/ — Block Devices

Symlinks into /sys/devices/ for all block devices. Each block device directory contains queue/ (scheduler, nr_requests, read_ahead_kb), stat (I/O counters), size (sectors), and partition subdirectories.

14.19.2.5.5 /sys/fs/ — Filesystem-Specific Controls

Directory	Content
cgroup/	cgroup filesystem mount controls (Section 17.2)
ext4/	Per-mount ext4 tuning (when ext4 driver is active)
fuse/	FUSE connection controls (Section 14.11)
selinux/	SELinux policy interface (duplicate of securityfs for compat)

14.19.2.5.6 /sys/kernel/ — Kernel Parameters

Directory	Content
mm/	Memory management controls (transparent_hugepage/, hugepages/, ksm/)
debug/	debugfs mount point (when debugfs is mounted here)
security/	securityfs mount point. security/ima/ascii_runtime_measurements output is scoped to the reading task's IMA namespace (Section 9.5).
tracing/	tracefs mount point
irq/	Default IRQ affinity
uevent_seqnum	Monotonic uevent sequence number (u64, for udev ordering)
kexec_loaded	Always 0 in UmkaOS. Live kernel evolution (Section 13.18) replaces kexec; the traditional kexec path is not implemented. Retained for compatibility with tools that read this file.
kexec_crash_loaded	Always 0 in UmkaOS. Crash recovery uses the live evolution infrastructure, not kexec-based kdump. Retained for compatibility.
vmcoreinfo	Crash dump layout information

14.19.2.5.7 /sys/module/ — Loaded Modules and Parameters

One directory per loaded module (including built-in modules that have parameters). Each module directory contains:

Entry	Content
parameters/	One file per module parameter. Read returns current value; write changes it (if the parameter is writable).
refcnt	Module reference count.
coresize	Size of the module's core section in bytes.
initsize	Size of the module's init section (0 after init completes).
holders/	Symlinks to modules that depend on this one.

14.19.2.5.8 /sys/power/ — Power Management

Entry	Mode	Description
state	0o644	Write mem, disk, freeze to trigger system suspend. Read returns available states. See Section 7.5.
wakeup_count	0o644	Wakeup event counter for suspend synchronization.
mem_sleep	0o644	Preferred suspend-to-RAM variant (s2idle, shallow, deep).
disk	0o644	Hibernate method (platform, shutdown, reboot, suspend).
pm_async	0o644	1 = async device suspend/resume (default), 0 = sequential.
image_size	0o644	Maximum hibernate image size in bytes.
resume	0o200	Write MAJOR:MINOR to set the resume device for hibernation.

14.19.2.6 uevent Mechanism

Kobject state changes generate uevents that are delivered to userspace via two channels: a netlink multicast socket (NETLINK_KOBJECT_UEVENT, group 1) and the /sys/*/uevent file.

/// Uevent actions. Each action generates a NETLINK_KOBJECT_UEVENT
/// message and updates the kobject's `uevent` sysfs file.
#[derive(Clone, Copy, Debug)]
pub enum KobjAction {
    /// Device/kobject added. udev creates /dev nodes and runs rules.
    Add,
    /// Device/kobject removed. udev removes /dev nodes.
    Remove,
    /// Device state changed (e.g., firmware loaded, link state changed).
    Change,
    /// Kobject moved to a different parent (renamed).
    Move,
    /// Device brought online (e.g., CPU, memory block).
    Online,
    /// Device taken offline.
    Offline,
    /// Device binding to a driver.
    Bind,
    /// Device unbound from a driver.
    Unbind,
}

/// Environment variables included in every uevent message.
pub struct UeventEnv {
    /// Key-value pairs. Standard keys:
    /// - `ACTION`: "add", "remove", "change", "move", "online",
    ///   "offline", "bind", "unbind"
    /// - `DEVPATH`: sysfs path relative to /sys (e.g., "/devices/pci0000:00/...")
    /// - `SUBSYSTEM`: bus or class name (e.g., "pci", "net", "block")
    /// - `SEQNUM`: monotonic u64 sequence number (never wraps in 50+ years
    ///   at 10 billion events/sec)
    /// - `DEVTYPE`: device type within subsystem (e.g., "disk", "partition")
    /// - `DRIVER`: driver name (present for bind/unbind)
    /// - `MAJOR`, `MINOR`: device numbers (present for char/block devices)
    /// - `DEVNAME`: device name for /dev (e.g., "sda", "ttyS0")
    ///
    /// Additional subsystem-specific keys are appended by the device's
    /// `uevent()` callback.
    pub vars: ArrayVec<UeventVar, 32>,
}

/// Single uevent environment variable.
pub struct UeventVar {
    pub key: &'static str,
    pub value: ArrayVec<u8, 256>,
}

/// Emit a uevent for a kobject.
///
/// 1. Increment the global uevent sequence number (AtomicU64).
/// 2. Build the UeventEnv: standard keys + subsystem-specific keys
///    from the kobject's `uevent()` callback.
/// 3. Format the netlink message: `ACTION@DEVPATH\0KEY=VALUE\0...`
/// 4. Multicast via NETLINK_KOBJECT_UEVENT to all listeners (udevd).
/// 5. Write the formatted uevent to `/sys/<devpath>/uevent` for
///    manual re-trigger (`echo add > /sys/devices/.../uevent`).
pub fn kobject_uevent(kobj: &Arc<Kobject>, action: KobjAction) -> Result<(), Errno>;

/// Emit a uevent with additional environment variables.
/// Used when the subsystem needs to include extra key-value pairs
/// beyond what the standard `uevent()` callback provides.
pub fn kobject_uevent_env(
    kobj: &Arc<Kobject>,
    action: KobjAction,
    extra_env: &[UeventVar],
) -> Result<(), Errno>;

14.19.2.7 uevent Delivery Path

1. Driver or subsystem calls kobject_uevent(kobj, action).
2. Kernel increments global UEVENT_SEQNUM (AtomicU64, visible at /sys/kernel/uevent_seqnum).
3. UeventEnv is built: ACTION, DEVPATH, SUBSYSTEM, SEQNUM, plus subsystem-specific variables from the kobject's uevent() method.
4. Netlink multicast: the formatted message is sent to NETLINK_KOBJECT_UEVENT group 1. udevd receives the message, matches it against udev rules, and creates/removes /dev nodes, loads firmware, sets permissions, runs RUN commands, etc.
5. The uevent file in the kobject's sysfs directory is updated. Writing an action name to this file re-triggers the uevent (e.g., echo change > /sys/devices/.../uevent forces udev to re-process the device).

14.19.2.8 sysfs Namespace Awareness

sysfs is network-namespace-aware for /sys/class/net/: each network namespace (Section 17.1) sees only its own interfaces. Other sysfs hierarchies are shared across all namespaces (devices, buses, and classes other than net/ are global). This matches Linux behavior. Device entries under /sys/class/net/ are tagged with their owning network namespace; readdir() filters entries to show only devices in the caller's net namespace.

14.19.2.9 sysfs Binary Attributes

For attributes that are not human-readable text (firmware blobs, ACPI tables, PCIe config space), sysfs provides binary attributes:

/// Binary attribute: arbitrary-length read/write with offset support.
/// Used for firmware upload, PCI config space, ACPI tables, etc.
pub trait SysfsBinAttribute: Send + Sync {
    /// Attribute file name.
    fn name(&self) -> &'static str;
    /// POSIX permission bits.
    fn mode(&self) -> u16;
    /// Maximum file size (for `stat()` reporting and write bounds checking).
    fn size(&self) -> usize;
    /// Read into `buf` at `offset`. Returns bytes read.
    fn read(&self, kobj: &Kobject, buf: &mut [u8], offset: u64) -> Result<usize, Errno>;
    /// Write from `buf` at `offset`. Returns bytes written.
    fn write(&self, kobj: &Kobject, buf: &[u8], offset: u64) -> Result<usize, Errno> {
        Err(Errno::EACCES)
    }
}

14.19.3 Boot Initialization Order

procfs and sysfs are among the earliest pseudo-filesystems mounted, as many subsequent boot steps depend on them:

Order	Filesystem	Dependency	Notes
1	sysfs	VFS core initialized	Mounted before device probing begins; bus/class directories must exist for device registration.
2	procfs	VFS core + PID allocator	Mounted before PID 1 executes; glibc requires /proc/self/.
3	devtmpfs	sysfs	Device nodes reference sysfs kobjects.

Both filesystems are kernel-mounted (not user-mounted): the kernel mounts them during early init before transferring control to PID 1. This is unlike debugfs, tracefs, and other pseudo-filesystems which are mounted by userspace init.