Chapter 17: Containers and Namespaces¶

Namespace architecture (8 types), cgroups v2, POSIX IPC, OCI runtime

Type Definitions Used in This Part¶

/// Unique identifier for a schedulable task within the kernel.
/// Globally unique, never reused (monotonically increasing from boot;
/// allocated from `NEXT_TASK_ID` in create_task step 5, 0 reserved as sentinel).
/// Used for PID translation in PID namespaces. Canonical model:
/// [Section 8.1](08-process.md#process-and-task-management--process-identity-model).
pub type TaskId = u64;

/// Unique identifier for a process (thread group): the TaskId of the
/// process's ORIGINAL thread-group leader. Globally unique, never reused,
/// and immutable across collapse_thread_group()/exec (the PID_TABLE alias entry is
/// re-pointed instead). Corresponds to the TGID in Linux terminology;
/// userspace sees it only after per-namespace translation — getpid()
/// returns `pid_nr_in()`'s namespace-local number, never this value.
/// Used in `Process.pid`, parent/child tracking, ptrace, and
/// deferred-operation targeting. Canonical model:
/// [Section 8.1](08-process.md#process-and-task-management--process-identity-model).
pub type ProcessId = u64;

/// A **non-owning** handle to a physical page frame — just its frame number.
///
/// **Ownership contract**: `PhysPage` is `Copy` and does NOT own the frame.
/// Dropping a `PhysPage` frees nothing. The frame's lifecycle is governed by
/// the allocator's per-frame refcount in its `Page` descriptor
/// ([Section 4.2](04-memory.md#physical-memory-allocator)); code that must keep a frame alive holds a
/// reference through that refcount (`page_get`/`page_put`, or an `Arc<Page>`),
/// NOT by retaining a `PhysPage`. Consequently:
/// - **Pipe gifting** (`PipePage.page`, [Section 17.3](#posix-ipc)): `vmsplice(SPLICE_F_GIFT)`
///   transfers a frame reference into the pipe by calling `page_get` on the
///   gifted frame; the reference is released (`page_put`) when the buffer is
///   consumed by a reader or the pipe is torn down. The `PhysPage` stored in the
///   buffer records WHICH frame; the pipe's ownership is the held refcount.
/// - **Snapshot lists** (`ArrayVec<PhysPage, N>` in crash-recovery /
///   live-evolution): these record frame numbers whose backing storage is kept
///   alive by a separate reservation; the `PhysPage` values are addresses, not
///   owners. `Vec<Arc<PhysPage>>` (PMU sample buffers) refcounts the *handle
///   list*, again distinct from the frame refcount.
///
/// A `PhysPage` therefore crosses interfaces freely (by copy) without transfer
/// of ownership — the same discipline Linux applies to a bare `struct page *`.
#[derive(Clone, Copy, PartialEq, Eq, Debug)]
pub struct PhysPage {
    /// Physical frame number (PFN).
    pub pfn: u64,
}

impl PhysPage {
    /// The NUMA node this frame belongs to, derived from the PFN via the
    /// physical memory map ([Section 4.2](04-memory.md#physical-memory-allocator)). O(1) — a single
    /// `Page.node_id` read at a fixed stride from `VMEMMAP_BASE` (see
    /// `phys_to_numa_node` below); no allocation, no locking. Used e.g. in
    /// `futex` shared-mapping node selection.
    pub fn numa_node(&self) -> u32 {
        phys_to_numa_node(self.pfn)
    }
}

/// Map a physical frame number to its NUMA node via the frame's `Page`
/// descriptor in the vmemmap ([Section 4.2](04-memory.md#physical-memory-allocator)). O(1) — a fixed
/// stride from `VMEMMAP_BASE`; `Page.node_id` is populated at boot from ACPI
/// SRAT / devicetree. No allocation, no locking.
fn phys_to_numa_node(pfn: u64) -> u32 {
    let vaddr =
        arch::current::mm::VMEMMAP_BASE + pfn as usize * core::mem::size_of::<Page>();
    // SAFETY: every present PFN has a mapped `Page` descriptor in the vmemmap;
    // callers pass PFNs of allocated frames.
    let page: &'static Page = unsafe { &*(vaddr as *const Page) };
    page.node_id
}

/// Wait queue head for blocking operations.
/// Used by pipe buffers to block readers/writers.
/// Defined in Section 3.1.6 (umka-nucleus/src/sync/wait.rs).
// WaitQueueHead is defined in Section 3.1.6.2 (03-concurrency.md).
// See that section for the full struct definition and wait/wake protocol.
pub use WaitQueueHead;

/// RCU-protected non-null cell for read-mostly data.
///
/// **NON-NORMATIVE STUB.** The canonical definition — struct, `new(T) -> Result`,
/// the `const` `new_static(&'static T)` const-construction constructor,
/// `read(&RcuReadGuard) -> &T`, `update(T, &impl WriterProof) -> Result`, and
/// `Drop` — lives at
/// [Section 3.4](03-concurrency.md#cumulative-performance-budget--rcu-protected-container-types).
/// This placeholder exists only so this chapter's field declarations name the
/// type; do NOT re-specify the API here (the earlier `new_empty`/`load() ->
/// Option<Arc<T>>`/`update(Arc<T>)` surface was divergent and never existed on
/// the canonical type). Reads return `&T` under an `RcuReadGuard`; writers
/// serialize on an external exclusive lock (`Mutex`, `SpinLock`, or the
/// `RwLock` write half) and pass its guard as the sealed `WriterProof`.
pub struct RcuCell<T> { _phantom: core::marker::PhantomData<T> }

/// Nullable single-owner RCU pointer (thinner than `RcuCell`; no internal `Arc`).
///
/// **NON-NORMATIVE STUB.** Canonical definition — `null()` (`const`),
/// `new(T) -> Result`, `read(&RcuReadGuard) -> Option<&T>`,
/// `update(Option<T>, &impl WriterProof) -> Result`, `Drop` — at
/// [Section 3.4](03-concurrency.md#cumulative-performance-budget--rcu-protected-container-types). Use for
/// values that may be absent (e.g. the maple-tree root, an uninitialized slot);
/// store an `Arc<U>` when readers must hold a reference past the RCU section.
pub struct RcuPtr<T> { _phantom: core::marker::PhantomData<T> }

/// Integer-to-object ID allocator (analogous to Linux `struct idr`).
/// NOT defined here — the CANONICAL definition (struct, concurrency model,
/// `IdrError`, and the full normative API: `new` / `idr_alloc` /
/// `idr_alloc_range` / `lookup(id, &RcuReadGuard)` / `idr_remove` /
/// `idr_replace` / `iter(&RcuReadGuard)`) lives at
/// [Section 3.6](03-concurrency.md#lock-free-data-structures--idrt-integer-id-allocator).
/// `Idr` is a lowest-free RECYCLING allocator (Linux IDR semantics) —
/// `PidNamespace.pid_map` requires exactly that discipline. Never-reused
/// kernel identifiers (TaskId, ProcessId) are monotonic `AtomicU64`
/// counters, not Idr instances.
pub struct Idr<T> { _phantom: core::marker::PhantomData<T> }

/// RCU-protected integer-KEYED map (radix tree) — DESPITE the `Idr` in
/// the name, this type is NOT an allocator: callers supply explicit u32
/// keys (there is no `idr_alloc` here). Its one consumer class is the
/// reverse-translation cache pattern (`PidNamespace.reverse_map`: key =
/// the LOWER 32 BITS of a u64 kernel id, truncated by the CALLER as
/// `id as u32` at every insert/lookup/remove site — the truncation point
/// is the call site, uniformly, so a u64-keyed insert can never silently
/// mismatch u32-keyed lookups).
///
/// Lookups are lock-free (RCU read guard only); insert/remove acquire an
/// internal write lock and publish via RCU. Values are `Copy` and are
/// returned BY VALUE from `lookup` (no guard-lifetime borrow to manage;
/// the canonical `Idr::lookup` returns a guard-borrowed reference — the
/// two APIs intentionally differ). Underlying RCU primitives:
/// [Section 3.4](03-concurrency.md#cumulative-performance-budget--rcu-protected-container-types).
///
/// # Canonical API (normative — this is the type's home)
/// ```rust
/// impl<T: Copy> RcuIdr<T> {
///     pub const fn new() -> Self;
///     /// Insert or overwrite the value at `key`. Internal write lock;
///     /// RCU-published — concurrent readers see old or new, never torn.
///     pub fn insert(&self, key: u32, value: T);
///     /// Lock-free read under the caller's RCU guard. Returns the
///     /// value by copy; `None` if the key is absent.
///     pub fn lookup(&self, key: u32, guard: &RcuReadGuard) -> Option<T>;
///     /// Remove the entry (no-op if absent). Internal write lock; the
///     /// removed node is freed after an RCU grace period.
///     pub fn remove(&self, key: u32);
/// }
/// ```
pub struct RcuIdr<T> { _phantom: core::marker::PhantomData<T> }

/// RCU-protected hash map: lock-free reads under an RCU guard, **per-bucket**
/// serialized writes.
///
/// **NON-NORMATIVE STUB.** Canonical definition — `new()` (`const`),
/// `lookup(&K, &RcuReadGuard) -> Option<V>`, `insert(K, V) -> Result`,
/// `remove(&K)`, `for_each`, the per-bucket `SpinLock` write discipline, and
/// the power-of-two resize policy — at
/// [Section 3.4](03-concurrency.md#cumulative-performance-budget--rcu-protected-container-types). Writers
/// take only the target bucket's lock (NOT a whole-table clone-and-swap), so
/// the map is correct for read-hot AND write-frequent consumers such as
/// `tcp_ehash` (per-connection insert/remove); it is NOT restricted to
/// "written rarely".
pub struct RcuHashMap<K, V> { _phantom: core::marker::PhantomData<(K, V)> }

/// RCU-protected immutable-snapshot vector: lock-free reads under an RCU guard,
/// whole-array replacement under an external exclusive writer lock.
///
/// **NON-NORMATIVE STUB.** Canonical definition — `new()` (`const`, null
/// snapshot), `load(&RcuReadGuard) -> &[T]` (single atomic load, no torn read),
/// `update(&[T], &impl WriterProof) -> Result`, the exact-length length-prefixed
/// snapshot allocation protocol, and `Drop` — at
/// [Section 3.4](03-concurrency.md#cumulative-performance-budget--rcu-protected-container-types). There is
/// no spare-capacity field: each snapshot is allocated to its exact length.
pub struct RcuVec<T> { _phantom: core::marker::PhantomData<T> }

/// Namespace type enumeration for hierarchy tracking.
/// UmkaOS implements all 8 Linux namespace types (see Section 8.1.6).
///
/// Uses sequential kernel-internal values (`#[repr(u8)]`). These do NOT
/// correspond to the CLONE_NEW* bitflags passed by userspace (e.g.,
/// CLONE_NEWPID = 0x20000000, CLONE_NEWNET = 0x40000000). Translation
/// from CLONE_NEW* bitflags happens at the syscall boundary via
/// `clone_flag_to_ns_type()` below.
#[repr(u8)]
pub enum NamespaceType {
    Pid    = 0,
    Net    = 1,
    Mnt    = 2,
    Uts    = 3,
    Ipc    = 4,
    User   = 5,
    Cgroup = 6,
    Time   = 7, // Linux 5.6+
}

/// Convert a single CLONE_NEW* bitflag (from clone(2) / unshare(2) flags)
/// to the kernel-internal `NamespaceType`.
///
/// Callers must iterate over each set bit in the `clone_flags` word and
/// call this function once per bit. Returns `None` for bits that are not
/// namespace flags (e.g., CLONE_VM, CLONE_FILES).
pub fn clone_flag_to_ns_type(bit: u64) -> Option<NamespaceType> {
    match bit {
        libc::CLONE_NEWPID    => Some(NamespaceType::Pid),
        libc::CLONE_NEWNET    => Some(NamespaceType::Net),
        libc::CLONE_NEWNS     => Some(NamespaceType::Mnt),
        libc::CLONE_NEWUTS    => Some(NamespaceType::Uts),
        libc::CLONE_NEWIPC    => Some(NamespaceType::Ipc),
        libc::CLONE_NEWUSER   => Some(NamespaceType::User),
        libc::CLONE_NEWCGROUP => Some(NamespaceType::Cgroup),
        libc::CLONE_NEWTIME   => Some(NamespaceType::Time),
        _                     => None,
    }
}

Note on Capability<T> syntax: This document uses Capability<NetStack> and Capability<VfsNode> as type hints indicating what resource a capability references. The underlying Capability struct (Section 9.1) is non-generic; the target type is determined by the object_id field. This notation is for documentation clarity only.

17.1 Namespace Architecture¶

Linux namespaces isolate global system resources. In UmkaOS, namespaces are not primitive kernel objects; rather, they are synthesized from UmkaOS's native Capability Domains (Section 9.1) and Virtual Filesystem (VFS) mounts.

17.1.1 Capability Domain Mapping¶

When a process creates a new namespace via clone(CLONE_NEW*), clone3(), or unshare() (CLONE_NEWTIME only via unshare(2)/clone3(2) — its flag value aliases the legacy-clone exit-signal byte; see the CLONE_NEWTIME bullet), UmkaOS allocates a new Capability Domain or modifies the existing one:

CLONE_NEWPID (PID Namespace): Creates a new PID translation table in the process's Capability Domain. The umka-sysapi layer translates local PIDs (e.g., PID 1) to global UmkaOS task IDs.
CLONE_NEWNET (Network Namespace): Creates an isolated network stack instance. On clone(CLONE_NEWNET), the new network namespace's user_ns is set to the creating credential's user namespace — child_cred.user_ns in create_task() (which is the NEW user namespace when CLONE_NEWUSER was combined in the same call, because the CLONE_NEWUSER credential transformation runs first), pending_cred.user_ns in sys_unshare() (same reason), and the caller's committed task.cred.user_ns otherwise. Never current_task().namespace_set.user_ns: during create_task() the current task is the PARENT, whose namespace_set still names the OLD user namespace — recording that as owner would leave the container init (uid 0 with CAP_FULL_SET in the new user ns, zero capabilities in the old one) unable to administer its own namespaces: every has_ns_cap(net_ns.user_ns, CAP_NET_ADMIN) check would resolve against a namespace where it holds nothing. This owner-assignment rule applies uniformly to EVERY namespace type created in the same call (net, pid, mount, uts, ipc, cgroup, time — see each type's user_ns field doc), and is the namespace-side twin of the has_ns_cap() rule that capability scope comes from cred.user_ns, not namespace_set.user_ns (Section 9.9).
The new namespace has no network interfaces except lo (loopback, 127.0.0.1/8)
No connectivity to the host or external network unless explicitly configured
Network interfaces (physical NICs, VETH pairs, bridges, VLANs) are owned by a specific namespace and cannot be accessed from other namespaces
Each namespace has its own routing table, iptables/nftables rules, and socket port space
Per-namespace network state is defined below; the umka-net subsystem (Section 16.1-38) implements the network stack that operates within these namespace boundaries:

/// Network interface table. Uses XArray for O(1) lookup on the
/// packet receive/transmit path (integer-keyed → XArray per collection policy).
/// Supports up to 2^32 interfaces; typical deployments have 2-16.
pub struct InterfaceTable {
    /// XArray indexed by InterfaceIndex (u32). O(1) lookup, RCU-compatible
    /// reads, ordered iteration for enumeration. Ascending-index XArray
    /// iteration IS the netlink `RTM_GETLINK` / `/proc/net/if_inet6`
    /// enumeration order (stable per namespace).
    table: XArray<Arc<NetDevice>>,
    /// Per-namespace ifindex rotor. **ifindex is a PER-NAMESPACE identity
    /// space** (Linux `struct net`'s `ifindex`, `dev_new_index()`): every
    /// namespace numbers its own devices from 1 (`lo`) independently, so the
    /// same small index recurs across containers — there is NO global device
    /// list or global index allocator. Seeded to 2 by `InterfaceTable::new()`
    /// (index 1 is reserved for the loopback assigned at namespace creation).
    /// u32 matches the netlink `ifi_index` wire width; the rotor is monotone
    /// and never resets, so an index freed by device removal is only reused
    /// after a full u32 wrap (a churn/50-year concern handled by the
    /// collision probe in `alloc_ifindex()`).
    next_ifindex: AtomicU32,
}

impl InterfaceTable {
    /// Empty table with the ifindex rotor seeded past the reserved loopback
    /// index (1). Used in the `NetNamespace` initial-state literal.
    pub fn new() -> Self {
        Self { table: XArray::new(), next_ifindex: AtomicU32::new(2) }
    }

    /// Allocate the next free per-namespace ifindex (Linux `dev_new_index()`).
    /// `fetch_add`-and-probe: bump the rotor, skip 0 and any index currently
    /// live in `table` (a live index recurs only after the u32 rotor wraps).
    /// The caller holds `NetNamespace.config_lock` (device add is a warm,
    /// config-locked path), so the bump + probe is atomic w.r.t. other
    /// registrations and the returned index cannot be double-assigned.
    pub fn alloc_ifindex(&self) -> u32 {
        loop {
            let idx = self.next_ifindex.fetch_add(1, Ordering::Relaxed);
            if idx == 0 {
                continue; // never hand out 0 (Linux: valid ifindex >= 1)
            }
            if self.table.load(idx as u64).is_none() {
                return idx;
            }
        }
    }

    /// Register a device under its already-assigned `dev.ifindex` (the
    /// loopback path, which pins index 1, and `register_netdev()`, which
    /// assigns via `alloc_ifindex()` first). Warm path; `config_lock` held.
    pub fn register(&self, dev: Arc<NetDevice>) {
        self.table.xa_store(dev.ifindex as u64, dev);
    }

    /// Erase every device from the table (namespace teardown,
    /// `net_ns_cleanup()`). Each erased `Arc<NetDevice>` drops its
    /// `NetDevice.net_ns` passive edge; the loopback goes with the rest.
    /// `config_lock` held (teardown is single-writer).
    pub fn remove_all(&self) {
        for idx in self.table.keys() {
            self.table.xa_erase(idx);
        }
    }

    /// Invoke `f` for each registered device, in ascending-ifindex order (the
    /// stable `RTM_GETLINK` / `/proc/net/if_inet6` enumeration order the
    /// `table` field documents). RCU read — readers are lock-free; concurrent
    /// registration/removal is serialized by `NetNamespace.config_lock` on the
    /// write side. Used by device-enumeration callers such as the TX qdisc
    /// rescan fallback ([Section 16.21](16-networking.md#traffic-control-and-queue-disciplines)) and
    /// netlink `RTM_GETLINK` dumps.
    pub fn for_each<F: FnMut(&NetDevice)>(&self, mut f: F) {
        for idx in self.table.keys() {
            if let Some(dev) = self.table.load(idx) {
                f(dev);
            }
        }
    }
}

/// Per-namespace packet-filtering anchor — a THIN handle into the BPF
/// hook-attachment machinery. **UmkaOS has no native nftables/iptables rule
/// engine** (design decision, 2026-07-05): one filtering engine exists — eBPF
/// — and every legacy ABI (nft netlink `NFT_MSG_*`, iptables `setsockopt`) is
/// translated to BPF programs at the syscall/netlink boundary. The machinery,
/// the packet-path invocation (`nf_hook_run()` and its ~1-2-cycle empty-hook
/// fast path), the conntrack builtins, and the normative translation surface
/// (what translates, what returns an honest error) live in
/// [Section 16.18](16-networking.md#packet-filtering-bpf-based) — this struct only anchors the per-netns
/// state. Merit: one verified, JIT-compiled engine instead of Linux's
/// parallel native-nf_tables VM + BPF/XDP data planes; legacy ABIs are
/// translated at the edge, and every installed rule runs as native code.
pub struct FirewallRules {
    /// The hook attachment slots (DATA PLANE): priority-ordered,
    /// RCU-published program lists per (family, hook) for the five IP hooks.
    /// `const`-constructed, allocation-free while empty; a namespace with no
    /// rules pays ~1-2 cycles per hook site
    /// ([Section 16.18](16-networking.md#packet-filtering-bpf-based--netfilter-hook-attachment-points)).
    /// Mutated only under `NetNamespace.config_lock`, whose `MutexGuard` is
    /// the `WriterProof` for the slot updates.
    pub hooks: NetfilterHooks,
    /// Control-plane shadow of the translated rulesets (nft tables/chains/
    /// rules/sets as round-trip netlink payloads, iptables table blobs) —
    /// the source for netlink dumps (`nft list ruleset`, `iptables-save`)
    /// and for recompilation on rule changes. Includes `next_handle`, the
    /// per-namespace monotonic u64 chain/rule/set handle allocator (never
    /// reused; 50-year safe). Cold path only — a sleeping `Mutex` is
    /// correct. Canonical type:
    /// [Section 16.18](16-networking.md#packet-filtering-bpf-based--legacy-abi-translation-surface).
    pub nft_shadow: Mutex<NftRulesetShadow>,
    /// Monotonically increasing generation counter. Incremented (Release)
    /// once per committed nft transaction / iptables table replace / direct
    /// BPF-link attach or detach. Read (Acquire) by `nft monitor`
    /// (`NFT_MSG_NEWGEN`/`GETGEN`) and by the conntrack re-evaluation path
    /// to detect stale rule evaluations.
    pub generation: AtomicU64,
}

/// Per-namespace network state.
pub struct NetNamespace {
    /// Namespace ID (unique across the system).
    pub ns_id: u64,

    /// Network interface table. Uses XArray for O(1) lookup on the
    /// packet receive/transmit path (integer-keyed → XArray per collection policy).
    ///
    /// **RCU-protected natively by XArray**: Interface lookup is on the per-packet
    /// hot path (every incoming and outgoing packet resolves its interface). XArray
    /// provides lock-free RCU reads natively — readers call `xa_load()` under
    /// `rcu_read_lock()`. Writers (interface add/remove, rare) call `xa_store()`
    /// / `xa_erase()` which publish entries via XArray's internal RCU mechanism.
    /// No clone-and-swap needed — per-entry O(log₆₄ N) updates. This matches
    /// Linux's RCU-protected `net_device` lookup exactly — lock-free reads,
    /// serialized writes. The write path holds `config_lock` (below) for
    /// serialization.
    pub interfaces: InterfaceTable,

    /// Loopback interface (always present, cannot be deleted).
    pub loopback: Arc<NetDevice>,

    /// Routing table (per-namespace, not shared).
    ///
    /// **RCU-protected natively by FIB trie internals**: Route lookup is on the
    /// per-packet forwarding path. The FIB trie uses per-entry RCU publishing
    /// for lock-free reads — `ip route add/del` modifies individual trie nodes
    /// with O(log N) cost, not O(total routes) clone-and-swap. The write path
    /// holds `config_lock` (below) for serialization. Linux uses RCU for FIB
    /// (Forwarding Information Base) lookup with the same per-entry pattern.
    pub routes: RouteTable,

    /// Packet filtering (the iptables/nftables ABI surface, BPF-only engine).
    /// Rules are scoped to this namespace only.
    ///
    /// Thin anchor into the BPF hook-attachment machinery
    /// ([Section 16.18](16-networking.md#packet-filtering-bpf-based--netfilter-hook-attachment-points)):
    /// the per-packet path calls `nf_hook_run()` at the five hook sites —
    /// one atomic-bitmask load (~1-2 cycles) when no programs are attached,
    /// one RCU slice read + JIT'd program calls otherwise. Writers (nft
    /// transactions, iptables replaces, direct BPF links) mutate the hook
    /// slots under `config_lock` (its guard is the `WriterProof`), then
    /// increment `firewall.generation` (Release).
    pub firewall: FirewallRules,

    /// Mutex for serializing configuration mutations (interface add/remove,
    /// route updates, firewall rule changes). Only the write side holds this;
    /// packet-path readers never touch it. Separating the write-side lock from
    /// the read-side RCU ensures that configuration changes do not block
    /// packet processing.
    pub config_lock: Mutex<()>,

    /// Port allocation bitmap (per-namespace).
    /// Allows the same port number to be bound in different namespaces.
    /// Mutex is correct here: port allocation happens on bind()/connect(),
    /// not on the per-packet path. See `PortAllocator` below.
    pub port_allocator: Mutex<PortAllocator>,

    /// Owning user namespace. Required for capability checks
    /// (CAP_NET_ADMIN, CAP_NET_RAW, CAP_NET_BIND_SERVICE) by networking
    /// subsystems that call `has_ns_cap(net_ns.user_ns, cap)`.
    pub user_ns: Arc<UserNamespace>,

    /// Handle to this network stack's capability (`Capability<NetStack>`
    /// in the doc notation — see the notation note in the chapter intro).
    /// Used for delegation; processes in this namespace implicitly hold
    /// the capability. Stored as `CapHandle` (index + generation into the
    /// capability table, [Section 9.1](09-security.md#capability-based-foundation)) — the handle
    /// is what `revoke_capability()` takes at namespace teardown.
    ///
    /// `OnceCell`: the capability registry entry wraps the
    /// `Arc<NetNamespace>` itself, so it can only be allocated AFTER
    /// `net_ns_alloc()` produced the Arc — a plain field would require a
    /// mutating write through the shared Arc (un-compilable). The cell is
    /// set exactly once by `net_stack_cap_create()` in the creation
    /// sequence (Phase 3 of the loopback registration steps below),
    /// before the namespace is shared. Readers use `.get()`; `Drop`
    /// revokes only when set (an alloc-failure rollback drops a
    /// namespace whose cell was never set — nothing to revoke).
    pub stack_cap: OnceCell<CapHandle>,

    /// Per-namespace connection tracking table. In Linux, conntrack state is
    /// per-network-namespace: each container's netns maintains its own connection
    /// tracking entries, independent of the host and other containers. NAT rules,
    /// stateful firewall decisions, and connection reuse all operate against this
    /// namespace-scoped table.
    pub conntrack: ConntrackTable,

    /// ARP neighbor table (IPv4 → MAC resolution). One `NeighborTable`
    /// ([Section 16.7](16-networking.md#neighbor-subsystem) — the canonical definition) per protocol
    /// per namespace, matching Linux's per-netns `arp_tbl` scoping.
    /// Entry lookups on the packet output path are RCU reads inside the
    /// table; creation/GC serialize on the table's internal per-bucket
    /// write locks, NOT on `config_lock`. Empty at namespace creation
    /// (initial-state item 5 below).
    pub neigh_v4: NeighborTable,

    /// NDP neighbor table (IPv6 → MAC resolution). Same structure and
    /// locking as `neigh_v4`; RFC 4861 state machine per
    /// [Section 16.7](16-networking.md#neighbor-subsystem).
    pub neigh_v6: NeighborTable,

    /// Active-user count — the number of `NamespaceSet` objects that
    /// currently name this namespace in their `net_ns` field (the tasks
    /// living in it). This is the **administrative liveness** signal, DISTINCT
    /// from the `Arc<NetNamespace>` strong count (which also includes the
    /// PASSIVE references held by every live socket's `SockCommon.net_ns` and
    /// every `NetDevice.net_ns`). Incremented by `net_ns_get()` when a
    /// `NamespaceSet` gains this net_ns, decremented by `net_ns_put()` when
    /// one drops it; the `1 → 0` transition triggers `net_ns_cleanup()` (the
    /// active teardown — see "Network Namespace Teardown" below). Two counts
    /// are REQUIRED because the socket/device passive edges form true refcount
    /// cycles with the `NetNamespace` (`socket_list`↔`SockCommon.net_ns`,
    /// `interfaces`↔`NetDevice.net_ns`): a pure strong-count-to-zero trigger
    /// is unreachable while any socket or device exists. The same
    /// active-vs-passive split resolves the identical cycle in Linux
    /// (`net/core/net_namespace.c`).
    /// `INIT_NET_NS` keeps `users >= 1` for the kernel's lifetime (the
    /// per-exit tombstone `NamespaceSet`s reference it), so it never tears down.
    pub users: AtomicU32,
    /// Death latch. `false` for the namespace's whole active life; set `true`
    /// — ONCE, never cleared — by `net_ns_cleanup()` at the `users → 0`
    /// transition, BEFORE the socket close-walk. A set latch is the
    /// "administratively dead" state the socket layer checks: operations on a
    /// socket whose `net_ns.dead` is set return `EIO`
    /// ([Section 16.3](16-networking.md#socket-abstraction); the "sockets bound to a destroyed namespace
    /// return EIO" rule). Sockets held open via `SCM_RIGHTS` in OTHER
    /// namespaces keep the `Arc<NetNamespace>` alive (passive) but see it dead.
    pub dead: AtomicBool,
    /// Enumeration index of every socket created in this namespace — the
    /// teardown close-walk target, NOT an ownership edge. Elements are
    /// **`Weak<dyn SocketOps>`** (the socket-abstraction registration type,
    /// [Section 16.3](16-networking.md#socket-abstraction)), not a strong `Arc`: an `Arc` here would form
    /// a refcount cycle with `SockCommon.net_ns: Arc<NetNamespace>` (the reverse
    /// strong edge), so the namespace could never reach a teardown state while
    /// any socket existed. Sockets register a `Weak` on creation (`socket()`
    /// syscall) and stale entries (`upgrade() == None`) are pruned lazily during
    /// the close-walk. Guarded by a real `Mutex<Vec<..>>` — NOT the detached
    /// `config_lock: Mutex<()>` — so the mutation is actually lock-protected
    /// (socket create/destroy is off the per-packet hot path). Unbounded `Vec`
    /// is acceptable per collection policy for this warm/cold path; the
    /// maximum is bounded by `ulimit -n` and the namespace's fd count.
    pub socket_list: Mutex<Vec<Weak<dyn SocketOps>>>,
}

impl NetNamespace {
    /// Select the per-protocol neighbor table — THE accessor used by
    /// `neighbor_resolve_and_xmit()` on the packet output path
    /// ([Section 16.2](16-networking.md#network-stack-architecture--neighbour-subsystem-arp-ndp)).
    ///
    /// Only IP families resolve L3→L2 neighbors: callers pass
    /// `AddressFamily::Inet` or `AddressFamily::Inet6`, derived from the
    /// next-hop address (never from user input). Any other family is a
    /// caller bug — debug-asserted; the release-build fallback to the ARP
    /// table is fail-safe (the lookup simply misses and resolution fails
    /// with `EHOSTUNREACH`, no state corruption).
    pub fn neigh_table(&self, family: AddressFamily) -> &NeighborTable {
        match family {
            AddressFamily::Inet6 => &self.neigh_v6,
            AddressFamily::Inet => &self.neigh_v4,
            _ => {
                debug_assert!(false, "neigh_table: non-IP address family");
                &self.neigh_v4
            }
        }
    }
}

/// Per-namespace connection tracking state. Each network namespace maintains its
/// own conntrack table so that NAT translations, stateful firewall decisions, and
/// connection reuse are fully isolated between containers.
///
/// **Internal structure is NOT the generic `RcuHashMap`.** The canonical
/// conntrack hash is the **per-bucket-spinlock chained hash with RCU-protected
/// lookup** specified as the deep design in
/// [Section 16.18](16-networking.md#packet-filtering-bpf-based) (boot-computed bucket count, EWMA-driven
/// RCU-safe resize, per-CPU slab entry allocation). A single internal writer
/// lock — which is all a generic `RcuHashMap` exposes — would serialize all
/// insertions and FAIL that section's 256-CPU contention analysis, whose write
/// path is per-bucket. This field DEFERS to that structure rather than
/// re-specifying it; the earlier `RcuHashMap<ConntrackKey, ConntrackEntry>`
/// re-spec was incorrect. Packet-path lookups are lock-free RCU reads; entry
/// create/destroy takes only the target bucket's spinlock.
pub struct ConntrackTable {
    /// Active connection tracking entries — `ConntrackHashTable`, the
    /// per-bucket-spinlock chained hash exported by name from
    /// [Section 16.18](16-networking.md#packet-filtering-bpf-based) (boot-computed bucket count,
    /// EWMA-driven RCU-safe resize, per-CPU slab entry allocation).
    pub entries: ConntrackHashTable,
    /// Total number of active entries (for `/proc/sys/net/netfilter/nf_conntrack_count`
    /// per-namespace accounting and early-drop threshold enforcement).
    pub count: AtomicU64,
}

/// The independent L4 ephemeral port spaces. TCP and UDP each own a FULL 16-bit
/// port space (Linux binds them through separate tables — `tcp_hashinfo.bhash`
/// vs Linux `udp_table`), so the SAME port number can be bound simultaneously in each:
/// TCP:53 + UDP:53 on every DNS server, TCP:443 + UDP:443 (QUIC) on every HTTP/3
/// server. A single shared bitmap would fail the second protocol's `bind()` with
/// EADDRINUSE — an ABI-visible break for unmodified userspace. RAW sockets do
/// not bind ports; only TCP and UDP carry ephemeral allocation, which is all
/// this allocator serves.
#[derive(Clone, Copy, PartialEq, Eq)]
#[repr(u8)]
pub enum PortSpace {
    /// TCP bind space (`tcp_hashinfo.bhash`).
    Tcp = 0,
    /// UDP bind space (its own protocol-specific bitmap).
    Udp = 1,
}

impl PortSpace {
    /// Number of independent port spaces — the `PortAllocator.spaces` array
    /// dimension.
    pub const COUNT: usize = 2;
}

/// Per-protocol ephemeral port state: an independent rotor + in-use bitmap for
/// one `PortSpace`.
///
/// The bitmap has 1024 words of 64 bits each = 65,536 bits. Bit N corresponds
/// to port N **in this protocol's space**. Ports 0-1023 (well-known) are set in
/// every space's bitmap at init time and never allocated by the rotor.
pub struct PortSpaceState {
    /// Next candidate port (rotor) for this protocol. Wraps within `range`.
    /// **Lock-free algorithm**: `next` is atomically incremented via
    /// `fetch_add(1, Relaxed)`. If the result exceeds `range.1`, it wraps
    /// to `range.0` via modular arithmetic. The `Mutex<PortAllocator>` in
    /// NetNamespace serializes the full allocate-and-set-bitmap operation
    /// (not the rotor read), so two concurrent allocators may read the same
    /// `next` value — the bitmap collision check resolves this (both try
    /// to set the same bit; the Mutex serializes the CAS on the bitmap word).
    pub next: AtomicU16,
    /// Bitmap of in-use ports for this protocol. 1024 words × 64 bits = 65,536
    /// ports. Bit set = port in use. Atomic words for lock-free read on
    /// bind-time collision check (write serialized by `Mutex<PortAllocator>`
    /// in `NetNamespace`).
    pub bitmap: [AtomicU64; 1024],
}

/// Per-namespace ephemeral port allocator. Covers the ephemeral port range
/// 1024-65535 (64,512 ports) **per protocol space**. Allocation uses a rotor
/// (`next`) for O(1) average-case and a bitmap for collision detection.
///
/// The TCP and UDP spaces are fully independent (see `PortSpace`); the `range`
/// field allows `/proc/sys/net/ipv4/ip_local_port_range` tunability per
/// namespace and applies to both spaces (Linux applies one range to both).
pub struct PortAllocator {
    /// Inclusive (low, high) ephemeral port range. Default: (32768, 60999).
    /// Configurable via `/proc/sys/net/ipv4/ip_local_port_range`. Shared by
    /// both protocol spaces (Linux applies one range to both).
    pub range: (u16, u16),
    /// One independent rotor+bitmap per L4 port space, indexed by `PortSpace`
    /// (`spaces[PortSpace::Tcp as usize]` / `spaces[PortSpace::Udp as usize]`).
    /// Separate spaces are what let TCP:53 and UDP:53 coexist.
    pub spaces: [PortSpaceState; PortSpace::COUNT],
}

PortAllocator provides namespace-scoped ephemeral port allocation. The per-port bitmap is the single canonical model — there is NO range-reservation layer — but the bitmap is per protocol space (PortSpace::Tcp / PortSpace::Udp), never one shared bitmap. Per-protocol tables (UdpTable.ephemeral_next, TcpTable.ephemeral_next) hold only a per-CPU rotor HINT into the ephemeral range; the authoritative allocation is a bit set in the MATCHING PortAllocator.spaces[space].bitmap under Mutex<PortAllocator>. There is no reserved-range list and no per-protocol lease granularity (the earlier "reserves ranges" wording is retracted).

Allocate/release protocol (the release side the two models both left unspecified). All bitmap operations below act on the socket's own PortSpace (spaces[space].bitmap / spaces[space].next), so a TCP bind and a UDP bind on the same port number touch different bitmaps and never collide: - Allocate (bind() to port 0, or connect() needing an ephemeral port): under the Mutex, advance THIS space's rotor within range, find the first clear bit in THIS space's bitmap, set it, return the port. Bit set ⇒ port owned in this protocol's space. Exhaustion (termination + errno): the scan terminates after ONE full rotor wrap over range — at most range.1 - range.0 + 1 probes; the Mutex makes wrap detection exact (snapshot the starting rotor position, stop when the probe returns to it with no clear bit found). Every bit in range set ⇒ no port available: bind() to port 0 fails with EADDRINUSE; an ephemeral-port connect() fails with EADDRNOTAVAIL — the two paths report DIFFERENT errnos (Linux parity: Linux inet_csk_get_port() bind exhaustion vs __inet_hash_connect() ephemeral exhaustion). Never an unbounded scan, never an unspecified error. - Explicit bind release: close() of the LAST socket bound to a port clears its bit in that socket's space. For UDP/raw this is immediate (Section 16.3, "releases port binding"). - TCP TIME_WAIT interlock: a TCP port entering TIME_WAIT stays OWNED (bit set in the TCP space) for the 2·MSL lifetime — the TIME_WAIT mini-socket, not the closed TcpCb, holds the bit; the bit is cleared when the TIME_WAIT timer fires. This prevents reuse of a 4-tuple whose old segments may still be in flight (the UDP space is unaffected). - SO_REUSEADDR / SO_REUSEPORT: multiple sockets may share ONE port, so a single bit cannot be one-socket-per-bit. The bit means "at least one owner"; a per-port owner refcount (in the per-protocol table's port entry, keyed by port) tracks how many sockets share it under the reuse rules. The bit is cleared only when that refcount reaches zero. SO_REUSEADDR (bind-time reuse of a TIME_WAIT port) additionally lets bind() succeed against a bit whose sole owner is a TIME_WAIT entry.

A new socket inherits the creating thread's network namespace (current_task().namespace_set.net_ns). setns(CLONE_NEWNET) affects future socket creation but does not migrate existing sockets. Sockets bound to a destroyed namespace return EIO on all operations. getsockopt(SO_NETNS_COOKIE) returns the namespace's unique 64-bit cookie for identification.

See Section 16.13 for NetDevice lifecycle, and Section 16.6 for RouteTable internals.

/// Fixed-size interface name (matching Linux IFNAMSIZ = 16).
/// Prevents unbounded heap allocation and OOM attacks via long names.
#[derive(Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash)]
pub struct InterfaceName([u8; 16]);

VETH pairs for inter-namespace connectivity: A VETH (virtual ethernet) pair connects two namespaces. Creation:

ip link add veth0 type veth peer name veth1
ip link set veth1 netns <target-namespace>

In UmkaOS, this creates two virtual interfaces that are cross-linked: - veth0 in the caller's namespace - veth1 in the target namespace - Packets sent to one end appear on the other (like a virtual patch cable)

Container networking flow: 1. Container runtime creates a new network namespace for the container 2. Creates a VETH pair: one end in host namespace (e.g., veth0), one in container (e.g., eth0) 3. Host end is attached to a bridge (e.g., docker0, cni0) for external connectivity 4. Container end is assigned an IP from the bridge's subnet 5. NAT/masquerading rules on the host allow container → external traffic 6. Port forwarding rules map host ports → container ports

Network namespace initial state: When a new network namespace is created via clone(CLONE_NEWNET) or unshare(CLONE_NEWNET), the kernel initializes the following state before returning to the caller:

Loopback interface (lo): auto-created, brought UP, assigned addresses 127.0.0.1/8 (IPv4) and ::1/128 (IPv6). The loopback device is permanent and cannot be deleted or moved to another namespace.

Construction ordering constraint: NetNamespace.loopback is a non-optional Arc<NetDevice>, so the loopback device must exist BEFORE the NetNamespace value handed to net_ns_alloc() is built — the device is created first, placed into the struct literal, and the post-allocation steps only REGISTER it (interface table, routes). All post-allocation mutation goes through interior-mutable fields (AtomicXX device fields, the InterfaceTable XArray, the RCU route trie) — never a plain field write through the shared Arc.

Registration steps (the CLONE_NEWNET creation sequence; every clone(CLONE_NEWNET) / unshare(CLONE_NEWNET) branch executes this):

// Phase 1 — build the loopback device (before the namespace value).
// NetDevice::new_loopback() returns a BARE NetDevice value
// ([Section 16.13](16-networking.md#network-device-interface-netdevice)). Plain write-once
// fields (`ifindex: u32`) are set while the value is still
// exclusively owned — BEFORE the Arc wrap. After the wrap, all
// remaining configuration goes through &self on interior-mutable
// fields (atomics + the locked address lists).
let mut lo_dev: NetDevice = NetDevice::new_loopback(ns_id);
lo_dev.ifindex = 1;             // Always 1 within the namespace.
let lo: Arc<NetDevice> = Arc::new(lo_dev);
lo.flags.store(IFF_LOOPBACK | IFF_UP | IFF_RUNNING, Release);
// Address-list mutation through the device's interior-mutable
// address lists ([Section 16.13](16-networking.md#network-device-interface-netdevice)).
lo.add_inet_addr(Inet4Addr::new(127, 0, 0, 1, 8));
lo.add_inet6_addr(Inet6Addr::loopback(128));

// Phase 2 — build the complete NetNamespace VALUE (loopback field
// populated) and move it into tracked storage.
let init = NetNamespace {
    ns_id,
    loopback: Arc::clone(&lo),
    user_ns: Arc::clone(owner_user_ns),  // creating cred's user_ns
    stack_cap: OnceCell::new(),          // set in Phase 3
    interfaces: InterfaceTable::new(),   // ifindex rotor seeded at 2
    users: AtomicU32::new(0),            // net_ns_get() by the caller
                                         // after publication into namespace_set
    dead: AtomicBool::new(false),
    socket_list: Mutex::new(Vec::new()),
    /* routes/firewall/conntrack/neigh_*: empty initial state;
       port_allocator: default range */
    ..
};
let ns: Arc<NetNamespace> = net_ns_alloc(init)?;   // ENOMEM on failure

// Phase 3 — register: capability entry, interface table, routes.
// net_stack_cap_create() allocates the Capability<NetStack> registry
// entry wrapping `ns` (fallible — ENOMEM propagates; the caller's
// CreatedNs rollback then drops `ns`, whose Drop revokes nothing
// because stack_cap is still unset). OnceCell::set() cannot fail:
// this is the only setter, running before `ns` is shared.
ns.stack_cap.set(net_stack_cap_create(&ns)?)
    .expect("stack_cap set exactly once at creation");
ns.interfaces.register(lo);   // ifindex 1 (pre-assigned); accessor,
                              // not a write through the private `table`
// Add loopback routes to ns.routes (see routing table init below) —
// FIB trie writers use per-entry RCU publication under config_lock.

The loopback driver is a Tier 0 software device (no hardware, no ring dispatch). NetDevice::new_loopback() is defined in Section 16.13.

Initial routing table: contains only loopback routes:
- 127.0.0.0/8 → lo (IPv4 loopback subnet)
- ::1/128 → lo (IPv6 loopback host route)
- No default gateway. The container runtime or CNI plugin is responsible for adding routes after creating veth pairs and assigning addresses (typically: default via <bridge-ip> dev eth0).
Empty firewall: no hook programs are attached (FirewallRules.hooks is the const allocation-free empty state; nft_shadow holds no tables) — the absence of programs IS the default ACCEPT policy, at ~1-2 cycles per hook site (Section 16.18). Built-in chains (INPUT, OUTPUT, FORWARD with policy ACCEPT) appear in the translated shadow the first time a legacy tool writes to the namespace, matching what iptables -L/nft list ruleset expect. The container runtime may install restrictive rules after namespace setup.
Empty socket table: no sockets exist. Sockets created in the new namespace bind to its port allocator (NetNamespace.port_allocator), independent of the host namespace's ports.
Empty FIB (Forwarding Information Base): except for the loopback routes above, no entries exist. No neighbor cache entries (the neigh_v4/neigh_v6 ARP/NDP tables start empty).
Empty conntrack table: no connection tracking state. Connections established after namespace creation are tracked independently from the host namespace's conntrack.
Sysctl defaults: per-namespace network sysctls (e.g., net.ipv4.ip_forward, net.core.rmem_default) are initialized to their kernel-default values, independent of the host namespace's sysctl settings. ip_forward defaults to 0 (disabled); the container runtime enables it if the container needs to forward traffic.

This design ensures that a new network namespace starts fully isolated with no connectivity. All external connectivity must be explicitly configured by the container runtime, matching Linux behavior exactly.

ifindex allocation and register_netdev(): Device registration (Section 16.13 step 5) assigns the device's ifindex from the DEVICE'S TARGET NAMESPACE, not from any global counter: dev.ifindex = target_ns.interfaces.alloc_ifindex(), then target_ns.interfaces.register(dev), both under target_ns.config_lock. The loopback is the sole exception — it is pre-assigned index 1 at namespace creation (above). Enumeration order (RTM_GETLINK, /proc/net/dev) is the target namespace's InterfaceTable ascending-index iteration.

Deferred handoff by symbol (Section 16.13, register_netdev() step 5): its "assigns ifindex, adds to global device list" wording is stale — there is no global device list; the device is added to its namespace's InterfaceTable and the index comes from that namespace's alloc_ifindex(). That prose should be updated to per-namespace scoping to agree with this section.

Tunnel device namespace scoping: Tunnel devices (GRE, VXLAN, GENEVE, WireGuard, IPIP, SIT) are scoped to the network namespace in which they are created. A tunnel's dev.net_ns field points to its owning namespace, and the tunnel's encap/decap processing uses that namespace's routing table and socket table. Moving a device between namespaces via ip link set <dev> netns <ns> (physical NIC, veth, or tunnel): remove the device from the source namespace's InterfaceTable (the source alloc_ifindex() rotor is NOT rewound — a monotone rotor guarantees no in-flight aliasing), then RE-ASSIGN a fresh ifindex from the target namespace's allocator (dev.ifindex = target_ns.interfaces.alloc_ifindex()) and register there — ifindex is NOT preserved across a namespace move, because the target namespace's index space is independent and may already have the old value live (Linux reassigns on move/collision). For tunnels this also re-binds the UDP encapsulation socket (if any) in the target namespace. This follows the same per-namespace InterfaceTable model as physical NICs and veth pairs (Section 16.13).

CLONE_NEWNS (Mount Namespace): Creates a private copy of the VFS mount tree for the process. Changes to this tree do not affect the parent domain unless explicitly marked shared. When unshare(CLONE_NEWNS) is called, a new mount namespace is created by COW-cloning the caller's mount tree. The task's root and pwd references are updated to point to the corresponding mount points in the new namespace. No filesystem data is copied — only the mount table is duplicated.
CLONE_NEWUTS (UTS Namespace): Creates an isolated hostname/domainname state. Stored as a reference-counted UtsNamespace struct in the task's NamespaceSet (see Section 17.1).
CLONE_NEWIPC (IPC Namespace): Isolates System V IPC objects and POSIX message queues. Flag validation (ABI): clone(CLONE_NEWIPC | CLONE_SYSVSEM) is rejected with EINVAL — CLONE_SYSVSEM shares the parent's sem-undo list, but after the child switches to a new IPC namespace the semaphore sets those undos reference are unreachable from it, so the combination is contradictory (Linux copy_namespaces(): (CLONE_NEWIPC | CLONE_SYSVSEM) == (CLONE_NEWIPC | CLONE_SYSVSEM) → -EINVAL, kernel/nsproxy.c, verified against torvalds/linux master). A plain clone(CLONE_NEWIPC) needs no undo detach: undo lists are per-process, and the child shares the parent's only under CLONE_SYSVSEM — without it the child starts with an empty list of its own. unshare(2) is different: BOTH flags together are legal there, and CLONE_NEWIPC IMPLIES the detach (see sys_unshare() below). (Deferred handoff by symbol: Section 8.1 create_task() entry validation — add this EINVAL pair-check before any namespace creation.)
CLONE_NEWUSER (User Namespace): Creates a new UID/GID mapping table within the Capability Domain.
CLONE_NEWTIME (Time Namespace): Creates isolated offsets for CLOCK_MONOTONIC and CLOCK_BOOTTIME. The container sees its own "boot time" starting from zero, independent of the host's actual boot time. The TimeNamespace struct with offset fields is defined in Section 17.1 below.

Flag-value caveat (ABI): CLONE_NEWTIME = 0x00000080 lies INSIDE CSIGNAL (0x000000ff, the legacy-clone exit-signal byte), so legacy clone(2) CANNOT express it: in a legacy clone() call, bit 0x80 is part of the child's exit-signal NUMBER and is NEVER interpreted as a namespace flag — the entry path extracts flags & CSIGNAL as the exit signal and masks the whole byte out of the namespace-flag view (a caller passing, e.g., a real-time signal number legitimately sets this bit). The flag is reachable ONLY via unshare(2) (its flag word carries no exit signal) and clone3(2) (the 64-bit clone_args.flags carries no CSIGNAL byte; clone3 rejects every OTHER CSIGNAL bit — kargs->flags & (CLONE_DETACHED | (CSIGNAL & ~CLONE_NEWTIME)) → the args are invalid, EINVAL). Linux include/uapi/linux/sched.h ("cloning flags intersect with CSIGNAL so can be used with unshare and clone3 syscalls only") + kernel/fork.c, verified against torvalds/linux master. (Deferred handoff by symbol: Section 8.1 create_task() — its flag handling must treat bit 0x80 as CSIGNAL when entered from legacy clone(2), and accept CLONE_NEWTIME as a namespace flag only from the clone3/unshare entry paths.)

Namespace creation rollback on partial failure: When clone() or unshare() requests multiple namespaces simultaneously (e.g., CLONE_NEWPID | CLONE_NEWNET | CLONE_NEWNS), namespace creation must be atomic — either all requested namespaces are created, or none are. If the Nth namespace allocation fails (ENOMEM, ENOSPC from namespace limits), the previously-created (N-1) namespaces must be rolled back:

Namespaces are created in a fixed order: user (if CLONE_NEWUSER, always first), then mount, PID, time, net, IPC, UTS, cgroup. The only ordering requirement for correctness is that CLONE_NEWUSER is processed first (see CLONE_NEWUSER ordering requirement above). The non-NEWUSER ordering is arbitrary -- these namespaces are independent of each other. The order listed here is the canonical implementation order used consistently in sys_unshare() and create_task() namespace creation paths.

Each successfully-created namespace is recorded in a local ArrayVec<CreatedNs, 8>:

/// One successfully-created namespace, recorded during the `sys_unshare()`
/// / `create_task()` namespace-creation loop so rollback (below) can undo a
/// partial failure. Each variant wraps the `Arc<T>` that step 4 assigned
/// into the staging `new_ns`; rollback drops these Arcs (not the staging
/// `new_ns` itself, which is not yet visible to any other task).
pub enum CreatedNs {
    User(Arc<UserNamespace>),
    Mount(Arc<MountNamespace>),
    Pid(Arc<PidNamespace>),
    Time(Arc<TimeNamespace>),
    Net(Arc<NetNamespace>),
    Ipc(Arc<IpcNamespace>),
    Uts(Arc<UtsNamespace>),
    Cgroup(Arc<CgroupNamespace>),
}

On failure at step K: a. Iterate the ArrayVec in reverse order. b. For each created namespace: drop(created_ns), decrementing its refcount. If the refcount reaches zero (no other task shares it), the namespace's impl Drop (see "Namespace Drop Semantics" below) runs and fully tears it down — PID IDR freed, net stack torn down and its capability revoked, mount tree dropped, etc. The rollback loop does NOT need type-specific cleanup logic: each namespace type's Drop impl is the single place that knows how to tear itself down, so the same drop-to-zero path is exercised whether the namespace dies via rollback, exit_task(), or ordinary namespace_set replacement. This also closes the capability-revocation gap: Capability<NetStack> (held in NetNamespace.stack_cap) is revoked from impl Drop for NetNamespace itself, not duplicated at each call site that might drop the last Arc<NetNamespace>. c. Return the error to the caller. The task's namespace_set is unchanged.
On success: atomically swap the task's namespace_set to the new NamespaceSet containing all newly-created namespaces. The old namespace_set is dropped (refcount decrement; shared namespaces remain alive via other tasks' references).

Namespace Drop semantics: a namespace's Arc refcount can reach zero via three independent paths — rollback (above), exit_task() detaching the last task, or namespace_set.store() replacing the last reference. All three paths must tear down identical state, so teardown logic lives in each namespace type's impl Drop, never duplicated at the call sites that happen to drop the last Arc. Two rules apply to every namespace type's Drop impl:

No live-namespace assumption. A namespace's Drop impl MUST NOT assume any other namespace it references is still alive — teardown order across independently-refcounted namespaces is not specified and callers (rollback, exit_task()) may drop namespaces in any relative order once each one's own refcount independently reaches zero. Cross-namespace references held for use during teardown are Weak<T> (e.g., UtsNamespace.user_ns, CgroupNamespace.user_ns, TimeNamespace.user_ns — all Weak<UserNamespace>) and MUST be accessed via .upgrade(), treating None as "target already destroyed, no-op" — never .unwrap(). (EXCEPTION: PidNamespace.user_ns is a STRONG Arc<UserNamespace> — a PID namespace deliberately PINS its owner alive, Linux get_user_ns() parity, so its Drop MAY assume the owner is live; the edge is acyclic because a user namespace never references a PID namespace, and it keeps the per-user PID-ns count leak-free. See the PidNamespace.user_ns field doc.) This is unconditionally safe by construction: Weak::upgrade() returns None once the target's strong count has reached zero, regardless of whether the target's own Drop has started, is running, or has finished — there is no window where a Weak observes a partially-torn-down target. (This resolves the concern that dependents "hold Weak refs that become invalid before their own drop runs": upgrade failure is the designed-for outcome, not a hazard, as long as callers actually check for None instead of unwrapping.)
Cross-namespace calls during teardown are effects, not assumptions. If a Drop impl needs to notify another subsystem (e.g., returning a kernel-memory charge to a cgroup), it must do so through the same Weak::upgrade() + None-is-noop pattern as rule 1, not by holding a strong reference that would itself prevent that subsystem's teardown.

impl Drop for NetNamespace (worked example — the namespace capability state, per the rollback note above):

type with href="#__codelineno-7-1">impl Drop for NetNamespace { fn drop(&mut self) { // Drop is the PASSIVE memory-reclaim tail, reached only after the last // strong `Arc<NetNamespace>` drops — i.e. after `net_ns_cleanup()` // (users→0, deferred to the `umkad-netns` workqueue) has already run // the ACTIVE teardown (RST close, device removal, flush) AND every // passive holder (SCM_RIGHTS-passed sockets in other namespaces, // devices) has released its `net_ns` reference. The ordering is // structural: the cleanup work item itself owns an `Arc` clone until // the walk completes, so this Drop cannot precede it. // Now that `socket_list` holds `Weak<dyn SocketOps>`, the Arc cycle is // broken, so this Drop is REACHABLE (the old strong-Arc `socket_list` // made it unreachable while any socket lived). Two things happen here: // // 1. Capability revoke — the SINGLE revoke site. The capability // service holds its own reference to the `Capability<NetStack>` // registry entry ([Section 17.1](#namespace-architecture--capability-domain-mapping)), // NOT just a strong-ref count on `NetNamespace`, so field drops do // not deregister it. `revoke_capability()` // ([Section 9.1](09-security.md#capability-based-foundation)) takes the `CapHandle`, bumps // the slot generation, and broadcasts to delegatees. Placing it // HERE (not in `net_ns_cleanup`) also covers the allocation-failure // ROLLBACK path, where the namespace is dropped before it ever // gained a user (`net_ns_cleanup` never ran): `stack_cap` is an // OnceCell set only in creation Phase 3, so a rollback-dropped // namespace whose cell was never set has nothing to revoke // (`get()` → None). if let Some(handle) = self.stack_cap.get() { let _ = revoke_capability(*handle); } // 2. Residual field reclaim. `socket_list` (now `Weak`) drops with no // active close — the close already happened at `net_ns_cleanup()`, // or (rollback) no socket was ever created. `interfaces`, `routes`, // `firewall`, `conntrack`, `neigh_*` drop their contents via their // own destructors. `user_ns: Arc<UserNamespace>` (a forward, // non-cyclic strong reference) drops last as an ordinary field. // `NetNamespace` caches no cgroup reference, so no `Weak::upgrade()` // cross-namespace effect (Drop rule 2) arises here. debug_assert!( self.stack_cap.get().is_none() || self.dead.load(Relaxed) || self.users.load(Relaxed) == 0, "NetNamespace dropped without net_ns_cleanup (or rollback)" ); } }

Network Namespace Teardown — the two-count model: NetNamespace has TWO lifetimes because sockets and devices hold reverse strong references (SockCommon.net_ns, NetDevice.net_ns) that form refcount cycles with the namespace's own socket_list/interfaces. A single Drop-at-strong-count-zero trigger can therefore NEVER fire while any socket or device exists, and a strong socket reference (Arc<dyn SocketOps>) passed to another namespace via SCM_RIGHTS would pin the whole network stack (routes, conntrack, devices) indefinitely under pod churn. The resolution splits liveness into two counts — the same active/passive split Linux uses for the identical cycle (net/core/net_namespace.c):

Passive references = the Arc<NetNamespace> strong count. Held by NamespaceSet.net_ns, every SockCommon.net_ns, and every NetDevice.net_ns. Keeps the memory alive; a fast pointer deref on the per-packet path (no scalar-id lookup cost). Reaching zero runs impl Drop (the passive tail above).
Active users = NetNamespace.users: AtomicU32. Counts only the NamespaceSet objects naming this net_ns. This is the administrative lifetime — "are there still tasks living in this namespace?".

/// Gain an active user. Called on construction sites that install THIS
/// net_ns under one of TWO preconditions — either makes the unconditional
/// `fetch_add` safe (it must never revive a dead namespace):
///
/// - **Caller already holds a live active user**: `Clone for NamespaceSet`
///   and the `create_task()` net-ns inheritance path. The SOURCE `NamespaceSet`
///   holds its own user on the same namespace for the whole call, so
///   `users >= 1` at entry and no concurrent `net_ns_put()` can reach `0`
///   underneath the increment.
/// - **The namespace is fresh and UNPUBLISHED**: `empty()`'s boot-time
///   INIT_NET_NS acquisition, the `sys_unshare()` CLONE_NEWNET
///   fresh-namespace assignment, and the CLONE_NEWNET creation sequence.
///   Here `users` IS `0` at entry (`AtomicU32::new(0)` in the initial-state
///   literal) — but the namespace is not yet visible to any other task, so
///   no `net_ns_put()` exists anywhere that could race the `0 → 1`
///   transition. Unpublishedness, not a nonzero count, is what makes these
///   sites safe (the earlier "users provably >= 1 at entry" wording was
///   FALSE for them).
///
/// Pairs 1:1 with `net_ns_put()`. The acquire-from-elsewhere paths
/// (`setns(CLONE_NEWNET)`, nsfs-fd open) — where NEITHER precondition holds
/// and the last task may be concurrently exiting — MUST use
/// `net_ns_get_not_dead()` instead.
pub fn net_ns_get(ns: &Arc<NetNamespace>) {
    ns.users.fetch_add(1, AcqRel);
}

/// Gain an active user ONLY if the namespace is still live — an
/// increment-if-not-dead CAS acquire. Used on the
/// acquire-from-elsewhere paths (`setns(CLONE_NEWNET)`, nsfs-fd open) where the
/// caller does NOT already hold an active user and the namespace's last task
/// may be concurrently running `net_ns_put()` 1 → 0.
///
/// A plain `net_ns_get()` on those paths is unsound: interleave
/// `net_ns_put()` 1 → 0 (which `net_ns_cleanup()` latches `dead`, force-closes
/// every socket, and `remove_all()`s every device) with a racing `setns()`,
/// and an unconditional `fetch_add` would revive `users` 0 → 1 on a GUTTED
/// namespace — then this task's own later exit runs `net_ns_cleanup()` a
/// SECOND time (the `1 → 0` trigger is unconditional; the once-only `dead`
/// latch is stored, not checked). The inc-not-zero CAS refuses the `0 → 1`
/// transition, returning `ENOENT` ("Namespace has been destroyed") instead.
/// (Linux's live-check acquire and nsfs-fd active counting resolve the same
/// race, `net/core/net_namespace.c`.)
///
/// **Release symmetry** — every successful acquire pairs with exactly one
/// `net_ns_put()`: the `setns(CLONE_NEWNET)` arm's acquired user is owned by
/// the new `NamespaceSet` and released by ITS `Drop` (the arm releases the
/// PREVIOUS namespace's user at the swap); the nsfs-open acquire is owned by
/// the `NsInode` and released by `impl Drop for NsInode` (see the nsfs
/// section, [Section 17.1](#namespace-architecture--namespace-hierarchy-and-inheritance)).
pub fn net_ns_get_not_dead(ns: &Arc<NetNamespace>) -> Result<(), Errno> {
    let mut cur = ns.users.load(Acquire);
    loop {
        if cur == 0 {
            // Last active user already gone — cleanup ran or is running.
            return Err(Errno::ENOENT);
        }
        match ns.users.compare_exchange_weak(cur, cur + 1, AcqRel, Acquire) {
            Ok(_) => return Ok(()),
            Err(observed) => cur = observed,
        }
    }
}

/// Dedicated netns-teardown workqueue (threads appear as `umkad-netns-N`,
/// [Section 3.11](03-concurrency.md#workqueue-deferred-work)). Created by network-stack initialization before
/// `INIT_NET_NS` — i.e., before the first namespace that could ever die.
/// Depth-provisioned for one outstanding item per live namespace: each
/// namespace enqueues AT MOST once in its lifetime (the `1 → 0` transition
/// is unique — `users` never revives past the `dead` latch), so
/// `queue_work()`'s ENOMEM backpressure arm is unreachable.
static NETNS_CLEANUP_WQ: BootOnceCell<WorkQueue> = BootOnceCell::new();

/// Drop an active user. Called from `impl Drop for NamespaceSet` (below) for
/// the net_ns it held. The `1 → 0` transition is the SOLE trigger for the
/// active teardown that `socket_list`'s "RST for TCP" promise and
/// [Section 16.2](16-networking.md#network-stack-architecture)'s last-task-exit cleanup both require.
///
/// **Context-safe BY CONSTRUCTION**: this function performs only an atomic
/// `fetch_sub`, an atomic `dead` store, and a non-blocking O(1) workqueue
/// submission — no sleeping, no locks. That matters because the `1 → 0`
/// transition is reached through `impl Drop for NamespaceSet`, whose call
/// sites include the `setns()` / `sys_unshare()` commit tails — where the
/// `ArcSwap::swap()` runs under `TASK_LOCK(20)`, but the old
/// `Arc<NamespaceSet>` is dropped only AFTER the lock is released (the
/// deliberate `drop(old_namespace_set)`-outside-the-lock discipline those commit
/// paths follow) — and, in principle, ANY OTHER context that drops an
/// `Arc<NamespaceSet>`. A synchronous close-walk here (sleeping socket
/// teardown, device removal) would be a latent sleep-under-spinlock hazard
/// for any unaudited drop site, and no call-site audit can close that
/// class: `Arc` drops are not confined to audited code. The sleeping body
/// is therefore ALWAYS deferred to the workqueue, never run in the
/// last-put caller's context. (Linux likewise never runs the final put's
/// cleanup inline at the last put — it always hands it to a workqueue,
/// `net/core/net_namespace.c`, verified against torvalds/linux master.)
pub fn net_ns_put(ns: &Arc<NetNamespace>) {
    if ns.users.fetch_sub(1, AcqRel) == 1 {
        // Last active user. Latch `dead` INLINE (Release) so socket ops
        // observe EIO immediately — before the deferred walk runs — then
        // hand the sleeping teardown body to `umkad-netns`. The work item
        // OWNS an `Arc<NetNamespace>` clone, so the passive tail
        // (`impl Drop for NetNamespace`) structurally cannot run before
        // the active teardown completes: active-before-passive ordering
        // needs no scheduling argument.
        ns.dead.store(true, Release);
        let work_ns: Arc<NetNamespace> = Arc::clone(ns);
        NETNS_CLEANUP_WQ.get().expect("created at net-stack init")
            .queue_work(WorkItem::new(
                net_ns_cleanup_work,
                Arc::into_raw(work_ns) as *mut (),
                0, // no deadline
            ))
            .expect("depth-provisioned: at most one item per live namespace");
    }
}

/// `umkad-netns` work entry point: reconstitute the `Arc` the enqueue
/// leaked and run the close-walk. Workqueue context — process context, may
/// sleep, holds no caller locks: the context `net_ns_cleanup()` requires is
/// guaranteed BY CONSTRUCTION rather than by auditing every
/// `Arc<NamespaceSet>` drop site.
fn net_ns_cleanup_work(data: *mut ()) {
    // SAFETY: `data` is the `Arc::into_raw` of the clone taken at enqueue;
    // exactly one work item exists per namespace lifetime (unique 1→0
    // transition), so the raw pointer is reconstituted exactly once.
    let ns: Arc<NetNamespace> = unsafe { Arc::from_raw(data as *const NetNamespace) };
    net_ns_cleanup(&ns);
    // `ns` drops here. If the work item held the LAST passive reference,
    // `impl Drop for NetNamespace` (capability revoke) runs now — strictly
    // after the active teardown, per the active-before-passive order.
}

/// Active teardown at `users → 0`. This IS the "last task exits a network
/// namespace" cleanup — the trigger substrate the refcount model could not
/// supply. Runs ONLY as `net_ns_cleanup_work` on `umkad-netns` (process
/// context, may sleep) — never inline in `net_ns_put()`, whose Drop-reached
/// call sites are not confined to audited, lock-free contexts (see
/// `net_ns_put`).
fn net_ns_cleanup(ns: &Arc<NetNamespace>) {
    // `dead` was latched (Release) by `net_ns_put()` at the `1 → 0`
    // transition, BEFORE this work was enqueued: subsequent socket ops
    // observe `dead` and return EIO ([Section 16.3](16-networking.md#socket-abstraction)); in-flight
    // packet paths that snapshot the namespace drain via the existing RCU
    // deferral ([Section 16.2](16-networking.md#network-stack-architecture)).
    debug_assert!(ns.dead.load(Acquire), "dead latched at the 1->0 transition");
    // ACTIVE close-walk — the real RST trigger. Upgrade each Weak; a live
    // socket is force-closed (TCP: send RST, transition to CLOSED; UDP/raw:
    // immediate free). fd-holders in OTHER namespaces cannot be force-closed
    // (owned elsewhere) — they keep the passive Arc alive and see `dead`.
    let mut list = ns.socket_list.lock();
    for weak in list.drain(..) {
        if let Some(sock) = weak.upgrade() {
            sock.abort_for_netns_teardown();   // RST/immediate-free path
        } // else: already-freed socket — pruned by the drain
    }
    drop(list);
    // Remove devices (each drops its `NetDevice.net_ns` passive edge), then
    // detach hook programs and flush routes and conntrack — the exact
    // ordered steps [Section 16.2](16-networking.md#network-stack-architecture) enumerates, now with a
    // defined trigger. The capability is revoked later, in `impl Drop`, once
    // the last passive reference is gone (single revoke site).
    ns.interfaces.remove_all();
    // Detach every packet-filtering hook program (translated rulesets AND
    // direct BPF links) — devices are gone, so no packet path can enter the
    // hooks; this drops the namespace's `Arc<BpfProg>` references promptly
    // (programs must not linger until the passive tail; a direct bpf_link's
    // program survives, defunct, until its link fd closes). The config_lock
    // guard is the WriterProof for the slot updates
    // ([Section 16.18](16-networking.md#packet-filtering-bpf-based--netfilter-hook-attachment-points)).
    {
        let cfg = ns.config_lock.lock();
        ns.firewall.hooks.detach_all(&cfg);
    }
    ns.routes.flush();
    ns.conntrack.flush();
    // PortAllocator drops with the namespace; nothing to actively release.
    // `firewall.nft_shadow` (control-plane bookkeeping) drops with the
    // namespace in the passive tail — it holds no packet-path state.
}

Deferred handoffs by symbol (this is the canonical teardown design; the named sites below carry the divergent/absent text): - Section 16.2 "Network namespace cleanup" (last-task-exit ordered steps) — its trigger is net_ns_put()'s users → 0 transition (which latches dead inline and defers net_ns_cleanup() to the umkad-netns workqueue), NOT an unsubstantiated "last task exits" with no mechanism. The step list (close sockets → remove devices → flush routes → flush conntrack → release PortAllocator) is authoritative; adopt this trigger and its workqueue context. - Section 16.3 SockCommon.net_ns — stays a strong PASSIVE reference (fast deref); the socket ALSO registers a Weak<dyn SocketOps> into net_ns.socket_list at socket() time, and every socket operation checks net_ns.dead → EIO. The close-walk calls SocketOps::abort_for_netns_teardown() (defined in Section 16.3 — TCP RST / UDP-raw immediate free). - Section 16.13 NetDevice.net_ns — strong PASSIVE reference; InterfaceTable::remove_all() (called here) drops those edges. (Resolves the NetDevice.net_ns ghost-flag design question together with the socket one — same passive-reference answer.)

CLONE_NEWUSER ordering requirement: When CLONE_NEWUSER is combined with other CLONE_NEW* flags in a single clone() or unshare() call, UmkaOS MUST create the user namespace first, before creating any other namespace. This ordering is a correctness requirement, not an optimization:

The new user namespace maps the caller's UID/GID to root (UID 0) inside the namespace, granting CAP_SYS_ADMIN within that namespace.
Creating other namespaces (PID, NET, MNT, IPC, UTS, CGROUP, TIME) requires CAP_SYS_ADMIN — either in the caller's current user namespace or in the newly created one.
If CLONE_NEWUSER is processed after other CLONE_NEW* flags, the capability check for those namespaces runs against the parent user namespace, which may deny the operation for unprivileged callers (rootless containers).
If CLONE_NEWUSER is absent from the flags, all other namespace creation operations require CAP_SYS_ADMIN in the caller's current user namespace.

Implementation: create_task() and sys_unshare() sort the namespace creation order internally. Regardless of the bit order in the flags argument, the processing sequence is: (1) CLONE_NEWUSER (if present), (2) all other CLONE_NEW* flags in the CANONICAL kernel order — mount, PID, time, net, IPC, UTS, cgroup — the single order defined in "Namespace creation rollback on partial failure" above and used identically by create_task() step 10 and sys_unshare() step 4. Only the user-first rule is a correctness requirement; the rest is a shared implementation convention (the CreatedNs LIFO rollback is order-insensitive). This matches Linux kernel behavior (Linux create_new_namespaces() always processes CLONE_NEWUSER first).

17.1.1.1 `sys_unshare()` — Standalone Namespace Disassociation¶

/// Create new instances of the specified namespace types for the calling task
/// without creating a new process. Equivalent to the namespace-creation effects
/// of `clone(flags)` but applied to the calling thread itself.
///
/// # Arguments
/// - `flags`: Bitmask. The `CLONE_NEW*` flags each create the corresponding
///   namespace type: `CLONE_NEWPID`, `CLONE_NEWNET`, `CLONE_NEWNS`,
///   `CLONE_NEWIPC`, `CLONE_NEWUTS`, `CLONE_NEWUSER`, `CLONE_NEWCGROUP`,
///   `CLONE_NEWTIME`. Flags may be combined.
///
///   **Non-namespace `unshare(2)` flags** (accepted by Linux
///   Linux `check_unshare_flags()` in `kernel/fork.c`, verified against torvalds/linux
///   master — `unshare(2)`'s input domain is NOT restricted to `CLONE_NEW*`;
///   container runtimes and glibc's `posix_spawn` pass these routinely):
///   - `CLONE_FILES`: unshare the file-descriptor table (the calling task gets
///     a private `FdTable` copy). Handled by the fork/clone fd-table machinery
///     — [Section 8.1](08-process.md#process-and-task-management--process-creation).
///   - `CLONE_FS`: unshare `root`/`cwd`/`umask` (private `FsStruct` copy).
///     Same machinery — [Section 8.1](08-process.md#process-and-task-management--process-creation).
///   - `CLONE_SYSVSEM`: detach from the shared SysV semaphore-adjustment
///     (`sem_undo`) list. Linux implements this as `exit_sem(current)` — the
///     task's accumulated undos are applied and it starts a fresh, empty
///     `Process.sysvsem_undo` list; equivalent to the SysV-sem half of
///     `sys_exit()` ([Section 8.2](08-process.md#process-lifecycle-teardown--step-4c-sysv-semaphore-undo-operations)).
///     **`CLONE_NEWIPC` IMPLIES this detach** even when CLONE_SYSVSEM is not
///     passed: after the namespace switch the old namespace's semaphore sets
///     are unreachable from the caller, so its accumulated undos must be
///     applied NOW, while they still resolve (Linux `ksys_unshare()`:
///     "CLONE_NEWIPC must also detach from the undolist" —
///     Linux `if (unshare_flags & (CLONE_NEWIPC|CLONE_SYSVSEM)) do_sysvsem = 1` →
///     Linux `exit_sem(current)` at the commit point, `kernel/fork.c`, verified
///     against torvalds/linux master). Unlike `clone()` (which rejects the
///     pair with EINVAL — see the CLONE_NEWIPC bullet in
///     [Section 17.1](#namespace-architecture--capability-domain-mapping)), passing BOTH
///     flags to `unshare()` is legal. The commit-point placement is step 5
///     of the algorithm below.
///   - `UNSHARE_EMPTY_MNTNS` (`0x0010_0000`): create the new mount namespace
///     EMPTY instead of COW-cloning the caller's mount tree. This is an
///     `unshare(2)`-only flag whose value REUSES the legacy-clone
///     `CLONE_PARENT_SETTID` bit (`0x0010_0000`), which carries no meaning for
///     `unshare()` — the same bit-aliasing trick `CLONE_NEWTIME` uses against
///     the exit-signal byte (Linux `include/uapi/linux/sched.h`
///     `#define UNSHARE_EMPTY_MNTNS 0x00100000` — `kernel/fork.c` only
///     *references* it in Linux `check_unshare_flags()` and `unshare_nsproxy_namespaces()`
///     Linux rewrites it to internal `CLONE_EMPTY_MNTNS` (`1u64 << 37`) before
///     namespace creation, `kernel/nsproxy.c`, verified against torvalds/linux
///     master). Setting it IMPLIES `CLONE_NEWNS` (which in turn implies
///     `CLONE_FS`) — the implementation ORs those bits in during validation, so
///     `UNSHARE_EMPTY_MNTNS` alone is sufficient. The resulting namespace is not
///     literally empty: it holds a SINGLE anonymous (nullfs) root mount over an
///     immutable empty directory, and the caller's `fs.root`/`fs.pwd` are reset
///     to that root. The intended workflow is to then mount a real rootfs onto
///     that root and `pivot_root()` — building a mount tree from scratch rather
///     than pruning an inherited copy. Requires `CAP_SYS_ADMIN` in the owning
///     user namespace, identical to `CLONE_NEWNS`. The empty-vs-cloned choice is
///     resolved in the `CLONE_NEWNS` creation step of the algorithm below.
///   Any bit outside the union of the `CLONE_NEW*` set and
///   `{CLONE_FILES, CLONE_FS, CLONE_SYSVSEM, CLONE_THREAD, CLONE_VM,
///   CLONE_SIGHAND, UNSHARE_EMPTY_MNTNS}` returns `EINVAL`.
///
///   **`CLONE_THREAD` / `CLONE_VM` / `CLONE_SIGHAND`**: accepted only when the
///   caller is single-threaded — `unshare(CLONE_THREAD|CLONE_SIGHAND|CLONE_VM)`
///   returns `EINVAL` if `thread_group` is non-empty (and `CLONE_SIGHAND`/
///   `CLONE_VM` additionally require the shared `SignalHandlers` refcount to be
///   1). A single-threaded caller has nothing to unshare for these bits, so
///   they are a successful no-op. Matches Linux `check_unshare_flags()`.
///
/// # Ordering
/// If `CLONE_NEWUSER` is present, it MUST be processed first (same ordering
/// as `clone()`). The implementation reorders internally regardless of the
/// caller's flag order.
///
/// # Semantics
/// - Creates new namespace instances for each specified flag.
/// - Replaces the calling task's `NamespaceSet` (`namespace_set`) with a new one
///   containing the newly created namespaces. Sibling threads are unaffected —
///   they retain the old `Arc<NamespaceSet>`.
/// - `CLONE_NEWPID`: Unlike `clone()`, the calling process does NOT enter the
///   new PID namespace itself. Instead, it is stored as `pending_pid_ns` and
///   future children will be created in the new namespace.
/// - Requires `CAP_SYS_ADMIN` in the caller's user namespace for all flags
///   except `CLONE_NEWUSER` (which is always allowed, subject to nesting limits).
///
/// # Returns
/// `Ok(0)` on success, or `Err(errno)` on failure:
/// - `EPERM`: Missing capability.
/// - `ENOSPC`: Namespace nesting depth exceeded (32 levels for both user and PID namespaces).
/// - `ENOMEM`: Insufficient memory to create namespace structures.
pub fn sys_unshare(flags: u64) -> Result<i64, Errno>;

sys_unshare() implementation algorithm:

Check permissions. Precedence rule (resolves the step-1 vs step-4 credential source): the CAP_SYS_ADMIN demand for the sibling namespaces (MNT, NET, IPC, UTS, PID, CGROUP, TIME) is evaluated HERE, against the caller's already-committed task.cred, ONLY when CLONE_NEWUSER is ABSENT from flags. When CLONE_NEWUSER IS present, step 1 performs NO per-namespace capability check — those checks are DEFERRED to step 4 and run against pending_cred (which already holds CAP_FULL_SET in the new user namespace). Running the committed-cred check here for a CLONE_NEWUSER combination would EPERM the canonical rootless case unshare(CLONE_NEWUSER|CLONE_NEWNS) — the caller has no CAP_SYS_ADMIN in the parent user namespace — which is exactly the case step 4's pending_cred design exists to allow (Linux creates the new user namespace FIRST via Linux unshare_userns(), then checks sibling namespaces against the new credentials, Linux kernel/fork.c ksys_unshare(), verified against torvalds/linux master). Step 1 unconditionally performs only the CLONE_NEWUSER-specific preconditions: nesting depth (rejected with ENOSPC when the parent is already at the maximum — UserNamespace::create's parent.level > 32 rule, deepest reachable level 33) and, if multi-threaded and CLONE_NEWUSER is set, return EINVAL (Linux requires single-threaded for user namespace unshare; returns EINVAL, not EPERM).
If CLONE_NEWUSER is present, PREPARE (but do not commit) credentials: a. Create new UserNamespace (child of current) via the canonical constructor: UserNamespace::create(&cred.user_ns, cred.euid, cred.egid, cred.cap_effective.contains(CAP_SETFCAP))? (defined in the User Namespaces section below; ENOSPC on nesting/count limits, ENOMEM; the fourth argument records parent_could_setfcap from the PRE-transform credential — the uid_map root-map gate's input), then allocate its companion IMA namespace and set the pairing cell: new_user_ns.ima_ns.set(ima_ns_create(&new_user_ns)?) (one ImaNamespace per user namespace — Section 9.5; the cell is what setns(CLONE_NEWUSER) reads to follow the user→IMA pairing). b. Prepare new credentials (stage_credentials) and apply the CANONICAL user-namespace credential transformation (Section 17.1): CAP_FULL_SET scoped to the new user namespace, ambient cleared, and the deferred-translation identity rule — the POSIX id fields keep their parent-namespace values with ids_ns = the parent namespace (Section 9.9); getuid() reports 65534 until uid_map is written, then the mapped value, with no stored-id rewrite at map-write time. Do NOT call install_credentials() yet — defer until step 5 to ensure rollback is possible if namespace allocation fails. Keep the prepared TrackedPtr<TaskCredential> as pending_cred — step 4 needs it for capability checks (see below); it is only promoted to task.cred at step 5. (stage_credentials is fallible — ENOMEM aborts the unshare with nothing to roll back yet.)
Clone the current NamespaceSet: let mut new_ns = old_ns.clone();
For each CLONE_NEW* flag, create the namespace and update new_ns. Capability-check source: if CLONE_NEWUSER was requested (step 2 prepared pending_cred), every capability check performed while creating the other namespaces (MNT, NET, IPC, UTS, PID, CGROUP, TIME) MUST resolve against pending_cred, NOT task.cred. task.cred still holds the caller's PARENT-namespace credentials until step 5 commits — checking against it would fail CAP_SYS_ADMIN for the common rootless-container case (unprivileged caller creating a user namespace specifically to gain privilege inside it). This matches Linux: the new namespace's creator UID becomes root inside the new user namespace before any sibling namespace is created. Use an explicit-credential variant of has_ns_cap() (Section 9.9), scoped to this call site: Same three checks as has_ns_cap(), presented in the same algorithm-step style — only the credential source differs (an explicit cred argument instead of reading task.cred from inside the function):
```
ns_capable_with_cred(cred: &TaskCredential, target_ns: &UserNamespace, cap: SystemCaps) -> bool:
  1. if !cred.cap_effective.contains(cap):
       return false             // pending_cred doesn't have the cap
  2. // Same namespace-hierarchy walk as has_ns_cap() step 4.
  3. if !is_same_or_ancestor(cred.user_ns, target_ns):
       return false             // cap not valid in target namespace
  4. if lsm_deny_capable(cred, target_ns, cap):  // [Section 9.8](09-security.md#linux-security-module-framework)
       return false
  5. return true
```
Called as ns_capable_with_cred(&pending_cred, &new_user_ns, cap) from each of step 4's namespace-creation branches that require a capability check. pending_cred.user_ns already equals new_user_ns (set in step 2b), so the hierarchy walk trivially passes the "same namespace" case; it exists so the check also correctly denies/allows for any nested ancestor relationship exactly like has_ns_cap() does. If CLONE_NEWUSER was NOT requested, step 4's checks use the normal has_ns_cap(current_task(), caller_user_ns, cap) against already-committed task.cred (no pending credentials exist in this path).
CLONE_NEWUSER (staged-set assignment — runs FIRST): the user namespace and its companion IMA namespace were CREATED in step 2; this arm installs them into the staging set so the published NamespaceSet agrees with the committed credential:
```
new_ns.user_ns = Arc::clone(&new_user_ns);
new_ns.ima_ns  = new_user_ns.ima_ns.get()
    .expect("ima_ns paired at user-ns creation (step 2a)").clone();
```
Without these two assignments, task.namespace_set.user_ns/ima_ns would permanently name the PARENT's namespaces while cred.user_ns names the new one — the exact cred/namespace_set divergence the setns(CLONE_NEWUSER) single-TASK_LOCK(20) protocol exists to prevent, made permanent. (/proc/self/ns/user readlinks, setns ownership checks, and every NamespaceSet-sourced user_ns consumer read the namespace_set copy.)

CLONE_NEWNS: Call copy_tree() to clone the mount tree. Signature (Section 14.6 — the canonical algorithm):

// Declaration only — the canonical algorithm (tree walk, propagation
// handling, PathRef translation, FsUpdate construction) is specified at
// [Section 14.6](14-vfs.md#mount-tree-data-structures-and-operations--copytree-clone-mount-tree-for-clonenewns).
fn copy_tree(
    source_ns: &Arc<MountNamespace>,       // walked under ITS mount_lock
    source_root_mount: &Arc<Mount>,
    source_root_dentry: &Arc<Dentry>,
    owner_user_ns: &Arc<UserNamespace>,    // pending_cred.user_ns here
    fs_snapshot: &FsStruct,                // whose root/pwd to translate
) -> Result<(Arc<MountNamespace>, FsUpdate)>;

When UNSHARE_EMPTY_MNTNS is set, copy_tree() is bypassed for the empty-namespace constructor (canonical home is THIS section — it is unshare(2)-specific and has no clone()/copy_tree() analogue):

// Allocate a fresh MountNamespace owned by `owner_user_ns` containing a
// single anonymous (nullfs) root mount over an immutable empty directory,
// and return the FsUpdate that resets fs.root/fs.pwd to that nullfs root.
// Fallible: ENOMEM on namespace/mount allocation, ENOSPC on the mount-ns
// ucount limit — same rollback contract as copy_tree (nothing applied to
// the task until step-5 commit).
fn new_empty_mount_ns(
    owner_user_ns: &Arc<UserNamespace>,
) -> Result<(Arc<MountNamespace>, FsUpdate)>;

The new MountNamespace is assigned to new_ns.mount_ns. The returned FsUpdate holds the TRANSLATED fs.root/fs.pwd PathRefs (computed inside copy_tree() via its internal mount_map, which stays unexposed) but does NOT apply them — the caller applies it at the step-5 COMMIT point, after every fallible step has succeeded. Applying eagerly here would corrupt the task's path-resolution state on a later-step failure (e.g. CLONE_NEWNET ENOMEM): fs.root/fs.pwd would point into a mount namespace that the rollback just destroyed while namespace_set.mount_ns still names the old one.

let fs_arc = task.fs.load();
let (mnt_ns, fs_update) = if flags & libc::UNSHARE_EMPTY_MNTNS != 0 {
    // Empty-namespace variant: skip copy_tree entirely. Build a fresh
    // MountNamespace holding a SINGLE anonymous (nullfs) root mount over
    // an immutable empty directory, and produce an FsUpdate that resets
    // fs.root/fs.pwd to that nullfs root (there is nothing to translate —
    // the caller mounts a real rootfs on top and pivot_root()s). Same
    // fallible/rollback discipline as copy_tree: nothing is applied to
    // the task until the step-5 commit.
    new_empty_mount_ns(owner_user_ns)?   // -> (Arc<MountNamespace>, FsUpdate)
} else {
    copy_tree(
        &old_ns.mount_ns,
        &old_ns.mount_ns.root_mount,
        &old_ns.mount_ns.root_dentry,
        owner_user_ns,               // pending_cred / task.cred user_ns
        &fs_arc.read(),
    )?
};
new_ns.mount_ns = mnt_ns;
// fs_update is applied in step 5 (commit), not here.

CLONE_NEWPID: Build the new PID namespace via the canonical constructor PidNamespace::create(&parent_pid_ns, owner_user_ns, owner_euid) (defined at Section 17.1; owner_euid is the creating credential's effective uid — resolved the SAME way as owner_user_ns above: pending_cred.euid when CLONE_NEWUSER is present, task.cred.euid otherwise — the ucount charge key (Linux inc_pid_namespaces uses current_euid()); parent_pid_ns = the caller's current pending_pid_ns if set, otherwise old_ns.pid_ns — the namespace the caller's future children would otherwise be born into; ENOSPC when the parent is already at the maximum nesting level 32 (a level-32 namespace is reachable; only its children are refused) or over /proc/sys/user/max_pid_namespaces, ENOMEM). Store as new_ns.pending_pid_ns (replaces any existing pending value — the old value's refcount is decremented). The calling task does NOT enter the new PID namespace.
CLONE_NEWTIME: Create new time namespace. Store as new_ns.pending_time_ns. Like CLONE_NEWPID, the calling task does NOT enter the new time namespace — only future children will. This matches Linux's behavior where Linux unshare(CLONE_NEWTIME) stores the namespace as time_ns_for_children.
CLONE_NEWNET: Create an empty network namespace by running the full CLONE_NEWNET creation sequence (the three-phase loopback registration steps in Section 17.1 — device build, net_ns_alloc(), then net_stack_cap_create() + registration), with owner_user_ns = pending_cred.user_ns / task.cred.user_ns. Update BOTH new_ns.net_ns AND new_ns.net_stack — the latter from the freshly set cell: new_ns.net_stack = *ns.stack_cap.get().expect("set in creation Phase 3") (dual-field invariant — see NamespaceSet definition). Rebalance the active-user count across the swap — step 3's old_ns.clone() already net_ns_get()'d the INHERITED net_ns, so release it before overwriting and acquire the fresh one:
```
net_ns_put(&new_ns.net_ns);          // release the cloned-in inherited ns
new_ns.net_ns = Arc::clone(&ns);     // the freshly created net namespace
net_ns_get(&new_ns.net_ns);          // this NamespaceSet now uses `ns`
```
The net_ns_put() here cannot prematurely tear the inherited ns down: the caller's CURRENT namespace_set still holds its own active user on it, so its users stays ≥ 1. Partial-write safety: these two field writes are not required to be atomic with each other. new_ns is a local staging variable, not yet visible to any other task (it is only published at step 5's single ArcSwap::store()), so a failure between the two writes (e.g., ENOMEM allocating the Capability<NetStack> for net_stack after net_ns was already assigned) cannot be observed with an inconsistent view by any other thread. It also cannot leak: when sys_unshare() returns Err, the local new_ns (and whichever of its fields were already assigned) drops normally, running impl Drop for NetNamespace on new_ns.net_ns if it was set — which revokes ITS OWN stack_cap (see "Namespace Drop semantics" above). Capability revocation is keyed to the NetNamespace object's own lifetime, not to whether new_ns.net_stack was successfully updated, so the two fields never need to be written atomically for correctness.
CLONE_NEWIPC: Create empty IPC namespace (no inherited IPC objects). Assign to new_ns.ipc_ns.
CLONE_NEWUTS: Create UTS namespace copying parent's hostname/domainname. Assign to new_ns.uts_ns.
CLONE_NEWCGROUP: Create cgroup namespace. The child's cgroup root is the caller's current cgroup. Assign to new_ns.cgroup_ns. See the Inheritance Rules table below for per-type semantics.

Commit credentials and swap namespace_set atomically: If CLONE_NEWUSER was requested, install_credentials() and namespace_set.store() are performed together inside a SINGLE TASK_LOCK(20) critical section — not two separate acquire/release pairs — matching the pattern used by setns(CLONE_NEWUSER) (Section 17.1):

// Pre-allocate before the lock: namespace_set_alloc moves new_ns into Nucleus
// tracked storage ([Section 17.1](#namespace-architecture--tracked-allocation-namespaceset-pidnamespace-netnamespace));
// its slow path may briefly take TRACKED_REGISTRY_LOCK(135), which must
// not nest inside `TASK_LOCK(20)` critical sections' O(1) budget.
let new_namespace_set = namespace_set_alloc(new_ns)?;   // ENOMEM → rollback
// IPC pre-commit effect — placed AFTER the last fallible step (a failed
// unshare must leave the caller's sem-undo state untouched) and OUTSIDE
// the `TASK_LOCK(20)` section below (detach_sysv_sem takes each referenced set's
// SemSet.lock and wakes its WaitQueue — neither belongs inside
// TASK_LOCK(20)'s O(1) critical-section budget). CLONE_NEWIPC or
// CLONE_SYSVSEM: apply-and-drop the caller's sem-undo list NOW
// (detach_sysv_sem, [Section 8.2](08-process.md#process-lifecycle-teardown--step-4c-sysv-semaphore-undo-operations))
// — after the namespace_set swap the OLD namespace's sets are unreachable
// from this task, so undos deferred to exit would be applied against a
// namespace the task long left. Linux ksys_unshare() runs the same
// detach at its commit point (kernel/fork.c, verified against
// torvalds/linux master).
if flags & (libc::CLONE_NEWIPC | libc::CLONE_SYSVSEM) != 0 {
    // The task's namespace_set still names the OLD namespace here — exactly
    // the namespace whose sets the accumulated undos reference
    // (detach_sysv_sem takes the ipc_ns explicitly; it must not read
    // task.namespace_set itself — its signature contract in
    // [Section 8.2](08-process.md#process-lifecycle-teardown--step-4c-sysv-semaphore-undo-operations)).
    detach_sysv_sem(&task.process, &task.namespace_set.load().ipc_ns);
}
// The shm half of Linux's CLONE_NEWIPC commit — `exit_shm(current)` +
// the Linux `shm_init_task(current)` step (kernel/fork.c) — re-anchors the per-task
// CREATED-segment list (`sysvshm.shm_clist`) whose only consumer is the
// `kernel.shm_rmid_forced` sysctl. UmkaOS keeps no per-task shm list:
// attachments hold `Arc<ShmSegment>` through their VMAs and detach via
// `shm_detach()` regardless of the owner's namespace membership, so
// existing shmat() mappings survive the switch unchanged (Linux parity —
// an IPC-namespace change never affects live attachments), and
// `kernel.shm_rmid_forced` is not currently specified (no consumer of a
// creator-task walk exists in the corpus). If that sysctl is added, the
// equivalent created-segment cleanup walk must be added HERE and at task exit.
let old_namespace_set;
{
    let _guard = task.task_lock();
    // TrackedPtr -> Arc::from_tracked, RCU pointer swap, O(1)
    install_credentials(task, pending_cred);
    // CLONE_NEWNS only: apply the deferred fs.root/pwd translation
    // computed by copy_tree() (step 4). Applied HERE — after the last
    // fallible step — so an earlier failure leaves the task's
    // path-resolution state untouched. FS_STRUCT_LOCK(42) nests
    // legally under TASK_LOCK(20) (20 → 42, ascending).
    if let Some(fs_update) = fs_update {
        fs_update.apply(&task.fs.load());   // write-locks the FsStruct
    }
    old_namespace_set = task.namespace_set.swap(new_namespace_set);
}
drop(old_namespace_set);   // Arc refcount decrement OUTSIDE the lock

Holding both operations under one guard prevents the window where credentials and namespace_set disagree: any concurrent has_ns_cap() call against this task observes either the fully-old or the fully-new state, never a mix. If CLONE_NEWUSER was not requested, the commit is the same minus install_credentials: namespace_set_alloc(new_ns)?, then the identical IPC pre-commit detach_sysv_sem (CLONE_NEWIPC/CLONE_SYSVSEM), then (CLONE_NEWNS only) fs_update.apply(...), then task.namespace_set.store(new_namespace_set) (ArcSwap::store() provides its own internal synchronization -- no explicit Ordering parameter; no TASK_LOCK(20) guard is needed since task.cred is not being modified in this path).

Drop the old Arc<NamespaceSet> (decrements refcounts on all old namespaces). When CLONE_NEWUSER was requested, this is the drop(old_namespace_set) already performed outside the lock at step 5; this step applies to the non-CLONE_NEWUSER path, where ArcSwap::store() drops the previous value internally.

If any step fails, roll back all namespaces created so far (same rollback protocol as clone() — see Section 17.1).

PID 1 signal protection within PID namespaces:

The first process created in a PID namespace gets PID 1 and acts as the namespace's init. PID 1 has special signal handling (critical for container correctness):

Default-disposition signals are silently dropped: Signals with default disposition (SIG_DFL) are NOT delivered to PID 1 unless PID 1 has explicitly installed a handler for that signal. This prevents accidental termination of the container init (e.g., SIGTERM with default disposition would kill a normal process but is dropped for namespace PID 1).
SIGKILL/SIGSTOP from within the namespace are dropped: Processes inside the same PID namespace cannot kill or stop their init. This prevents a misbehaving container process from bringing down the container.
Parent namespace CAN send any signal: The parent namespace (or any ancestor namespace) can send any signal including SIGKILL to PID 1 of a child namespace. This is how the container runtime stops a container — it sends SIGKILL from outside the namespace.

These rules are enforced in send_signal() by checking whether the target is whether the target is PID 1 of its namespace and whether the sender is in the same or an ancestor namespace. See Section 8.6.

17.1.2 Namespace Implementation¶

Namespaces are implemented entirely within the umka-sysapi layer. The core microkernel (umka-nucleus) is unaware of namespaces; it only understands Capability Domains and object access rights.

/// Opaque per-container identity, used to scope resources that must not leak or
/// collide across containers — notably the Windows NT object namespace
/// ([Section 19.6](19-sysapi.md#windows-emulation-acceleration)), where two containers must not be able
/// to squat each other's named objects (`\BaseNamedObjects\...`).
///
/// A "container" is not a distinct kernel object in UmkaOS (as in Linux, it
/// is a bundle of namespaces), so the kernel never observes a "container
/// create" event it could stamp an id at. `ContainerId` is therefore
/// DERIVED, never stored: **the container anchor is the task's active PID
/// namespace**, and the id is that namespace's never-reused `ns_id` —
/// `ContainerId::of(namespace_set)` below returns
/// `ContainerId(namespace_set.pid_ns.ns_id)`, with the init PID namespace mapped
/// to the reserved host value `ContainerId::HOST` (0). No `NamespaceSet`
/// field carries it (a stored copy could only go stale across
/// `setns`/`unshare`), and "the container's root PID namespace" needs no
/// nesting heuristic — EVERY PID-namespace level IS its own container for
/// isolation purposes:
///
/// - All tasks sharing a PID namespace observe the same `ContainerId`;
///   container runtimes create one PID namespace per container, so a
///   container's tasks agree on the id.
/// - A NESTED container (new PID namespace inside the outer one) gets a
///   DISTINCT id — required: NT-object isolation means the nested container
///   must not be able to squat the OUTER container's names either.
/// - A task that changed only its other namespaces (`setns(CLONE_NEWNET)`
///   etc.) keeps its PID namespace and therefore its container identity.
/// - The failure direction is CLOSED: a sandbox that unshares a PID
///   namespace inside a container becomes a SEPARATE object-namespace scope
///   (over-isolation — its NT objects are invisible to the outer
///   container), never a shared one (cross-container squatting is
///   structurally impossible).
///
/// u64 — `ns_id` comes from `NEXT_NS_ID` (never reused within the 50-year
/// uptime envelope; longevity analysis at the static's definition below);
/// `0` is reserved for the host: `NEXT_NS_ID` starts at 1, so no namespace
/// ever collides with `HOST`.
#[repr(transparent)]
#[derive(Clone, Copy, PartialEq, Eq, Hash, PartialOrd, Ord)]
pub struct ContainerId(pub u64);

impl ContainerId {
    /// The host / init-PID-namespace scope (reserved id 0).
    pub const HOST: ContainerId = ContainerId(0);

    /// Derive the container identity of a namespace set — the ONLY
    /// constructor besides `HOST`. O(1) (one pointer compare + one field
    /// read). Consumers (the NT object namespace,
    /// [Section 19.6](19-sysapi.md#windows-emulation-acceleration)) call it with the acting
    /// task's `namespace_set` at object creation/open and compare ids.
    pub fn of(namespace_set: &NamespaceSet) -> ContainerId {
        let init = INIT_PID_NS.get().expect("init namespaces set at boot");
        if Arc::ptr_eq(&namespace_set.pid_ns, init) {
            ContainerId::HOST
        } else {
            ContainerId(namespace_set.pid_ns.ns_id)
        }
    }
}

/// Per-task namespace set. Owned by each `Task` (not `Process`) so that
/// `setns(2)` and `unshare(2)` can change namespaces for a single thread
/// without affecting siblings.
/// Wrapped in `Arc<NamespaceSet>` in the Task struct; threads that share
/// namespaces share the same `Arc`. `unshare()`/`setns()` replaces the
/// calling task's `Arc` with a new one.
pub struct NamespaceSet {
    /// PID namespace for this task. Determines the PID number space
    /// visible to the task: `getpid()` returns this namespace's local
    /// PID, not the global TaskId. The `PidNamespace` struct (defined
    /// below) contains the per-namespace IDR allocation map (`pid_map`)
    /// and the reverse map (global TaskId -> local pid_t).
    ///
    /// Shared via `Arc` across all tasks in the same PID namespace.
    /// A new `Arc<PidNamespace>` is created only by `clone(CLONE_NEWPID)`
    /// or `unshare(CLONE_NEWPID)`.
    pub pid_ns: Arc<PidNamespace>,

    /// Pending PID namespace for future children (set by setns(CLONE_NEWPID)).
    /// When set, fork()/clone() creates children in this namespace rather than
    /// the current task's PID namespace. The task's own PID is unchanged.
    ///
    /// Protected by SpinLock to serialize a `setns()`/`unshare()` writer
    /// against a concurrent `clone()` reader in multi-threaded processes.
    ///
    /// **PERSISTENT, NOT one-shot** (Linux `pid_ns_for_children`): `clone()`
    /// READS `pending_pid_ns` WITHOUT clearing it (`create_task()` step 10:
    /// `parent_ns.pending_pid_ns.lock().clone()`,
    /// [Section 8.1](08-process.md#process-and-task-management--process-creation)), so
    /// EVERY future child is born into the pending namespace until the value
    /// is REPLACED by a subsequent `setns(CLONE_NEWPID)` / `unshare(CLONE_NEWPID)`.
    /// A clear-on-read (one-shot) rule would drop the SECOND post-unshare
    /// fork back into the parent's old PID namespace — an ABI-visible break of
    /// every runtime that forks more than once after `unshare(CLONE_NEWPID)`.
    /// The SpinLock exists only to make the read/replace race safe, never to
    /// clear the field.
    pub pending_pid_ns: SpinLock<Option<Arc<PidNamespace>>>,

    /// Mount namespace containing the mount tree, mount hash table,
    /// and all mount metadata for this task's VFS view.
    /// See [Section 14.6](14-vfs.md#mount-tree-data-structures-and-operations) for `MountNamespace` definition.
    pub mount_ns: Arc<MountNamespace>,

    /// Network stack capability handle (`Capability<NetStack>` in the doc
    /// notation) — used to resolve the concrete namespace via
    /// `cap_resolve()`. `CapHandle` per [Section 9.1](09-security.md#capability-based-foundation).
    pub net_stack: CapHandle,

    /// Cached resolved network namespace. Populated from `net_stack.cap_resolve()`
    /// at creation time (clone/unshare/setns). Provides direct `Arc<NetNamespace>`
    /// access without capability resolution on every socket/routing operation.
    /// All networking code uses `namespace_set.net_ns` (not `net_stack`) for lookups.
    ///
    /// **INVARIANT**: `net_ns` must always equal `net_stack.cap_resolve()`. Updated
    /// in lockstep with `net_stack` at every NamespaceSet construction site:
    /// `clone()`, `unshare()`, `setns(CLONE_NEWNET)`, and the `CLONE_NEWUSER` path.
    /// A stale `net_ns` silently routes network operations (socket creation, routing
    /// lookups, neighbor resolution) to the wrong namespace.
    ///
    /// **Circular reference prevention**: `net_stack` (Capability<NetStack>) holds
    /// an `Arc<NetNamespace>` internally via capability resolution. `net_ns` is a
    /// separate `Arc<NetNamespace>` clone (not a second ownership path — both point
    /// to the same allocation). There is no cycle: `NamespaceSet` → `Arc<NetNamespace>`
    /// is a one-way ownership edge. The `NetNamespace` does NOT hold a reference back
    /// to `NamespaceSet`. There is no cycle because `UserNamespace` does not hold
    /// a reference back to `NetNamespace` or `NamespaceSet`. The `Arc<UserNamespace>`
    /// in `NetNamespace.user_ns` is a forward reference (child namespace pointing
    /// to parent user namespace), not a back-edge.
    pub net_ns: Arc<NetNamespace>,

    /// UTS namespace (hostname, domainname).
    pub uts_ns: Arc<UtsNamespace>,

    /// IPC namespace (SysV semaphores, message queues, shared memory).
    pub ipc_ns: Arc<IpcNamespace>,

    /// Cgroup namespace (cgroup root view).
    pub cgroup_ns: Arc<CgroupNamespace>,

    /// Time namespace offsets (CLOCK_MONOTONIC, CLOCK_BOOTTIME).
    pub time_ns: Arc<TimeNamespace>,

    /// Pending time namespace for future children (set by setns(CLONE_NEWTIME)).
    /// When set, fork()/clone() creates children with the target time offsets.
    /// Follows Linux 5.6+ semantics where CLONE_NEWTIME affects children only
    /// (time namespaces shipped in Linux 5.6; the affects-children-only rule
    /// held from day one — `uapi/linux/sched.h`, verified vs torvalds/linux).
    ///
    /// Protected by SpinLock (same rationale as `pending_pid_ns` above).
    pub pending_time_ns: SpinLock<Option<Arc<TimeNamespace>>>,

    /// User namespace governing UID/GID mappings and capability scope.
    pub user_ns: Arc<UserNamespace>,

    /// IMA namespace (per-container integrity measurement policy and log).
    /// Created alongside the user namespace. See Section 9.4.3 for ImaNamespace struct.
    pub ima_ns: Arc<ImaNamespace>,
}

impl NamespaceSet {
    /// Construct a "tombstone" NamespaceSet referencing the init namespaces.
    /// Used by `exit_task()` Step 11 to detach the exiting task from its
    /// namespaces without leaving dangling references. Each field points to
    /// the system's init namespace instance (created at boot, never destroyed).
    /// (Linux's init task holds the same init-namespace instances through its
    /// own init namespace set.)
    pub fn empty() -> Self {
        // Each INIT_*_NS is a `OnceCell<Arc<T>>`, not a bare `Arc<T>`.
        // `.get().expect(...)` extracts the inner `&Arc<T>` from the
        // initialized OnceCell. This panics only if called before
        // `init_namespaces()` completes (which runs early in boot,
        // before any task can call `exit_task()`).
        let ns = NamespaceSet {
            pid_ns: Arc::clone(INIT_PID_NS.get().expect("INIT_PID_NS not initialized")),
            pending_pid_ns: SpinLock::new(None),
            mount_ns: Arc::clone(INIT_MOUNT_NS.get().expect("INIT_MOUNT_NS not initialized")),
            net_stack: INIT_NET_STACK.get().expect("INIT_NET_STACK not initialized").clone(),
            net_ns: Arc::clone(INIT_NET_NS.get().expect("INIT_NET_NS not initialized")),
            uts_ns: Arc::clone(INIT_UTS_NS.get().expect("INIT_UTS_NS not initialized")),
            ipc_ns: Arc::clone(INIT_IPC_NS.get().expect("INIT_IPC_NS not initialized")),
            cgroup_ns: Arc::clone(INIT_CGROUP_NS.get().expect("INIT_CGROUP_NS not initialized")),
            time_ns: Arc::clone(INIT_TIME_NS.get().expect("INIT_TIME_NS not initialized")),
            pending_time_ns: SpinLock::new(None),
            user_ns: Arc::clone(INIT_USER_NS.get().expect("INIT_USER_NS not initialized")),
            ima_ns: Arc::clone(INIT_IMA_NS.get().expect("INIT_IMA_NS not initialized")),
        };
        // Active-user accounting: this NamespaceSet names INIT_NET_NS.
        // Balanced by `impl Drop for NamespaceSet`. (INIT_NET_NS therefore
        // never reaches users→0 — correct; the init net namespace is
        // permanent.)
        net_ns_get(&ns.net_ns);
        ns
    }
}

/// Clone implementation for NamespaceSet. For each Arc field, performs
/// `Arc::clone()` (cheap refcount increment). For SpinLock fields
/// (pending_pid_ns, pending_time_ns), acquires the lock, clones the inner
/// value (`Option<Arc<T>>`), and creates a new unlocked SpinLock protecting
/// the cloned value. The new NamespaceSet is an independent copy that can
/// be mutated without affecting the original.
impl Clone for NamespaceSet {
    fn clone(&self) -> Self {
        let ns = NamespaceSet {
            pid_ns: Arc::clone(&self.pid_ns),
            pending_pid_ns: SpinLock::new(self.pending_pid_ns.lock().clone()),
            mount_ns: Arc::clone(&self.mount_ns),
            net_stack: self.net_stack.clone(),
            net_ns: Arc::clone(&self.net_ns),
            uts_ns: Arc::clone(&self.uts_ns),
            ipc_ns: Arc::clone(&self.ipc_ns),
            cgroup_ns: Arc::clone(&self.cgroup_ns),
            time_ns: Arc::clone(&self.time_ns),
            pending_time_ns: SpinLock::new(self.pending_time_ns.lock().clone()),
            user_ns: Arc::clone(&self.user_ns),
            ima_ns: Arc::clone(&self.ima_ns),
        };
        // A cloned NamespaceSet is a NEW active user of the same net_ns.
        // If a later `unshare(CLONE_NEWNET)` replaces `ns.net_ns`, that arm
        // does `net_ns_put(&ns.net_ns)` before the swap and `net_ns_get()`
        // after (see step 4's CLONE_NEWNET branch), keeping the count exact.
        net_ns_get(&ns.net_ns);
        ns
    }
}

/// Active-user accounting for the network namespace: every `NamespaceSet`
/// that names a net_ns holds exactly one active user (`net_ns_get()` at
/// construction), released here when the NamespaceSet object is destroyed
/// (its last `Arc<NamespaceSet>` dropped). The `users → 0` transition inside
/// `net_ns_put()` runs `net_ns_cleanup()` — the sole active-teardown trigger.
///
/// **Invariant** (deferred handoff to `create_task()` /
/// [Section 8.1](08-process.md#process-and-task-management--process-creation)): any
/// site that builds a child `NamespaceSet` by other than `clone()`/`empty()`
/// (e.g. the fork inheritance path, or a `setns(CLONE_NEWNET)` swap) MUST
/// `net_ns_get()` the net_ns it installs so this Drop's `net_ns_put()` stays
/// balanced. Only `net_ns` carries an active count; the other namespace types
/// are torn down purely by their `Arc` strong count (no socket/device cycle).
impl Drop for NamespaceSet {
    fn drop(&mut self) {
        net_ns_put(&self.net_ns);
    }
}

17.1.2.1 Init Namespace Initialization¶

All init namespace instances are stored as static globals initialized once at boot via BootOnceCell. Initialization order matters — user_ns must be first because all other namespaces reference it via user_ns fields. Init namespaces are never destroyed (their refcount never reaches zero).

/// Init (root) PID namespace. Level 0, no parent.
static INIT_PID_NS: BootOnceCell<Arc<PidNamespace>> = BootOnceCell::new();
/// Init mount namespace. Contains the root filesystem mount tree.
static INIT_MOUNT_NS: BootOnceCell<Arc<MountNamespace>> = BootOnceCell::new();
/// Init network namespace. Contains the host network stack.
static INIT_NET_NS: BootOnceCell<Arc<NetNamespace>> = BootOnceCell::new();
/// Network stack capability handle — a type alias for the erased `CapHandle`
/// ([Section 9.1](09-security.md#capability-based-foundation)); `Capability<NetStack>` in the doc model.
/// Aliasing (not a distinct newtype) keeps `INIT_NET_STACK` and the
/// `NamespaceSet.net_stack` field (declared `CapHandle`) the SAME type, so the
/// boot-time `INIT_NET_STACK.get()...clone()` into `net_stack` type-checks.
type NetStackHandle = CapHandle;
/// Init network stack capability handle.
static INIT_NET_STACK: BootOnceCell<NetStackHandle> = BootOnceCell::new();
/// Init UTS namespace. Contains the host's hostname and domainname.
static INIT_UTS_NS: BootOnceCell<Arc<UtsNamespace>> = BootOnceCell::new();
/// Init IPC namespace. Contains system-wide SysV/POSIX IPC objects.
static INIT_IPC_NS: BootOnceCell<Arc<IpcNamespace>> = BootOnceCell::new();
/// Init cgroup namespace. Root cgroup view.
static INIT_CGROUP_NS: BootOnceCell<Arc<CgroupNamespace>> = BootOnceCell::new();
/// Init time namespace. Zero offsets (no adjustment).
static INIT_TIME_NS: BootOnceCell<Arc<TimeNamespace>> = BootOnceCell::new();
/// Init user namespace. Root user namespace (uid_map = identity).
static INIT_USER_NS: BootOnceCell<Arc<UserNamespace>> = BootOnceCell::new();
/// Init IMA namespace.
static INIT_IMA_NS: BootOnceCell<Arc<ImaNamespace>> = BootOnceCell::new();

INIT_PID_NS and INIT_NET_NS are produced by the tracked-storage constructors pid_ns_alloc() / net_ns_alloc() (Section 17.1); ns_register_tracked_types() runs before init_namespaces(), so even the boot-time instances are enumerable by the evolution orchestrator. A registration or allocation failure here is a boot-time panic directing the operator to raise umka.tracked_storage_size — deterministic at boot, never a runtime surprise.

Initialization order (called from init_namespaces() during boot, after memory allocator and slab are online): 1. INIT_USER_NS — first, because all others reference it. 2. INIT_PID_NS — PID 1 (init) is the child reaper. 3. INIT_MOUNT_NS — requires VFS and root filesystem to be mounted. 4. INIT_NET_NS, INIT_NET_STACK — network stack initialization. 5. INIT_UTS_NS, INIT_IPC_NS, INIT_CGROUP_NS, INIT_TIME_NS, INIT_IMA_NS — order among these is arbitrary (no inter-dependencies).

/// PID namespace. Each namespace has its own PID number space: a process visible
/// in a child namespace has a different pid_t than in the parent namespace.
///
/// # Nesting
/// PID namespaces form a tree. The root (init) namespace is the global root.
/// A process in namespace N with pid=5 may appear as pid=105 in namespace N's parent.
/// Translation traverses `parent` pointers up the tree.
///
/// # PID allocation
/// Each namespace allocates PIDs from an IDR (integer allocation map). The global PID
/// (used internally in the kernel) is always allocated from the root namespace.
/// Every namespace in the path from root to the process's namespace gets one entry
/// in the translation map.
///
/// # /proc visibility and PID enumeration
/// `/proc/[pid]` uses the PID from the reading process's namespace, not the global
/// TaskId. A process reading `/proc/5/status` in namespace N will see the task whose
/// local pid_t in N equals 5; the same task may have a different pid_t in the parent
/// namespace. If no task with that local pid exists in the reader's namespace,
/// the entry is absent from `/proc`.
///
/// **PID enumeration scoping**: `readdir("/proc")` enumerates only the PIDs visible
/// in the calling process's PID namespace. A container's init (PID 1) sees only its
/// own descendants; the host's root namespace sees all PIDs. The procfs `readdir`
/// implementation iterates `PidNamespace.pid_map` (below) for the caller's namespace
/// level, yielding only entries that have a valid local pid_t at that level.
pub struct PidNamespace {
    /// Unique namespace identifier (for /proc/self/ns/pid).
    pub ns_id: u64,
    /// Owning user namespace. Required by the `Namespace` trait for
    /// `setns()` capability checks. Set at creation to the CREATING
    /// CREDENTIAL's user namespace (`child_cred.user_ns` in fork /
    /// `pending_cred.user_ns` in unshare — the post-CLONE_NEWUSER-
    /// transform namespace; see the owner-assignment rule in
    /// [Section 17.1](#namespace-architecture--capability-domain-mapping)).
    ///
    /// STRONG `Arc` — a PID namespace PINS its owning user namespace alive
    /// (Linux `copy_pid_ns()` → `get_user_ns()`). This is NOT a cycle: a
    /// `UserNamespace` never references a `PidNamespace`, so the edge is
    /// one-way. Unlike `child_reaper` below (a Task back-reference that
    /// WOULD cycle, hence a scalar id), there is nothing to break here. The
    /// strong edge also makes the per-user PID-namespace count leak-free:
    /// the owner's `UserEntry` ancestor chain — where the count lives —
    /// is guaranteed alive when `impl Drop for PidNamespace` uncharges it
    /// ([Section 17.1](#namespace-architecture--user-namespace-uidgid-mapping-security)).
    pub user_ns: Arc<UserNamespace>,
    /// TaskId of the namespace's init process (PID 1). `0` = not yet set
    /// (namespace created, first task not yet published) or init has been
    /// released. Set by `create_task` step 16 (`store(child.tid, Release)`)
    /// when `CLONE_NEWPID` gave this namespace its PID 1; cleared
    /// (`store(0, ...)`) by the fork rollback rows and at init's
    /// `reap_task()`. Used by signal delivery to implement PID 1
    /// signal protection and by `exit_task()`'s zap gate.
    ///
    /// **Scalar `AtomicU64` id, not `Weak<Task>`** — the canonical
    /// back-reference pattern for cycle-threatened parent links
    /// (`Process.parent`, ptrace tracer link): TaskIds are never reused,
    /// so a stale id simply fails resolution — no ABA. Readers resolve
    /// via `find_task_by_tid(ns.child_reaper.load(Acquire))` (RCU
    /// lookup-and-pin, lock-free); `None` means the init task has exited
    /// (namespace teardown in progress via `terminate_members()`) —
    /// callers fall back to the parent-namespace reaper walk, never
    /// `.expect()`. A `Weak<Task>` here would additionally be a plain
    /// non-atomic field mutated through a shared `Arc<PidNamespace>`
    /// (un-compilable — `Arc` has no `DerefMut`) and racy against
    /// concurrent signal-delivery readers; the atomic id is both
    /// compilable through `&self` and data-race-free.
    pub child_reaper: AtomicU64,
    /// Parent namespace. `None` only for the root PID namespace.
    pub parent: Option<Arc<PidNamespace>>,
    /// Nesting level. Root = 0; maximum = 32 (matches Linux `MAX_PID_NS_LEVEL`).
    /// Both PID and user namespace nesting depth exceeded return ENOSPC.
    /// PID nesting is bounded by the real Linux constant `MAX_PID_NS_LEVEL`
    /// (32; Linux `create_pid_namespace()` checks `level > MAX_PID_NS_LEVEL`,
    /// `kernel/pid_namespace.c`). User-namespace nesting has NO such named
    /// constant in Linux: `create_user_ns()` uses a literal check
    /// `parent_ns->level > 32` (`kernel/user_namespace.c`, verified against
    /// torvalds/linux master). There is no `MAX_USER_NS_LEVEL`.
    pub level: u32,
    /// PID allocation map for this namespace level.
    /// Key: pid_t value in this namespace (allocated by IDR); Value: global TaskId.
    ///
    /// IDR (integer-ID radix-tree allocator) provides O(log n) pid allocation with
    /// RCU read-side protection. `pid_lookup()` is lock-free on the read path
    /// (kill(), waitpid(), /proc/[pid] traversal). Writes (fork/exit) are serialized
    /// by the Idr's internal SpinLock. Integrated next-ID allocation eliminates a
    /// separate PID counter and separate "find a free PID" logic.
    pub pid_map: Idr<TaskId>,
    /// Reverse map: global TaskId → local pid_t in this namespace.
    /// Used by `local_nr()` (TaskId → local pid_t) — a MEMBERSHIP-EXPECTED
    /// lookup only (see `local_nr()`'s caller contract): callers such as
    /// `collapse_thread_group`'s PID-number swap already know the task lives at this
    /// level, so the cache hit is the overwhelming case. The
    /// namespace-VISIBILITY hot flows — signal delivery, `waitpid()`,
    /// `/proc/[pid]` translation — do NOT probe this cache: they translate
    /// through the task's captured `pid_links` chain (`pid_nr_in()`
    /// below, "no reverse_map probe"), where a non-membership verdict is
    /// O(depth), not a scan.
    ///
    /// RCU-protected read path: `local_nr()` takes only an RCU read guard (~1-3 cycles,
    /// no spinning). Write path: insert at fork, remove at exit — both serialized by
    /// `pid_map`'s existing SpinLock (held anyway for IDR allocation/deallocation).
    /// This eliminates the separate `SpinLock<HashMap>` that previously serialized
    /// every signal delivery on the read path.
    ///
    /// Implementation: sparse radix tree (same Idr structure as pid_map) keyed on
    /// the lower 32 bits of TaskId. **Longevity analysis**: At 1 million forks/sec
    /// (sustained, far beyond any practical workload), the lower 32 bits of TaskId
    /// wrap after ~4295 seconds (~72 minutes). However, the reverse_map only contains
    /// LIVE tasks in this namespace — entries are removed at task exit. A collision
    /// requires two live tasks in the same namespace whose TaskId lower-32 bits match,
    /// which requires >4 billion cumulative forks within the namespace lifetime with
    /// both tasks still alive. For short-lived containers this is impossible; for
    /// long-running namespaces with extreme fork rates, the full 64-bit TaskId is
    /// used for authoritative identification (the reverse_map is a fast-path cache).
    ///
    /// **Collision fallback**: After `reverse_map.lookup(task_id as u32)` returns
    /// a candidate `pid_t`, the caller MUST verify `pid_map.lookup(pid_t).task_id ==
    /// task_id` (full 64-bit check). On mismatch, fall back to linear scan of
    /// `pid_map` entries. This ensures correctness even when two live tasks' lower-32
    /// bits collide. The linear scan is O(N) where N = live tasks in the namespace,
    /// but the mismatch case is astronomically rare under normal workloads.
    pub reverse_map: RcuIdr<u32>,
    /// Process-group number registry for this level: ns-local pgid number →
    /// the live group. One entry per live `ProcessGroup` whose captured
    /// `pid_chain` includes this namespace, inserted at group creation and
    /// removed when the group empties. This is the caller-namespace
    /// resolution path for `setpgid`/`tcsetpgrp`/`kill(pid <= 0)`/
    /// `waitpid(0, < -1)` group arguments, and the reason a live group's
    /// numbers are PINNED against recycling (see
    /// [Section 17.1](#namespace-architecture--process-group-session-number-registries-and-pid-pinning)).
    /// Reads are RCU/lock-free; mutations happen only inside the pin/unpin
    /// compound operations, serialized with `pid_map` by PID_MAP_LOCK(47).
    pub pgids: XArray<u32, Arc<ProcessGroup>>,
    /// Session twin of `pgids`: ns-local sid number → live session.
    /// Same lifecycle and locking rules.
    pub sids: XArray<u32, Arc<Session>>,
    /// Maximum PID value in this namespace. Default: 32768
    /// (`0x8000`, Linux `include/linux/threads.h`). This is externally visible:
    /// a fresh `/proc/sys/kernel/pid_max` MUST read 32768 to match Linux — the
    /// value 4,194,304 is NOT the default but `PID_MAX_LIMIT` (`4 * 1024 * 1024`
    /// on 64-bit), the sysctl *ceiling*. "PID_MAX" is not a Linux constant.
    /// **Invariant**: pid_max <= i32::MAX (2,147,483,647). POSIX pid_t is signed
    /// i32; values above i32::MAX would be interpreted as negative PIDs by
    /// userspace (violating the `kill(-pid)` process-group convention). A sysctl
    /// write to `/proc/sys/kernel/pid_max` is clamped to `PID_MAX_LIMIT`
    /// (Linux parity); UmkaOS additionally never permits a value above i32::MAX.
    /// Reduced-max namespaces allow container runtimes to limit PID exhaustion attacks.
    pub pid_max: u32,
    /// Number of active tasks in this namespace.
    /// **u32 justification**: Bounded by `pid_max` (max i32::MAX ≈ 2.1 billion),
    /// which is well within u32 range. Unlike cgroup task counters (which use
    /// AtomicU64 because cgroups can span multiple PID namespaces and accumulate
    /// across namespace boundaries), a single PID namespace's task count is
    /// strictly bounded by its `pid_max`.
    pub nr_tasks: AtomicU32,
    /// Namespace-death latch. `false` for the namespace's whole live
    /// lifetime; set to `true` — ONCE, never cleared — by
    /// `terminate_members()` step 0, under this namespace's
    /// PID_MAP_LOCK(47), BEFORE the SIGKILL broadcast. Checked by
    /// `create_task()` step 10a inside each level's allocation critical
    /// section (same lock): a set flag fails the fork with ENOMEM.
    ///
    /// This is the mechanism that closes the window neither the
    /// fork-side `fatal_signal_pending(parent)` check nor the zap
    /// re-broadcast can: a CROSS-NAMESPACE parent (forking into this
    /// namespace via `pending_pid_ns`) is not a member and is never
    /// signalled by the broadcast, and once zap has returned there is
    /// no re-broadcast left to kill its child — without this latch such
    /// a fork would deliver a live task into a namespace whose init is
    /// gone (unreapable orphan; a fresh task could even be handed PID 1
    /// of a dead namespace). ABI parity: Linux fails such forks with
    /// `-ENOMEM` (Linux `alloc_pid()` after `disable_pid_allocation()` cleared
    /// `PIDNS_ADDING` in Linux — `kernel/pid.c` / `kernel/pid_namespace.c`).
    /// UmkaOS deliberately uses an explicit named flag rather than
    /// folding a state bit into a counter word (Linux's
    /// packed state bit and member count) — same behavior, no
    /// flags-word import; the lock-serialized explicit latch costs one
    /// load under a lock already held.
    ///
    /// Why the lock matters: setting and checking under the SAME
    /// per-namespace PID_MAP_LOCK(47) section that publishes the
    /// `pid_map` entry and increments `nr_tasks` makes the two orders
    /// exhaustive — a fork's level either completes its locked section
    /// BEFORE the latch store (its entry and count are then visible to
    /// zap's broadcast and reap-and-wait loop) or observes the latch
    /// and unwinds. No third interleaving exists.
    pub dying: AtomicBool,
    /// This namespace's own per-user creation charge
    /// (`NsUcountKind::PidNs`), returned by `ns_ucount_charge` at
    /// `create()`. Held for the namespace's whole life; its `Drop`
    /// uncharges every ancestor level. Because the token holds a strong
    /// `Arc<UserEntry>` per charged level, teardown decrements exactly what
    /// creation incremented — the uncharge needs no live-owner walk, so it
    /// is leak-free regardless of `user_ns` lifetime
    /// ([Section 17.1](#namespace-architecture--user-namespace-uidgid-mapping-security)).
    pub ucount_charge: NsUcountCharge,
}

/// Translates a global `TaskId` to the local pid_t visible in `ns`.
/// Returns `None` if the task is not visible in `ns` (created in a sibling namespace).
///
/// **Caller contract — membership-expected lookups ONLY.** Absence is not
/// cheap here: the reverse_map is a lossy lower-32 cache, so EVERY
/// true-negative (task not in `ns`) — and every collision or
/// value-verified-eviction miss — falls through to `local_nr_slow()`'s
/// O(live-tasks-in-namespace) forward scan, bounded only by `pid_max`.
/// `local_nr()` is therefore for callers that already know the task is (or
/// until a moment ago was) a member of `ns` — e.g. `collapse_thread_group`'s PID-number
/// swap — where the fast path hits and the scan is a rare cache-repair. Do
/// NOT use it as a visibility PREDICATE: signal delivery, `waitpid()`, and
/// `/proc/[pid]` translation ask "is this task visible in `ns`?" as a
/// routine question whose answer is often "no", and must use
/// `pid_nr_in()` (below) — its captured-chain walk answers
/// non-membership in O(depth), never a scan. A caller that routinely feeds
/// non-members to `local_nr()` turns its worst case into its common case.
///
/// Uses an RCU read-side guard — no spinning, no lock acquisition.
pub fn local_nr(task_id: TaskId, ns: &PidNamespace) -> Option<u32> {
    let guard = rcu_read_lock();
    // Truncating cast: reverse_map is keyed on the lower 32 bits of TaskId
    // (TaskId is a u64 type alias — plain `as u32` IS the lower-32 extraction).
    //
    // ABSENCE is NOT authoritative not-visible. The reverse_map is a
    // fast-path cache keyed on lower-32; a live task's slot can be lost when
    // a COLLIDING task's value-verified eviction (`pid_allocator_free`,
    // below) removes the shared lower-32 slot after the collider exits —
    // leaving this still-live task with no cache entry. A missing slot must
    // therefore fall through to the authoritative forward-map scan, NEVER
    // return `None` on its own (a `?` here would make such a task
    // permanently invisible to signal delivery / waitpid / /proc — reachable
    // only past >4G forks in one namespace, i.e. within the 50-year horizon).
    let candidate_pid = match ns.reverse_map.lookup(task_id as u32, &guard) {
        Some(p) => p,
        None => return local_nr_slow(task_id, ns, &guard),
    };
    // Full 64-bit verification: at 1M forks/sec the lower 32 bits wrap after
    // ~72 minutes. If two live tasks collide on lower32, the reverse_map
    // returns the wrong pid_t. Verify via the forward map.
    //
    // Forward-map ABSENCE is ALSO not authoritative-None: the cached
    // candidate pid can be stale (a colliding task's locked release section
    // removed the forward slot after we read the reverse cache — RCU readers
    // interleave freely with that section). A `?` here would return
    // authoritative-`None` for a live, visible task. Fall through to the
    // scan, exactly as reverse_map absence does above.
    let entry = match ns.pid_map.lookup(candidate_pid, &guard).copied() {
        Some(e) => e,
        None => return local_nr_slow(task_id, ns, &guard),
    };
    if entry == task_id {
        Some(candidate_pid)
    } else {
        // Collision: fall back to linear scan (cold path).
        local_nr_slow(task_id, ns, &guard)
    }
}

/// Cold-path linear scan for local_nr when lower-32 collision is detected.
/// Iterates all entries in the pid_map to find the one matching the full
/// 64-bit TaskId. O(N) where N = live tasks in this namespace, bounded
/// by pid_max. Expected frequency: near-zero under normal workloads.
#[cold]
fn local_nr_slow(task_id: TaskId, ns: &PidNamespace, guard: &RcuReadGuard) -> Option<u32> {
    for (pid, tid) in ns.pid_map.iter(guard) {
        if *tid == task_id {
            return Some(pid);
        }
    }
    None
}

/// Translates a task's PID as seen from an arbitrary target namespace.
///
/// This is the primary cross-namespace PID translation function, equivalent to
/// Linux's `task_pid_nr_ns()`. Used by:
/// - `kill()` to translate the target PID from the caller's namespace
/// - `waitpid()` to report child PIDs in the caller's namespace
/// - `getppid()` to report the parent's PID in the caller's namespace
/// - `/proc/[pid]/status` fields (PPid, NSpid, etc.)
/// - `io_uring` PID namespace resolution
///
/// # Algorithm
/// Resolve the task's namespace membership through its IMMUTABLE captured PID
/// chain (`Task.pid_links`, [Section 8.1](08-process.md#process-and-task-management--task-model)) —
/// NEVER through `task.namespace_set`. The chain runs root→leaf; each `PidLink`
/// carries `{ ns, nr }` = the namespace and the pid_t `create_task()` step 10a
/// allocated at that level. The task is visible in `ns` iff some link's
/// namespace IS `ns`; the captured `nr` is authoritative, so no reverse_map
/// probe, level compare, or parent walk is needed:
/// 1. Snapshot `pid_links` under its LEAF SpinLock (no other lock is taken).
/// 2. Return the `nr` of the link whose `ns.ns_id == ns.ns_id`.
/// 3. No such link ⇒ the task was created in a PARENT of `ns` (no link at
///    `ns`'s depth) or in a SIBLING at the same level ⇒ `None` — the same
///    verdict the old level-compare + sibling-check produced.
///
/// **Why not `task.namespace_set`**: on every release path `exit_task()` Step 11
/// tombstones `namespace_set` to the init namespace set (`NamespaceSet::empty()`,
/// level 0). A post-Step-11 zombie read through `namespace_set` would report
/// `task_ns.level == 0`, so any container observer (`ns.level > 0`) took the
/// `ns.level > task_ns.level → None` branch: `si_pid` collapsed to 0 in
/// `do_notify_parent` and `wait4(pid)` never matched — SIGCHLD/waitpid broken
/// in EVERY non-root PID namespace. `pid_links` survives until
/// `reap_task()` and is stable for a live/zombie task (mutated only by
/// `collapse_thread_group` Phase 3(a2)), so it is the correct observer substrate.
///
/// Hot path: O(depth), depth = chain length (typically 1-3).
pub fn pid_nr_in(task: &Task, ns: &PidNamespace) -> Option<u32> {
    // Snapshot the captured chain under its LEAF SpinLock, then release it
    // before returning (the clone is ≤ 33 `PidLink`s — Arc + i32 each; the
    // leaf-lock contract forbids nesting any other lock under it).
    let links: ArrayVec<PidLink, 33> = task.pid_links.lock().clone();
    for link in links.iter() {
        // Compare by ns_id, not Arc::ptr_eq: `ns` is `&PidNamespace` (not
        // `&Arc<PidNamespace>`), and `ns_id: u64` is unique per namespace for
        // the kernel's lifetime — u64 equality IS pointer identity here.
        if link.ns.ns_id == ns.ns_id {
            return Some(link.nr as u32);
        }
    }
    None // task not a member of `ns` (parent-of or sibling) — not visible
}

17.1.2.1.1 `pid_nr_in_lockless()` — Interrupt/NMI-Context Task Translation¶

pid_nr_in() above takes the pid_links leaf SpinLock. That is correct and required for process-context callers, but it makes the function unusable from a context that may not spin on a lock the interrupted code could already hold — hard IRQ, and on architectures that have them, NMI. The PMU sample-record writer runs in exactly that context (Section 20.8) and still has to translate ids.

Task.pid_links is mutated in only two published-task cases — collapse_thread_group() Phase 3(a2) and pid_ns_release()'s reap-time drain — and both now bracket their write with the task's pid_links_seq odd/even bump (Section 8.1). That makes a retrying lock-free read well-defined:

/// Lock-free counterpart of `pid_nr_in()` for interrupt/NMI context.
/// Same answer, same `None`-means-not-visible semantics; never spins on a
/// lock, never allocates, never sleeps.
///
/// # Reader discipline (normative — a violation is a use-after-free)
/// The snapshot window is UNVALIDATED until the closing seqcount check, so
/// inside it the reader may ONLY read plain inline bytes of the `Task`. It
/// MUST NOT clone an `Arc<PidNamespace>` (a refcount bump on a stale pointer
/// is unrecoverable) and MUST NOT dereference one. Namespace identity is
/// therefore decided by comparing the `Arc`'s POINTER VALUE against `ns` —
/// the same deref-free comparison `pgid_nr_ns()` uses
/// ([Section 8.7](08-process.md#process-groups-and-sessions--per-namespace-numbering-pgidnrns-sidnrns))
/// — never by loading `link.ns.ns_id`, which `pid_nr_in()` can afford
/// because it holds the lock and this function cannot.
///
/// Both writer cases keep every slot the reader may touch byte-valid:
/// Phase 3(a2) exchanges chains between two tasks of the SAME process and
/// therefore the same chain depth (entry-for-entry overwrite, length
/// unchanged), and the reap drain only shortens a chain whose entries were
/// moved — not freed — before the closing bump. A torn read is consequently
/// a wrong answer, never an invalid pointer, and the retry discards it.
///
/// **Algorithm** (BOUNDED seqlock retry loop; the payload is read WITHOUT
/// taking the `pid_links` `SpinLock` — this is the one sanctioned unlocked
/// reader of that field, which is why the writer-side bumps are normative):
/// ```
/// 1. s1 = task.pid_links_seq.load(Acquire).
///    If s1 is ODD: a write is in progress — count one failed attempt,
///    spin_loop(), and restart at 1. After 8 failed attempts, give up and
///    return None (translation unavailable now).
/// 2. Walk the chain entries, comparing each link's Arc POINTER VALUE with
///    `ns` (no dereference, no clone) and copying out `link.nr as u32` on
///    the first match; `None` if no entry matches. Bound the walk by the
///    chain length observed in this same window.
/// 3. If task.pid_links_seq.load(Acquire) != s1, the snapshot was torn or
///    superseded — discard the result, count one failed attempt, and
///    restart at 1. After 8 failed attempts, give up and return None
///    (translation unavailable now).
/// 4. Otherwise the result is stable: return it.
/// ```
///
/// O(depth) per attempt. Retries are BOUNDED — not merely "expected to
/// finish" — because the reader may BE the event that interrupted the
/// writer: a same-CPU PMU NMI taken inside the `collapse_thread_group`
/// Phase 3(a2) bracket, while sampling the exec-ing task itself, can never
/// observe the seqcount go even, since the interrupted writer makes no
/// progress beneath the handler. Waiting there is a livelock (a hard
/// lockup), not a delay. Cross-CPU writers, by contrast, complete their
/// O(depth) odd window well inside 8 attempts, so the bound never fires in
/// the ordinary case. The give-up `None` must be treated by callers as
/// not-visible-NOW, which `perf_record_ids()`
/// ([Section 20.8](20-observability.md#performance-monitoring-unit--pid-translation-in-perf-records))
/// already maps to the contract's 0 (alive) / `(u32)-1` (dead) encoding — a
/// sample taken mid-exec-collapse of the sampled task is genuinely
/// ambiguous.
pub fn pid_nr_in_lockless(task: &Task, ns: &PidNamespace) -> Option<u32>;

/// Lock-free (immutable data). Process-level sibling of `pid_nr_in()`.
///
/// Returns the PROCESS's pid_t as visible in `ns` — the thread-group id, the
/// value perf's `pid` field and `/proc` report — or `None` when the process
/// has no number at that level.
///
/// No seqcount and no lock: `Process.pid_chain`
/// ([Section 8.1](08-process.md#process-and-task-management--task-model)) is written once, before
/// the `Arc<Process>` is published, and never mutated. A process's
/// namespace-local ids are invariant from fork to reap: `execve()` does not
/// change them, and `collapse_thread_group()` mutates only the surviving
/// TASK's chain. Deriving this from `ThreadGroup::leader()` instead would
/// require `PROCESS_LOCK(25)`/`SIGLOCK(40)`, which interrupt and NMI context
/// cannot take — that is why the chain is captured on the `Process`.
///
/// O(depth) walk of the immutable chain (≤ 33 entries; typically 1-3), same
/// shape as `pgid_nr_ns()`.
pub fn process_nr_in(process: &Process, ns: &PidNamespace) -> Option<u32> {
    process.pid_chain.iter()
        .find(|link| core::ptr::eq(Arc::as_ptr(&link.ns), ns))
        .map(|link| link.nr as u32)
}

/// UTS namespace state.
///
/// Hostname and domainname are read on every `uname()` syscall (glibc calls
/// this once per process, but short-lived processes — container health checks,
/// shell scripts — call it frequently). Writes (`sethostname`, `setdomainname`)
/// are rare (typically once at container creation). RCU gives lock-free reads.
pub struct UtsNamespace {
    /// Unique namespace ID (same value as the nsfs inode number).
    /// Allocated from a global `AtomicU64` counter at creation time.
    pub ns_id: u64,
    /// Owning user namespace. Required by the `Namespace` trait for
    /// `setns()` capability checks. Weak reference avoids cycles.
    pub user_ns: Weak<UserNamespace>,
    /// Current hostname and domainname. `RcuPtr<T>` is the canonical nullable
    /// single-owner RCU pointer defined in
    /// [Section 3.4](03-concurrency.md#cumulative-performance-budget--rcu-protected-container-types)
    /// (read under an `RcuReadGuard` → `Option<&T>` via one `Acquire` load;
    /// writers swap under an external `WriterProof` guard; old value reclaimed
    /// after a grace period). The chapter-intro `RcuPtr` is a forward
    /// declaration whose own doc already defers to that canonical home — this
    /// field is buildable against the full API below.
    ///
    /// **uname() read** (lock-free): `let g = rcu_read_lock(); let s =
    /// self.strings.read(&g);` returns `Option<&Arc<UtsStrings>>`. `Arc` as the
    /// pointee is deliberate: the reader does `Arc::clone(s?)` BEFORE dropping
    /// `g`, so the clone outlives the RCU grace period and the subsequent
    /// `copy_to_user` (which may sleep on a page fault) reads a pinned
    /// `UtsStrings` even if `sethostname()` swaps concurrently.
    ///
    /// **sethostname()/setdomainname() write** (clone-and-swap): under
    /// `update_lock`, build a fresh `Arc<UtsStrings>` with the mutated field
    /// and publish it — `self.strings.update(Some(new), &update_guard)` — which
    /// swaps the pointer (Release) and defers the old `Arc`'s free past the
    /// grace period. `update()` takes the `MutexGuard<'_, ()>` from
    /// `update_lock` as its `WriterProof` (any exclusive guard satisfies the
    /// sealed trait); two unsynchronized writers would otherwise double-free.
    pub strings: RcuPtr<Arc<UtsStrings>>,
    /// Serializes `sethostname()` / `setdomainname()` updates. Its guard is
    /// the writer-proof argument to `strings.update()` (above).
    pub update_lock: Mutex<()>,
}

/// UTS string pair (hostname + domainname). Immutable once published;
/// updates create a new `UtsStrings` and swap the RCU pointer.
pub struct UtsStrings {
    /// Hostname (max 64 bytes, NUL-terminated).
    pub hostname: [u8; 65],
    /// NIS domain name (max 64 bytes, NUL-terminated).
    pub domainname: [u8; 65],
}

**IPC namespace**: Per-IPC-namespace state for SysV and POSIX IPC objects.
Canonical definition with full field documentation:
[Section 17.3](#posix-ipc--ipc-namespace-dispatch-sysv-ipc). Created by `clone(CLONE_NEWIPC)` or
`unshare(CLONE_NEWIPC)`. Integer-keyed SysV maps use `Idr<T>` (O(1),
RCU-protected reads); POSIX message queues use `RwLock<BTreeMap>` (string keys,
cold path).

/// SysV shared memory segment (shmget/shmat/shmctl).
///
/// **Interior mutability**: a `ShmSegment` is shared as `Arc<ShmSegment>` and
/// reached through the IPC namespace `shm` RwLock READ path
/// ([Section 17.3](#posix-ipc--ipc-namespace-dispatch-sysv-ipc)), so every field that
/// changes after `shmget()` MUST be interior-mutable — a plain field write
/// through the shared `Arc` is uncompilable (`Arc` has no `DerefMut`). The
/// per-attach/detach timestamps, the `IPC_SET`-mutable owner/mode, the attach
/// count, and the RMID mark are therefore all atomics.
pub struct ShmSegment {
    /// Unique key (from shmget; IPC_PRIVATE = 0 means anonymous). Immutable.
    pub key: i32,
    /// Segment identifier (returned by shmget). Signed i32 matching
    /// Linux shmid_ds.shm_perm.id and the shmget() return type. `AtomicI32`
    /// ONLY so `IpcIdTable::insert` can stamp it ONCE, post-allocation, under
    /// the id-table write lock (the id `(seq << 15) | idx` is unknown until the
    /// slot is allocated). Set once,
    /// then immutable; `IPC_STAT` reads it. See [Section 17.3](#posix-ipc) `IpcObject`.
    pub shmid: AtomicI32,
    /// Size in bytes (rounded up to page boundary at creation). Immutable.
    pub size: usize,
    /// Physical pages backing this segment (reference-counted). See
    /// `PhysPages` for the demand-allocated, memcg-charged, free-claim model
    /// (swap-out is a documented future milestone — see `PhysPages`).
    pub pages: Arc<PhysPages>,
    /// Owner UID/GID. Mutable via `shmctl(IPC_SET)` — hence atomic.
    pub uid: AtomicU32,
    pub gid: AtomicU32,
    /// Permission mode bits (lower 9 bits, like file mode). Mutable via
    /// `shmctl(IPC_SET)`. Stored in an `AtomicU32` (holds the u16 value) for
    /// interior mutability through the shared `Arc`.
    pub mode: AtomicU32,
    /// **RMID mark** (Linux `SHM_DEST` in `shm_perm.mode`). `false` until
    /// removed; set `true` — never cleared — by exactly TWO setters, both
    /// idempotent stores of the same value: `shmctl(IPC_RMID)` and
    /// `impl Drop for IpcNamespace` (namespace teardown,
    /// [Section 17.3](#posix-ipc--ipc-namespace-dispatch-sysv-ipc)) — the same
    /// syscall-plus-namespace-Drop setter pattern as the sem/msg `removed`
    /// latches (`*_rmid_mark_and_wake`). BOTH setters MUST use the `SeqCst`
    /// store: the free protocol's missed-free proof (below) argues over the
    /// single SeqCst total order of the `shm_dest` store/load and `nattach`
    /// fetch_sub/load — a weaker teardown-side store would reopen the
    /// missed-free interleaving against a concurrent last `shm_detach()`
    /// from a task that `setns`'d away. A set mark means "removed: no new
    /// shmat, free pages at last detach". UmkaOS uses an explicit named
    /// flag rather than folding `SHM_DEST` into the mode word (same
    /// "explicit latch, no flags-word packing" convention as
    /// `PidNamespace.dying`). The last-detach free protocol checks THIS.
    pub shm_dest: AtomicBool,
    /// **Free-claim latch**. Guards the ONE-TIME `shm_free_pages()` against the
    /// RMID-immediate-free vs. concurrent-last-detach double-free: the last
    /// `shm_detach()` (nattach 1→0 after the mark) and the IPC_RMID immediate
    /// free (mark set while nattach already 0) can BOTH observe
    /// "removed && no attachments" and both call `shm_free_pages()`.
    /// `shm_free_pages()` CASes `false→true` on this latch and only the winner
    /// frames-frees + memcg-uncharges; losers no-op. Set once, never cleared
    /// (the segment is gone after the free). The two triggering conditions are
    /// evaluated with `SeqCst` (see the free protocol) so exactly ONE call
    /// reaches `shm_free_pages()` first — no double free, and no missed free.
    pub freed: AtomicBool,
    /// Attachment count (number of active shmat() mappings). Incremented at
    /// `shmat()`, decremented at `shmdt()` AND at process-exit VMA teardown
    /// of a shm mapping (see Lifecycle §Destruction).
    pub nattach: AtomicU32,
    /// Creation time and last attach/detach timestamps (monotonic nanoseconds).
    /// `atime`/`dtime` are updated per attach/detach; `ctime` on `IPC_SET`.
    pub ctime: AtomicU64,
    pub atime: AtomicU64,
    pub dtime: AtomicU64,
}

/// Physical page backing for SysV shared memory segments.
///
/// Shared between all processes that `shmat()` the segment. Each `shmat()`
/// creates a VMA in the calling process's address space that maps these
/// shared physical pages. The same `Pfn` appears in multiple page tables
/// simultaneously.
///
/// # Lifecycle
///
/// 1. **Creation** (`shmget`): `PhysPages` is allocated with `pages` pre-sized
///    to `ceil(size / PAGE_SIZE)` entries, all initialized to `None`.
///    If `SHM_HUGETLB` is set, huge pages are allocated eagerly at creation.
/// 2. **First attach** (`shmat`): for non-hugetlb segments, physical pages are
///    allocated lazily on first page fault (demand paging). The faulting task
///    acquires `pages.lock`, checks if `pages[idx]` is `None`, allocates a
///    zeroed page frame, and stores the `Pfn`. The lock is held only for the
///    duration of the page allocation (cold path, not per-access).
/// 3. **Detach** (`shmdt`, AND process-exit VMA teardown): removes the shm VMA
///    from the calling process and calls `shm_detach(seg)`:
///    `seg.dtime.store(now(), Relaxed); if seg.nattach.fetch_sub(1, SeqCst) == 1
///    && seg.shm_dest.load(SeqCst) { shm_free_pages(seg); }`. Process exit tears
///    down VMAs without an explicit `shmdt()`, so the VMA-teardown path MUST
///    invoke this same `shm_detach()` for any shm-backed VMA — otherwise
///    `nattach` never reaches 0 and an IPC_RMID'd segment leaks. (Deferred
///    handoff by symbol: [Section 4.8](04-memory.md#virtual-memory-manager) VMA teardown / `detach_mm`
///    calls `shm_detach()` for `VmaKind::SysvShm` mappings.)
/// 4. **Destruction** (`shmctl IPC_RMID`): sets `seg.shm_dest.store(true, SeqCst)`
///    (the RMID mark) and removes `shmid` from the IPC namespace's `shm` table
///    (under its RwLock write), preventing new `shmat()`. Then, if
///    `seg.nattach.load(SeqCst) == 0` (a segment created and marked with no live
///    attachments), IPC_RMID calls `shm_free_pages(seg)` itself. Otherwise the
///    free is driven by the last detach (step 3) once the mark is set.
///
///    **Single free-claim (double-free-safe AND missed-free-safe).** Both the
///    step-3 last-detach path and this step-4 immediate-free path can observe
///    "removed && nattach == 0" and both call `shm_free_pages()`. That helper
///    is idempotent — it CASes `seg.freed` `false→true` and only the winner
///    performs the actual `PhysPages` frame free + `memcg_uncharge`:
///    ```
///    fn shm_free_pages(seg: &ShmSegment) {
///        // Claim the one-time free; a loser (the other racing path) returns.
///        if seg.freed.compare_exchange(false, true, AcqRel, Acquire).is_err() {
///            return;
///        }
///        // Winner: free every present Pfn in seg.pages, then uncharge the
///        // segment's memcg for the resident bytes.
///        ...
///    }
///    ```
///    The CAS latch rules out the DOUBLE free. The MISSED free (neither path
///    frees) is ruled out by evaluating the two triggering writes/reads —
///    the `shm_dest` store/load and the `nattach` fetch_sub/load — with
///    `SeqCst`: in the single SeqCst total order the four ops form a cycle if
///    both the detacher's `shm_dest.load` misses the mark AND RMID's
///    `nattach.load` misses the decrement, which is impossible, so at least one
///    path observes the condition and claims the free.
///
/// # Reclaim and memory accounting
///
/// Segment pages are **memcg-charged** to the faulting task's memory cgroup at
/// fault-in (`memcg_charge()` in step 2's page-fault handler, uncharged at
/// `shm_free_pages()`). This is what bounds the otherwise-unbounded growth risk
/// of the `shmmax`/`shmall` defaults (`u64::MAX - (1<<24)`,
/// [Section 17.3](#posix-ipc--ipc-namespace-dispatch-sysv-ipc)), which an unprivileged task
/// can reset by `unshare(CLONE_NEWIPC)`: the memcg limit fails the charge (fault
/// → cgroup OOM) long before such a task can pin all RAM. Segments persisting
/// after creator exit (SysV semantics, until IPC_RMID) stay charged to their
/// charging cgroup until freed.
///
/// **Swap-out is a documented future milestone, NOT claimed here.** In this
/// design the `PhysPages` table owns bare `Pfn`s (`Option<Pfn>` per slot, with
/// exactly two states: not-yet-faulted vs. resident) — there is no swap-entry
/// slot state and no shmem `address_space`/rmap linkage through which reclaim
/// could find and rewrite the multi-process PTEs. The pages are therefore
/// pinned resident (like Linux `SHM_LOCK`ed shm), bounded by memcg rather than
/// reclaimed. Full shmem/tmpfs backing — matching Linux's swappable SysV shm,
/// where `shmget` pages live in a tmpfs inode's `address_space` — requires
/// `PhysPages` to hold a shmem-mapping reference instead of bare `Pfn`s and to
/// participate in rmap; that is deferred to the shmem-backing milestone
/// ([Section 4.4](04-memory.md#page-cache)). Swappability is not ABI-observable to userspace, so this
/// de-scope does not affect Linux compatibility.
///
/// # Design note
///
/// `Vec<Option<Pfn>>` inside `SpinLock` is acceptable because:
/// - The `Vec` is pre-allocated to full size at segment creation (no realloc
///   under the lock). `Vec::with_capacity(npages)` followed by `resize(npages, None)`.
/// - The lock protects only the page-present state during fault handling.
/// - Hot-path page table lookups do NOT acquire this lock — they read the PTE
///   directly. The lock is only needed when populating a previously-absent page.
pub struct PhysPages {
    /// Physical page frames backing this shared memory segment.
    /// Indexed by page offset within the segment. `None` = not yet faulted in.
    /// Pre-allocated to `ceil(size / PAGE_SIZE)` entries at creation.
    pub pages: SpinLock<Vec<Option<Pfn>>>,
    /// Total size in bytes (rounded up to page boundary at creation).
    pub size: usize,
}

/// SysV semaphore set (semget/semop/semctl).
///
/// # semop() blocking protocol (the sleep side `detach_sysv_sem()` wakes)
///
/// `semop()` applies an ARRAY of up to `SEMOPM` operations ATOMICALLY —
/// all-or-nothing across the array (POSIX). The `sems: Box<[AtomicU16]>` is
/// storage only (lock-free `GETVAL`); the atomicity comes from holding
/// `SemSet.lock` across the whole evaluate-then-apply of the op array:
/// 1. Take `lock`. Evaluate every op against the current `sems`: a
///    `sem_op < 0` that would drive a value below 0, or a `sem_op == 0`
///    (wait-for-zero) on a nonzero value, means the batch cannot proceed NOW.
/// 2. If it can proceed: apply all ops (each `sems[i]` under `lock`), record
///    `SEM_UNDO` inverses, bump `otime`, then `waiters.wake_up_all()` (a value
///    increase may satisfy other blocked batches). Drop `lock`.
/// 3. If it cannot and `IPC_NOWAIT` is set: drop `lock`, return `EAGAIN`.
/// 4. Otherwise enqueue the task on `waiters`, DROP `lock`, and sleep. On
///    wake, RE-EVALUATE the full array under `lock` (the wake predicate is
///    "the whole batch is now satisfiable"). Wakes come from: another
///    `semop()` that raised a value (step 2), `semctl(SETVAL/SETALL)`,
///    `detach_sysv_sem()` applying undo adjustments
///    ([Section 8.2](08-process.md#process-lifecycle-teardown--step-4c-sysv-semaphore-undo-operations)), or a `semtimedop()` timer.
/// 5. Termination conditions during sleep: signal → `EINTR`;
///    `semctl(IPC_RMID)` destroying the set → the destroyer sets the `removed`
///    latch (below) `true` under `lock`, then `waiters.wake_up_all()`; every
///    woken task re-checks `removed` under `lock` FIRST and, seeing it set,
///    returns `EIDRM` (before any op re-evaluation); `semtimedop()` timeout →
///    `EAGAIN`.
///
/// **Interior mutability**: reached through `Arc<SemSet>` on the IPC-namespace
/// `sem` RwLock read path, so `IPC_SET`-mutable owner/mode and the per-op
/// timestamps are atomics (a plain field write through the shared `Arc` is
/// uncompilable), and multi-op semval atomicity is provided by `lock`.
pub struct SemSet {
    pub key: i32,
    /// Set identifier (returned by semget). Linux ABI: `semget()` returns int
    /// (i32). `AtomicI32` only so `IpcIdTable::insert` can stamp it ONCE,
    /// post-allocation, under the id-table write lock. Set once, then
    /// immutable. See [Section 17.3](#posix-ipc) `IpcObject`.
    pub semid: AtomicI32,
    /// Number of semaphores in this set (1–SEMMSL; Linux default SEMMSL=32000).
    pub nsems: u16,
    /// Semaphore values (one per semaphore in the set). Atomic storage for
    /// lock-free `GETVAL`; multi-op atomicity is provided by `lock`.
    pub sems: Box<[AtomicU16]>,
    /// **Set lock**: serializes the atomic multi-op semop evaluate+apply and
    /// all `semctl` mutations. THIS is the `sem_set.lock` `detach_sysv_sem()` takes
    /// ([Section 8.2](08-process.md#process-lifecycle-teardown--step-4c-sysv-semaphore-undo-operations)) before applying undos.
    pub lock: SpinLock<()>,
    /// **Wait queue** for tasks blocked in `semop()` (step 4 above). THIS is
    /// the `sem_set.waiters` `detach_sysv_sem()` wakes after applying undo
    /// adjustments. Woken tasks re-evaluate their full op array under `lock`.
    pub waiters: WaitQueue,
    /// **RMID/removed latch** (semop step 5). `false` for the set's whole life;
    /// set `true` — once, never cleared — under `lock` by `semctl(IPC_RMID)`
    /// and by `IpcNamespace::Drop` (via `sem_rmid_mark_and_wake`,
    /// [Section 17.3](#posix-ipc--ipc-namespace-dispatch-sysv-ipc)) BEFORE
    /// `waiters.wake_up_all()`. A woken `semop()` sleeper loads this under
    /// `lock` before re-evaluating its op array; a set latch means the set was
    /// destroyed → return `EIDRM`. This is the storage `sem_rmid_mark_and_wake`
    /// defers to; same explicit-latch convention as `PidNamespace.dying`.
    pub removed: AtomicBool,
    /// Undo table: per-task pending undos (restored on task exit).
    /// SpinLock because semop undo list is accessed in task-exit path.
    pub undo_list: SpinLock<Vec<SemUndo>>,
    /// Owner UID/GID. Mutable via `semctl(IPC_SET)` — hence atomic.
    pub uid: AtomicU32,
    pub gid: AtomicU32,
    /// Permission mode (lower 9 bits). Mutable via `semctl(IPC_SET)`; the
    /// `AtomicU32` holds the u16 value for interior mutability.
    pub mode: AtomicU32,
    /// Last `semctl(IPC_SET)` time and last successful `semop()` time
    /// (monotonic nanoseconds). Updated through the shared `Arc` — atomic.
    pub ctime: AtomicU64,
    pub otime: AtomicU64,
}

/// SysV semaphore undo entry. Tracks adjustments that must be reversed
/// on process exit to prevent semaphore value leakage.
///
/// **Per-process sharing**: SysV semaphore undos are per-process (thread
/// group), not per-thread. All threads in a thread group share a single
/// `undo_list` (stored in `Process.sysvsem_undo`, not `Task`). This matches
/// Linux's `struct sem_undo_list` which is shared across all threads via
/// `current->sysvsem.undo_list`. `detach_sysv_sem()` runs once when the thread
/// group leader exits (last thread in the group), applying all accumulated
/// undo adjustments atomically against each referenced semaphore set.
///
/// When a `semop()` call includes `SEM_UNDO`, the kernel records the
/// inverse of each semaphore adjustment in a `SemUndo` entry associated
/// with the calling process. On process exit (`detach_sysv_sem()`), the kernel
/// iterates the process's `undo_list` and applies all recorded adjustments
/// atomically per semaphore set, restoring semaphore values to their
/// pre-operation state.
///
/// # Storage design
///
/// Uses a sparse representation: only semaphores that were actually modified
/// with `SEM_UNDO` are tracked. This avoids allocating a dense array of
/// 32000 entries (Linux SEMMSL) for the common case where a process touches
/// only a few semaphores in a set.
pub struct SemUndo {
    /// PID of the process that owns this undo entry. Required for
    /// `detach_sysv_sem()` to identify entries belonging to the dying process
    /// when scanning from the semaphore-set side (`SemSet.undo_list`).
    /// The per-process `sysvsem_undo` list provides the reverse index
    /// (O(1) traversal from the process side).
    pub process_id: ProcessId,
    /// Semaphore set this undo applies to. **`i32`, matching the i32 id held in
    /// `SemSet.semid`** (an `AtomicI32` carrying the `i32` semget() return
    /// value) — a `u32` here forced an unspecified cast at every cross-index
    /// site (`detach_sysv_sem()` indexes the set by this id). Negative semids never
    /// occur, but the types must agree.
    pub sem_id: i32,
    /// Sparse list of (semaphore_index, adjustment) pairs.
    /// Only semaphores modified with SEM_UNDO are tracked. On process exit,
    /// `semval[idx] += adj` is applied for each `(idx, adj)` entry.
    ///
    /// **Capacity**: Capped at `semset.nsems` (the semaphore count of the
    /// owning set), matching Linux behavior where the undo array is sized to
    /// SEMMSL. Since `sem_nsems <= SEMMSL` (Linux default 32000), this is the
    /// natural upper bound. If the limit is reached, subsequent SEM_UNDO
    /// operations on new semaphore indices in this set return ENOSPC.
    ///
    /// **Allocation**: `Vec` instead of `ArrayVec<256>` — the `semop()` path is
    /// warm (bounded frequency), so heap allocation is acceptable per collection
    /// policy (Ch 3.1.13). Each push calls `memcg_charge()` against the
    /// CHARGING process's memory cgroup (recorded as `charged_memcg` below so
    /// the uncharge cannot drift to a different cgroup), preventing
    /// unprivileged processes from consuming unbounded kernel memory via
    /// SEM_UNDO. The cap is Evolvable policy (not a Nucleus type parameter): a
    /// live evolution can adjust the limit without data structure migration.
    ///
    /// **Uncharge sites** (the charge-anchor the per-push charge previously
    /// lacked): `detach_sysv_sem()` drains this list and, per drained `SemUndo`,
    /// calls `memcg_uncharge(undo.charged_memcg, bytes)`
    /// ([Section 8.2](08-process.md#process-lifecycle-teardown--step-4c-sysv-semaphore-undo-operations)); `semctl(IPC_RMID)`
    /// destroying a set with live undos uncharges each against its own
    /// `charged_memcg` before dropping it. Uncharging the CHARGING cgroup
    /// (not the freeing task's) is required because a `SemSet` outlives the
    /// process that pushed undos, and `detach_sysv_sem()` may run in a different
    /// cgroup context (charge-anchor-drift).
    pub adjustments: Vec<(u16, i16)>,
    /// The memory cgroup charged for `adjustments` at push time — the uncharge
    /// anchor. A `Weak<MemCgroup>` (`upgrade() == None` ⇒ cgroup already gone,
    /// uncharge is a no-op; never `.unwrap()`), following the `Weak`
    /// back-reference discipline.
    pub charged_memcg: Weak<MemCgroup>,
}

/// SysV message queue (msgget/msgsnd/msgrcv).
///
/// **Allocation strategy**: Message bodies are allocated from a per-IPC-namespace
/// slab cache (`msg_slab`) BEFORE acquiring the message queue SpinLock. The slab
/// cache uses fixed-size buckets (64, 256, 1024, 4096, `MSGMAX` bytes) to avoid
/// heap fragmentation. The caller allocates a `SysVMessage` from the slab, copies
/// the userspace data, then acquires `lock` and pushes the pre-allocated message
/// into the ring. This ensures zero heap allocation under the SpinLock.
pub struct MsgQueue {
    /// Immutable identity (fixed at msgget()).
    pub key: i32,
    /// Queue identifier (returned by msgget). Linux ABI i32. `AtomicI32` only
    /// so `IpcIdTable::insert` can stamp it ONCE, post-allocation, under the
    /// id-table write lock. Set once, then immutable. See [Section 17.3](#posix-ipc)
    /// `IpcObject`.
    pub msqid: AtomicI32,
    /// Mutable state protected by the SpinLock. All fields that change
    /// after msgget() live inside `MsgQueueInner` so the lock actually
    /// protects the data it guards (no `SpinLock<()>` anti-pattern). This
    /// INCLUDES `uid`/`gid`/`mode`: `msgctl(IPC_SET)` mutates exactly those
    /// (confirmed by `MsgQueueInner.ctime`'s "Last msgctl IPC_SET" role), and
    /// the msgsnd/msgrcv permission check reads them — leaving them outside
    /// `inner` made them unwritable through the shared `Arc<MsgQueue>` and
    /// racy against the permission check, violating this struct's own
    /// invariant.
    pub inner: SpinLock<MsgQueueInner>,
}

/// Mutable MsgQueue state, protected by `MsgQueue.inner` SpinLock.
///
/// # msgsnd/msgrcv algorithm (why the message store is a LIST, not a ring)
///
/// `msgrcv(msgtyp)` is NOT FIFO-only: `msgtyp == 0` takes the head; `msgtyp > 0`
/// takes the FIRST message whose `mtype == msgtyp` (or, with `MSG_EXCEPT`, the
/// first `mtype != msgtyp`); `msgtyp < 0` takes the message with the LOWEST
/// `mtype <= |msgtyp|`. All three remove from the MIDDLE of the queue while
/// later messages remain — which a FIFO `BoundedRing` cannot express. The store
/// is therefore an intrusive doubly-linked list threaded through the
/// slab-allocated `SysVMessage` nodes, so
/// mid-queue removal is O(1) and no separate container is allocated (the node
/// memory is the per-namespace `msg_slab` allocation already made before the
/// lock). Capacity is bounded by BYTES (`max_bytes`), not a message count, so
/// no count-based pre-allocation is possible or needed.
/// - **msgsnd**: block on `send_wait` while `current_bytes + len > max_bytes`
///   (unless `IPC_NOWAIT` → `EAGAIN`). Else link the pre-allocated node at the
///   tail, `current_bytes += len`, `stime = now()`, `recv_wait.wake_up_all()`.
/// - **msgrcv**: scan `messages` for the first node matching `msgtyp` per the
///   rules above; if found, unlink it, `current_bytes -= len`, `rtime = now()`,
///   `send_wait.wake_up_all()`, return it (`MSG_NOERROR` truncates an
///   over-long message; without it an over-long message is `E2BIG` and stays
///   queued). If none matches: `IPC_NOWAIT` → `ENOMSG`; else block on
///   `recv_wait`. `msgctl(IPC_RMID)` sets the `removed` flag (below) `true`
///   under `inner` and wakes both queues; every woken sender/receiver re-checks
///   `removed` under `inner` FIRST and, seeing it set, returns `EIDRM`.
pub struct MsgQueueInner {
    /// Messages stored as an intrusive doubly-linked list (see algorithm
    /// above). Nodes are `SysVMessage` (slab-allocated before the lock);
    /// insertion at tail, removal from anywhere. Bounded by `max_bytes`.
    pub messages: MsgList,
    /// Owner UID/GID and permission mode — mutated by `msgctl(IPC_SET)`,
    /// read by the msgsnd/msgrcv permission check. Inside `inner` so both are
    /// serialized (the struct invariant).
    pub uid: u32,
    pub gid: u32,
    pub mode: u16,
    /// Current total size of all messages in bytes.
    pub current_bytes: usize,
    /// Maximum bytes in queue (default: MSGMNB = 16384).
    pub max_bytes: usize,
    /// Tasks waiting to send (queue full).
    pub send_wait: WaitQueue,
    /// Tasks waiting to receive (queue empty or no matching type).
    pub recv_wait: WaitQueue,
    /// Last msgsnd time (monotonic nanoseconds).
    pub stime: u64,
    /// Last msgrcv time (monotonic nanoseconds).
    pub rtime: u64,
    /// Last msgctl IPC_SET or msgget creation time (monotonic nanoseconds).
    pub ctime: u64,
    /// **RMID/removed latch** (msgsnd/msgrcv algorithm above). `false` for the
    /// queue's whole life; set `true` — once, never cleared — under `inner` by
    /// `msgctl(IPC_RMID)` and by `IpcNamespace::Drop` (via
    /// `msg_rmid_mark_and_wake`, [Section 17.3](#posix-ipc--ipc-namespace-dispatch-sysv-ipc))
    /// BEFORE waking `send_wait`/`recv_wait`. A woken blocked sender/receiver
    /// re-checks this under `inner` before retrying; a set latch means the queue
    /// was destroyed → return `EIDRM`. A plain `bool` (not atomic) because every
    /// access is already under the `inner` SpinLock. This is the storage
    /// `msg_rmid_mark_and_wake` defers to.
    pub removed: bool,
}

/// Intrusive doubly-linked list of queued `SysVMessage` nodes — the
/// `MsgQueueInner.messages` store. Enables O(1) mid-queue removal for typed
/// `msgrcv` (a FIFO ring cannot). The list owns each node's memory (the
/// pre-lock `msg_slab` allocation); no separate container allocation grows
/// under the lock. Head/tail are `None` when empty. All link manipulation is
/// confined to this type's methods (`push_back`, `remove`, `scan_first`), the
/// standard safe boundary for an intrusive kernel list.
pub struct MsgList {
    head: Option<NonNull<SysVMessage>>,
    tail: Option<NonNull<SysVMessage>>,
    len: usize,
}

/// A single SysV message, and its own intrusive list node.
/// Allocated from the per-IPC-namespace `msg_slab` cache before acquiring
/// the MsgQueue SpinLock. Freed back to the slab after msgrcv() returns.
pub struct SysVMessage {
    /// Message type (from msgsnd mtype; must be > 0). The `msgrcv(msgtyp)`
    /// match key.
    pub mtype: i64,
    /// Intrusive list links (managed only by `MsgList`). `None` while the
    /// node is detached (owned by the caller between slab alloc and enqueue,
    /// or between dequeue and free).
    pub next: Option<NonNull<SysVMessage>>,
    pub prev: Option<NonNull<SysVMessage>>,
    /// Message data — slab-allocated fixed-size buffer. `data_len` holds
    /// the actual message length; the slab bucket may be larger.
    pub data: SlabBox<[u8]>,
    /// Actual data length in bytes (≤ slab bucket size).
    pub data_len: usize,
}

/// POSIX message queue (mq_open/mq_send/mq_receive; mqueue filesystem).
/// Linux-compatible: /dev/mqueue filesystem, mq_notify(3) supported.
///
/// This struct is the CANONICAL backing object for a POSIX message queue.
/// The per-IPC-namespace
/// `mqueuefs` filesystem is the VFS surface over it: each mqueuefs inode's
/// private data is the `Arc<PosixMqueue>`, and the dedicated
/// mq_send/mq_receive/mq_timedsend/mq_timedreceive syscalls dispatch to it
/// via `file.mqueue_inner()`. Syscall dispatch, the fd-status `read(2)`
/// format, and the mq_notify ownership/exit-cleanup contract:
/// [Section 17.3](#posix-ipc--posix-message-queues-mqueuefs).
pub struct PosixMqueue {
    /// Queue name (from mq_open; unique within the mqueue namespace).
    pub name: ArrayString<256>,
    /// Immutable attributes (fixed at mq_open): max messages, max message
    /// size. The CURRENT message count is not stored — it is
    /// `inner.lock().queue.len()` (a `BinaryHeap` maintains its length;
    /// a separate counter could only desynchronize). `mq_getattr()` reads
    /// it under the `inner` lock.
    pub attr: MqueueAttr,
    /// All mutable queue state — protected by a single SpinLock (same
    /// pattern as `MsgQueue` above — no `SpinLock<()>` anti-pattern).
    /// Every field modified during mq_send()/mq_receive()/mq_notify()/
    /// close() is inside this lock; `PosixMqueue` itself is shared as
    /// `Arc<PosixMqueue>` and mutated only through `inner` (interior
    /// mutability — never `&mut self` through the Arc).
    ///
    /// Queue memory: the BinaryHeap is pre-allocated at mq_open() time with
    /// `BinaryHeap::with_capacity(attr.maxmsg)`, so no heap allocation
    /// occurs under the SpinLock during mq_send(). Maximum memory per
    /// queue: maxmsg * (size_of::<PosixMessage>() + msgsize). Bounded by
    /// RLIMIT_MSGQUEUE per-user limit (default: 819200 bytes on Linux);
    /// mq_open() validates the requested queue size against the caller's
    /// remaining RLIMIT_MSGQUEUE allowance.
    ///
    /// **RLIMIT_MSGQUEUE accounting anchor**: the charge
    /// ([Section 8.8](08-process.md#resource-limits-and-accounting--rlimitmsgqueue-posix-mq-bytes-per-uid),
    /// `mq_check_and_add`) is against the CREATOR's UID — which is `PosixMqueue.uid`
    /// (POSIX mqueues have no chown, so `uid` is stable = the accounting anchor).
    /// The charge is the queue's WORST-CASE footprint, applied ONCE at
    /// `mq_open(O_CREAT)` — send/receive NEVER touch `mq_bytes` (matching
    /// Linux `ipc/mqueue.c` `mqueue_get_inode()` creation-time ucounts
    /// charging; Linux `do_mq_timedsend()` has no RLIMIT_MSGQUEUE check). The
    /// single uncharge — the same creation-time amount, via
    /// `mq_uncharge(mq.uid, ..)` — happens in the queue destructor
    /// (`mq_unlink()` + last close, `mq_drain_and_uncharge`) and MUST
    /// target `mq.uid`'s `UserEntry.mq_bytes` regardless of the acting
    /// task's UID. The destructor drain frees any still-queued message
    /// memory, but the uncharge amount is footprint-based, not
    /// occupancy-based — nothing leaks from `mq_bytes` however many
    /// messages remain queued at destruction.
    pub inner: SpinLock<PosixMqueueInner>,
    /// Tasks blocked on mq_receive (queue empty).
    pub recv_waiters: WaitQueue,
    /// Tasks blocked on mq_send (queue full).
    pub send_waiters: WaitQueue,
    pub uid: u32,
    pub gid: u32,
    pub mode: u16,
}

/// Interior mutable state of a POSIX message queue, protected by `PosixMqueue.inner`.
pub struct PosixMqueueInner {
    /// Priority queue: messages ordered by descending priority, then FIFO within
    /// equal priority (see `PosixMessage::cmp`). Pre-allocated at mq_open() time
    /// with `BinaryHeap::with_capacity(attr.maxmsg)`. `queue.len()` IS the
    /// queue's current message count (`mq_curmsgs`).
    pub queue: BinaryHeap<PosixMessage>,
    /// Monotonically increasing sequence counter. Assigned to each message on
    /// mq_send() to provide stable FIFO ordering within equal-priority messages.
    pub next_seq: u64,
    /// Notification registration (mq_notify), single-owner. Inside `inner`
    /// because it is mutated through the shared `Arc<PosixMqueue>` from
    /// three concurrent contexts — mq_notify(2), the mq_send() delivery
    /// path (one-shot auto-deregister), and `mqueue_flush` on close
    /// ([Section 17.3](#posix-ipc--mqnotify-ownership-and-exit-cleanup)) — matching this rule:
    /// Linux guards `notify_owner` with `info->lock`.
    pub notify: Option<MqueueNotify>,
}

/// Immutable POSIX mqueue attributes, fixed at mq_open().
pub struct MqueueAttr {
    /// Maximum number of messages (mq_maxmsg; default 10, max 65536).
    pub maxmsg: u32,
    /// Maximum message size in bytes (mq_msgsize; default 8192,
    /// max 16 MiB = Linux `HARD_MSGSIZEMAX` = 16*1024*1024, `ipc/mqueue.c`,
    /// verified against torvalds/linux master; the `msgsize_max` sysctl is
    /// raisable up to this hard ceiling). `mq_open()` validation rejects a
    /// requested `msgsize` above this bound with EINVAL — the SAME limit the
    /// `PosixMessage.data` doc cites, so the two agree.
    pub msgsize: u32,
}

/// A POSIX message queue message. Ordering is by descending priority, then
/// ascending sequence number (FIFO within equal priority), as required by
/// POSIX.1-2017 mq_receive(3).
///
/// # Ordering (BinaryHeap is a max-heap)
/// ```rust
/// impl Ord for PosixMessage {
///     fn cmp(&self, other: &Self) -> Ordering {
///         self.priority.cmp(&other.priority)
///             .then(other.seq.cmp(&self.seq)) // reverse seq: lower seq = older = wins
///     }
/// }
/// ```
/// A higher priority beats a lower priority. Within the same priority, the
/// message with the smaller `seq` (sent earlier) has a *larger* `Ord` value
/// so the max-heap dequeues it first — preserving FIFO order.
pub struct PosixMessage {
    /// Priority (0–MQ_PRIO_MAX-1; higher = delivered first).
    pub priority: u32,
    /// Per-queue send sequence number. Assigned from `PosixMqueueInner.next_seq`
    /// at mq_send() time. Breaks ties within equal-priority messages: lower
    /// seq means the message was sent earlier and must be dequeued first.
    pub seq: u64,
    /// Message payload. `Box<[u8]>` is a warm-path allocation: one allocation
    /// per `mq_send()` call. Size is bounded by `PosixMqueue.attr.msgsize`
    /// (max 16 MiB per Linux, default 8192 bytes). The BinaryHeap owns the
    /// Box; deallocation happens on `mq_receive()` when the message is consumed.
    pub data: Box<[u8]>,
}

/// Namespace-local process identifier. Matches Linux `pid_t` (always `i32` on
/// all architectures). Meaningful ONLY relative to a PID namespace (allocated
/// per-namespace by `pid_map`, recycled per-namespace) — never a global kernel
/// identity. See [Section 8.1](08-process.md#process-and-task-management--process-identity-model) for
/// the canonical three-level identity model (TaskId / ProcessId / Pid).
pub type Pid = i32;

/// POSIX message queue notification registration (mq_notify(3)).
///
/// At most one process may register for notification on a given queue at any
/// time. Notification fires **once** when a message arrives on a previously
/// empty queue, then auto-deregisters (matching POSIX.1-2017 semantics).
/// The process must call `mq_notify()` again to re-register.
///
/// **Blocked-receiver precedence (POSIX.1-2017 / Linux `pipelined_send`)**: if
/// a thread is ALREADY blocked in `mq_receive()` when a message arrives, the
/// message is handed directly to that receiver and **NO notification fires** —
/// from the notifier's viewpoint the queue never transitioned empty→non-empty,
/// so the one-shot registration is PRESERVED (not consumed). The `mq_send()`
/// delivery step therefore fires the notification only when `recv_waiters` is
/// empty at arrival. Firing it while a receiver was waiting would burn the
/// registrant's one-shot on a message it never had a chance to receive —
/// an observable ABI divergence. (Linux `ipc/mqueue.c`: `__do_notify` is
/// skipped when Linux `pipelined_send` handed the message to a waiter, verified
/// against torvalds/linux master.)
///
/// Registration: `mq_notify(mqd, &sigevent)` with `sigev_notify` = SIGEV_SIGNAL
/// or SIGEV_THREAD. Passing a null sigevent pointer deregisters the current
/// notification. Returns EBUSY if another process is already registered.
pub struct MqueueNotify {
    /// Notification delivery mode.
    pub mode: MqueueNotifyMode,
    /// Kernel identity of the process that registered for notification —
    /// captured as `current.process.pid` at `mq_notify()` time. A global
    /// `ProcessId` (u64, never reused), NOT a namespace-local `Pid`: the
    /// notification fires from an arbitrary `mq_send()` context (possibly a
    /// different PID namespace), where an ns-local number would be ambiguous.
    /// Delivery resolves via `find_process_by_pid()` and silently drops the
    /// notification if the registrant has exited
    /// ([Section 8.1](08-process.md#process-and-task-management--process-identity-model)).
    pub pid: ProcessId,
}

/// Length of the SIGEV_THREAD notification cookie (Linux `NOTIFY_COOKIE_LEN`,
/// `ipc/mqueue.c`).
pub const NOTIFY_COOKIE_LEN: usize = 32;

/// Notification delivery mode for POSIX message queues.
///
/// Two kernel-side mechanisms, matching the Linux `ipc/mqueue.c` ABI (verified
/// against torvalds/linux master): `SIGEV_SIGNAL` delivers a signal; the
/// glibc `SIGEV_THREAD` decomposition delivers a **cookie over a netlink
/// socket** — the kernel NEVER creates a user thread. glibc's `mq_notify(3)`
/// wrapper, when the app requests `SIGEV_THREAD`, spins up its OWN helper
/// thread and an `AF_NETLINK` socket, then registers with the kernel using the
/// netlink socket + a cookie; on delivery the kernel writes the cookie to that
/// socket and glibc's helper thread runs the app's notify function. UmkaOS must
/// serve THAT protocol (an unmodified glibc registers it) — the previous
/// "kernel creates a thread via internal clone" model was mechanism-free
/// (no stack/TLS/trampoline/failure path) and served no real glibc.
pub enum MqueueNotifyMode {
    /// Deliver a signal to the registered process.
    /// `si_code` is set to `SI_MESGQ`, `si_value` carries the `sigev_value`
    /// from the original `mq_notify()` registration.
    Signal {
        /// Signal number to deliver (e.g., SIGRTMIN+0).
        signo: u8,
        /// User-provided value passed back in siginfo_t.si_value.
        sigev_value: usize,
    },
    /// glibc SIGEV_THREAD path: notify by writing `cookie` to `notify_sock`.
    /// The registration `sigevent` carries the netlink socket fd in
    /// `sigev_signo` and the cookie via `sigev_value.sival_ptr` (the glibc
    /// ABI). On delivery the kernel does the equivalent of Linux's
    /// Linux `netlink_sendskb(notify_sock, cookie)`; glibc's helper thread receives
    /// it and spawns the app's handler. No kernel thread, stack, or TLS is
    /// created.
    Netlink {
        /// Kernel-held reference to the registrant's netlink socket (resolved
        /// from `sigev_signo` at registration; the canonical AF_NETLINK endpoint
        /// type, [Section 16.17](16-networking.md#netlink-socket-interface)). Delivery unicasts the cookie
        /// here. The `Arc` keeps the endpoint alive until the notify
        /// registration is cleared (Linux `sock_hold` on `notify_sock`).
        notify_sock: Arc<NetlinkSocket>,
        /// Opaque cookie bytes (from `sigev_value.sival_ptr`) that glibc's
        /// helper thread matches to decide which handler to run.
        cookie: [u8; NOTIFY_COOKIE_LEN],
    },
}

/// Cgroup namespace state.
///
/// **Capture rule**: `CLONE_NEWCGROUP` captures the CALLER's cgroup at creation
/// time as `cgroup_root` (specified at [Section 17.2](#control-groups), the cgroup-namespace
/// section). Path reporting for a reader whose cgroup is NOT under `cgroup_root`
/// (outside-subtree) emits Linux's `/..` relative form, also specified there
/// ([Section 17.2](#control-groups), `cgroup_path_from_reader`) — this struct just holds
/// the root; it does not re-specify those.
///
/// **rmdir of a namespace-pinned root**: `cgroup_rmdir()` MAY succeed on a
/// cgroup that is some namespace's `cgroup_root` (Linux parity: the css stays
/// pinned; namespace processes see a DEAD root). Mechanism here: `cgroup_root`
/// is a strong `Arc<Cgroup>`, so after `rmdir` erases the cgroup from
/// `CGROUP_REGISTRY` and the tree ([Section 17.2](#control-groups)), this Arc keeps the
/// `Cgroup` OBJECT alive — the object is freed only after BOTH the tree
/// reference and this namespace pin drop (RCU grace period). Processes in the
/// namespace continue to see `cgroup_root` as "/" in `/sys/fs/cgroup`, but it
/// is an offline (dead) cgroup: no new children can be created under it and
/// controller files reflect the offline state. This is the namespace-side
/// counterpart to `rmdir`'s emptiness/offline gate.
pub struct CgroupNamespace {
    /// Unique namespace ID (same value as the nsfs inode number).
    pub ns_id: u64,
    /// Owning user namespace. Required by the `Namespace` trait for
    /// `setns()` capability checks.
    pub user_ns: Weak<UserNamespace>,
    /// Root cgroup directory visible to processes in this namespace.
    /// Processes see this as "/" in /sys/fs/cgroup. Strong `Arc` — pins the
    /// `Cgroup` object across `rmdir` (see the rmdir-of-pinned-root note above).
    pub cgroup_root: Arc<Cgroup>,
}

/// Time namespace state (Linux 5.6+). **This is the CANONICAL `TimeNamespace`
/// definition** (the namespace chapter is its home): it carries `ns_id` and
/// `user_ns` in addition to the offsets and the seal. The
/// [Section 7.8](07-scheduling.md#timekeeping-and-clock-management) time-namespace section holds the
/// clock-read formulas and the per-namespace vDSO `VvarPage` (into which these
/// offsets are pre-applied so userspace `clock_gettime()` never enters the
/// kernel).
///
/// **Offset write/seal protocol** (the write path the offsets are useless
/// without): offsets are written via `/proc/[pid]/timens_offsets` AFTER
/// `unshare(CLONE_NEWTIME)` but BEFORE the namespace is first ENTERED — i.e.
/// before the first `fork()`/`clone()` that creates a child in it (Linux
/// Linux `timens_on_fork()` in `kernel/time/namespace.c`) or a `setns(CLONE_NEWTIME)`
/// join (Linux `timens_install()`). `exec()` is NOT an entry/seal event —
/// Linux commits `time_ns_for_children → time_ns` only at fork and setns; the
/// exec path (`fs/exec.c`) contains no time-namespace commit. Once entered,
/// `offsets_sealed` is set and further writes return `EACCES` — Linux
/// Linux `frozen_offsets` semantics (`kernel/time/namespace.c`). The offsets are `AtomicI64` (NOT plain `i64`)
/// because the `/proc` writer mutates them through the shared
/// `Arc<TimeNamespace>` — interior mutability is REQUIRED; the seal gates WHEN
/// a write is allowed, the atomics make the write compilable through `&self`.
///
/// > **Deferred handoff by symbol**: [Section 7.8](07-scheduling.md#timekeeping-and-clock-management)
/// > also defines a `TimeNamespace` (plain `i64` offsets + a `frozen` bool).
/// > That duplicate is uncompilable for the `/proc` write path (plain `i64`
/// > cannot be written through a shared `Arc`) and lacks `ns_id`/`user_ns`; it
/// > should DEFER to this canonical struct. (Its CRIU "frozen ⇒ all clocks
/// > return freeze-time" reinterpretation is a separate timekeeping concern,
/// > not merged here.)
pub struct TimeNamespace {
    /// Unique namespace ID (same value as the nsfs inode number).
    pub ns_id: u64,
    /// Owning user namespace. Required by the `Namespace` trait for
    /// `setns()` capability checks.
    pub user_ns: Weak<UserNamespace>,
    /// Offset added to CLOCK_MONOTONIC for processes in this namespace.
    /// Allows containers to see a "boot time" starting from 0. Written only
    /// while `!offsets_sealed` (see protocol above).
    pub monotonic_offset_ns: AtomicI64,
    /// Offset added to CLOCK_BOOTTIME for processes in this namespace.
    pub boottime_offset_ns: AtomicI64,
    /// Write seal (Linux `frozen_offsets`). `false` until the namespace is
    /// first ENTERED; then set `true` — never cleared — after which
    /// `/proc/[pid]/timens_offsets` writes return `EACCES`. Exactly TWO
    /// setter sites, both idempotent `store(true, Release)` on this
    /// set-once latch, both in
    /// [Section 17.1](#namespace-architecture--joining-namespaces-setns2-and-nsenter):
    /// the `setns(CLONE_NEWTIME)` Time arm (joining an existing namespace —
    /// Linux `timens_install()`), and the fork-side pending-namespace
    /// consumption in `NamespaceSet::clone_for_fork()` (first child created into the
    /// namespace — Linux `timens_on_fork()`). Nothing seals at
    /// `unshare(CLONE_NEWTIME)` itself — that is what keeps the canonical
    /// unshare → write-offsets → fork container-runtime sequence legal.
    /// The `/proc` write path checks it with Acquire before mutating an
    /// offset.
    pub offsets_sealed: AtomicBool,
}

17.1.2.2 PID Number Release — `pid_allocator_free()` / `pid_ns_release()`¶

The release-side counterparts of create_task() step 10a's per-level PID allocation. Both operate on the task's immutable captured PID chain (Task.pid_links, Section 8.1) — NEVER on task.namespace_set, which is the init-namespace tombstone on every release path (exit_task Step 11's post-swap invariant). Each is called exactly once per task, from reap_task() (Section 8.2).

/// reap_task() step 2: return the task's pid_t number at every
/// namespace level to that level's allocator. The numbers become
/// reusable by future fork() calls; the TaskId is never reused.
///
/// An EMPTY chain is a no-op by construction — the collapse_thread_group
/// contract: a PID-swapped old leader's numbers were transferred to the
/// exec-ing task and its chain cleared (collapse_thread_group Phase 3(a2),
/// [Section 8.1](08-process.md#process-and-task-management--program-execution-exec)), so
/// releasing the old leader must not (and structurally cannot) free the
/// tgid numbers the live process still uses.
pub fn pid_allocator_free(task: &Task) {
    // SNAPSHOT the chain under the leaf lock, then DROP the lock before
    // any other lock is taken. `Task.pid_links` is a LEAF SpinLock
    // ("never held across any other lock acquisition" — its field
    // contract); the per-level operations below each take that
    // namespace's PID_MAP_LOCK(47) internally, so iterating the live
    // chain under the leaf lock would nest 47 under it — exactly the
    // contract violation the leaf classification forbids. The clone is
    // cheap (≤ 33 `Arc` clones + i32s) and exact: the chain is immutable
    // between fork step 15b and this call (reap_task runs once, and
    // the only other mutator — collapse_thread_group Phase 3(a2) — completed before
    // the old leader became releasable).
    let links: ArrayVec<PidLink, 33> = task.pid_links.lock().clone();
    let rcu = rcu_read_lock();
    // Leaf→root, mirroring the allocation-failure rollback order. The
    // chain entries are AUTHORITATIVE (captured at allocation) — no
    // local_nr() probe is needed to find the number, and no lower-32
    // collision can misdirect the free.
    for link in links.iter().rev() {
        // ONE PID_MAP_LOCK(47) critical section per level: the pin probe
        // and the retarget/free must be atomic against the pin/unpin
        // compound ops and against allocation — `Idr::with_lock` holds
        // the Idr's internal lock (which IS this namespace's
        // PID_MAP_LOCK(47), master-table row 47) across the closure and
        // exposes a locked view with non-relocking accessors
        // ([Section 3.6](03-concurrency.md#lock-free-data-structures--idrt-integer-id-allocator)).
        link.ns.pid_map.with_lock(|map| {
            debug_assert!(map.lookup(link.nr as u32).copied()
                == Some(task.tid), "chain entry must still map to this task");
            // Pin check: if a live ProcessGroup or Session still carries
            // this number in its captured chain, the NUMBER must survive
            // the task — retarget the slot to the TaskId 0 sentinel
            // (allocated-but-resolves-to-nothing) instead of freeing it.
            // Linux parity: `struct pid` stays alive via its
            // Linux keeps the number pinned through its group/session attachment after the task
            // detaches. The number is finally freed by
            // `pid_unpin_chain()` when the group/session empties
            // ([Section 17.1](#namespace-architecture--process-group-session-number-registries-and-pid-pinning)).
            if link.ns.pgids.get(link.nr as u32).is_some()
                || link.ns.sids.get(link.nr as u32).is_some() {
                let _ = map.replace(link.nr as u32, 0);
            } else {
                // Idr deallocation — the number is immediately reallocatable.
                map.remove(link.nr as u32);
            }
            // Value-verified reverse_map eviction, INSIDE this same
            // PID_MAP_LOCK(47) section. The reverse_map field's
            // write-serialization contract requires BOTH writes — insert
            // at fork (create_task step 10a) and remove at exit (here) — under
            // this per-namespace lock. The lower-32 cache slot may hold a
            // COLLIDING live task's translation (reverse_map is keyed on
            // the lower 32 bits of TaskId), so evict only if it still maps
            // to OUR number. Serializing this check-and-remove with the
            // fork-side insert under one lock makes the pair atomic: a
            // concurrent colliding child's insert either fully precedes
            // this section (our value-verify then sees the child's nr and
            // skips the remove) or fully follows it (the child re-inserts
            // after our remove) — either way the surviving slot is the
            // live task's, never the dead one's. Doing the eviction
            // OUTSIDE the lock (the pre-fix form) left a lookup/remove
            // TOCTOU that could delete a live colliding task's entry.
            if link.ns.reverse_map.lookup(task.tid as u32, &rcu)
                == Some(link.nr as u32) {
                link.ns.reverse_map.remove(task.tid as u32);
            }
        });
    }
    // Task.pid_links itself is retained (not drained) — pid_ns_release()
    // below consumes it.
}

/// reap_task() step 7: drop the task's namespace MEMBERSHIP — the
/// per-level nr_tasks counts (incremented at create_task step 10a) and the
/// chain's Arc<PidNamespace> references. Drains the chain; afterwards
/// Task::leaf_pid_ns()/leaf_pid_nr() return None. A namespace whose
/// nr_tasks reaches 0 has a pid_map empty of task entries
/// (pid_allocator_free ran first for every member; TaskId-0 sentinel
/// slots pinned by a still-live cross-namespace group/session may
/// remain, and the pinner's chain Arc keeps the namespace object alive
/// until unpin) and is destroyed when its last Arc drops
/// (tracked-storage Drop). Empty chain (collapse_thread_group old leader): no-op —
/// its membership travelled to the exec-ing task with the numbers.
pub fn pid_ns_release(task: &Task) {
    // Drain into a local under the leaf lock, DROP the lock, then
    // consume. The final `Arc<PidNamespace>` drop of a dying namespace
    // runs its tracked-storage teardown (`free_tracked` →
    // TRACKED_REGISTRY_LOCK(135)) — a lock acquisition that must not
    // happen under the leaf `pid_links` SpinLock (same discipline as
    // pid_allocator_free above).
    let drained: ArrayVec<PidLink, 33> = {
        let mut links = task.pid_links.lock();
        // Seqcount bracket: the drain is a mutation of a PUBLISHED task's
        // chain, so an interrupt/NMI reader must retry rather than walk a
        // half-emptied ArrayVec (`Task.pid_links_seq`,
        // [Section 8.1](08-process.md#process-and-task-management--task-model)). Plain load+store,
        // single writer — the leaf SpinLock is the writer exclusion.
        let s = task.pid_links_seq.load(Relaxed);
        task.pid_links_seq.store(s + 1, Release);   // odd
        let taken = core::mem::take(&mut *links);
        task.pid_links_seq.store(s + 2, Release);   // even
        taken
    };
    for link in drained {
        link.ns.nr_tasks.fetch_sub(1, AcqRel);
        // link.ns (Arc<PidNamespace>) drops here — with no lock held.
    }
}

17.1.2.3 Process-Group / Session Number Registries and PID Pinning¶

Process groups and sessions are identified by the creating leader's ProcessId (u64, never reused) and translated to per-namespace numbers via their captured pid_chain (Section 8.7). Two per-namespace obligations follow, both served by the PidNamespace.pgids / PidNamespace.sids registries:

Reverse resolution: a caller-supplied pgid_t (e.g. kill(-pgid), setpgid, tcsetpgrp, waitpid(< -1)) must resolve to the group in the CALLER's namespace: ns.pgids.get(pgid as u32) — O(1), RCU-read.
Number pinning: POSIX permits addressing a group whose leader has exited (killpg() on a shell job whose leader died). The ns-local number must therefore stay ALLOCATED — un-recyclable by fork — for as long as the group is non-empty. Linux gets this from struct pid refcounting (a Linux process-group/session attachment keeps the number alive); UmkaOS gets it from the registry entries plus a sentinel in pid_map.

Compound operations (each level's mutation runs under that namespace's PID_MAP_LOCK(47), the same lock that serializes pid_map writes — pin, unpin, task-number free, and allocation are therefore mutually ordered; registry READS are RCU/lock-free). The lock acquisition is EXPLICIT in the pseudocode: Idr::with_lock(f) holds the Idr's internal SpinLock — which IS this namespace's PID_MAP_LOCK(47) (master-table row 47) — across the closure and passes a locked view whose lookup/remove/replace accessors do not re-lock (Section 3.6). Without the spanning lock, the separate get/lookup/remove calls in unpin would be a check-then-act race: between sids.get(nr) == None and remove(nr), a concurrent setsid() could pin the same nr — the unpin would then free a slot the new session just pinned, and idr_alloc_range could hand the number to a fresh task while the live session's registries still map it, breaking the no-recycling invariant below.

/// Group/session creation (setsid(), setpgid()-new-group, boot init):
/// register every level of the captured chain in the level's registry.
/// The chain was cloned from the LIVE leader's `Task.pid_links`, so every
/// number is currently allocated in its `pid_map` — registration pins it.
pub fn pid_pin_chain_pgrp(chain: &[PidLink], group: &Arc<ProcessGroup>) {
    for link in chain {
        link.ns.pid_map.with_lock(|_map| {   // PID_MAP_LOCK(47)
            link.ns.pgids.insert(link.nr as u32, Arc::clone(group));
        });
    }
}
/// Session twin (setsid() calls BOTH: once for the session, once for the
/// new group — the two lifetimes end independently).
pub fn pid_pin_chain_session(chain: &[PidLink], session: &Arc<Session>) {
    for link in chain {
        link.ns.pid_map.with_lock(|_map| {   // PID_MAP_LOCK(47)
            link.ns.sids.insert(link.nr as u32, Arc::clone(session));
        });
    }
}

/// Group/session destruction (the group emptied: reap_task() step 5c
/// or setpgid() departure; the session's last group left: same sites):
/// deregister, then free any number whose owning task is already gone.
///
/// `registry` is the matching table (pgids for a group, sids for a
/// session). For each level: after removing the registry entry, the
/// pid_map slot is freed IFF it holds the TaskId 0 sentinel (the leader
/// exited first and pid_allocator_free retargeted the slot) AND the twin
/// registry does not still pin it (a setsid() leader's number is pinned
/// once as a sid and once as a pgid). A slot still holding a live TaskId
/// belongs to the still-running leader task and is freed by ITS
/// pid_allocator_free. The remove/probe/free triple is ONE
/// PID_MAP_LOCK(47) critical section per level.
pub fn pid_unpin_chain_pgrp(chain: &[PidLink]) {
    for link in chain {
        let nr = link.nr as u32;
        link.ns.pid_map.with_lock(|map| {    // PID_MAP_LOCK(47)
            link.ns.pgids.remove(nr);
            let still_pinned = link.ns.sids.get(nr).is_some();
            if !still_pinned && map.lookup(nr).copied() == Some(0) {
                map.remove(nr);
            }
        });
    }
}
/// Session twin: checks `pgids` as the remaining pinner.
pub fn pid_unpin_chain_session(chain: &[PidLink]) {
    for link in chain {
        let nr = link.nr as u32;
        link.ns.pid_map.with_lock(|map| {    // PID_MAP_LOCK(47)
            link.ns.sids.remove(nr);
            let still_pinned = link.ns.pgids.get(nr).is_some();
            if !still_pinned && map.lookup(nr).copied() == Some(0) {
                map.remove(nr);
            }
        });
    }
}

Invariants:

A registered number's pid_map slot is always allocated: it holds either the live leader's TaskId or the 0 sentinel. idr_alloc_range() therefore can never hand it to a new task — no recycling while the group/session lives.
The 0 sentinel resolves to nothing: resolve_pid()'s find_task_by_tid(0) returns None (TaskId 0 is reserved), so kill(pid) on a dead-leader number yields ESRCH while kill(-pid) on the same number still finds the group through ns.pgids — exactly Linux's observable behavior for a dead group leader.
terminate_members()'s pid_map broadcast iteration skips sentinel entries naturally (find_task_by_tid(0) → None → skip).
No double-free: a number is freed EITHER by pid_allocator_free (task exits unpinned) OR by the unpin path (pinned at task exit; freed when the last pinner leaves) — the sentinel value distinguishes the two states under PID_MAP_LOCK(47).
Registry entries hold Arc<ProcessGroup>/Arc<Session>; the group's own pid_chain holds Arc<PidNamespace>. This is Arc-cyclic only via the registry (namespace → group → chain → namespace), and the cycle is broken deterministically at group/session destruction (unpin removes the registry Arc) — the group cannot be destroyed without unpinning, and unpin is part of the single destruction path (empty-group removal, mirrored at setpgid step 3g and reap_task() step 5c).

50-year note: registries are bounded by live groups/sessions (≤ live processes); entries are removed on the same path that removes the group from PROCESS_GROUPS/SESSIONS — zero residual growth.

17.1.2.4 `terminate_members()` — PID-Namespace Teardown¶

When the init process (PID 1, child_reaper) of a non-root PID namespace exits, every other process in the namespace must be killed and reaped BEFORE init completes its own teardown — once init is gone, nothing inside the namespace can reap namespace orphans. The trigger is an explicit step in exit_task()'s last-thread gate — before address-space teardown and before the namespace_set tombstone swap (Section 8.2) — NOT a refcount hook: other members hold Arc<PidNamespace> references, so PID 1's own reference drop can never be the trigger.

/// Kill and reap every process in `ns` except the calling init.
/// Called by exit_task()'s last-thread gate when the dying process IS
/// ns.child_reaper's process and ns is not the root namespace.
///
/// Preconditions: caller is the namespace init's last live thread with
/// PF_EXITING set; caller's namespace_set is still live (signal-permission
/// context); caller holds no locks. May sleep.
pub fn terminate_members(ns: &Arc<PidNamespace>, caller: &Task);

Procedure:

Close the namespace to new members: set the death latch — ns.pid_map.with_lock(|_| ns.dying.store(true, Ordering::Release)) — under the namespace's PID_MAP_LOCK(47), BEFORE the broadcast below. From this point every create_task() step-10a allocation targeting this namespace fails with ENOMEM (the per-level check runs inside the same lock, so a racing fork either published its pid_map entry and nr_tasks increment before this store — making it visible to the broadcast and the wait loop — or observes the latch and unwinds; no third interleaving). Never cleared: the namespace is on its one-way path to destruction. (Linux closes the same window by clearing a Linux PIDNS_ADDING flag before its kill loop, kernel/pid_namespace.c; the dedicated dying latch keeps that state out of the member counter.)
SIGKILL broadcast: iterate ns.pid_map under RCU; for every entry whose TaskId is not the caller's, resolve via find_task_by_tid() (lookup-and-pin; TaskId-0 sentinel entries left by pinned group/session numbers resolve to None and are skipped) and signal_wake_up(t, true) — the same fatal-wake primitive as the exit_group zap (SIGKILL into the per-task pending set, then the BYPASS_CBS wake with the full fatal mask including TASK_STOPPED/TASK_TRACED — Section 8.6), each enqueue+wake performed under that target's own SIGLOCK(40) (signal_wake_up()'s precondition; targets here span DIFFERENT processes, so the lock is per-target, not hoisted around the loop). PID-1 signal protection does not apply: the sender IS init.

Mid-fork children — the three-sided closure. A child between create_task step 10a (pid_map published, nr_tasks incremented) and step 16 (PID_TABLE published) resolves to None here and is SKIPPED — this one-shot broadcast alone cannot kill it, and its forking parent may not yet have observed its own SIGKILL. Three mechanisms close the window, and ALL THREE are required: - the step-0 dying latch rejects every allocation that has not yet completed its locked step-10a section — and, crucially, every FUTURE fork into this namespace, including by a CROSS-NAMESPACE parent (pending_pid_ns) that is not a member and is therefore never signalled by this broadcast, even one arriving after zap returned. - create_task step 17's under-lock re-check includes fatal_signal_pending(parent) — a member parent signalled by this broadcast unwinds the fork (releasing the step-10a nr_tasks increment) before the child ever runs (Section 8.1). - the reap-and-wait loop below RE-RUNS this broadcast every iteration — a child whose step-10a completed before the latch is by the next iteration in PID_TABLE (step 16) and is killed by the re-broadcast. None is redundant: without the latch a cross-namespace fork lands an unreapable orphan in a dead namespace (its parent unsignalled, no re-broadcast running); without the fork-side check every zap pays a full 1 s timeout round for a racing member fork; without the re-broadcast a fork whose step-17 check ran before the broadcast's SIGKILL landed would leave an unsignalled member and hang the loop forever. 2. Reap-and-wait loop, until ns.nr_tasks.load(Acquire) == 1: a. Re-broadcast: re-run step 1's pid_map iteration. SIGKILL delivery is idempotent (a bit in the pending set; already-dying tasks no-op), so re-sending is safe and cheap — O(members) RCU reads per iteration, and the loop runs only during namespace teardown. b. Reap: scan the caller's children with the wait-scan skeleton (Section 8.2), claiming fully-dead zombies via the ZOMBIE → DEAD CAS and reap_task()-ing them (which is where nr_tasks is decremented — the loop's exit condition is met at REAP, not at death). Members whose parent is IN this namespace, or whose parent has since exited, reparent to this still-findable init as their in-namespace ancestors die, and init reaps them here. c. Wait on caller.process.wait_chldexit (woken by every child's do_notify_parent()), with a bounded (1 s) per-iteration timeout so a victim stuck in an unbounded uninterruptible kernel wait degrades to periodic re-scan (including the re-broadcast in (a)) instead of a silent hang. A SIGKILL re-aimed at init itself does not abort the loop; teardown must complete.

External-reaper dependence (member with a live cross-namespace parent). The reap in (b) sees only init's children. A member created by a CROSS-NAMESPACE parent P in an ancestor namespace — the pending_pid_ns fork population (P did setns(CLONE_NEWPID) into this namespace, then fork()ed; discussed in the three-sided closure of step 1) — is NOT init's child. The step-1 broadcast SIGKILLs it and it becomes a zombie, but reparenting happens only when a task's PARENT dies, and P is alive, so the zombie stays P's and is claimable ONLY by P's wait4/ waitid in the ancestor namespace. It never surfaces in init's wait-scan, and its nr_tasks contribution is released only when P reaps it. Init's teardown therefore DEPENDS on those external parents reaping their members: a member whose live cross-namespace parent neither reaps nor sets SIGCHLD to SIG_IGN/SA_NOCLDWAIT keeps ns.nr_tasks > 1 and blocks this loop until P acts. This is intentional and Linux-identical — Linux kernel/pid_namespace.c zap_pid_ns_processes() documents the same dependence (cross-namespace children are reaped by their real parent, not by the dying init). The 1 s timeout keeps the loop interruptible and re-scanning; it is NOT an autoreap point, and init does NOT steal or reparent P's zombie (that would hand P's wait4 a spurious ECHILD, an ABI break). The block is bounded to this one dying init task (sitting in S state) — the rest of the system is unaffected; it is a userspace responsibility, never a kernel hang. (An implementing agent must NOT "fix" this with an autoreap-on-zap: the cross-namespace parent owns the reap.) 3. Return. ns.nr_tasks == 1 (only init). The remaining Arc<PidNamespace> references — init's namespace_set (released in exit_task Step 11) and its pid_links entry (released in reap_task() step 7) — gate the namespace's destruction.

Even with zap, exit paths must still handle a DEAD child_reaper: between init's release and a victim's last reparent/notify attempt, find_task_by_tid(ns.child_reaper.load(Acquire)) resolves to None (the id was cleared at init's reap_task(), and TaskIds are never reused) — find_live_reaper()'s parent-namespace walk (Section 8.2) is the required fallback, never an .expect().

17.1.2.5 Tracked Allocation — NamespaceSet, PidNamespace, NetNamespace¶

NamespaceSet, PidNamespace, and NetNamespace are migration-tracked types (Decision 1, Section 13.18): long-lived container control structures whose layout must be evolvable over the kernel's operational lifetime. Every instance allocates from Nucleus tracked storage through the canonical constructors below — never through bare Arc::new or a slab cache. The other namespace types (UTS/IPC/Cgroup/Time/User/IMA) are not on the Decision-1 list and keep their existing allocation; MountNamespace is VFS-owned and already converted (Section 14.6).

All creation sites are warm-to-cold paths (clone(CLONE_NEW*), unshare, setns — container-lifecycle frequency); the tracked hot path is a per-CPU magazine pop, cycle-equivalent to a slab fast path, so the conversion is cost-neutral.

Registration — one #[module_init] constructor for the containers module, running during boot BEFORE init_namespaces() so the init instances (INIT_PID_NS, INIT_NET_NS) and the EMPTY_NSPROXY tombstone themselves live in tracked storage:

// umka-nucleus/src/ns/ns_init.rs — Evolvable

impl TrackedType for NamespaceSet {
    fn type_id() -> TypeId {
        NS_SET_TYPE_ID.get().copied().expect("NamespaceSet not yet registered")
    }
}
impl TrackedType for PidNamespace {
    fn type_id() -> TypeId {
        PID_NS_TYPE_ID.get().copied().expect("PidNamespace not yet registered")
    }
}
impl TrackedType for NetNamespace {
    fn type_id() -> TypeId {
        NET_NS_TYPE_ID.get().copied().expect("NetNamespace not yet registered")
    }
}

static NS_SET_TYPE_ID: BootOnceCell<TypeId> = BootOnceCell::new();
static PID_NS_TYPE_ID: BootOnceCell<TypeId> = BootOnceCell::new();
static NET_NS_TYPE_ID: BootOnceCell<TypeId> = BootOnceCell::new();

#[module_init]
fn ns_register_tracked_types() {
    let set_template = TypeDescriptorTemplate {
        size: core::mem::size_of::<NamespaceSet>() as u32,
        alignment: core::mem::align_of::<NamespaceSet>() as u32,
        // Runtime-derived: at most one INSTALLED NamespaceSet per task
        // (threads sharing a set via Arc only lower the live count), ×2
        // for the staging window where the replacement exists while the
        // old set is still installed (unshare/setns pre-allocate via
        // namespace_set_alloc BEFORE `TASK_LOCK(20)`; fork step 10 builds the child's
        // set before publication), + 1 for the boot-allocated
        // EMPTY_NSPROXY tombstone ([Section 8.2](08-process.md#process-lifecycle-teardown)).
        max_instances: 2 * cpu_count() * MAX_TASKS_PER_CPU + 1,
        migration_fn: Some(nsset_migrate),
        checker_id: None,
    };
    NS_SET_TYPE_ID.set(register_tracked_type(set_template)
        .expect("NamespaceSet descriptor registration failed"))
        .expect("NamespaceSet type_id already set");

    let pid_ns_template = TypeDescriptorTemplate {
        size: core::mem::size_of::<PidNamespace>() as u32,
        alignment: core::mem::align_of::<PidNamespace>() as u32,
        // 0 = unbounded (registry grows on demand) — PidNamespace is a
        // slow-changing type, exactly the case the TypeDescriptor contract
        // names as safe for an unbounded registry (alongside NetNamespace).
        // Creation rate is bounded externally by the per-user
        // /proc/sys/user/max_pid_namespaces limit and the nesting-depth
        // cap (32); every live PID namespace transitively pins at least
        // one Task, so the Task instance budget indirectly bounds
        // namespace-driven storage growth.
        max_instances: 0,
        migration_fn: Some(pid_ns_migrate),
        checker_id: None,
    };
    PID_NS_TYPE_ID.set(register_tracked_type(pid_ns_template)
        .expect("PidNamespace descriptor registration failed"))
        .expect("PidNamespace type_id already set");

    let net_ns_template = TypeDescriptorTemplate {
        size: core::mem::size_of::<NetNamespace>() as u32,
        alignment: core::mem::align_of::<NetNamespace>() as u32,
        // 0 = unbounded — NetNamespace is the type the TypeDescriptor
        // contract itself names as the safe unbounded example. Bounded
        // externally by /proc/sys/user/max_net_namespaces.
        max_instances: 0,
        migration_fn: Some(net_ns_migrate),
        checker_id: None,
    };
    NET_NS_TYPE_ID.set(register_tracked_type(net_ns_template)
        .expect("NetNamespace descriptor registration failed"))
        .expect("NetNamespace type_id already set");
}

Migration decisions — all three register Some (full Shadow-and-Migrate available), following the task_migrate field-copy template (Section 13.18). Rationale: populations are small (NamespaceSet bounded by task count; namespaces number in the hundreds/thousands per host), so the Phase B stop-the-world walk is microseconds. External Arc identity is handled by the tracked slot's side header exactly as for Task/MmStruct/Mount. XArray-rooted maps (PidNamespace.pid_map/reverse_map, NetNamespace.interfaces) migrate as root-pointer field copies — interior nodes are ordinary slab allocations, not part of the tracked instance.

Allocation bridges (Arc-owned pattern, same as mount_alloc()):

/// Move a fully-built NamespaceSet into Nucleus tracked storage and bridge
/// it to the kernel-wide Arc handle. Canonical constructor for EVERY
/// published Arc<NamespaceSet>: fork step 10
/// ([Section 8.1](08-process.md#process-and-task-management)), sys_unshare step 5, setns(), and
/// the EMPTY_NSPROXY boot initialization.
fn namespace_set_alloc(ns: NamespaceSet) -> Result<Arc<NamespaceSet>, Errno> {
    let mut ptr: TrackedPtr<NamespaceSet> =
        alloc_tracked::<NamespaceSet>()
            // Both arms map to ENOMEM (clone(2)/unshare(2)/setns(2)
            // document ENOMEM for kernel allocation failure); the FMA
            // framework records the distinct OutOfInstances cause.
            .map_err(|_| Errno::ENOMEM)?;
    ptr.as_uninit().write(ns);
    // SAFETY: freshly allocated, fully initialized, never shared. The
    // last Arc strong-ref drop runs the payload Drop (decrementing each
    // contained namespace Arc), then free_tracked::<NamespaceSet>().
    Ok(unsafe { Arc::from_tracked(ptr) })
}

/// Same bridge for PidNamespace — called ONLY by `PidNamespace::create()`
/// (the canonical constructor below, reached from fork step 10 and
/// unshare step 4's pending_pid_ns path) and, with a root-namespace
/// literal (parent = None, level = 0), for INIT_PID_NS at boot.
/// OutOfInstances is unreachable (max_instances = 0); OutOfStorage → ENOMEM.
fn pid_ns_alloc(init: PidNamespace) -> Result<Arc<PidNamespace>, Errno>;

/// Same bridge for NetNamespace — used by every "create empty network
/// namespace" step (clone(CLONE_NEWNET), unshare(CLONE_NEWNET)) and for
/// INIT_NET_NS at boot. The post-allocation initialization sequence
/// (loopback registration Phase 3, port allocator, sysctl defaults — see
/// "Network namespace initial state" above) runs on the returned Arc
/// through interior-mutable fields only; the loopback device itself is
/// built BEFORE this call and arrives inside `init` (non-optional field).
fn net_ns_alloc(init: NetNamespace) -> Result<Arc<NetNamespace>, Errno>;

/// Allocate the `Capability<NetStack>` registry entry wrapping a freshly
/// allocated network namespace ([Section 9.1](09-security.md#capability-based-foundation)). Runs
/// in creation Phase 3, immediately after `net_ns_alloc()`; the returned
/// handle is stored into `ns.stack_cap` (OnceCell, set exactly once) and
/// copied into every `NamespaceSet.net_stack` that adopts this namespace.
/// Fallible: ENOMEM on capability-table exhaustion — the caller's
/// rollback drops the namespace Arc; `Drop` sees the unset cell and
/// revokes nothing.
fn net_stack_cap_create(ns: &Arc<NetNamespace>) -> Result<CapHandle, Errno>;

17.1.2.6 PidNamespace::create — Canonical Constructor¶

pid_ns_alloc() above is only the tracked-storage BRIDGE — it moves a fully-built value. The value itself is built by exactly one constructor, the PID twin of UserNamespace::create():

impl PidNamespace {
    /// Build and allocate a child PID namespace. Called from `create_task()`
    /// step 10's CLONE_NEWPID branch and `sys_unshare()` step 4's
    /// CLONE_NEWPID branch — the ONLY creation sites (INIT_PID_NS at boot
    /// uses a root-namespace literal through `pid_ns_alloc()` directly:
    /// `parent = None`, `level = 0`).
    ///
    /// - `parent`: the namespace the caller's future children would
    ///   otherwise be born into — the resolved `pending_pid_ns` when a
    ///   prior `setns(CLONE_NEWPID)`/`unshare(CLONE_NEWPID)` set one,
    ///   the caller's `namespace_set.pid_ns` otherwise. (Linux parity:
    ///   Linux `copy_pid_ns()` nests under `pid_ns_for_children`.)
    /// - `owner`: the CREATING CREDENTIAL's user namespace
    ///   (`child_cred.user_ns` in fork, `pending_cred.user_ns` in
    ///   unshare — see the owner-assignment rule in
    ///   [Section 17.1](#namespace-architecture--capability-domain-mapping)).
    /// - `owner_uid`: the creator's EFFECTIVE uid expressed in `owner`'s
    ///   terms — the ucount charge key (Linux `inc_pid_namespaces` uses
    ///   Linux `current_euid()`). Callers (create_task step 10, unshare) pass it.
    ///
    /// # Errors
    /// - `ENOSPC`: `parent.level >= 32` (the child would land at level 33,
    ///   exceeding the PID-namespace depth limit of 32 — the deepest CREATABLE namespace is
    ///   level 32, matching Linux `create_pid_namespace()`'s
    ///   Linux checks `if (level > 32)` with `level = parent->level + 1`,
    ///   `kernel/pid_namespace.c`, verified against torvalds/linux master;
    ///   a namespace AT level 32 is permitted, only its children are refused).
    ///   The struct-doc nesting invariant (`maximum = 32`) is ENFORCED here,
    ///   at the single constructor. Also `ENOSPC` if any ANCESTOR level's
    ///   `/proc/sys/user/max_pid_namespaces` is exceeded
    ///   (`ns_ucount_charge(owner, owner_uid, PidNs)` — the same
    ///   ancestor-charged accounting as user namespaces; the returned token
    ///   in `ucount_charge` decrements on Drop, leak-free on rollback AND
    ///   teardown).
    /// - `ENOMEM`: tracked-storage exhaustion (from `pid_ns_alloc`) — the
    ///   charge token drops and uncharges (no leak).
    pub fn create(
        parent: &Arc<PidNamespace>,
        owner: &Arc<UserNamespace>,
        owner_uid: u32,
    ) -> Result<Arc<PidNamespace>, Errno> {
        if parent.level >= 32 {
            // PID namespace depth limit = 32: a parent already at the maximum depth
            // cannot spawn a level-33 child. Parent levels 0..=31 are
            // permitted, producing children at levels 1..=32 — a level-32
            // namespace exists and is reachable (Linux parity).
            return Err(Errno::ENOSPC);
        }
        // Charge the per-(user ns, uid) PID-namespace count against the
        // OWNER's ancestor chain (ancestor-charged; ENOSPC over any level's
        // /proc/sys/user/max_pid_namespaces). The RAII token is stored in
        // `ucount_charge`; if `pid_ns_alloc` then fails, the token drops and
        // every charged level is uncharged. `owner_uid` is the creator's
        // effective uid in `owner`'s terms (Linux `inc_pid_namespaces` keys
        // by the effective uid).
        let ucount_charge = ns_ucount_charge(owner, owner_uid, NsUcountKind::PidNs)?;
        pid_ns_alloc(PidNamespace {
            ns_id: NEXT_NS_ID.fetch_add(1, Ordering::Relaxed),
            user_ns: Arc::clone(owner),   // STRONG — pins the owner (Linux get_user_ns)
            child_reaper: AtomicU64::new(0),   // set at create_task step 16
            parent: Some(Arc::clone(parent)),
            level: parent.level + 1,
            pid_map: Idr::new(),               // empty — first alloc yields 1
            reverse_map: RcuIdr::new(),
            pgids: XArray::new(),
            sids: XArray::new(),
            // Inherit the parent's ceiling: a child namespace never
            // exposes more numbers than its parent permits; lowerable
            // per-namespace via the namespace-scoped pid_max sysctl.
            pid_max: parent.pid_max,
            nr_tasks: AtomicU32::new(0),
            dying: AtomicBool::new(false),
            ucount_charge,   // Drop uncharges every level (rollback + teardown)
        })
    }
}

Drop semantics unchanged: impl Drop for NetNamespace (capability revocation — see "Namespace Drop semantics" above) still runs exactly once, when the last Arc<NetNamespace> strong reference drops: the payload's Drop executes in place inside the tracked slot, THEN the embedded TrackedPtr drops, invoking free_tracked::<NetNamespace>() — returning the slot to the per-type free list and removing the instance from the Nucleus live registry. The same order applies to PidNamespace and NamespaceSet. There is no separate free path.

17.1.3 Container Root Filesystem: pivot_root(2)¶

Container runtimes (runc, containerd, crun) require a mechanism to change the root filesystem after setting up the mount namespace. UmkaOS implements the standard pivot_root(2) syscall:

/// pivot_root(new_root: &CStr, put_old: &CStr) -> Result<()>
///
/// Atomically swaps the root mount with another mount point. Required for
/// OCI-compliant container creation.
///
/// # Prerequisites (checked by syscall)
/// - new_root must be a mount point
/// - put_old must be at or under new_root
/// - Caller must be in a mount namespace (CLONE_NEWNS or unshare(CLONE_NEWNS))
/// - Caller must have CAP_SYS_ADMIN in its user namespace
///
/// # Operation
/// 1. Attach new_root to the root of the mount namespace
/// 2. Move the old root to put_old
/// 3. The process's root directory is now new_root
/// 4. Subsequent umount(put_old) removes the old root from the namespace
///
/// # Container Runtime Usage
/// ```
/// // Standard OCI container creation sequence:
/// unshare(CLONE_NEWNS);                    // New mount namespace
/// mount("none", "/", NULL, MS_REC | MS_PRIVATE, NULL);  // Make all private
/// mount("/var/lib/container/rootfs", "/var/lib/container/rootfs",
///       NULL, MS_BIND | MS_REC, NULL);     // Bind-mount rootfs onto itself
/// pivot_root("/var/lib/container/rootfs", "/var/lib/container/rootfs/.oldroot");
/// chdir("/");                              // Ensure we're in new root
/// umount2("/.oldroot", MNT_DETACH);        // Detach old root
/// // Process now sees container rootfs as /
/// ```
///
/// # Difference from chroot(2)
/// pivot_root is fundamentally different from chroot:
/// - chroot only affects the process's view of the root directory
/// - pivot_root actually moves the mount point, affecting all processes in the namespace
/// - chroot can be escaped via mount namespace tricks; pivot_root cannot
/// - Container runtimes MUST use pivot_root for secure isolation
///
/// # Error codes
/// - EBUSY: new_root is not a mount point, or put_old is not under new_root
/// - EINVAL: new_root and put_old are the same
/// - ENOENT: path component does not exist
/// - ENOTDIR: path component is not a directory
/// - EPERM: Caller lacks CAP_SYS_ADMIN, or not in mount namespace
/// - ENOSYS: Not implemented (will not occur in UmkaOS)
fn sys_pivot_root(new_root: UserPtr<u8>, put_old: UserPtr<u8>) -> Result<(), Errno> { ... }

Mandatory umount2(put_old, MNT_DETACH) after pivot_root: After pivot_root() succeeds, the host filesystem is still mounted at put_old inside the container's mount namespace. This is a security requirement — the container init process MUST call umount2(put_old, MNT_DETACH) to detach the host filesystem before executing the container entrypoint. Without this step:

The entire host filesystem remains visible and traversable inside the container at the put_old mount point (e.g., /.oldroot/etc/shadow, /.oldroot/proc).
A container process with sufficient capabilities could read host secrets, modify host files, or escape the container entirely by chdir-ing into the host filesystem tree.
MNT_DETACH (lazy unmount) is used instead of a synchronous unmount because in-flight path lookups may still hold references to the old root mount; lazy unmount detaches the mount from the namespace immediately (invisible to new lookups) and the actual cleanup occurs after the last reference is released (RCU grace period).

OCI-compliant container runtimes (runc, containerd, crun) all perform this step. UmkaOS does not enforce it automatically (the kernel cannot know when the container setup sequence is complete), but the container creation documentation, examples, and the pivot_root(2) man page MUST document this as a mandatory step. The container creation sequence in the seccomp-bpf section below reflects this ordering.

Effect on other processes: pivot_root only affects processes whose root is the old root mount within the same mount namespace. Processes in other mount namespaces are unaffected. Within the same mount namespace, processes whose root directory points to the old root mount will see the new root after the RCU-published pointer swap (step 7 above). Processes that have already chroot-ed to a subdirectory of the old root are also unaffected because their root is not the namespace root mount.

Interaction with other namespaces: - pivot_root operates on the caller's mount namespace - The root change is visible to all processes sharing that mount namespace - Combined with PID namespace: the container's init (PID 1) sees only the new root - Combined with User namespace: unprivileged processes can pivot_root within their own user namespace if they have CAP_SYS_ADMIN there

Implementation notes: The VFS layer (Section 14.1) handles the mount tree manipulation. The Mount struct, MountNamespace, mount hash table, and the complete pivot_root algorithm using these types are defined in Section 14.6 (13-vfs.md). The summary below is retained for context; the authoritative specification is Section 14.6.

Lookup new_root and verify it's a mount point
Lookup put_old and verify it's under new_root
Lock the mount tree for modification (holds mount_lock)
Detach the current root from the namespace's mount list
Attach new_root as the new namespace root
Reattach the old root at put_old position
Publish the new root via RCU: rcu_assign_pointer(namespace->root, new_root)
Unlock the mount tree

Atomicity with respect to path lookups: Steps 4–6 are performed while holding mount_lock, and the old root pointer remains valid in the RCU-published slot until step 7 overwrites it. Path lookups (open(), stat(), readlink(), etc.) take an RCU read-side reference to the namespace root at the start of lookup via rcu_dereference(namespace->root). This ensures: - In-flight path lookups that started before pivot_root complete with the old root (consistent view) - New path lookups that start after step 7 see the new root - No path lookup can see a partially-updated state (no torn reads, no null pointer) - Between steps 4–6, the data structures are modified under mount_lock, but lookups still see the old root via RCU The RCU grace period after step 7 ensures that by the time umount(put_old) completes, no in-flight lookups hold references to the old root.

17.1.4 Joining Namespaces: setns(2) and nsenter¶

Container operations like docker exec require joining an existing namespace. UmkaOS implements setns(2) for this purpose:

/// setns(fd: RawFd, nstype: c_int) -> Result<()>
///
/// Reassociates the calling thread with the namespace referenced by fd.
///
/// # Parameters
/// - fd: File descriptor referring to a namespace (obtained from /proc/[pid]/ns/[type])
/// - nstype: Namespace type (CLONE_NEW* constant) or 0 to auto-detect from fd
///
/// # Prerequisites
/// - Caller must have CAP_SYS_ADMIN in the target namespace's owning user namespace
/// - For PID namespaces: No restriction (affects future children only, per Linux 3.8+)
/// - For user namespaces: Caller must not be in a chroot environment
/// - The namespace must still exist (owning process hasn't exited)
///
/// # Container Runtime Usage (docker exec)
/// ```
/// // Join a running container's namespaces:
/// int fd = open("/proc/[container_pid]/ns/mnt", O_RDONLY | O_CLOEXEC);
/// setns(fd, CLONE_NEWNS);  // Join mount namespace
/// close(fd);
///
/// fd = open("/proc/[container_pid]/ns/net", O_RDONLY | O_CLOEXEC);
/// setns(fd, CLONE_NEWNET); // Join network namespace
/// close(fd);
///
/// // PID namespace must be joined via clone(), not setns()
/// // (kernel limitation: can't change PID namespace of running process)
/// // exec() into container: now running in container's namespaces
/// execve("/bin/sh", ["/bin/sh"], envp);
/// ```
///
/// # Two fd kinds
/// `fd` is either an **nsfs namespace fd** (from `/proc/[pid]/ns/*` or an
/// `open()` of a bind-mounted namespace file) — which joins the SINGLE
/// namespace it names, `nstype` used only to cross-check the type — or a
/// **pidfd**, which joins all of the target process's namespaces selected
/// by the `nstype` CLONE_NEW* mask (Linux 5.8+, `nsenter --all`). Only the
/// pidfd form joins more than one namespace, and it applies the USER
/// namespace FIRST because entering it rewrites the caller's credentials
/// and thereby gates the CAP_SYS_ADMIN checks of every other join (Linux
/// Linux `validate_nsset()` validates the user namespace first, `kernel/nsproxy.c`).
/// That multi-namespace pidfd form is a documented Phase-3 milestone —
/// specified in "Multi-namespace setns via pidfd" below; the single nsfs-fd
/// form implemented here joins one namespace and has NO intra-call ordering.
/// There is NO cross-`setns()`-call reordering either: each `setns()` is an
/// independent syscall carrying ONE fd, and the kernel cannot reorder across
/// separate syscalls.
///
/// # Namespace file descriptors
/// Each namespace type is exposed via /proc/[pid]/ns/:
/// ```
/// /proc/[pid]/ns/cgroup         → Cgroup namespace
/// /proc/[pid]/ns/ipc            → IPC namespace
/// /proc/[pid]/ns/mnt            → Mount namespace
/// /proc/[pid]/ns/net            → Network namespace
/// /proc/[pid]/ns/pid            → PID namespace (current)
/// /proc/[pid]/ns/pid_for_children → PID namespace for future children (after setns)
/// /proc/[pid]/ns/time           → Time namespace (Linux 5.6+)
/// /proc/[pid]/ns/time_for_children → Time namespace for future children (Linux 5.6+)
/// /proc/[pid]/ns/user           → User namespace
/// /proc/[pid]/ns/uts            → UTS namespace
/// ```
///
/// The `*_for_children` symlinks reveal the pending namespace set by
/// `setns(CLONE_NEWPID)` or `setns(CLONE_NEWTIME)`. They differ from the
/// main symlinks when a process has called `setns()` but not yet forked.
/// Container introspection tools (`lsns`, `nsenter --target`) use these.
///
/// These are magic links: reading them returns the namespace type, and
/// opening them gives a file descriptor that can be passed to setns().
///
/// # Error codes
/// - EBADF: Invalid fd
/// - EINVAL: fd is neither an nsfs namespace fd nor a pidfd; `nstype`
///           doesn't match a single nsfs fd's type; a pidfd `nstype` mask
///           is zero or has unknown bits; (CLONE_NEWUSER) the caller is
///           multi-threaded or the target is the caller's current, or the
///           init, user namespace; (CLONE_NEWPID) the target is not the
///           caller's active PID namespace or a descendant of it
/// - EPERM: Caller lacks CAP_SYS_ADMIN in the required user namespace
///          (target's owning/parent ns, and — for PID — the caller's own),
///          or is chrooted and joining a user namespace
/// - ENOENT: Namespace has been destroyed
fn sys_setns(fd: RawFd, nstype: c_int) -> Result<(), Errno> { ... }

PID namespace special case: A process cannot change its own PID namespace via setns() — the process's PID in its original namespace remains unchanged. However, setns(fd, CLONE_NEWPID) is valid since Linux 3.8: it sets the PID namespace for future children created by fork()/clone(). The caller's own PID view is unchanged, but newly created children will be in the target PID namespace.

This is why docker exec uses nsenter with --fork flag: it joins other namespaces via setns(), sets the target PID namespace for children, then forks a child that inherits all joined namespaces and has the correct PID view.

TOCTOU safety: setns() acquires a reference count on the target namespace before validating it, then holds that reference across the join operation. The namespace cannot be destroyed while setns() holds its reference — this prevents the use-after-free TOCTOU that would otherwise exist between checking namespace validity and joining it. The reference is released after the join completes or if validation fails.

Implementation:

/// True iff `target` is `active` or a DESCENDANT of it — the
/// setns(CLONE_NEWPID) reachability rule (Linux `pidns_is_ancestor(target,
/// active)`: `active` must be ancestor-or-equal of `target`). Walks
/// `target` up its strong `parent` chain until its level matches `active`'s;
/// they are related iff that ancestor IS `active`. Bounded by
/// the PID namespace depth limit (32).
fn pid_ns_is_self_or_descendant(
    target: &Arc<PidNamespace>,
    active: &Arc<PidNamespace>,
) -> bool {
    if target.level < active.level {
        return false;   // target is shallower — an ancestor, not reachable
    }
    let mut ns = Arc::clone(target);
    while ns.level > active.level {
        // Every non-root level (> 0) has a Some parent.
        let parent = Arc::clone(ns.parent.as_ref().expect("non-root PID ns has a parent"));
        ns = parent;
    }
    Arc::ptr_eq(&ns, active)
}

fn sys_setns(fd: RawFd, nstype: c_int) -> Result<()> {
    let file = current_task().files.get(fd)?;
    // nsfs single-namespace form: `fd` must name exactly ONE namespace. A
    // pidfd (the multi-namespace form) fails this NsInode downcast and
    // yields EINVAL — `setns(pidfd, mask)` is a documented Phase-3 milestone
    // (see "Multi-namespace setns via pidfd" below). This single-fd form has
    // NO intra-call ordering: it joins one namespace.
    let ns_inode = file.inode.downcast_ref::<NsInode>()
        .ok_or(Errno::EINVAL)?;

    // Verify nstype matches (if specified). The NsInode field is
    // `ns_type: NamespaceType` (kernel-internal enum); the syscall's
    // `nstype` is a CLONE_NEW* bitflag — translate before comparing.
    if nstype != 0
        && clone_flag_to_ns_type(nstype as u64) != Some(ns_inode.ns_type) {
        return Err(Errno::EINVAL);
    }
    // Exactly one namespace is joined here, so there is no intra-call
    // ordering to resolve — the user-first ordering the pidfd form needs
    // lives in setns_pidfd() above. (A pidfd was already dispatched away.)

    // Check CAP_SYS_ADMIN in target namespace's user namespace.
    // `user_ns()` is the Namespace TRAIT METHOD (returns
    // Weak<UserNamespace>) — `namespace` is `Arc<dyn Namespace>`, which
    // has no `user_ns` field.
    let target_user_ns = ns_inode.namespace.user_ns().upgrade().ok_or(Errno::ENOENT)?;
    if !has_ns_cap(current_task(), &target_user_ns, CAP_SYS_ADMIN) {
        return Err(Errno::EPERM);
    }

    // Join the namespace: clone the current NamespaceSet, replace the target
    // namespace field, and swap the task's `namespace_set` Arc to point to the
    // new set. This is per-task:
    // sibling threads are unaffected (they hold their own Arc<NamespaceSet>).
    let task = current_task();
    // RCU load + structural clone. This clones the entire NamespaceSet even
    // though setns() only modifies one field. The clone cost is ~8 Arc::clone
    // operations (one per namespace field) + SpinLock init for pending_pid_ns
    // and pending_time_ns. This is acceptable for setns() (cold path, ~1-10
    // calls per container lifecycle). A future optimization could use a
    // COW NamespaceSet that defers cloning until the second mutation, but
    // the complexity is not justified given the low call frequency.
    let old_ns = task.namespace_set.load().as_ref().clone();

    // Build a new NamespaceSet from the clone. All arms modify `new_ns`
    // in-place rather than using struct update syntax (`..old_ns`), which
    // avoids Rust borrow-checker issues: struct update moves the source,
    // making it incompatible with arms that need `old_ns` intact (TIME, PID).
    //
    // **SpinLock::Clone contract**: Cloning a `NamespaceSet` clones the
    // protected DATA inside each SpinLock field (e.g., pending_pid_ns,
    // pending_time_ns), not the lock state. The new `SpinLock` is a fresh,
    // unlocked instance protecting a clone of the inner value. Calling
    // `.lock().replace()` on `new_ns.pending_pid_ns` below acquires the
    // NEW lock (which is uncontended — `new_ns` is a local variable with
    // no concurrent accessors). The lock acquisition is technically
    // vacuous for the local variable, but it is required by the SpinLock
    // API and ensures the code compiles correctly with the same type
    // signatures used when operating on a shared NamespaceSet.
    let mut new_ns = old_ns;
    // CLONE_NEWNS records its fs.root/pwd reset here and applies it in
    // the commit tail, AFTER the fallible namespace_set_alloc() (see the Mnt
    // arm's ordering rationale).
    let mut pending_fs_root: Option<PathRef> = None;
    // CLONE_NEWIPC records its sem-undo detach the same way — the commit
    // tail runs detach_sysv_sem() once nothing can fail (see the Ipc arm).
    let mut pending_ipc_detach: bool = false;

    // Exhaustive match on the kernel-internal NamespaceType (the
    // NsInode field) — the CLONE_NEW* bitflags exist only at the syscall
    // boundary and were translated above.
    match ns_inode.ns_type {
        NamespaceType::Mnt => {   // CLONE_NEWNS
            // Mount namespace join requires CAP_SYS_ADMIN in the caller's
            // own user namespace AND CAP_SYS_CHROOT in the caller's own
            // user namespace (because joining a mount namespace resets
            // root/pwd, equivalent to a chroot). The CAP_SYS_ADMIN check
            // in the target namespace's user_ns was already performed above.
            // This matches Linux kernel behavior (verified via setns(2) man page).
            let caller_user_ns = &task.namespace_set.load().user_ns;
            if !has_ns_cap(current_task(), caller_user_ns, CAP_SYS_ADMIN) {
                return Err(Errno::EPERM);
            }
            if !has_ns_cap(current_task(), caller_user_ns, CAP_SYS_CHROOT) {
                return Err(Errno::EPERM);
            }

            // Switch to the target mount namespace. Entering a mount
            // namespace adopts its root: task.fs.root and task.fs.pwd are
            // reset to the target's root mount (Linux setns(2) behavior).
            //
            // The fs write is DEFERRED to the commit tail (below, after
            // the fallible `namespace_set_alloc()` succeeds) — record the
            // intent here only. Writing fs.root/pwd inside this arm and
            // then failing `namespace_set_alloc()` with ENOMEM would leave the
            // task in a mixed state: `namespace_set.mount_ns` still the OLD
            // namespace while root/pwd point at a mount absent from its
            // hash table — the exact cross-namespace path-resolution
            // corruption `copy_tree()` step 6's rationale forbids.
            //
            // Open file descriptors are NOT affected — they retain their
            // original dentry/vfsmount references (not re-resolved). Only
            // future path resolutions (open, stat, etc.) use the new root.
            let target_mnt = ns_inode.namespace.as_mnt_ns()
                .expect("MNT namespace");
            pending_fs_root = Some(target_mnt.root_mount());
            new_ns.mount_ns = target_mnt;
        }
        NamespaceType::Net => {   // CLONE_NEWNET
            let target_net = ns_inode.namespace.as_net_ns().expect("NET namespace");
            // Acquire an active user on the join target BEFORE releasing the
            // current net_ns, and refuse a namespace whose last task is
            // concurrently exiting. `net_ns_get_not_dead()` is the inc-not-zero
            // CAS: a plain `net_ns_get()` here could
            // revive a namespace whose `users` already reached 0 — one whose
            // `net_ns_cleanup()` already latched `dead` and gutted the stack —
            // and this task's later exit would then re-run cleanup a second
            // time. On failure the pending `new_ns` is simply discarded (no
            // shared state mutated yet), so early-return is safe.
            net_ns_get_not_dead(&target_net)?;
            // stack_cap is a OnceCell<CapHandle>, set exactly once in the
            // namespace's creation Phase 3 — always present for a live,
            // joinable namespace (guaranteed live by the successful acquire).
            let target_stack = *target_net.stack_cap.get()
                .expect("stack_cap set at namespace creation");
            // Release the cloned-in current net_ns now that the target is
            // pinned. The current namespace_set still holds its own user on the
            // released ns, so its `users` stays ≥ 1.
            net_ns_put(&new_ns.net_ns);
            new_ns.net_ns = target_net;
            new_ns.net_stack = target_stack;
        }
        NamespaceType::Uts => {   // CLONE_NEWUTS
            new_ns.uts_ns = ns_inode.namespace.as_uts_ns().expect("UTS namespace");
        }
        NamespaceType::Ipc => {   // CLONE_NEWIPC
            new_ns.ipc_ns = ns_inode.namespace.as_ipc_ns().expect("IPC namespace");
            // Record the sem-undo detach for the commit tail. Linux
            // Linux `commit_nsset()` runs `exit_sem(me)` for CLONE_NEWIPC
            // (`kernel/nsproxy.c`, verified against torvalds/linux master):
            // after the switch, the OLD namespace's semaphore sets are
            // unreachable from this task, so its accumulated SEM_UNDO
            // adjustments must be applied while they still resolve. Without
            // the detach they would sit stale until exit and then be
            // applied against a namespace the task long left — mutating
            // live sets shared with tasks that stayed behind, at an
            // arbitrary later time. Deferred (not run here) so a later
            // failure (namespace_set_alloc ENOMEM) leaves the caller's IPC state
            // untouched — the bare assignment above mutates only the local
            // staging `new_ns`.
            pending_ipc_detach = true;
        }
        NamespaceType::User => {  // CLONE_NEWUSER
            let target_user = ns_inode.namespace.as_user_ns().expect("USER namespace");
            // setns(CLONE_NEWUSER) gates — Linux `userns_install()`
            // (`kernel/user_namespace.c`, verified against torvalds/linux
            // master). All are EINVAL (structural), distinct from the EPERM
            // capability failure the generic CAP_SYS_ADMIN check above
            // already screened: because `UserNamespace::user_ns()` returns
            // the target's PARENT (see the `Namespace` trait impls below),
            // that generic check enforced "CAP_SYS_ADMIN in the target user
            // namespace's PARENT" — step 1 of the credential-transformation
            // prose below — so no separate parent check is needed here.
            //
            // (i) Cannot re-enter the namespace already in effect
            //     (Linux `user_ns == current_user_ns()` → EINVAL).
            if Arc::ptr_eq(&target_user, &task.namespace_set.load().user_ns) {
                return Err(Errno::EINVAL);
            }
            // (ii) Cannot enter the INIT user namespace: it has no parent,
            //      so "privileged in the parent" is undefined and every
            //      caller is already its descendant. This is also the gate
            //      that makes `user_ns_transform_ids`' init-target
            //      `expect()` unreachable (the transform re-expresses ids
            //      through `target.parent`, which the init namespace lacks).
            //      Normally the generic check above already returned EPERM
            //      for a descendant caller — `has_ns_cap(current, INIT_USER_NS,
            //      CAP_SYS_ADMIN)` is false unless the caller is IN init —
            //      but this keeps the reject authoritative and local.
            if target_user.parent.is_none() {
                return Err(Errno::EINVAL);
            }
            // (iii) Single-thread only: every thread shares `task.cred`, so a
            //       credential-rewriting user-namespace change is safe only
            //       when the caller is alone in its thread group. Multi-
            //       threaded callers must use `clone(CLONE_NEWUSER)`.
            //       (`thread_group.count` is the live-thread count; == 1
            //       means single-threaded — Linux `thread_group_empty()`.)
            if task.process.thread_group.count.load(Ordering::Acquire) > 1 {
                return Err(Errno::EINVAL);
            }
            // Chroot'd processes cannot join user namespaces (the fresh
            // full-capability grant could be used to escape the chroot).
            if task.is_chrooted() {
                return Err(Errno::EPERM);
            }
            // Update IMA namespace alongside user namespace. IMA measurement
            // policy is scoped per user namespace — switching user_ns without
            // updating ima_ns would cause IMA policy lookups to resolve against
            // the wrong namespace, potentially bypassing container-specific
            // integrity requirements or logging measurements to the wrong log.
            // The pairing lives in `UserNamespace.ima_ns` (OnceCell, set at
            // user-namespace creation — see the field doc below): this IS
            // the structural user-ns→IMA-ns association setns needs.
            let target_ima = target_user.ima_ns.get()
                .expect("ima_ns paired at user-ns creation").clone();

            // Atomicity: namespace_set.user_ns and task.cred MUST be updated together
            // under a single `TASK_LOCK(20)` hold. Without this, a window exists where
            // namespace_set.user_ns points to the new namespace but cred.user_ns still
            // references the old namespace (or vice versa). During that window,
            // has_ns_cap() checks would resolve against the wrong namespace,
            // potentially granting or denying capabilities incorrectly.
            //
            // Protocol (Solution B — true atomic swap):
            //   1. Prepare new credentials with cred.user_ns = target_user.
            //   2. Pre-allocate new namespace_set Arc BEFORE taking `TASK_LOCK(20)`.
            //   3. Under `TASK_LOCK(20)`: install_credentials(new_cred) AND swap namespace_set
            //      atomically. Both are O(1) pointer swaps, safe under spinlock.
            //   4. Drop old_namespace_set OUTSIDE lock scope (Arc refcount decrement
            //      may involve deallocation, which must not happen under spinlock).
            //
            // Invariant: install_credentials() must not sleep or acquire locks that
            // nest outside alloc_lock (confirmed: UmkaOS install_credentials is
            // rcu_assign_pointer only).
            //
            // Better than Linux: zero-width window. Linux accepts a brief
            // cred/namespace_set disagreement and relies on the soft invariant that
            // Linux `ns_capable()` reads cred.user_ns. Our approach is defense-in-depth.
            // stage_credentials() returns TrackedPtr<TaskCredential> (uniquely
            // owned, mutably deref-able, Nucleus tracked storage), so the
            // field writes below compile. install_credentials promotes it to Arc
            // via Arc::from_tracked at RCU publication. ENOMEM → setns fails.
            let mut new_cred = stage_credentials(&task.cred)?;
            new_cred.user_ns = target_user.clone();
            new_cred.cap_effective = CAP_FULL_SET;
            new_cred.cap_permitted = CAP_FULL_SET;
            new_cred.cap_inheritable = 0;
            new_cred.cap_bounding = CAP_FULL_SET;
            new_cred.cap_ambient = 0;
            // Identity: the CANONICAL user-namespace transformation
            // ([Section 17.1](#namespace-architecture--user-namespace-credential-transformation-canonical))
            // — deferred-translation form. When the target's maps are
            // already written (the common setns case: joining a running
            // container), this normalizes immediately: every POSIX id
            // field is translated parent→target and `ids_ns` = target.
            // Unmapped ids translate to 65534 (overflow), matching Linux
            // Linux overflow-substitution semantics (outer→inner). When the target's maps are NOT yet
            // written, the id VALUES are kept in parent-namespace terms
            // with `ids_ns` = target's parent — getuid() then reports
            // 65534 via read-boundary translation until the map is
            // written ([Section 9.9](09-security.md#credential-model-and-capabilities--deferred-translation-credentials-unmapped-user-namespaces)).
            user_ns_transform_ids(&mut new_cred, &target_user);

            new_ns.user_ns = target_user;
            new_ns.ima_ns = target_ima;

            // Pre-allocate namespace_set before taking the lock. namespace_set_alloc
            // moves new_ns into Nucleus tracked storage; ENOMEM → setns fails
            // (new_cred drops via TrackedPtr::drop — nothing published yet).
            let new_namespace_set = namespace_set_alloc(new_ns)?;
            let old_namespace_set;
            {
                let _guard = task.task_lock();
                // TrackedPtr -> Arc::from_tracked, RCU pointer swap, O(1)
                install_credentials(task, new_cred);
                // Task.namespace_set is ArcSwap<NamespaceSet> — interior mutability
                // allows mutation through &Task. The ArcSwap::store() is an atomic
                // pointer exchange (O(1)), safe under `TASK_LOCK(20)`.
                old_namespace_set = task.namespace_set.swap(new_namespace_set);
            }
            drop(old_namespace_set);  // Arc refcount decrement OUTSIDE lock
            return Ok(());      // early return — skip common namespace_set swap below
        }
        NamespaceType::Cgroup => {  // CLONE_NEWCGROUP
            new_ns.cgroup_ns = ns_inode.namespace.as_cgroup_ns()
                .expect("CGROUP namespace");
        }
        NamespaceType::Time => {  // CLONE_NEWTIME
            // Time namespace affects future children, not the caller (Linux 5.6+ semantics).
            // Set pending_time_ns so fork()/clone() children use the target time offsets.
            // pending_time_ns is SpinLock-protected — interior mutability.
            let target_time = ns_inode.namespace.as_time_ns().expect("TIME namespace");
            // SEAL the target's offsets at setns install — setter site 1 of 2
            // for `offsets_sealed` (Linux `timens_install()` sets
            // Linux `frozen_offsets = true` when a task joins an existing time
            // namespace, `kernel/time/namespace.c`, verified against
            // torvalds/linux master): once ANY joiner may compute clock
            // reads against these offsets, a later
            // /proc/[pid]/timens_offsets write must be refused (EACCES).
            // Idempotent set-once latch — re-joining an already-sealed
            // namespace just re-stores `true`.
            target_time.offsets_sealed.store(true, Ordering::Release);
            new_ns.pending_time_ns.lock().replace(target_time);
        }
        NamespaceType::Pid => {   // CLONE_NEWPID
            let target_pid = ns_inode.namespace.as_pid_ns().expect("PID namespace");
            // PID-namespace join gates — Linux `pidns_install()`
            // (`kernel/pid_namespace.c`, verified against torvalds/linux
            // master). NB: there is NO single-thread restriction here
            // (namespace_set is per-task, so sibling threads keep their own set —
            // see the error table); the multi-thread EINVAL is a
            // CLONE_NEWUSER property only.
            //
            // Dual CAP_SYS_ADMIN: the generic check above already required
            // it in the TARGET's owning user namespace; Linux additionally
            // requires it in the CALLER's OWN user namespace.
            let caller_user_ns = Arc::clone(&task.namespace_set.load().user_ns);
            if !has_ns_cap(current_task(), &caller_user_ns, CAP_SYS_ADMIN) {
                return Err(Errno::EPERM);
            }
            // Ancestor ordering: children's PID namespace may only be set to
            // the caller's active namespace or a DESCENDANT of it (Linux
            // Linux `pidns_is_ancestor(new, active)` → EINVAL otherwise) — future
            // children must be reachable from the caller's current namespace
            // so their numbers can be allocated at every intervening level.
            let active_pid = Arc::clone(&task.namespace_set.load().pid_ns);
            if !pid_ns_is_self_or_descendant(&target_pid, &active_pid) {
                return Err(Errno::EINVAL);
            }
            // PID namespace affects future children, not the caller. Set
            // pending_pid_ns so fork()/clone() creates children in the
            // target NS. pending_pid_ns is SpinLock-protected — interior
            // mutability.
            new_ns.pending_pid_ns.lock().replace(target_pid);
        }
        // No wildcard arm: NamespaceType is exhaustively covered (8
        // variants). An invalid nstype ARGUMENT was rejected above.
    };

    // Atomically replace the task's namespace_set under `TASK_LOCK(20)` to prevent
    // concurrent setns() races. Without locking, two concurrent setns()
    // calls (thread A sets NET, thread B sets UTS) could race — B's store
    // would overwrite A's net_ns change because both cloned from the same
    // old namespace_set. `TASK_LOCK(20)` serializes the entire clone-modify-swap.
    // namespace_set_alloc: Nucleus tracked storage; ENOMEM → setns fails —
    // and, because the fs update below has not yet run, the failure
    // leaves the task's path-resolution state fully in the OLD namespace
    // (no mixed state).
    let new_namespace_set = namespace_set_alloc(new_ns)?;
    // CLONE_NEWIPC deferred sem-undo detach — after the last fallible step,
    // OUTSIDE the `TASK_LOCK(20)` section (detach_sysv_sem takes each referenced set's
    // SemSet.lock and wakes its WaitQueue — neither belongs inside
    // TASK_LOCK(20)'s O(1) critical-section budget). Runs BEFORE the
    // namespace_set store, while the old namespace's sets are still this task's
    // resolvable view — Linux commit_nsset() order (exit_sem, then
    // publishing the new namespace set).
    if pending_ipc_detach {
        // task.namespace_set still names the OLD namespace (the store below has
        // not run) — the namespace whose sets the undos reference. detach_sysv_sem
        // takes it explicitly; it must not read task.namespace_set itself
        // ([Section 8.2](08-process.md#process-lifecycle-teardown--step-4c-sysv-semaphore-undo-operations)).
        detach_sysv_sem(&task.process, &task.namespace_set.load().ipc_ns);
    }
    let old_namespace_set;
    {
        let _guard = task.task_lock();
        // CLONE_NEWNS deferred fs reset — now that nothing can fail.
        // FS_STRUCT_LOCK(42) nests legally under TASK_LOCK(20).
        if let Some(new_root) = pending_fs_root {
            // Load-then-write-lock discipline: bind the ArcSwap guard to
            // a local so it outlives the RwLock guard. Both fields are
            // reset in one write section (readers see both-old or
            // both-new).
            let fs_arc = task.fs.load();
            let mut fs = fs_arc.write();
            fs.root = new_root.clone();
            fs.pwd = new_root;
        }
        // Swap under the lock, but do NOT drop the old set here: a
        // last-member Drop of the old Arc<NamespaceSet> may SLEEP (a final
        // CLONE_NEWNS join umounts the old mount ns's tree; a final
        // CLONE_NEWIPC join takes each SemSet.lock and wakes its WaitQueue —
        // see the NamespaceSet Drop table), which is illegal inside
        // TASK_LOCK(20)'s spinlocked O(1) critical section. Defer the drop
        // to outside the guard, mirroring the CLONE_NEWUSER arm above and
        // unshare step 5.
        old_namespace_set = task.namespace_set.swap(new_namespace_set);
    }
    drop(old_namespace_set);  // Arc refcount decrement OUTSIDE lock — Drop may sleep

    Ok(())
}

Multi-namespace setns via pidfd (Phase 3 milestone):

setns(pidfd, mask) — where fd is a pidfd rather than an nsfs fd and mask is an OR of CLONE_NEW* bits — joins EVERY one of the target process's namespaces named in mask in a single call (Linux 5.8+; nsenter --all -t PID). It is the only setns form that joins more than one namespace, and thus the only one with intra-call ordering. The single-fd form above is complete; this multi-namespace form is deferred to Phase 3 (util-linux falls back to per-namespace setns() calls when it is unavailable, so the compatibility gap is bounded). Until then a pidfd fails the NsInode downcast in sys_setns and yields EINVAL. Its specification, for the implementing agent:

Mask validation (Linux check_setns_flags()): reject mask == 0 or any bit outside the OR of the eight CLONE_NEW* bits with EINVAL.
Target: resolve the pidfd to its Arc<Process>; the source namespaces are the group leader's namespace_set snapshot, kept alive across the joins by the pidfd's Arc<Process> (the TOCTOU safety the nsfs file gives the single-fd form).
Two-phase, user-first: stage every selected namespace into ONE accumulator (a clone of the caller's NamespaceSet, at most one prepared credential, and a deferred fs.root reset), applying CLONE_NEWUSER FIRST so the subsequent joins' CAP_SYS_ADMIN gates observe the new credential, then commit the whole accumulator ONCE under TASK_LOCK(20) (install_credentials if a cred was prepared + one namespace_set.swap + the deferred fs reset, and — when mask includes CLONE_NEWIPC — the same deferred detach_sysv_sem sem-undo detach as the single-fd Ipc arm, run at the commit exactly as Linux commit_nsset() does, outside the task_lock section). The old Arc<NamespaceSet> returned by that namespace_set.swap is dropped OUTSIDE the TASK_LOCK(20) section for the same reason as the single-fd form: a last-member Drop may sleep. A permission failure on ANY namespace aborts BEFORE commit, leaving the caller entirely in its original set (Linux Linux validate_nsset() → commit_nsset(): no partial application). Each per-type stage runs the SAME gates and field swap as the matching single-fd match arm above, sourced from the target's set instead of an NsInode. Fixed canonical order (Linux validate_nsset()): user, mnt, uts, ipc, pid, cgroup, net, time.

chroot + setns interaction:

When setns(CLONE_NEWNS) switches the task's mount namespace, task.fs.root and task.fs.pwd are reset to the new namespace's root mount (lines above). This effectively escapes any previous chroot boundary -- the task's root is now the new namespace's root, not the chroot directory. This is Linux-compatible behavior (man 2 setns: "A process reassociating itself with a new mount namespace... will have its root and current working directory reset to the root of the mount namespace"). For user namespaces, setns(CLONE_NEWUSER) is denied for chrooted tasks (returns EPERM) to prevent chroot escapes via user namespace capability grants.

setns(CLONE_NEWUSER) credential transformation:

When setns(fd, CLONE_NEWUSER) is called, the kernel performs a credential transformation to grant the caller capabilities within the target user namespace:

Validation: Verify the caller has CAP_SYS_ADMIN in the target user namespace's parent namespace. This prevents unprivileged users from joining arbitrary user namespaces — only a process that is already privileged in the parent context can adopt a child user namespace's identity.

Credential update (install_credentials path — used by nsenter --user): When setns(CLONE_NEWUSER) is called directly (not via pending+fork), the kernel performs an immediate credential transformation — the CANONICAL transformation shared by clone(CLONE_NEWUSER), unshare(CLONE_NEWUSER), and this path (see Section 17.1):

// stage_credentials() -> TrackedPtr<TaskCredential> (uniquely owned,
// DerefMut, Nucleus tracked storage). ENOMEM → setns fails.
let mut new_cred = stage_credentials(&current_task().cred)?;
new_cred.user_ns = target_ns.clone();
// Grant full capability set within the target namespace.
// These capabilities are namespace-scoped: has_ns_cap() checks
// resolve against cred.user_ns, so CAP_FULL_SET here does NOT
// grant capabilities in the parent or init namespace.
new_cred.cap_effective = CAP_FULL_SET;
new_cred.cap_permitted = CAP_FULL_SET;
new_cred.cap_inheritable = 0;
// Reset bounding set to full — all capabilities are permitted
// in the new namespace (matching Linux behavior).
new_cred.cap_bounding = CAP_FULL_SET;
// Clear ambient capabilities — ambient caps do not cross user
// namespace boundaries (matching Linux 4.8+ behavior).
new_cred.cap_ambient = 0;
// Identity (canonical deferred-translation rule): if the target's
// maps are written, every POSIX id field is translated
// parent→target (unmapped → 65534 overflow, Linux from_kuid_munged
// semantics, outer→inner) and ids_ns = target; if unwritten, the id VALUES stay
// in parent terms with ids_ns = target's parent, and the 65534
// view is produced at the read boundary
// ([Section 9.9](09-security.md#credential-model-and-capabilities--deferred-translation-credentials-unmapped-user-namespaces)).
user_ns_transform_ids(&mut new_cred, &target_ns);
install_credentials(current_task(), new_cred);

Multi-threaded restriction: setns(CLONE_NEWUSER) fails with EINVAL if the calling process has more than one thread (thread_group_count > 1). This matches Linux behavior: changing the user namespace affects credential resolution for all threads (they share task.cred), so it is only safe when the process is single-threaded. Multi-threaded processes must use clone(CLONE_NEWUSER) to create a child in the new namespace instead.

Fork/clone consumption of pending namespaces:

When fork() or clone() creates a child process, it reads (but does NOT consume) the pending PID and time namespaces. The lock().clone() pattern ensures that concurrent setns() + clone() in a multi-threaded process cannot race. The pending value is NOT consumed on fork -- this matches Linux's pid_ns_for_children semantics where unshare(CLONE_NEWPID) affects ALL future children, not just the next one. The pending value is only cleared by a subsequent setns() that replaces it, or by the parent's exit:

// In NamespaceSet::clone_for_fork() during fork/clone:
let pending_pid = current_task().namespace_set.pending_pid_ns.lock().clone();
let child_pid_ns = pending_pid.unwrap_or_else(|| Arc::clone(&current_task().namespace_set.pid_ns));

let pending_time = current_task().namespace_set.pending_time_ns.lock().clone();
let child_time_ns = pending_time.unwrap_or_else(|| Arc::clone(&current_task().namespace_set.time_ns));

// SEAL on first entry — setter site 2 of 2 for `offsets_sealed` (Linux
// Linux `timens_on_fork()`: a child created into a time namespace DIFFERENT from
// its parent's freezes that namespace's offsets, `kernel/time/namespace.c`,
// verified against torvalds/linux master). The unshare(CLONE_NEWTIME) →
// write /proc/[pid]/timens_offsets → fork() sequence works BECAUSE the seal
// happens here, at consumption, not at unshare time. Idempotent set-once
// latch; the != guard keeps plain same-namespace forks from touching the
// parent namespace's flag.
if !Arc::ptr_eq(&child_time_ns, &current_task().namespace_set.time_ns) {
    child_time_ns.offsets_sealed.store(true, Ordering::Release);
}

17.1.5 Namespace Hierarchy and Inheritance¶

Namespaces form a hierarchical tree with parent-child relationships. When a process creates a new namespace via clone() or unshare(), the new namespace is a child of the caller's namespace:

Root Namespace (init)
  ├── PID NS 1 (container A)          ← child of root PID NS
  │   └── PID NS 1.1 (nested container) ← child of PID NS 1
  ├── PID NS 2 (container B)          ← child of root PID NS
  └── User NS 1 (unprivileged container)
      └── User NS 1.1 (child of User NS 1)

Parent-child link semantics:

Hierarchical namespaces track their parent with a per-type STRONG Option<Arc<T>> field — UserNamespace.parent: Option<Arc<UserNamespace>> and PidNamespace.parent: Option<Arc<PidNamespace>> (None only at the root). There is NO generic NamespaceHierarchy struct and NO parent→children back-list: a parent does not track its children, and a child PINS its parent alive. This is Linux parity (get_user_ns() / get_pid_ns() on the parent) and the correct model:

A child CANNOT outlive its parent. Because the child holds a strong Arc to the parent, the parent's refcount stays ≥ 1 for the child's whole life; is_same_or_ancestor() therefore safely walks parent links without any upgrade() (the earlier "children become orphans when the parent is dropped" model was WRONG for these strong edges and is retired). A parent CAN outlive its children (it does not reference them).
No cycle. The parent does not reference the child, so the strong child→parent edge is one-way; the chain is bounded by the nesting limit (32), so ancestor walks and the child→parent Arc drop chain are bounded.
Flat namespaces (net, uts, ipc, mnt, cgroup, time) have NO parent/child hierarchy at all — they are independent instances; only their owning user_ns link relates them to the user-namespace tree.

Namespace destruction remains independent and Drop-based (below): each namespace type's impl Drop runs when its own Arc refcount reaches zero. The strong parent edge simply means a hierarchical child's drop is what eventually releases its parent reference — never the reverse.

/// Inode backing `/proc/[pid]/ns/*` entries in the nsfs pseudo-filesystem.
///
/// Each namespace type is exposed as a magic symlink under `/proc/[pid]/ns/`.
/// Opening such a symlink returns a file descriptor backed by an `NsInode`.
/// This fd can be passed to `setns(fd, nstype)` to join the namespace, or
/// held open to keep the namespace alive (preventing destruction even after
/// all member processes have exited).
///
/// The nsfs pseudo-filesystem is mounted internally at boot and is not
/// visible in the mount tree. Its sole purpose is to provide inode objects
/// for namespace file descriptors.
pub struct NsInode {
    /// Namespace type (Pid, Net, Mnt, Uts, Ipc, User, Cgroup, Time).
    /// Used by `setns()` to verify the `nstype` argument matches the fd.
    pub ns_type: NamespaceType,
    /// Reference to the actual namespace object. Downcasted to the concrete
    /// type (`Arc<NetNamespace>`, `Arc<PidNamespace>`, etc.) by `setns()`
    /// and other namespace operations. Holding this `Arc` keeps the namespace
    /// OBJECT alive as long as the fd (or a bind mount of it) is open. For
    /// every type except Net, object liveness IS namespace liveness. For
    /// `NamespaceType::Net` the `Arc` alone is only the PASSIVE (memory)
    /// reference — administrative liveness is the separate `users` count —
    /// so the inode ALSO holds one ACTIVE user (the active-user contract
    /// below this struct).
    pub namespace: Arc<dyn Namespace>,
    /// Inode number = namespace ID (unique u64, assigned at namespace creation
    /// from a global atomic counter). This is the value returned by `stat()`
    /// on `/proc/[pid]/ns/*` and used by `lsns(1)` to identify namespaces.
    pub ino: u64,
}

/// nsfs inodes are STASHED per namespace (Linux `ns->stashed`): the first
/// open of `/proc/[pid]/ns/<type>` creates the inode; subsequent opens and
/// bind mounts of the same namespace share it; it is freed when its last
/// reference (fd or bind mount) drops. One inode per namespace, never one
/// per open — so the active-user accounting below is per namespace-inode
/// lifetime, not per fd.
///
/// **Net namespaces: the nsfs inode holds one ACTIVE user.** An
/// `Arc<NetNamespace>` alone is passive — it keeps the MEMORY alive but not
/// the STACK: at `users → 0` the namespace is gutted (`net_ns_cleanup()`
/// force-closes every socket and removes every device) regardless of how
/// many passive holders remain. Without an active user here, `ip netns add`
/// — a bind mount of a transient process's `/proc/self/ns/net` — would pin
/// an empty but DEAD namespace: every namespace created that way would stop
/// functioning the moment its creating process exited. The nsfs inode for a
/// Net namespace therefore participates in the ACTIVE count (Linux's nsfs
/// likewise takes an active reference at nsfs-inode creation, so a
/// bind-mounted namespace stays live past its creator's exit, `fs/nsfs.c`,
/// verified against torvalds/linux master):
///
/// - **Acquire — at inode creation**: the nsfs-open path for a Net
///   namespace calls `net_ns_get_not_dead(&net)?` BEFORE publishing the
///   inode; `ENOENT` ("Namespace has been destroyed") if the namespace's
///   last active user is concurrently exiting — a dead network namespace is
///   not re-openable. (Deliberate divergence: Linux master can RESURRECT a
///   pinned-but-inactive netns subtree via `SIOCGSKNS` on a surviving
///   socket. UmkaOS cannot and must not — its active teardown is
///   destructive (sockets closed, devices removed), so a gutted stack has
///   nothing to resurrect, and UmkaOS socket references are PASSIVE by
///   design (`socket_list` doc above), so the Linux resurrection case has
///   no analogue.)
/// - **Release — at inode teardown**: `impl Drop for NsInode` below.
///   Exactly one `net_ns_put()` per inode lifetime, symmetric with the
///   acquire — this is the release site whose absence made the active
///   count unreturnable-to-zero for any namespace ever opened via nsfs.
impl Drop for NsInode {
    fn drop(&mut self) {
        // Release the active user the nsfs-open path acquired for a Net
        // namespace. The 1→0 transition inside net_ns_put() only latches
        // `dead` and enqueues the deferred cleanup work — an atomic store
        // plus a non-blocking submission, safe from any drop context.
        if self.ns_type == NamespaceType::Net {
            if let Some(net) = self.namespace.as_net_ns() {
                net_ns_put(&net);
            }
        }
    }
}

/// Trait implemented by all namespace types. Provides type identification
/// and the unique namespace ID for nsfs inode generation.
// `use core::any::Any;` — the `Any` supertrait provides the runtime type
// identity that makes the typed downcasts below possible.
pub trait Namespace: Any + Send + Sync {
    /// Returns the type of this namespace.
    fn ns_type(&self) -> NamespaceType;
    /// Returns the unique namespace ID (same value as the nsfs inode number).
    fn id(&self) -> u64;
    /// Returns the OWNING user namespace as a `Weak` — the capability-check
    /// target in `setns()`. For every namespace type this is the user ns the
    /// namespace was created in, EXCEPT `UserNamespace`, whose owner is its
    /// PARENT (Linux `userns_owner()` returns `ns->parent`, `kernel/user_namespace.c`);
    /// the parentless init user namespace returns a weak handle to itself so
    /// the check resolves (a descendant caller gets EPERM, never a dangling
    /// upgrade). This is precisely what makes the ONE generic setns CAP check
    /// enforce "CAP_SYS_ADMIN in the target's PARENT" for CLONE_NEWUSER.
    fn user_ns(&self) -> Weak<UserNamespace>;
    /// Type-erasing upcast for downcasting. Every concrete impl is the
    /// trivial `{ self }` — the compiler inserts the unsizing coercion
    /// `Arc<ConcreteNs> → Arc<dyn Any + Send + Sync>`. An `Arc<Self>`
    /// receiver is object-safe (`Arc: DispatchFromDyn`), so this is callable
    /// on `Arc<dyn Namespace>`; the vtable dispatches to the concrete impl,
    /// which unsizes from its OWN type (no `dyn Namespace → dyn Any` trait
    /// upcast is involved).
    fn into_any_arc(self: Arc<Self>) -> Arc<dyn Any + Send + Sync>;
}

/// Typed downcasts for the `Arc<dyn Namespace>` that `NsInode.namespace`
/// holds. `setns()` and other nsfs consumers call these to recover the
/// concrete `Arc<ConcreteNs>` (returning `None` on a type mismatch). Each
/// clones the fat pointer, upcasts to `Arc<dyn Any>` via `into_any_arc()`,
/// and applies `Arc::downcast`. A trait method on `&self` CANNOT produce an
/// owned `Arc<ConcreteNs>` (no namespace struct stores a self-`Weak`), which
/// is why these live here, on the `Arc`, rather than on the `Namespace`
/// trait. The methods resolve on `NsInode.namespace` because it is exactly
/// `Arc<dyn Namespace>`.
pub trait NamespaceDowncast {
    fn as_mnt_ns(&self) -> Option<Arc<MountNamespace>>;
    fn as_pid_ns(&self) -> Option<Arc<PidNamespace>>;
    fn as_net_ns(&self) -> Option<Arc<NetNamespace>>;
    fn as_uts_ns(&self) -> Option<Arc<UtsNamespace>>;
    fn as_ipc_ns(&self) -> Option<Arc<IpcNamespace>>;
    fn as_user_ns(&self) -> Option<Arc<UserNamespace>>;
    fn as_cgroup_ns(&self) -> Option<Arc<CgroupNamespace>>;
    fn as_time_ns(&self) -> Option<Arc<TimeNamespace>>;
}

impl NamespaceDowncast for Arc<dyn Namespace> {
    fn as_mnt_ns(&self) -> Option<Arc<MountNamespace>> {
        Arc::clone(self).into_any_arc().downcast::<MountNamespace>().ok()
    }
    fn as_pid_ns(&self) -> Option<Arc<PidNamespace>> {
        Arc::clone(self).into_any_arc().downcast::<PidNamespace>().ok()
    }
    fn as_net_ns(&self) -> Option<Arc<NetNamespace>> {
        Arc::clone(self).into_any_arc().downcast::<NetNamespace>().ok()
    }
    fn as_uts_ns(&self) -> Option<Arc<UtsNamespace>> {
        Arc::clone(self).into_any_arc().downcast::<UtsNamespace>().ok()
    }
    fn as_ipc_ns(&self) -> Option<Arc<IpcNamespace>> {
        Arc::clone(self).into_any_arc().downcast::<IpcNamespace>().ok()
    }
    fn as_user_ns(&self) -> Option<Arc<UserNamespace>> {
        Arc::clone(self).into_any_arc().downcast::<UserNamespace>().ok()
    }
    fn as_cgroup_ns(&self) -> Option<Arc<CgroupNamespace>> {
        Arc::clone(self).into_any_arc().downcast::<CgroupNamespace>().ok()
    }
    fn as_time_ns(&self) -> Option<Arc<TimeNamespace>> {
        Arc::clone(self).into_any_arc().downcast::<TimeNamespace>().ok()
    }
}

// The eight concrete implementations. `into_any_arc` is always `{ self }`;
// `id()` returns `self.ns_id`; `user_ns()` returns the owning user ns as a
// Weak — cloning a `Weak` field (Uts/Cgroup/Time), downgrading a strong
// `Arc` field (Mnt/Net/Ipc/Pid), or, for UserNamespace, downgrading the
// PARENT (init → self).
impl Namespace for MountNamespace {
    fn ns_type(&self) -> NamespaceType { NamespaceType::Mnt }
    fn id(&self) -> u64 { self.ns_id }
    fn user_ns(&self) -> Weak<UserNamespace> { Arc::downgrade(&self.user_ns) }
    fn into_any_arc(self: Arc<Self>) -> Arc<dyn Any + Send + Sync> { self }
}
impl Namespace for PidNamespace {
    fn ns_type(&self) -> NamespaceType { NamespaceType::Pid }
    fn id(&self) -> u64 { self.ns_id }
    fn user_ns(&self) -> Weak<UserNamespace> { Arc::downgrade(&self.user_ns) }
    fn into_any_arc(self: Arc<Self>) -> Arc<dyn Any + Send + Sync> { self }
}
impl Namespace for NetNamespace {
    fn ns_type(&self) -> NamespaceType { NamespaceType::Net }
    fn id(&self) -> u64 { self.ns_id }
    fn user_ns(&self) -> Weak<UserNamespace> { Arc::downgrade(&self.user_ns) }
    fn into_any_arc(self: Arc<Self>) -> Arc<dyn Any + Send + Sync> { self }
}
impl Namespace for UtsNamespace {
    fn ns_type(&self) -> NamespaceType { NamespaceType::Uts }
    fn id(&self) -> u64 { self.ns_id }
    fn user_ns(&self) -> Weak<UserNamespace> { self.user_ns.clone() }
    fn into_any_arc(self: Arc<Self>) -> Arc<dyn Any + Send + Sync> { self }
}
impl Namespace for IpcNamespace {
    fn ns_type(&self) -> NamespaceType { NamespaceType::Ipc }
    fn id(&self) -> u64 { self.ns_id }
    fn user_ns(&self) -> Weak<UserNamespace> { Arc::downgrade(&self.user_ns) }
    fn into_any_arc(self: Arc<Self>) -> Arc<dyn Any + Send + Sync> { self }
}
impl Namespace for CgroupNamespace {
    fn ns_type(&self) -> NamespaceType { NamespaceType::Cgroup }
    fn id(&self) -> u64 { self.ns_id }
    fn user_ns(&self) -> Weak<UserNamespace> { self.user_ns.clone() }
    fn into_any_arc(self: Arc<Self>) -> Arc<dyn Any + Send + Sync> { self }
}
impl Namespace for TimeNamespace {
    fn ns_type(&self) -> NamespaceType { NamespaceType::Time }
    fn id(&self) -> u64 { self.ns_id }
    fn user_ns(&self) -> Weak<UserNamespace> { self.user_ns.clone() }
    fn into_any_arc(self: Arc<Self>) -> Arc<dyn Any + Send + Sync> { self }
}
impl Namespace for UserNamespace {
    fn ns_type(&self) -> NamespaceType { NamespaceType::User }
    fn id(&self) -> u64 { self.ns_id }
    /// A user namespace is OWNED by its parent (Linux `userns_owner()`); the
    /// parentless init user namespace is self-owned and returns a weak handle
    /// to the global `INIT_USER_NS` so the setns CAP check resolves (a
    /// descendant caller lacks CAP over init → EPERM) rather than dangling.
    fn user_ns(&self) -> Weak<UserNamespace> {
        match &self.parent {
            Some(p) => Arc::downgrade(p),
            None => Arc::downgrade(INIT_USER_NS.get()
                .expect("INIT_USER_NS set at boot")),
        }
    }
    fn into_any_arc(self: Arc<Self>) -> Arc<dyn Any + Send + Sync> { self }
}

17.1.5.1 nsfs ioctls (`NS_GET_*`)¶

Namespace fds are not only setns() handles — lsns(1), util-linux nsenter, CRIU, and systemd introspect them through an ioctl family on the nsfs file itself (ioctl_ns(2)). The request codes are ABI (magic NSIO = 0xb7, include/uapi/linux/nsfs.h, verified against torvalds/linux master) — including the historical quirk that the four original requests use the _IO (no-payload) encoding even though NS_GET_OWNER_UID writes a uid_t through arg. The NsInode file's ioctl file operation dispatches them; an unrecognized request returns ENOTTY.

/// nsfs ioctl magic (Linux `NSIO`).
pub const NSIO: u32 = 0xb7;

/// New fd for the OWNING user namespace of the fd's namespace.   _IO(NSIO, 0x1)
pub const NS_GET_USERNS: u32          = 0x0000_B701;
/// New fd for the PARENT namespace (user/pid only).              _IO(NSIO, 0x2)
pub const NS_GET_PARENT: u32          = 0x0000_B702;
/// Returns the namespace's CLONE_NEW* constant as the ioctl
/// return value.                                                 _IO(NSIO, 0x3)
pub const NS_GET_NSTYPE: u32          = 0x0000_B703;
/// Owner uid of a USER namespace, written through `arg` as a
/// `*mut uid_t` (encoded `_IO`, not `_IOR` — Linux ABI quirk;
/// keep exact).                                                  _IO(NSIO, 0x4)
pub const NS_GET_OWNER_UID: u32       = 0x0000_B704;
/// Mount-namespace 64-bit id through `arg: *mut u64`.     _IOR(NSIO, 5, __u64)
pub const NS_GET_MNTNS_ID: u32        = 0x8008_B705;
/// PID-translation quartet (pid-namespace fds only; `arg` carries
/// the pid to translate; the result is the ioctl return value).
pub const NS_GET_PID_FROM_PIDNS: u32  = 0x8004_B706; // _IOR(NSIO, 0x6, int)
pub const NS_GET_TGID_FROM_PIDNS: u32 = 0x8004_B707; // _IOR(NSIO, 0x7, int)
pub const NS_GET_PID_IN_PIDNS: u32    = 0x8004_B708; // _IOR(NSIO, 0x8, int)
pub const NS_GET_TGID_IN_PIDNS: u32   = 0x8004_B709; // _IOR(NSIO, 0x9, int)
/// Mount-namespace enumeration (extensible-struct ioctls: the
/// `_IOC_SIZE` bits carry the CALLER's struct size, which may
/// exceed 16 as the struct grows — dispatch matches on `_IOC_NR`
/// and requires `size >= MNT_NS_INFO_SIZE_VER0`).
pub const NS_MNT_GET_INFO: u32        = 0x8010_B70A; // _IOR(NSIO, 10, mnt_ns_info)
pub const NS_MNT_GET_NEXT: u32        = 0x8010_B70B; // _IOR(NSIO, 11, mnt_ns_info)
pub const NS_MNT_GET_PREV: u32        = 0x8010_B70C; // _IOR(NSIO, 12, mnt_ns_info)
/// The fd's namespace id (u64 = `NsInode.ino` = `ns_id`), any type.
pub const NS_GET_ID: u32              = 0x8008_B70D; // _IOR(NSIO, 13, __u64)

/// Userspace-ABI result struct for the `NS_MNT_GET_*` ioctls (Linux
/// `struct mnt_ns_info`; `MNT_NS_INFO_SIZE_VER0` = 16). Extensible: the
/// kernel fills `min(caller size, current size)` bytes and stamps `size`
/// with the size it filled.
#[repr(C)]
pub struct MntNsInfo {
    /// Size the kernel filled in (VER0 = 16).
    pub size: u32,
    /// Number of mounts currently in the namespace.
    pub nr_mounts: u32,
    /// The mount namespace's 64-bit id (same value as NS_GET_MNTNS_ID).
    pub mnt_ns_id: u64,
}
const_assert!(size_of::<MntNsInfo>() == 16); // MNT_NS_INFO_SIZE_VER0

Dispatch semantics (Linux fs/nsfs.c ns_ioctl(), verified against torvalds/linux master):

Request	Applies to	Effect	Errors
`NS_GET_USERNS`	any	Opens a NEW `O_RDONLY \\| O_CLOEXEC` nsfs fd for the namespace's OWNING user namespace — the `Namespace::user_ns()` target (for a user-namespace fd this is its PARENT, Linux `userns_owner()` parity).	`EPERM` if the owner is outside the caller's scope (see the scope gate below); `EMFILE`/`ENFILE` fd-table limits.
`NS_GET_PARENT`	user, pid	New nsfs fd for the parent namespace (walks the strong `parent` edge).	`EINVAL` for non-hierarchical types (net/mnt/uts/ipc/cgroup/time — no `parent`); `EPERM` if the fd names the root namespace (no parent) or the parent is outside the caller's scope.
`NS_GET_NSTYPE`	any	Returns the `CLONE_NEW*` constant for `ns_type` as the ioctl RETURN VALUE (the syscall-boundary inverse of `clone_flag_to_ns_type()`).	—
`NS_GET_OWNER_UID`	user only	Writes the namespace's `owner_uid`, RE-EXPRESSED in the CALLER's user namespace, through `arg` (Linux `from_kuid_munged(current_user_ns(), owner)`). `owner_uid` is stored in the target's PARENT's terms: walk it up to init-absolute via each ancestor's `map_uid_to_parent()`, then down into the caller's namespace via the caller-chain `map_uid_from_parent()` hops; ANY unmapped hop → the overflow uid (65534).	`EINVAL` if the fd is not a user namespace.
`NS_GET_ID`	any	Writes the u64 `ns_id` (== nsfs inode number) through `arg`.	—
`NS_GET_MNTNS_ID`	mnt only	Same u64 id write, mount namespaces only.	`EINVAL` otherwise.
`NS_GET_PID_FROM_PIDNS` / `NS_GET_TGID_FROM_PIDNS`	pid only	`arg` = a pid IN THE FD's namespace; returns that task's (or its thread-group leader's) pid in the CALLER's active PID namespace. Both translations use the captured-chain walk (`pid_nr_in()`), never `task.namespace_set`.	`EINVAL` non-pid fd; `ESRCH` if no such task, or the task is invisible in the destination namespace (translation returned no link).
`NS_GET_PID_IN_PIDNS` / `NS_GET_TGID_IN_PIDNS`	pid only	Inverse direction: `arg` = a pid in the CALLER's namespace; returns the pid in the FD's namespace.	Same.
`NS_MNT_GET_INFO`	mnt only	Fills the caller's `MntNsInfo`.	`EINVAL` non-mnt fd, null `arg`, or caller size < 16.
`NS_MNT_GET_NEXT` / `NS_MNT_GET_PREV`	mnt only	Opens an nsfs fd for the next/previous mount namespace in `ns_id` order (a kernel-wide walk across FOREIGN namespaces), optionally filling `MntNsInfo` when `arg` is non-null.	`EPERM` unless the caller passes the see-all gate below; `ENOENT` at either end of the list; `EINVAL` size < 16.

Two privilege gates, both Linux-exact:

Scope gate (NS_GET_USERNS, NS_GET_PARENT): the returned namespace's owning user namespace must satisfy is_same_or_ancestor(caller_user_ns, owner) — i.e. the caller's user namespace must be the owner or an ancestor of it (Linux ns_get_owner()'s parent-walk from the owner to Linux current_user_ns()), else EPERM ("requested namespace is outside of the caller's namespace scope", ioctl_ns(2)). This is what stops a container from minting fds for namespaces it could never setns() into anyway.
See-all gate (NS_MNT_GET_NEXT/PREV only): enumerating FOREIGN mount namespaces requires visibility over all of them — the caller's ACTIVE PID namespace must be the init PID namespace AND the caller must hold CAP_SYS_ADMIN over the init user namespace (Linux Linux may_see_all_namespaces() requires task_active_pid_ns(current) == &init_pid_ns and Linux ns_capable_noaudit(init_pid_ns.user_ns, CAP_SYS_ADMIN), kernel/nscommon.c, verified against torvalds/linux master), else EPERM. Per-namespace ioctls on an fd the caller already legitimately holds are NOT gated this way.

The fd-returning requests (NS_GET_USERNS, NS_GET_PARENT, NS_MNT_GET_NEXT/PREV) go through the same nsfs-open path as /proc/[pid]/ns/*: they resolve or create the STASHED per-namespace NsInode — so a returned Net-namespace fd would acquire its active user exactly like a proc-path open (moot today: neither request can produce a Net fd — net is non-hierarchical and its owner is a user namespace — but the invariant is stated so a future fd-returning request cannot bypass it).

Inheritance rules:

Namespace Type	Child Inherits	Modification Scope
`CLONE_NEWPID`	No (child starts fresh with PID 1)	Child's PID 1 = child init process
`CLONE_NEWNET`	No (child gets isolated network stack)	Child has no interfaces except loopback
`CLONE_NEWNS`	Yes (copy-on-write mount tree)	Child's mounts are private unless marked shared
`CLONE_NEWUTS`	Yes (copies parent's hostname/domainname)	Container runtimes typically overwrite via `sethostname()`
`CLONE_NEWIPC`	No (child gets empty IPC namespace)	Child has isolated SysV/POSIX IPC
`CLONE_NEWUSER`	No (child starts with empty UID/GID mappings)	Parent must write `/proc/PID/uid_map` and `gid_map` to grant subordinate ranges
`CLONE_NEWCGROUP`	No (child gets own cgroup root)	Child's cgroup is a child of caller's cgroup
`CLONE_NEWTIME`	No (child gets zero offsets)	Child's time offsets are independent

Note on CLONE_NEWUTS: The child namespace initially inherits the parent's hostname and domainname (copy, not reference). Container runtimes (runc, containerd) typically overwrite this immediately with the container ID via sethostname().

Note on CLONE_NEWUSER: A newly created user namespace starts with empty UID/GID mappings — all UIDs/GIDs resolve to nobody/nogroup (65534) until mappings are written to /proc/PID/uid_map and /proc/PID/gid_map by a privileged process in the parent namespace. This is a critical security property: children do not automatically inherit the parent's full UID range. Instead, the parent explicitly grants a subordinate range (typically from /etc/subuid and /etc/subgid).

User namespace nesting limit: User namespaces can be nested to a maximum depth of 32 (matching Linux's compile-time limit). clone() or unshare() with CLONE_NEWUSER returns ENOSPC if the nesting depth would exceed 32. This prevents resource exhaustion attacks via deeply nested namespaces.

Namespace reference counting: Each namespace is reference-counted via Arc<Namespace>. A namespace is destroyed when: 1. All processes in the namespace have exited (process count → 0) 2. All file descriptors referring to /proc/PID/ns/* are closed 3. All bind mounts of the namespace file have been unmounted

Reference chain for bind-mounted namespace files: When a namespace file (/proc/PID/ns/net, etc.) is bind-mounted to keep the namespace alive, the reference chain is: Mount → Dentry → NsInode → Arc<Namespace>. Each link holds an Arc reference. Unmounting drops the Mount, which drops the Dentry reference, which drops the NsInode, which decrements the Arc<Namespace> refcount. Only when ALL three conditions above are met does the refcount reach zero, triggering the destruction protocol below. For a Net namespace the NsInode link additionally holds one ACTIVE user (the nsfs active-user contract above), released by impl Drop for NsInode at the end of this chain — so an nsfs fd or bind mount keeps the network namespace FUNCTIONAL (sockets creatable, devices movable in), not merely allocated, until it is closed/unmounted. This is what makes ip netns add persistence work: the bind mount under /run/netns/ holds the empty namespace's sole active user after the creating process exits.

Note: Namespace destruction is per-refcount, but the parent/child relationship differs by type. For the HIERARCHICAL namespaces (user, pid) a child holds a STRONG Arc to its parent, so a parent CANNOT be destroyed while any child exists, and a child cannot outlive its parent (Linux Linux get_user_ns() / get_pid_ns() parity — see "Parent-child link semantics" above). For the FLAT namespaces (net, uts, ipc, mnt, cgroup, time) there is no parent/child relationship at all; each is destroyed independently when its own refcount reaches zero. There are no orphaned hierarchical namespaces.

Namespace Destruction — Drop-Based (the ONLY teardown model):

Each namespace is destroyed independently when its own Arc refcount drops to zero. Namespaces do not all reach zero simultaneously — a task's namespace_set drop decrements each namespace's refcount independently. When a single task exits and drops its Arc<NamespaceSet>, each contained Arc<FooNamespace> is decremented; whichever of them reaches zero has its impl Drop run in NamespaceSet field-declaration order. Other namespaces in the same namespace_set may still have non-zero refcounts (held by other tasks, bind mounts, or open /proc/PID/ns/* fds).

There is NO namespace_put() function, NO manual refcount check, and NO registered-cleanup-callback list: teardown logic lives exclusively in each namespace type's impl Drop (the "Namespace Drop semantics" rules in Section 17.1 — Weak::upgrade() with None-is-no-op for every cross-namespace reference; no relative-order assumptions between independently refcounted namespaces). The same Drop runs identically whether the last reference died via creation rollback, exit_task(), namespace_set replacement, a closed /proc/PID/ns/* fd, or an unmounted bind mount.

A last-reference Drop may SLEEP — the Mount arm umounts the whole tree and the IPC arm takes each SemSet.lock and wakes its WaitQueue (per the table below). Therefore no Arc<NamespaceSet> may be dropped inside a spin-locked section (notably TASK_LOCK(20)). Every site that replaces a task's namespace_set swaps the new set in under the lock and drops the returned old set OUTSIDE it (sys_setns common commit, the CLONE_NEWUSER arm, the pidfd multi-setns commit, and unshare step 5 all use this swap-then-drop idiom).

Per-type Drop responsibilities:

Type	`impl Drop` does
Time	Nothing beyond field drops (two atomics).
Net	Revoke `stack_cap` if set; sockets/interfaces/routes/firewall/conntrack drop via field destructors (worked example above).
Cgroup	Drop the `cgroup_root` view reference (the cgroup tree itself is owned by `CgroupRoot`, not the namespace).
PID	Free the (empty) `pid_map`/`reverse_map`/`pgids`/`sids` structures; the `ucount_charge` token's `Drop` uncharges the per-(user ns, uid) PID-namespace count at every ancestor level (`ns_ucount_uncharge`). Never kills or waits: member teardown happened long before — `terminate_members()` is an EXPLICIT `exit_task()` step on the namespace init, NOT a refcount hook (Section 17.1). A refcount-triggered kill would be structurally unreachable anyway: `nr_tasks > 0` implies member `pid_links` chains hold `Arc<PidNamespace>` references, so the refcount cannot reach zero with live members. Pinned TaskId-0 sentinel slots cannot outlive the namespace either — the pinning group/session's `pid_chain` holds the namespace `Arc`.
IPC	Per-object RMID-mark teardown, NOT a bare "destroy all" (a destroy under a live mapping would be a use-after-free — attachments do not pin the namespace): sem sets and msg queues get `sem_rmid_mark_and_wake`/`msg_rmid_mark_and_wake` (blocked waiters return `EIDRM`; queued messages drain back to `msg_slab`); shm segments get the RMID mark (`shm_dest.store(true, SeqCst)` — the second setter the `shm_dest` field doc names): a segment with live attachments (`nattach > 0`, held by tasks that `setns`'d away or by exiting tasks' not-yet-torn-down VMAs) keeps its pages until the LAST `shm_detach()` runs the `freed`-CAS free protocol, exactly as after `shmctl(IPC_RMID)`; an unattached segment frees when its table `Arc<ShmSegment>` drops. POSIX mqueues are woken, drained, and uncharged. Canonical impl: `impl Drop for IpcNamespace`, Section 17.3.
UTS	Field drops (the RCU string pair).
Mount	Umount all mounts in the namespace's tree (reverse mount order). MNT_DETACH orphans: detached mounts with open fds stay alive until the last fd closes (new opens rejected); the last file release runs the final mount cleanup. Overlayfs ordering: workdir cleanup before upper-layer unmount (Section 14.8).
User	The `ucount_charge` token's `Drop` uncharges the per-(user ns, uid) user-namespace count at every ancestor level (`ns_ucount_uncharge`); drop the frozen map arrays (uid/gid/projid) and the companion `ima_ns` Arc. No active capability revocation: capabilities scoped to this namespace become unexercisable because `is_same_or_ancestor()` walks can no longer reach it.

Ordering within a single NamespaceSet drop: Rust drops fields in declaration order — pid_ns, …, mount_ns, net_ns, uts_ns, ipc_ns, cgroup_ns, time_ns, …, user_ns, ima_ns. In particular net_ns precedes cgroup_ns, so when both die in the same set-drop, network teardown (sockets closed, conntrack purged) completes before the cgroup namespace view detaches. No teardown logic may DEPEND on any relative order (Drop rule 1): namespaces held by other references die at arbitrary later times, so every cross-namespace effect in a Drop goes through Weak::upgrade() + None-is-no-op regardless.

17.1.6 User Namespace UID/GID Mapping Security¶

User namespaces allow unprivileged users to have "root" (UID 0) within a namespace while mapping to an unprivileged UID outside. This is the foundation of rootless containers.

Security model:

/// A single contiguous range in a UID or GID mapping.
/// Maps `count` IDs starting at `inner_start` (inside namespace) to
/// `outer_start` (in parent namespace).
pub struct IdMapEntry {
    pub inner_start: u32,
    pub outer_start: u32,
    pub count: u32,
}

/// Maximum ID mapping entries per user namespace (matches Linux's limit of 340
/// per /proc/PID/uid_map and /proc/PID/gid_map).
const MAX_ID_MAPPINGS: usize = 340;

/// Global namespace-id counter shared by all namespace types. u64:
/// never wraps in operational lifetime (~585k years at 1M ns-creations/s).
static NEXT_NS_ID: AtomicU64 = AtomicU64::new(1);

/// `/proc/sys/kernel/overflowuid` — the UID reported when a real UID cannot
/// be represented in the current user namespace (unmapped id, NFS/UID
/// squash). Linux exposes this as a WRITABLE sysctl (default 65534, range
/// 0..=65535); UmkaOS backs it with a registered
/// typed-sysctl parameter, not a hardcoded constant. Read via `.load()` on
/// every unmapped-id path (`map_uid_from_parent`, `ns_ucount_charge`).
static OVERFLOW_UID: AtomicU32 = AtomicU32::new(65534);
/// `/proc/sys/kernel/overflowgid` — GID twin of `OVERFLOW_UID`.
static OVERFLOW_GID: AtomicU32 = AtomicU32::new(65534);

kernel_param! {
    name:          "kernel.overflowuid",
    schema:        ParamSchema::U32 { min: 0, max: 65535, default: 65534 },
    description:   "UID reported when a real UID has no user-namespace mapping \
                    (unmapped id, NFS/UID squash).",
    privileged:    true,
    per_namespace: false,
    getter:        || ParamValue::U32(OVERFLOW_UID.load(Ordering::Relaxed)),
    setter:        |v| match v {
        ParamValue::U32(n) => { OVERFLOW_UID.store(n, Ordering::Release); Ok(()) }
        _ => Err(ParamError::TypeMismatch),
    },
}
kernel_param! {
    name:          "kernel.overflowgid",
    schema:        ParamSchema::U32 { min: 0, max: 65535, default: 65534 },
    description:   "GID reported when a real GID has no user-namespace mapping \
                    (unmapped id, NFS/GID squash).",
    privileged:    true,
    per_namespace: false,
    getter:        || ParamValue::U32(OVERFLOW_GID.load(Ordering::Relaxed)),
    setter:        |v| match v {
        ParamValue::U32(n) => { OVERFLOW_GID.store(n, Ordering::Release); Ok(()) }
        _ => Err(ParamError::TypeMismatch),
    },
}

/// Per-(user-namespace, uid) namespace-creation counts, each bounded by its
/// own `/proc/sys/user/max_*_namespaces` sysctl (Linux `ucount_type`, the
/// namespace subset). Discriminants index the per-level count and limit
/// arrays. Extensible by appending a variant.
#[derive(Clone, Copy)]
pub enum NsUcountKind {
    UserNs = 0,
    PidNs = 1,
}
const NS_UCOUNT_KINDS: usize = 2;

/// Charge ONE creation of `kind`
/// against `(owner-chain, uid)`. The count lives per-(user ns, uid) in each
/// level's `UserEntry` (`UserNamespace.users` —
/// [Section 8.8](08-process.md#resource-limits-and-accounting--uid-level-accounting)), charged at
/// EVERY ancestor level from `owner` up to init, so a level's sysctl
/// transitively bounds creation in its whole subtree. A single-level (flat
/// `owner_uid`) charge would let a container with a wide id map defeat the
/// sysctl by rotating owner uids — the charge-anchor drift the bare
/// `user_ns_count_check(owner_uid)` signature suffered.
///
/// `uid` is the creator's effective uid in `owner`'s terms; at each hop it is
/// re-expressed in the parent's terms via `map_uid_to_parent()` (inner→outer;
/// an unmapped hop degrades to the overflow uid — the lenient variant). Every
/// ancestor is alive because `UserNamespace.parent` is a strong `Arc`.
///
/// Success → an `NsUcountCharge` RAII token the created namespace STORES;
/// dropping it (namespace teardown, or construction failure before it is
/// moved into the namespace — this is what makes the ENOMEM path leak-free)
/// uncharges every level it paid. Exceeding a level's limit → the increments
/// done so far are rolled back and `ENOSPC` returned. Bounded to ≤ 33 levels.
pub fn ns_ucount_charge(owner: &Arc<UserNamespace>, uid: u32, kind: NsUcountKind)
    -> Result<NsUcountCharge, Errno>
{
    let mut level_uid = uid;
    let mut ns = Arc::clone(owner);
    let mut charged: ArrayVec<Arc<UserEntry>, 33> = ArrayVec::new();
    loop {
        // Get-or-create the per-(this ns, level_uid) entry
        // ([Section 8.8](08-process.md#resource-limits-and-accounting--uid-level-accounting)).
        let entry = ns.get_user_entry(level_uid);
        // Increment only if strictly below this level's limit (a CAS
        // loop, race-free).
        if !inc_below(entry.ns_count(kind), ns.ns_ucount_max(kind)) {
            for e in charged.iter().rev() {          // undo charged ancestors
                e.ns_count(kind).fetch_sub(1, AcqRel);
            }
            return Err(Errno::ENOSPC);
        }
        charged.push(entry);
        match &ns.parent {
            None => return Ok(NsUcountCharge { chain: charged, kind }),
            Some(p) => {
                // Re-express uid one level up (inner→outer); unmapped → overflow.
                level_uid = ns.map_uid_to_parent(level_uid)
                    .unwrap_or_else(|| OVERFLOW_UID.load(Ordering::Relaxed));
                ns = Arc::clone(p);
            }
        }
    }
}

/// Increment `counter` iff it is `< max`; returns whether it did. CAS loop —
/// correct under concurrent charges.
fn inc_below(counter: &AtomicU32, max: u32) -> bool {
    let mut cur = counter.load(Ordering::Relaxed);
    loop {
        if cur >= max { return false; }
        match counter.compare_exchange_weak(cur, cur + 1, AcqRel, Ordering::Relaxed) {
            Ok(_) => return true,
            Err(observed) => cur = observed,
        }
    }
}

/// Paired decrement: decrement every level the token
/// charged. Called from `impl Drop for NsUcountCharge`. Leak-free without a
/// live-owner walk: the token holds a strong `Arc<UserEntry>` per level, so
/// the EXACT entries incremented at creation are decremented at teardown,
/// even if their `UserNamespace.users` maps have since been torn down or the
/// entries GC-detached.
fn ns_ucount_uncharge(charge: &NsUcountCharge) {
    for entry in &charge.chain {
        entry.ns_count(charge.kind).fetch_sub(1, AcqRel);
    }
}

/// RAII token returned by `ns_ucount_charge`, stored as a field by the
/// namespace it paid for (`UserNamespace.ucount_charge`,
/// `PidNamespace.ucount_charge`). Holds a strong `Arc<UserEntry>` per charged
/// level so teardown decrements exactly what creation incremented — this is
/// why neither namespace needs a live-owner `Weak` walk to uncharge. This is
/// independent of why `PidNamespace.user_ns` is a strong `Arc`: that edge is
/// load-bearing beyond the ucount. A PID namespace can outlive every task in
/// it — an nsfs bind-mount of `/proc/[pid]/ns/pid` pins a taskless namespace
/// with zero members — and every later `setns()`/`has_ns_cap()` decision on
/// it resolves `user_ns` through the owner/creator rule; a dead `Weak` owner
/// would leave those checks unanswerable for a live, reachable namespace. The
/// edge is acyclic (a `UserNamespace` never points back at a `PidNamespace`),
/// so the strong ref leaks nothing. (Linux `get_user_ns()` mirrors this.)
pub struct NsUcountCharge {
    chain: ArrayVec<Arc<UserEntry>, 33>,
    kind: NsUcountKind,
}
impl NsUcountCharge {
    /// Empty charge for the INIT/ROOT namespaces (`parent = None`), which are
    /// bootstrapped at boot and charged against no ancestor. Holds nothing;
    /// its `Drop` is a no-op. The init-namespace construction in boot code
    /// (`init_namespaces()`, umka-nucleus) uses this for `INIT_USER_NS`'s and the
    /// root `PidNamespace`'s `ucount_charge` field.
    pub fn empty() -> Self {
        Self { chain: ArrayVec::new(), kind: NsUcountKind::UserNs }
    }
}
impl Drop for NsUcountCharge {
    fn drop(&mut self) { ns_ucount_uncharge(self); }
}

/// User namespace: defines UID/GID translation mappings and capability scope.
/// Each user namespace has an owner (the uid in the parent namespace of the
/// process that created it) and an ordered list of ID mappings. Capabilities
/// held by a process are relative to its user namespace — CAP_SYS_ADMIN in
/// a child user namespace does not grant privilege in the parent.
///
/// **Write-once ID mappings (lock-free reads):** Linux enforces that
/// `/proc/PID/uid_map` and `/proc/PID/gid_map` can each be written **exactly
/// once** per user namespace lifetime. UmkaOS mirrors this: `uid_map` and
/// `gid_map` use a write-once-then-frozen model. Before the map is written,
/// all UIDs/GIDs resolve to `nobody`/`nogroup` (65534). After the single
/// write, the map is frozen and all subsequent reads are **lock-free** — a
/// plain pointer dereference to an immutable `IdMapArray`. No RwLock, no
/// atomic RMW, zero contention on the hottest path in the kernel (`stat()`,
/// `open()`, `access()`, `kill()`, every permission check).
///
/// The write path uses `map_lock` to serialize the single write and publish
/// the frozen map via a Release store on the `Arc` pointer. Reads use an
/// Acquire load — on x86 this compiles to a plain `MOV` (TSO).
///
/// /proc/PID/uid_map write restrictions (summary — the normative protocol,
/// including WHOSE credential each check uses, is the "uid_map, gid_map,
/// and setgroups — Write Protocol" section below; the checks are anchored
/// on the OPENER's `f_cred`, not the write-time caller):
/// - Opener must be in the target namespace or its parent, and hold
///   CAP_SYS_ADMIN over the target namespace
/// - Privileged multi-entry maps need CAP_SETUID/CAP_SETGID in the parent
///   held by BOTH the opener and the current writer; otherwise only the
///   rootless single-entry self-map is allowed
/// - Mapped outer IDs must be valid in the parent namespace from the
///   opener's perspective
/// - Mapping parent uid 0 additionally requires the CAP_SETFCAP root-map
///   gate (fscap forgery prevention)
/// - Can only be written ONCE; second write returns EPERM
// kernel-internal, not KABI — contains Arc, OnceCell, Mutex, RcuHashMap; a
// plain Rust struct (deliberately NOT #[repr(C)]) that never crosses a
// KABI/wire/userspace boundary.
pub struct UserNamespace {
    /// Unique namespace identifier, monotonically increasing. Used for
    /// cross-namespace permission checks, procfs display, and uevent
    /// attribution. Allocated from the global `NEXT_NS_ID` counter above.
    pub ns_id: u64,

    /// Parent user namespace (`None` for the root user namespace).
    parent: Option<Arc<UserNamespace>>,

    /// Nesting depth of this user namespace. The initial (root) user namespace
    /// has `level = 0`. Each child increments by 1. Linux `create_user_ns`
    /// rejects only `parent_ns->level > 32` (`kernel/user_namespace.c`, with
    /// `ns->level = parent_ns->level + 1`), so `unshare(CLONE_NEWUSER)` and
    /// `clone(CLONE_NEWUSER)` return `-ENOSPC` when `parent.level > 32` — the
    /// deepest reachable user namespace is level 33 (one deeper than PID
    /// namespaces, whose Linux check bounds the CHILD level at 32).
    /// This limit prevents unbounded recursion in `is_same_or_ancestor()` checks
    /// and caps the O(depth) capability lookup chain used during cross-namespace
    /// permission checks (see `compute_effective_caps` below).
    pub level: u32,

    /// Frozen UID mappings. `None` before `/proc/PID/uid_map` is written
    /// (all UIDs resolve to 65534). `Some(...)` after the single write —
    /// immutable thereafter. Reads are lock-free (Acquire load on the
    /// `Option` discriminant). The `Arc` ensures the backing array lives
    /// as long as any namespace referencing it.
    ///
    /// `OnceCell<T>`: write-once cell. `get()` returns `Option<&T>` via
    /// Acquire load (lock-free). `set(value)` initializes exactly once
    /// (returns `Err` if already set). Equivalent to `std::sync::OnceLock`
    /// but `no_std`-compatible.
    uid_map: OnceCell<Arc<IdMapArray>>,

    /// Frozen GID mappings. Same write-once-then-frozen semantics as uid_map.
    gid_map: OnceCell<Arc<IdMapArray>>,

    /// Frozen PROJECT-ID mappings (`/proc/[pid]/projid_map`). Same
    /// write-once-then-frozen model as uid_map/gid_map; `None` until the
    /// single write, so no project id is translatable (→ `None`) before then.
    /// Consumed by filesystem project quotas
    /// ([Section 14.15](14-vfs.md#disk-quota-subsystem)) via `map_projid_to_init()`
    /// (ns-relative→init-absolute — a per-hop WALK to init, since UmkaOS maps
    /// are parent-relative, not pre-composed like Linux `make_kprojid`). Project
    /// ids are not principals, so there is NO self-map case — the write
    /// protocol's privilege rule always requires CAP_SYS_ADMIN in the parent.
    projid_map: OnceCell<Arc<IdMapArray>>,

    /// Serializes the single write to uid_map/gid_map AND the
    /// `/proc/PID/setgroups` state transition (the two interlock — see
    /// the map-write protocol below), AND the first-use creation of a
    /// per-uid `UserEntry` in `users` (`get_user_entry`, so two
    /// concurrent first-chargers resolve to one shared entry). Only held
    /// during those rare write/creation paths (a handful of times per
    /// namespace lifetime). Never contended after initialization.
    map_lock: Mutex<()>,

    /// `/proc/[pid]/setgroups` state: 1 = "allow" (default), 0 = "deny".
    /// Writable (under `map_lock`) only while `gid_map` is unwritten;
    /// frozen thereafter. Read by `setgroups(2)`'s user-namespace gate
    /// and by the unprivileged `gid_map` write rule (see the map-write
    /// protocol below — Linux `USERNS_SETGROUPS_ALLOWED` semantics,
    /// `kernel/user_namespace.c`). AtomicU8 for lock-free reads on the
    /// setgroups(2) path.
    setgroups_allowed: AtomicU8,

    /// Companion IMA namespace — one `ImaNamespace` per user namespace
    /// ([Section 9.5](09-security.md#runtime-integrity-measurement)). Set exactly once,
    /// immediately after construction, by every creation site
    /// (`create_task()` step 10 branch (1a), `sys_unshare()` step 2a, boot
    /// init). This cell is the structural user-ns→IMA-ns association
    /// that `setns(CLONE_NEWUSER)` reads to switch `namespace_set.ima_ns` in
    /// lockstep. Forward-owning `Arc` edge; the ImaNamespace's
    /// back-reference to its user namespace is `Weak` — no cycle.
    pub ima_ns: OnceCell<Arc<ImaNamespace>>,

    /// Owner's UID in the parent namespace.
    ///
    /// Recorded from the creator's effective UID expressed in the parent
    /// namespace's terms. Consumed by `has_ns_cap()`'s owner/creator rule
    /// (Linux `cap_capable`'s `uid_eq(ns->owner, cred->euid)` clause,
    /// [Section 9.9](09-security.md#credential-model-and-capabilities)): when this namespace's parent
    /// IS the checking credential's user namespace, `owner_uid` is compared
    /// against `cred.euid` (both in the parent's terms) to grant the
    /// unprivileged creator all caps here. Corner case: a creator whose own
    /// credential is still in deferred-translation form across an UNWRITTEN
    /// intermediate map records the overflow id (65534) here — the owner rule
    /// then simply does not fire for that rare case (fails CLOSED: no spurious
    /// capability grant), and the map-write permission rules below are
    /// capability checks that do not depend on it either.
    owner_uid: u32,
    /// Owner's GID in the parent namespace.
    owner_gid: u32,
    /// Recorded at creation: whether the CREATING credential held
    /// `CAP_SETFCAP` (effective) in the parent namespace at `create()` time
    /// (Linux `user_namespace.parent_could_setfcap`, set in
    /// Linux `create_user_ns()` from the pre-transform credential —
    /// `kernel/user_namespace.c`, verified against torvalds/linux master).
    /// Consumed ONLY by the uid_map root-map gate (write-protocol step 5a
    /// below): a process writing its OWN namespace's uid_map may map parent
    /// uid 0 only if its creator could have set file capabilities in the
    /// parent anyway. Immutable after creation. Kernel-internal `bool`
    /// (this struct is not KABI).
    parent_could_setfcap: bool,

    /// Per-UID accounting entries FOR THIS LEVEL: `uid` (in this namespace's
    /// terms) → its `UserEntry`. This is the storage home for both the
    /// per-UID rlimit counters (task_count/sigpending/mq_bytes) and the
    /// per-(this ns, uid) namespace-creation counts that `ns_ucount_charge`
    /// increments. The namespace-local `Uid` key newtype and the `UserEntry`
    /// layout are both defined in
    /// [Section 8.8](08-process.md#resource-limits-and-accounting--uid-level-accounting); the field is
    /// declared here so the canonical `UserNamespace` struct is complete (it
    /// was previously prose-only in that chapter). `get_user_entry`
    /// inserts a zeroed entry on first use.
    pub users: RcuHashMap<Uid, Arc<UserEntry>>,

    /// Per-level limits for each `NsUcountKind`, the `/proc/sys/user/
    /// max_{user,pid}_namespaces` surface for THIS namespace (a per-namespace
    /// sysctl set). Read by
    /// `ns_ucount_max()`. Inherited from the parent at creation; the init
    /// namespace seeds a large default. Writable (CAP_SYS_ADMIN over this ns)
    /// via the typed-sysctl registry.
    ucount_max: [AtomicU32; NS_UCOUNT_KINDS],

    /// This namespace's OWN creation charge (`NsUcountKind::UserNs`),
    /// returned by `ns_ucount_charge` at `create()`. Held for the
    /// namespace's whole life; its `Drop` uncharges every ancestor level —
    /// the leak-free replacement for the old "decremented from impl Drop"
    /// prose that named no counter.
    ucount_charge: NsUcountCharge,
}

impl UserNamespace {
    /// This namespace's limit for `kind` — the `/proc/sys/user/max_*_namespaces`
    /// value for this level.
    fn ns_ucount_max(&self, kind: NsUcountKind) -> u32 {
        self.ucount_max[kind as usize].load(Ordering::Relaxed)
    }
    /// Look up (or atomically create, zeroed) the `UserEntry` for `uid` in
    /// THIS namespace's terms. Shares the entry lifecycle with the
    /// on-first-task-creation path
    /// ([Section 8.8](08-process.md#resource-limits-and-accounting--uid-level-accounting)); both need
    /// insert-if-absent. A concurrent creator MUST resolve to a SINGLE shared
    /// entry, never two — the split-limit correctness of `ns_ucount_charge`
    /// depends on it. The canonical `RcuHashMap::insert`
    /// ([Section 3.4](03-concurrency.md#cumulative-performance-budget--rcu-protected-container-types))
    /// OVERWRITES on a duplicate key, so a bare RCU-miss-then-`insert` would
    /// let a second creator discard the first's entry (and its charge). Creation
    /// is therefore serialized under the namespace's `map_lock` (the same rare
    /// warm/cold write lock the id-map writes use) with a re-check inside the
    /// lock; the fast path stays lock-free.
    pub fn get_user_entry(&self, uid: u32) -> Arc<UserEntry> {
        // Fast path: lock-free RCU lookup.
        {
            let guard = rcu_read_lock();
            if let Some(e) = self.users.lookup(&Uid(uid), &guard) {
                return e;
            }
        }
        // Slow path: serialize creation, re-check (another creator may have
        // won between the RCU miss and acquiring the lock).
        let _g = self.map_lock.lock();
        {
            let guard = rcu_read_lock();
            if let Some(e) = self.users.lookup(&Uid(uid), &guard) {
                return e;
            }
        }
        let entry = Arc::new(UserEntry::zeroed());
        // insert allocates a node; this warm namespace-creation path treats
        // that as infallible (matching the surrounding accounting model — a
        // node-alloc failure here surfaces as the caller's ENOMEM).
        self.users.insert(Uid(uid), Arc::clone(&entry))
            .expect("UserEntry node allocation");
        entry
    }
}

/// Frozen, immutable ID mapping array. Created once when `/proc/PID/uid_map`
/// (or `gid_map`) is written, never modified thereafter.
pub struct IdMapArray {
    /// The mapping extents, sorted by `outer_start` (the PARENT-side key).
    /// This is the canonical order: `map_from_parent` (parent→child = global→
    /// ns-relative — the `map_from_parent` direction) binary-searches it
    /// by `outer_start`, the parent-side key.
    /// Typical container: 1 entry (e.g. inner 0..65535 → outer 100000..165535).
    /// For ≤5 entries both directions linear-scan (faster than binary search —
    /// no branch mispredict); for >5 entries `map_from_parent` binary-searches
    /// these and `map_to_parent` binary-searches `inner_sorted`.
    entries: ArrayVec<IdMapEntry, MAX_ID_MAPPINGS>,
    /// Permutation of `entries` indices ordered by `inner_start` (the
    /// CHILD-side key), enabling the INVERSE lookup `map_to_parent`
    /// (child→parent = ns-relative→global — the `map_to_parent` translation
    /// direction). Populated
    /// at freeze ONLY when `entries.len() > 5`
    /// (≤5 uses a linear scan and leaves this empty); same length as
    /// `entries` when populated. `u16` indices — MAX_ID_MAPPINGS = 340 < 65536.
    inner_sorted: ArrayVec<u16, MAX_ID_MAPPINGS>,
    /// Cached: true ONLY for the FULL-COVERAGE 1:1 identity map — a single
    /// extent `{ outer_start: 0, inner_start: 0, count: u32::MAX }` that maps
    /// every id to itself. A PARTIAL identity map (e.g. `0 0 1000`) is NOT
    /// identity: it does not cover ids >= its count, so those must return
    /// `None` (Linux `from_kuid`/`make_kuid` fail outside coverage). Setting
    /// `is_identity` for a partial map was a bug — the fast-path bypass returns
    /// the input id unchanged in BOTH directions, so it is only sound when
    /// coverage is total. Partial maps fall through to the range lookup, which
    /// correctly returns `None` out of coverage.
    is_identity: bool,
}

impl IdMapArray {
    /// Translate a parent-namespace ID (outer) to a child-namespace ID (inner).
    ///
    /// Returns `Some(inner)` if the parent ID falls within a mapped range, or
    /// `None` if no mapping covers it. The caller must substitute `overflow_uid`
    /// (65534) when `None` is returned — this outer→inner (global→ns-relative)
    /// lookup is the `map_from_parent` direction, and the overflow substitution
    /// returns the configured overflow value when the id is unmapped.
    ///
    /// For ≤5 entries a linear scan is O(n) and faster than binary search
    /// (avoids branch misprediction overhead). For >5 entries a binary search
    /// on the `outer_start`-sorted `entries` is used.
    pub fn map_from_parent(&self, outer: u32) -> Option<u32> {
        if self.is_identity {
            return Some(outer);
        }
        let hit = if self.entries.len() <= 5 {
            self.entries.iter().find(|e|
                outer >= e.outer_start
                && outer < e.outer_start.saturating_add(e.count))
        } else {
            // entries are outer_start-sorted with disjoint ranges: the first
            // range whose end > outer is the only candidate.
            let i = self.entries.partition_point(|e|
                e.outer_start.saturating_add(e.count) <= outer);
            self.entries.get(i).filter(|e| outer >= e.outer_start)
        };
        hit.map(|e| e.inner_start + (outer - e.outer_start))
    }

    /// Translate a CHILD-namespace ID (inner) to a PARENT-namespace ID (outer)
    /// — the INVERSE of `map_from_parent`, the `map_to_parent` direction
    /// (ns-relative→global). `None` if `inner` is not covered by any
    /// range: a strict `map_to_parent` caller treats `None` as "unmappable" (an
    /// INVALID id), a lenient caller substitutes the overflow value. ≤5
    /// entries: linear over
    /// `entries` by `inner_start`; >5: binary search on the `inner_sorted`
    /// (`inner_start`-ordered) index.
    pub fn map_to_parent(&self, inner: u32) -> Option<u32> {
        if self.is_identity {
            // FULL-COVERAGE identity only (see `is_identity` doc): every id
            // maps to itself, so `Some(inner)` is correct for all coverage.
            // A partial identity map leaves `is_identity == false` and takes
            // the range lookup below, which returns `None` out of coverage.
            return Some(inner);
        }
        let hit = if self.entries.len() <= 5 {
            self.entries.iter().find(|e|
                inner >= e.inner_start
                && inner < e.inner_start.saturating_add(e.count))
        } else {
            let j = self.inner_sorted.partition_point(|&idx| {
                let e = &self.entries[idx as usize];
                e.inner_start.saturating_add(e.count) <= inner
            });
            self.inner_sorted.get(j)
                .map(|&idx| &self.entries[idx as usize])
                .filter(|e| inner >= e.inner_start)
        };
        hit.map(|e| e.outer_start + (inner - e.inner_start))
    }
}

impl UserNamespace {
    /// Create a child user namespace of `parent`. Canonical constructor
    /// for CLONE_NEWUSER — called from `create_task()` step 10 (branch (1) of
    /// the credential transformation) and `sys_unshare()` step 2a.
    ///
    /// - **Nesting limit**: `parent.level > 32` → `ENOSPC` (Linux
    ///   Linux `create_user_ns` rejects only `parent_ns->level > 32`, verified vs
    ///   torvalds/linux master `kernel/user_namespace.c`; deepest reachable
    ///   user namespace is level 33 — one deeper than the PID-namespace limit,
    ///   whose Linux check bounds the child level at 32).
    /// - **Per-user limit**: `ns_ucount_charge(parent, owner_uid,
    ///   UserNs)` charges the ancestor chain and returns `ENOSPC` if any
    ///   level's `/proc/sys/user/max_user_namespaces` is exceeded. The
    ///   returned token is stored in `ucount_charge`; its `Drop` is the
    ///   paired decrement (leak-free on rollback AND teardown).
    /// - `ns_id` is drawn from the global namespace-id counter (the
    ///   `NEXT_NS_ID` static above); `level = parent.level + 1`.
    /// - `owner_uid` / `owner_gid` record the creator's EFFECTIVE ids in
    ///   the PARENT namespace (fork passes the parent credential's
    ///   euid/egid).
    /// - `creator_setfcap` records whether the creating credential held
    ///   `CAP_SETFCAP` (effective) in the PARENT namespace — callers
    ///   evaluate it on the PRE-transform credential:
    ///   `cred.cap_effective.contains(CAP_SETFCAP)` for `task.cred` /
    ///   the not-yet-transformed `pending_cred` (Linux `create_user_ns()`
    ///   samples `new->cap_effective` BEFORE `set_cred_user_ns()`). Stored
    ///   as `parent_could_setfcap` for the uid_map root-map gate.
    ///   (Deferred handoff by symbol:
    ///   [Section 8.1](08-process.md#process-and-task-management--process-creation) — the
    ///   `create_task()` step-10 CLONE_NEWUSER branch's `UserNamespace::create`
    ///   call gains this same fourth argument, evaluated on the parent
    ///   credential before the CLONE_NEWUSER transformation.)
    /// - `uid_map` / `gid_map` / `projid_map` start unwritten: all ids
    ///   resolve to the overflow value until the corresponding
    ///   `/proc/[pid]/*_map` is written.
    /// - `ucount_max` is inherited from the parent's per-level limits.
    ///   (There is NO per-namespace capability mask here: what a process
    ///   in the namespace can hold lives in `TaskCredential`'s five sets,
    ///   scoped by `cred.user_ns` — the former `cap_permitted` field had
    ///   no reader and is retired.)
    /// - `ENOMEM` on allocation failure — the charge token drops and every
    ///   charged level is uncharged (no leak).
    pub fn create(
        parent: &Arc<UserNamespace>,
        owner_uid: u32,
        owner_gid: u32,
        creator_setfcap: bool,
    ) -> Result<Arc<UserNamespace>, Errno> {
        if parent.level > 32 {
            // Linux `create_user_ns` rejects only `parent_ns->level > 32`
            // (`kernel/user_namespace.c`, verified vs torvalds/linux master),
            // with `ns->level = parent_ns->level + 1`. Parent levels 0..=32 are
            // permitted, producing children at levels 1..=33 — the deepest
            // reachable user namespace is level 33. (One deeper than PID
            // namespaces, whose Linux check bounds the CHILD level at 32; the
            // two subsystems genuinely differ.)
            return Err(Errno::ENOSPC);
        }
        // Enforce /proc/sys/user/max_user_namespaces for owner_uid
        // (per-user namespace accounting; ENOSPC when over the sysctl).
        // Charge the per-(user ns, uid) creation count against the PARENT
        // chain (ancestor-charged; ENOSPC over any level's
        // /proc/sys/user/max_user_namespaces). The returned RAII token is
        // moved into `ucount_charge` below; if the fallible allocation then
        // fails, the token drops and every charged level is uncharged —
        // closing the ENOMEM leak the old charge-then-Arc::new order had.
        let ucount_charge = ns_ucount_charge(parent, owner_uid, NsUcountKind::UserNs)?;
        Arc::try_new(UserNamespace {
            ns_id: NEXT_NS_ID.fetch_add(1, Ordering::Relaxed),
            parent: Some(Arc::clone(parent)),
            level: parent.level + 1,
            uid_map: OnceCell::new(),
            gid_map: OnceCell::new(),
            projid_map: OnceCell::new(),
            map_lock: Mutex::new(()),
            setgroups_allowed: AtomicU8::new(1),   // "allow" until denied
            ima_ns: OnceCell::new(),   // paired by the caller immediately
            owner_uid,
            owner_gid,
            parent_could_setfcap: creator_setfcap,
            users: RcuHashMap::new(),
            ucount_max: [
                AtomicU32::new(parent.ns_ucount_max(NsUcountKind::UserNs)),
                AtomicU32::new(parent.ns_ucount_max(NsUcountKind::PidNs)),
            ],
            ucount_charge,   // drops (→ uncharge every level) if try_new fails
        }).map_err(|_| Errno::ENOMEM)
        // Every caller allocates the companion ImaNamespace right after
        // this returns and sets `ima_ns` exactly once:
        //   ns.ima_ns.set(ima_ns_create(&ns)?)... — see the fork/unshare
        // creation branches. `ima_ns_create` builds the per-namespace
        // ImaNamespace (empty local measurement log, fresh virtual PCR
        // bank, read-only inherited global policy —
        // [Section 9.5](09-security.md#runtime-integrity-measurement)).
    }

    /// Map a UID from the parent namespace into this namespace.
    ///
    /// Called by `setns(CLONE_NEWUSER)` and `unshare(CLONE_NEWUSER)` to
    /// translate the caller's UID into its equivalent in the target user
    /// namespace. If no mapping exists for the UID, `overflow_uid` (65534,
    /// matching Linux `overflowuid`) is returned — the process appears as
    /// "nobody" for that UID within this namespace.
    ///
    /// Thread-safe: reads `uid_map` via `OnceCell::get()` which is lock-free
    /// after the single write. No RwLock or Mutex held on the read path.
    ///
    /// Linux equivalent: `from_kuid_munged()` in `kernel/user_namespace.c`
    /// (outer→inner / `map_id_up`; substitutes the overflow uid when unmapped).
    pub fn map_uid_from_parent(&self, outer_uid: u32) -> u32 {
        let overflow = OVERFLOW_UID.load(Ordering::Relaxed);
        match self.uid_map.get() {
            None => overflow,   // no mapping written yet → nobody
            Some(map) => map.map_from_parent(outer_uid).unwrap_or(overflow),
        }
    }

    /// Map a GID from the parent namespace into this namespace.
    ///
    /// Identical semantics to `map_uid_from_parent` but operates on the
    /// `gid_map`. Returns the overflow gid when no mapping exists.
    ///
    /// Linux equivalent: `from_kgid_munged()` in `kernel/user_namespace.c`
    /// (outer→inner / `map_id_up`).
    pub fn map_gid_from_parent(&self, outer_gid: u32) -> u32 {
        let overflow = OVERFLOW_GID.load(Ordering::Relaxed);
        match self.gid_map.get() {
            None => overflow,
            Some(map) => map.map_from_parent(outer_gid).unwrap_or(overflow),
        }
    }

    /// Translate a PROJECT ID that is in THIS namespace's terms into the
    /// canonical **init-namespace-absolute** project id used by the quota subsystem.
    ///
    /// UNLIKE `map_uid_to_parent`/`map_gid_to_parent`, which express an id in
    /// the IMMEDIATE parent's terms (a single `map_to_parent` hop
    /// per level), this is a to-INIT WALK. UmkaOS stores each level's
    /// `projid_map` in its PARENT's terms: the write protocol's step 4 only
    /// VALIDATES each `outer_start..+count` range against the parent — it does
    /// NOT pre-compose the outer column into init-absolute space the way Linux
    /// Linux `map_write` does (`lower_first = map_id_range_down(parent_map, ...)`,
    /// `kernel/user_namespace.c` master). So a single hop yields only the
    /// immediate parent's projid; at user-namespace nesting depth ≥ 2 that is
    /// an intermediate-namespace value, NOT the canonical id (storing it into
    /// storing that intermediate value in an inode would alias project quotas). The
    /// walk therefore applies each ancestor's map in turn — per-hop, mirroring
    /// `ns_ucount_charge` — until it reaches the init user namespace
    /// (`parent == None`), where the accumulated value IS init-absolute. Each
    /// hop is one `map_to_parent` operation per level.
    ///
    /// STRICT: any unmapped hop → `None` (no overflow substitution — a projid
    /// not fully mappable to init has no canonical form). This is the ONE
    /// translation the quota subsystem needs: the inode stores the
    /// init-absolute project id, and the caller's user-namespace projid is
    /// converted ONCE at the `FS_IOC_FSSETXATTR` write boundary
    /// ([Section 14.15](14-vfs.md#disk-quota-subsystem)); `None` → the handler rejects (`EINVAL`),
    /// matching Linux `make_kprojid` returning an invalid id. The `projid_map`
    /// is unmapped (nothing translatable upward) until `/proc/[pid]/projid_map`
    /// is written. The walk is bounded to ≤ 33 levels (the user-ns nesting
    /// cap). (There is no `map_projid_from_init`/`map_projid_from_parent`:
    /// projids are not principals — no setns/getuid display path needs the
    /// inverse (outer→inner) direction.)
    pub fn map_projid_to_init(&self, inner_projid: u32) -> Option<u32> {
        // First hop: express `inner_projid` (in self's terms) in self's
        // parent's terms; if self IS init, it is already init-absolute.
        let mut projid = inner_projid;
        let mut ns = match &self.parent {
            None => return Some(projid),        // self is init: already absolute
            Some(p) => {
                projid = self.projid_map.get()?.map_to_parent(projid)?;
                Arc::clone(p)
            }
        };
        // Remaining hops: climb until the init user namespace, where the
        // accumulated value is expressed in init-absolute terms. Only non-init
        // levels (those WITH a parent) contribute a map application.
        loop {
            match &ns.parent {
                None => return Some(projid),    // ns is init: projid is absolute
                Some(p) => {
                    projid = ns.projid_map.get()?.map_to_parent(projid)?;
                    ns = Arc::clone(p);
                }
            }
        }
    }

    /// Express a UID that is in THIS namespace's terms in the PARENT's terms
    /// — Linux `make_kuid`/`map_id_down` direction (ns-relative→global), the
    /// INVERSE of `map_uid_from_parent`. `None` if the inner uid is unmapped
    /// (strict `map_to_parent` → `None`); callers wanting a lenient result
    /// substitute the overflow uid. Used by
    /// ancestor-directed id expression: ns-ucount charging, and — via the
    /// credential-model helper `cred_uid_in_ancestor()`
    /// ([Section 9.9](09-security.md#credential-model-and-capabilities)) — VFS/overlayfs inode-owner
    /// writes onto a mount whose user namespace is an ancestor.
    pub fn map_uid_to_parent(&self, inner_uid: u32) -> Option<u32> {
        match self.uid_map.get() {
            None => None,   // unwritten map: nothing is mappable upward
            Some(map) => map.map_to_parent(inner_uid),
        }
    }

    /// GID twin of `map_uid_to_parent` (Linux `make_kgid`/`map_id_down`).
    pub fn map_gid_to_parent(&self, inner_gid: u32) -> Option<u32> {
        match self.gid_map.get() {
            None => None,
            Some(map) => map.map_to_parent(inner_gid),
        }
    }

    /// STRICT translation of a UID from the kernel-internal GLOBAL
    /// (init-namespace) representation — the representation stored in
    /// `Inode::i_uid` ([Section 14.1](14-vfs.md#virtual-filesystem-layer)) — into THIS
    /// namespace's terms.
    ///
    /// Walks the namespace chain from the init namespace down to `self`,
    /// applying each hop's written `uid_map` strictly (the `Option`-returning
    /// `IdMapArray::map_from_parent`; NO overflow substitution). Returns
    /// `None` if this namespace or any ancestor hop lacks a covering
    /// mapping — i.e. the global uid is not representable here.
    ///
    /// Strictness is the point: a caller that wants lenient "munged"
    /// behaviour substitutes the overflow uid itself, at its own call site,
    /// where the substitution is visible and reviewable. A primitive that
    /// substituted silently would make every "is this id representable?"
    /// security predicate built on it vacuously true — see the copy-up guard
    /// in [Section 14.8](14-vfs.md#overlayfs-union-filesystem-for-containers).
    ///
    /// The walk is bounded by user-namespace nesting depth (≤ 33 levels) and
    /// reads each level's map through a lock-free `OnceCell::get()`.
    ///
    /// Recursive definition:
    ///   init namespace => Some(global_uid)
    ///   otherwise      => parent.uid_from_global(global_uid)
    ///                       .and_then(|u| self.uid_map.get()?.map_from_parent(u))
    pub fn uid_from_global(&self, global_uid: u32) -> Option<u32> {
        match &self.parent {
            None => Some(global_uid),   // self is init: already global
            Some(parent) => {
                let outer = parent.uid_from_global(global_uid)?;
                self.uid_map.get()?.map_from_parent(outer)
            }
        }
    }

    /// GID twin of `uid_from_global` — identical chain-walk semantics over
    /// `gid_map`, `None` on any non-covering hop, no overflow substitution.
    pub fn gid_from_global(&self, global_gid: u32) -> Option<u32> {
        match &self.parent {
            None => Some(global_gid),
            Some(parent) => {
                let outer = parent.gid_from_global(global_gid)?;
                self.gid_map.get()?.map_from_parent(outer)
            }
        }
    }
}

17.1.6.1 User-Namespace Credential Transformation (Canonical)¶

clone(CLONE_NEWUSER) (create_task step 10), unshare(CLONE_NEWUSER) (step 2b), and setns(CLONE_NEWUSER) perform ONE credential transformation. The capability half is identical at all three sites (CAP_FULL_SET effective/permitted/bounding scoped to the new namespace, inheritable and ambient cleared, validated against the five install_credentials() invariants). The IDENTITY half is the single function below — there are no per-path identity rules, and in particular fork does NOT "set euid = 0": with an unwritten map there is no mapping to make the creator root, and getuid() after clone(CLONE_NEWUSER) MUST report 65534 until the map is written, then the mapped value (Linux ABI, man 7 user_namespaces).

/// Canonical identity transformation on user-namespace entry
/// (creation or join). Implements the DEFERRED-TRANSLATION credential
/// rule ([Section 9.9](09-security.md#credential-model-and-capabilities--deferred-translation-credentials-unmapped-user-namespaces)):
/// the eight POSIX id fields (uid/gid/euid/egid/suid/sgid/fsuid/fsgid)
/// are always stored in the terms of `cred.ids_ns`, and this function
/// decides whether they can be normalized to the target's terms yet.
///
/// Precondition: `new_cred.user_ns` was already set to `target`, and the
/// old ids are expressed in `new_cred.ids_ns` terms (invariant: same as
/// or an ancestor of the PREVIOUS user_ns — re-expressed toward
/// `target.parent` by `reexpress_ids_toward()` when the caller joined
/// from a namespace whose ids_ns is a higher ancestor: each hop through
/// a WRITTEN map translates exactly; an unwritten/unmapped hop degrades
/// to the overflow id, matching Linux `from_kuid()` for an unmappable
/// id).
pub fn user_ns_transform_ids(
    new_cred: &mut TaskCredential,   // TrackedPtr deref — uniquely owned
    target: &Arc<UserNamespace>,
) {
    // Bring the ids into the target's PARENT terms first (no-op in the
    // overwhelmingly common case: creator/joiner already lives in the
    // target's parent and its ids are normalized there).
    //
    // `target.parent` is ALWAYS `Some` at every call site: the creation
    // paths (create_task step 10, unshare 2a/2b) build a fresh CHILD namespace
    // whose parent is the creator's user ns, and setns(CLONE_NEWUSER)
    // rejects a parentless (init) target with EINVAL — gate (ii) in the
    // User arm — BEFORE reaching here. The `expect()` documents an
    // invariant the callers enforce; it is unreachable, not a live panic.
    let target_parent = target.parent.as_ref()
        .expect("user_ns_transform_ids never runs for the init user ns: \
                 creation paths make a child; setns rejects init (gate ii)");
    reexpress_ids_toward(new_cred, target_parent);
    if target.uid_map.get().is_some() && target.gid_map.get().is_some() {
        // Maps written (common setns case): NORMALIZE now.
        for id in new_cred.uid_fields_mut() {
            *id = target.map_uid_from_parent(*id);   // unmapped → 65534
        }
        for id in new_cred.gid_fields_mut() {
            *id = target.map_gid_from_parent(*id);
        }
        new_cred.ids_ns = Arc::clone(target);
    } else {
        // Maps not (fully) written — creation case: KEEP the id values in
        // parent terms; record the expression namespace. The 65534 view
        // exists only at the read boundary; the creator's true identity
        // is preserved for file-permission and signal checks.
        new_cred.ids_ns = Arc::clone(target_parent);
    }
}

Why deferred translation (design decision — UmkaOS-native). UmkaOS credentials store namespace-LOCAL ids (unlike Linux's global kuid_t). Storing the literal 65534 at creation would DESTROY the creator's identity — file access against its own files, signal-permission checks, and the post-map-write getuid() transition would all break. The two candidate repairs were (a) rewriting every member credential when the map is written, or (b) keeping the outer ids and translating at the read boundary. UmkaOS chooses (b), because (a) is structurally impossible under the credential model: install_credentials() runs only for current (single-writer-per-task publication), so a map-freeze pass cannot commit other tasks' credentials. Under (b):

Zero cost for every normal task: ids_ns == user_ns (pointer-equal fast path — one predicted-taken compare at the read boundary).
ABI transitions fall out automatically: getuid() translates ids_ns-terms → user_ns-terms on read; unwritten map → 65534, written map → the mapped value. The 65534→mapped flip at map-write time requires NO stored-id rewrite and no member enumeration.
Identity is exact throughout: permission checks translate from ids_ns (the parent-terms truth) to whatever namespace the object lives in — equivalent to Linux's kuid comparisons.
Self-normalization: a task in deferred form normalizes its OWN credential at its next credential-mutating syscall once the map is written (the prepared clone is translated and ids_ns set to user_ns); id-changing syscalls (setuid family) against an unwritten map fail EINVAL (nothing is mapped), matching Linux.

The full read-boundary and field-level specification lives in Section 9.9.

17.1.6.2 uid_map, gid_map, and setgroups — Write Protocol¶

The /proc/[pid]/uid_map, /proc/[pid]/gid_map, /proc/[pid]/projid_map, and /proc/[pid]/setgroups files (procfs registry rows: Section 14.19) are the userspace surface of the write-once map model above. Their semantics are ABI-mandated — every rootless runtime (newuidmap/newgidmap, podman, rootless Docker) is built on them; corroborated against Linux kernel/user_namespace.c (map_write(), userns_may_setgroups()) on torvalds/linux master and man 7 user_namespaces.

Write handler for uid_map / gid_map (the target namespace is the user namespace of the process named by the /proc/[pid]/ directory; all steps run under the namespace's map_lock).

Credential source — the OPENER, not "the writer". Every placement and privilege check below runs against the OPENER's credential — the f_cred snapshot the VFS captured when /proc/[pid]/*_map was opened (OpenFile.f_cred, Section 14.1) — and, where noted, ALSO against the current writer. Checking only the write-time caller opens the fd-passing confused deputy: an unprivileged process opens the map file and hands the fd to a privileged writer (setuid helper, privileged service) whose write would then pass checks the opener could never pass. Linux deliberately anchors these checks on file->f_cred (Linux map_write(), Linux new_idmap_permitted() and seq_user_ns() — all opener-derived; kernel/user_namespace.c, verified against torvalds/linux master):

Once-only: if the corresponding OnceCell is already set → EPERM (each map is writable exactly once per namespace lifetime).
Parse: up to MAX_ID_MAPPINGS (340) lines of inner_start outer_start count (decimal, whitespace-separated — format %10u %10u %10u); reject on: count == 0, inner/outer range overflow past u32::MAX, overlapping inner or outer ranges, or more than 340 entries → EINVAL.
Opener placement + admin: the OPENER's user namespace (f_cred.user_ns) must be the target namespace OR its parent → EPERM otherwise (Linux seq_ns != ns && seq_ns != ns->parent, where Linux seq_user_ns() is the opener's namespace). The opener must additionally hold CAP_SYS_ADMIN over the TARGET namespace (the Linux file_ns_capable(file, map_ns, CAP_SYS_ADMIN) check) → EPERM.
Outer validity: every outer_start..+count range must be mapped in the PARENT namespace from the OPENER's perspective (ids the opener cannot name cannot be delegated) → EPERM.
Privilege rule (Linux new_idmap_permitted(), cred = file->f_cred):
Privileged: BOTH the current writer holds CAP_SETUID (CAP_SETGID for gid_map) in the target's PARENT namespace (has_ns_cap) AND the opener passes the Linux file_ns_capable check → any ranges satisfying steps 4 and 5a are allowed. The dual check is load-bearing: current-only reopens the confused deputy above; opener-only would let a privileged opener's leaked fd empower an unprivileged writer.
Rootless self-map (otherwise): exactly ONE entry with count == 1; the OPENER's euid must equal the target namespace's owner_uid (the creator — Linux uid_eq(ns->owner, cred->euid)); and the mapped OUTER id must equal the OPENER's own effective id (euid for uid_map, egid for gid_map) → anything else EPERM. gid_map additionally requires /proc/[pid]/setgroups to have been set to "deny" first → EPERM otherwise. (This is the CVE-2014-8989 gate: without it, an unprivileged process could use a self-written gid_map plus setgroups() to DROP a supplementary group that a !group ACL relies on.)

5a. Root-map gate — uid_map only (fscap-forgery prevention): if any entry's OUTER range covers uid 0 of the parent namespace, the map would let the target namespace's root author file capabilities that VALIDATE in ancestor namespaces (fscaps are interpreted relative to the file's owning user namespace — the "File capability interpretation" rules below). Gate (Linux verify_root_map(), verified against torvalds/linux master): - OPENER's user namespace == the target namespace (the unshared process writing its own map): allowed only if the target's parent_could_setfcap was recorded at creation — i.e. the CREATOR held CAP_SETFCAP (effective) in the parent when it created the namespace (see the UserNamespace.parent_could_setfcap field); - otherwise (a parent-side writer mapping root into a child): the OPENER must hold CAP_SETFCAP over the target's PARENT namespace (Linux file_ns_capable(file, map_ns->parent, CAP_SETFCAP)); → EPERM on either failure. 6. Freeze: build the IdMapArray — entries sorted by outer_start (the CANONICAL key that map_from_parent binary-searches); for > 5 entries ALSO build the inner_start-ordered inner_sorted index that map_to_parent binary-searches; compute is_identity — set true ONLY when the map is the single full-coverage extent { outer_start: 0, inner_start: 0, count: u32::MAX } (a partial identity map like 0 0 1000 leaves it false so out-of-coverage ids correctly resolve to None). OnceCell::set() it — all reads are lock-free from here on. No member-credential rewrite occurs: the deferred-translation model above makes the overflow→mapped transition a pure read-boundary effect.

/proc/[pid]/setgroups (0o644): reads return "allow\n" or "deny\n" (setgroups_allowed load). Writes ("allow"/"deny", under map_lock): permitted only while gid_map is unwritten → EPERM after; writer must be in the target namespace or its parent and hold CAP_SYS_ADMIN over the target namespace. "deny" is one-way once gid_map is written (the flag freezes with the map).

setgroups(2) gate inside a user namespace: in addition to has_ns_cap(cred.user_ns, CAP_SETGID), the call requires setgroups_allowed == 1 AND gid_map written — exact Linux behavior: Linux userns_may_setgroups() returns failure otherwise. (Cross-reference: Section 9.9.)

projid_map: same parser, placement, and once-only rules as uid_map, freezing into the projid_map OnceCell (privilege rule checks CAP_SYS_ADMIN in the parent — project ids have no self-map case); consumed by filesystem project quotas via map_projid_to_init() (ns-relative→init-absolute). NOTE: unlike Linux map_write, step 4 above only VALIDATES the outer column against the parent — it does NOT pre-compose it to init-absolute space, so each level's stored map is PARENT-relative; map_projid_to_init() composes the chain per-hop to init at read time.

Capability interactions with user namespaces:

A process with UID 0 inside a user namespace has full capabilities within that namespace
Capabilities are NOT granted against resources owned by ancestor namespaces
Example: A process with "root" in User NS 1 cannot mount() a filesystem from the host
The cap_effective mask is computed at syscall entry time based on:
The process's current UID within its user namespace
The target object's owning user namespace
The intersection of the process's capability bounding set with capabilities valid for the target

Determining the owning user namespace for kernel objects:

Object Type	Owning User Namespace	Mechanism
File (VFS inode)	User namespace of the mount	Each mount has `Mount.user_ns` set at mount time. Files inherit from their mount.
Socket	User namespace of the creating process	Stored through `SockCommon.net_ns.user_ns` at socket creation
IPC object (shm, sem, msg)	User namespace of the creating namespace	IPC namespace → User namespace mapping at IPC NS creation
Capability token	User namespace of the issuing process	Stored in capability header
Process (for signals)	User namespace of the process	Stored in `task_struct->user_ns`
Device node	User namespace of the initial mount	Device nodes are always in the initial namespace

cap_effective computation algorithm:

The effective capability set for a process operating on an object is the intersection of: 1. The process's current effective capabilities (cap_effective) 2. The capabilities valid for the target object's namespace

This ensures that a process which has dropped capabilities via capset() does not regain them when accessing child namespace objects.

compute_effective_caps(process, object):
  1. proc_ns = process.user_namespace
  2. obj_ns = object.owning_user_namespace
  3. proc_caps = process.cap_effective  // NOT cap_bounding — use current effective set

  4. // Check if process's NS is an ancestor of object's NS (or same NS)
  5. if is_same_or_ancestor(proc_ns, obj_ns):
  6.     // Process is in a parent (or same) namespace — capabilities apply
  7.     // Return intersection of process's effective caps and caps valid for target
  8.     return intersection(proc_caps, capabilities_valid_for(obj_ns))

  9. // Check if process's NS is a descendant of object's NS
 10. if is_ancestor(obj_ns, proc_ns):
 11.     // Process is in a child namespace — no capabilities against parent objects
 12.     return EMPTY_CAP_SET

 13. // Unrelated namespaces (neither ancestor nor descendant)
 14. // This happens with sibling containers
 15. return EMPTY_CAP_SET

is_same_or_ancestor(potential_ancestor, potential_descendant):
  // Walk up the hierarchy from potential_descendant toward root.
  // Return true if potential_ancestor is encountered (including if they're the same).
  // The parent link is a per-type STRONG `Option<Arc<UserNamespace>>`, so the
  // walk dereferences `cursor.parent` directly — NO `Weak::upgrade()`. A child
  // CANNOT outlive its parent (the strong edge pins every ancestor for the
  // child's whole life), so every ancestor on the chain is unconditionally
  // alive and there is no destroyed-ancestor case to handle. (The earlier
  // Weak-upgrade / "orphaned on parent drop" model was WRONG for these strong
  // edges and is retired — see the parent-child link semantics above.)
  cursor = potential_descendant
  while cursor != None:
      if cursor == potential_ancestor:
          return true
      cursor = cursor.parent  // Arc<UserNamespace> — parent is always alive while child exists
  return false

capabilities_valid_for(namespace):
  // Returns the set of capabilities valid for objects in this namespace.
  // Capabilities are restricted based on namespace ownership rules:
  let mut valid = ALL_CAPS
  // CAP_SYS_ADMIN operations that affect global kernel state (e.g., swapon,
  // mount --bind outside the mount namespace) are not valid in non-init namespaces.
  if namespace.is_non_init_user_ns():
      valid &= ~CAP_SYS_ADMIN_GLOBAL  // Remove host-affecting subset
      // Distributed/cluster-wide capabilities are stripped for non-init user
      // namespaces. Containers must not issue cluster-wide operations —
      // a compromised container should not be able to join/leave the cluster,
      // create DSM regions, or manage peer membership.
      valid &= ~CAP_CLUSTER_ADMIN     // Cluster join/leave, topology changes
      valid &= ~CAP_DSM_CREATE        // Create/destroy DSM regions
      valid &= ~CAP_PEER_MANAGE       // Peer membership, capability delegation
  // NOTE: network-namespace scoping is deliberately NOT expressed here.
  // `capabilities_valid_for` is a function of ONE user namespace; the
  // CAP_NET_ADMIN / CAP_NET_RAW scope is a property of the OBJECT's
  // NETWORK namespace — a `UserNamespace` has no `net_ns` field, and the
  // object is not a parameter of this function (the earlier
  // `namespace.net_ns != target_object.net_ns` clause here referenced a
  // nonexistent field and an out-of-scope identifier; it is retired).
  // Net-scoped objects are gated at their call sites instead: every
  // networking privilege check is `has_ns_cap(object_net_ns.user_ns,
  // CAP_NET_ADMIN)` against the OBJECT's owning network namespace's owner
  // (the `NetNamespace.user_ns` field contract in
  // [Section 17.1](#namespace-architecture--capability-domain-mapping)), so a
  // capability held in the caller's own user namespace reaches a foreign
  // namespace's devices/routes ONLY when that namespace's owning user ns
  // is reachable under has_ns_cap()'s ancestor rule.
  return valid

/// `CAP_SYS_ADMIN_GLOBAL` — a real UmkaOS capability bit (bit 87), defined in
/// [Section 9.2](09-security.md#permission-and-acl-model). It authorizes operations with cluster-wide or
/// host-global scope that go beyond what `CAP_SYS_ADMIN` permits.
///
/// In namespace context: tasks in non-init user namespaces have
/// `CAP_SYS_ADMIN_GLOBAL` stripped from their effective set (see
/// `capabilities_valid_for` above), preventing them from exercising
/// host-affecting operations regardless of other capabilities held.
///
/// Operations requiring `CAP_SYS_ADMIN_GLOBAL` (forbidden in non-init user namespaces):
/// - Creating new user namespaces when `user_namespaces_max` system limit is exceeded
/// - Mounting filesystems with `MS_STRICTATIME` in any namespace other than init
/// - Modifying HOST-GLOBAL kernel parameters — those not scoped to any
///   namespace — via `sysctl(2)`. Per-namespace sysctls remain writable by a
///   namespace-privileged caller (per-netns `net.*`, per-pidns
///   `kernel.pid_max`, per-userns `/proc/sys/user/max_*`, per-ipcns
///   `kernel.shmmax`/`kernel.sem`); only knobs backing shared host state are
///   gated here.
/// - Cluster-wide DLM lockspace management, shared overlay topology changes
///
/// `setns(2)` is deliberately NOT on this list. Joining another process's
/// PID/UTS/IPC/net/mnt namespace is gated by the per-namespace
/// `has_ns_cap(target-owner user ns, CAP_SYS_ADMIN)` checks in `sys_setns`
/// ([Section 17.1](#namespace-architecture--joining-namespaces-setns2-and-nsenter)) —
/// namespace-scoped, never init-only. Nested-container `docker exec`
/// depends on this: a container-privileged caller attaches to PID/UTS/IPC
/// namespaces owned by ITS OWN user namespace, which an init-only
/// CAP_SYS_ADMIN_GLOBAL gate would break outright (every `docker exec`
/// inside a container would fail). The Linux `pidns_install`/`utsns_install`/
/// Linux `ipcns_install`/`netns_install` paths likewise demand only namespace-scoped
/// CAP_SYS_ADMIN (`kernel/pid_namespace.c`, `kernel/utsname.c`,
/// `ipc/namespace.c`, `net/core/net_namespace.c`, verified against
/// torvalds/linux master).

Key invariant: The intersection() at line 8 ensures that if a process drops CAP_NET_ADMIN via capset(), it cannot exercise CAP_NET_ADMIN against any object, including objects in child namespaces. This upholds the guarantee in Section 9.9: "a dropped privilege can never be regained."

File capability interpretation:

File capabilities (set via setcap) are interpreted relative to the file's owning user namespace: 1. When execve() loads a binary with file capabilities, the kernel checks if the file's owning user namespace is the same as or an ancestor of the process's user namespace. 2. If the file is in a descendant namespace (i.e., the file was created inside a child namespace), its capability bits are ignored when executed from the parent — prevents a child namespace from granting capabilities in the parent. 3. If the file is in the same or an ancestor namespace, the file's capabilities are added to the process's permitted/effective sets, subject to the usual cap_bounding restrictions. This matches Linux semantics: a host binary with file caps is honored inside a container, but a container binary with file caps is not honored on the host.

Setuid/setgid binary behavior in nested namespaces:

Binary Location	Setuid Behavior	Rationale
Initial namespace (host)	UID changes in initial namespace	Traditional Unix behavior
Child namespace	UID changes within child namespace only	Cannot escalate to parent namespace UIDs
Mounted from host into container	Setuid bit ignored	Prevents host→container privilege escalation

Privilege escalation prevention:

A process in a child user namespace cannot modify the parent's UID mappings
setuid() inside a user namespace only affects the inner UID, not the outer UID
File capability bits (setcap) are interpreted relative to the file's owning user namespace
Signals from a less-privileged namespace to a more-privileged namespace are blocked unless explicitly allowed

17.1.7 User Namespace Mount Restrictions¶

Not all filesystem types are safe to mount from within an unprivileged user namespace. A filesystem that reads raw block device data (ext4, XFS, Btrfs) could exploit a crafted disk image to trigger kernel vulnerabilities. UmkaOS restricts which filesystems are mountable in non-init user namespaces via the FS_USERNS_MOUNT flag on the filesystem type registration.

bitflags! {
    /// Filesystem type flags, set at fs_type registration time.
    pub struct FsTypeFlags: u32 {
        /// This filesystem is safe to mount in a non-init user namespace.
        /// Only filesystems that do NOT interpret raw block device data
        /// and cannot be used to escalate privileges should set this flag.
        const FS_USERNS_MOUNT  = 1 << 0;
        /// Filesystem does not require a backing device (pseudo-fs: proc,
        /// sysfs, tmpfs, cgroup2). `mount_filesystem` IGNORES the `source` argument
        /// for such a filesystem (step 2b below). Mutually exclusive with
        /// FS_REQUIRES_DEV.
        const FS_NO_DEV        = 1 << 1;
        /// Filesystem requires a block device as `source`. `mount_filesystem`
        /// rejects a MISSING source with ENODEV and a NON-block source with
        /// ENOTBLK (step 2a below). Mutually exclusive with FS_NO_DEV.
        const FS_REQUIRES_DEV  = 1 << 2;
    }
}

Mount permission check in mount_filesystem():

mount_filesystem(source, target, fs_type, flags):
  1. Resolve fs_type by name from the registered filesystem table.
  2. Source resolution (enforces FS_REQUIRES_DEV / FS_NO_DEV):
     a. If fs_type.flags contains FS_REQUIRES_DEV: `source` MUST name a
        block device. If `source` is absent -> ENODEV; if it resolves to a
        path that is not a block device -> ENOTBLK. The resolved block
        device is handed to the filesystem's mount routine.
     b. If fs_type.flags contains FS_NO_DEV: `source` is IGNORED (pseudo-fs);
        no device is resolved and any `source` string is accepted, unused.
     c. FS_REQUIRES_DEV and FS_NO_DEV are mutually exclusive per fs_type; a
        registration setting both is rejected as a bug.
  3. If the calling task is NOT in the init user namespace:
     a. Check: fs_type.flags contains FS_USERNS_MOUNT.
        If not: return EPERM — this filesystem cannot be mounted in
        a non-init user namespace.
     b. Check: has_ns_cap(current.namespace_set.mount_ns.user_ns, CAP_MOUNT).
        The caller must have CAP_MOUNT (bit 70) within the mount namespace's
        owning user namespace (not the init user namespace).
        See [Section 14.6](14-vfs.md#mount-tree-data-structures-and-operations--mountfilesystem-mount-a-filesystem) for the
        canonical algorithm — the FS_REQUIRES_DEV / FS_NO_DEV source
        resolution of step 2 is enforced THERE, at the canonical mount_filesystem
        source-resolution step (this section defines the flags; that section
        is the enforcement site).
  4. If the calling task IS in the init user namespace:
     Check: has_cap(CAP_MOUNT).
  5. Proceed with mount.

Filesystems with FS_USERNS_MOUNT (safe for unprivileged user namespace mounts):

Filesystem	Rationale
`proc`	Virtual; no raw device access; per-PID-namespace view
`sysfs`	Virtual; read-only for non-init namespaces
`tmpfs`	Memory-backed; no device access
`overlayfs`	Layer composition; lower layers already mounted
`FUSE`	Userspace filesystem; kernel only relays operations
`devpts`	PTY slave filesystem; namespace-scoped
`mqueue`	POSIX message queue filesystem; namespace-scoped
`cgroup2`	Cgroup filesystem; namespace-scoped view

Filesystems WITHOUT FS_USERNS_MOUNT (require CAP_SYS_ADMIN in init user namespace):

Filesystem	Rationale
`ext4`, `xfs`, `btrfs`	Parse untrusted on-disk data structures
`nfs`	Network filesystem; requires kernel credential management
`zfs`	Complex on-disk format with kernel-level decompression
`fat`, `exfat`, `ntfs`	Parse untrusted on-disk data
`iso9660`	Parse untrusted on-disk data

This matches Linux behavior (since Linux 3.8+ user namespace mount restrictions) and is required for rootless container runtimes (Podman rootless, Docker rootless) to mount proc/sysfs/tmpfs inside unprivileged containers while preventing privilege escalation via crafted filesystem images.

17.1.8 Devtmpfs Namespace Awareness¶

Devtmpfs (Section 14.5) is a kernel-managed tmpfs that auto-populates /dev with device nodes. In a namespace-aware kernel, containers must not see all host devices.

Design: Devtmpfs itself is a single global instance (the kernel needs exactly one authoritative device registry). Container isolation of /dev is achieved through the mount namespace and device cgroup mechanism, not by creating per-namespace devtmpfs instances:

Mount namespace filtering: The container runtime creates a new mount namespace (CLONE_NEWNS), mounts a fresh tmpfs on /dev inside the container, and bind-mounts only the specific device nodes the container needs from the host's devtmpfs. Typically: /dev/null, /dev/zero, /dev/random, /dev/urandom, /dev/full, /dev/tty, /dev/ptmx, and any explicitly granted devices.
Device cgroup enforcement: The BPF_PROG_TYPE_CGROUP_DEVICE program (Section 17.2) attached to the container's cgroup denies open() on device nodes not in the allow-list. Even if a container process mknods a device node it has no cgroup permission for, the open will be denied by the BPF hook in device_node_open().
Net effect: A container sees a minimal /dev with only bind-mounted devices, and the device cgroup prevents access to any device not explicitly granted. This matches Docker/Kubernetes behavior exactly: docker run creates a restricted /dev with ~15 entries from the default device allow-list.

17.1.9 Security Policy Integration¶

Container isolation requires multiple defense layers beyond namespaces and capabilities. UmkaOS integrates with security policy mechanisms at specific points in the container lifecycle:

seccomp-bpf (Syscall Filtering): OCI-compliant container runtimes (Docker, containerd, CRI-O) require seccomp-bpf to restrict the syscall surface available to containerized processes. UmkaOS's seccomp implementation is part of the eBPF subsystem described in Section 19.2, which covers eBPF program types including seccomp-bpf for per-process syscall filtering. The typical container creation sequence is:

--- Privileged path (root or CAP_SYS_ADMIN) ---
1.  clone(CLONE_NEWPID | CLONE_NEWNS | CLONE_NEWNET | CLONE_NEWUSER | ...)
    — CLONE_NEWUSER is processed first internally (see ordering requirement above).
      The child inherits full capabilities within the new user namespace.

--- Rootless (unprivileged) path ---
1.  unshare(CLONE_NEWUSER)
    — Creates a new user namespace FIRST. The calling process gains CAP_SYS_ADMIN
      within the new user namespace, enabling subsequent namespace creation.
1a. Write UID/GID mappings to /proc/self/uid_map and /proc/self/gid_map.
1b. clone(CLONE_NEWPID | CLONE_NEWNS | CLONE_NEWNET | ...)
    — Now succeeds because the calling process has CAP_SYS_ADMIN in its
      (new) user namespace.

--- Common steps (both paths, in the child process) ---
2.  mount("overlay", new_root, "overlay",
         "lowerdir=<image_layers>,upperdir=<container_rw>,workdir=<overlay_work>")
    — Mount overlayfs with lowerdir=image layers (read-only), upperdir=container
      writable layer, workdir=overlay work directory. This assembles the container's
      root filesystem from the OCI image layer stack before pivot_root.
3.  pivot_root(new_root, put_old) — change filesystem root
4.  umount2(put_old, MNT_DETACH) — MANDATORY: detach host filesystem (security)
5.  Place the container init process in its cgroup. Two approaches:
    a. **Two-step (legacy)**: Write the PID to `cgroup.procs` in the target cgroup
       hierarchy, moving it from the parent's cgroup to the container-specific cgroup.
    b. **`CLONE_INTO_CGROUP` (preferred, Linux 5.7+)**: Pass the target cgroup fd
       via `clone3(CLONE_INTO_CGROUP)` at step 1, which places the child directly
       into the target cgroup at fork time — no post-fork migration needed. See
       [Section 8.1](08-process.md#process-and-task-management) for `CLONE_INTO_CGROUP` specification.
    Both approaches enforce per-container resource limits (memory, CPU, I/O) from
    this point forward.
6.  seccomp(SECCOMP_SET_MODE_FILTER, ...) — install syscall filter
7.  drop_capabilities() — reduce capability set
8.  execve() — exec container entrypoint

drop_capabilities — container-runtime userspace sequence (step 7):

drop_capabilities is NOT a kernel function or syscall — it is the USERSPACE sequence runc/crun perform at step 7 of the container-creation flow above, built entirely from the prctl(2) and capset(2) syscalls. It restricts the calling task's capability sets to those permitted by the OCI runtime spec process.capabilities, and is the last privilege reduction before execve() (without it a container process keeps all capabilities inherited from the root-running runtime). It is documented here, like the numbered flow around it, because it is load-bearing for container security; the KERNEL side is the capset/prctl semantics in Section 9.9. The runtime applies the OCI allowed sets in this order:

Bounding set: for each capability NOT in allowed.bounding, call prctl(PR_CAPBSET_DROP, cap) — permanently removed, unregainable even via setuid binaries; bounds which caps can enter the permitted set after execve().
Permitted set: intersect the current permitted set with allowed.permitted; drop the rest via capset(). Once dropped, a cap cannot be re-acquired (the bounding set prevents it).
Effective set: set to allowed.effective (typically = permitted, or empty for unprivileged containers that raise caps explicitly).
Inheritable set: set to allowed.inheritable — controls which caps survive execve() when combined with file caps (Docker default: empty; Kubernetes may set inheritable caps for init containers).
Ambient set: set to allowed.ambient — auto-added to permitted + effective on execve() without file caps; each ambient cap must also be in permitted AND inheritable (kernel enforced).

capset() returns EPERM if the runtime requests a capability it does not itself hold (a runtime bug). After the sequence, the task's effective set is a subset of the OCI-allowed set and no capability outside it can be regained by the task or its descendants (bounding-set guarantee). The OCI config the runtime builds (mirroring process.capabilities) is:

/// OCI capability configuration the runtime builds from the OCI spec's
/// `process.capabilities` object (userspace type; the five sets the step-7
/// sequence above applies via prctl/capset).
pub struct OciCapabilities {
    /// Capabilities retained in the bounding set.
    pub bounding: CapabilitySet,
    /// Capabilities in the effective set after drop.
    pub effective: CapabilitySet,
    /// Capabilities in the inheritable set.
    pub inheritable: CapabilitySet,
    /// Capabilities in the permitted set.
    pub permitted: CapabilitySet,
    /// Capabilities in the ambient set.
    pub ambient: CapabilitySet,
}

/// Capability set for OCI config — a TYPE ALIAS for the canonical
/// `SystemCaps` ([Section 9.1](09-security.md#capability-based-foundation)), NOT a second type.
/// `SystemCaps` is the `u128` bitflags already used by the whole
/// credential model (`TaskCredential`'s five capability sets) and by
/// accelerators (`caller_caps: &CapabilitySet`); the OCI
/// `process.capabilities` sets are expressed in that same layout (bits 0-63
/// Linux-compatible CAP_CHOWN..CAP_CHECKPOINT_RESTORE, bits 64-127
/// UmkaOS-native). Keeping it an alias means one canonical type, no From/Into
/// conversion, and no duplicate bit-layout to drift.
pub type CapabilitySet = SystemCaps;

Docker's default capability set retains 14 of the 41 capabilities: CAP_CHOWN, CAP_DAC_OVERRIDE, CAP_FSETID, CAP_FOWNER, CAP_MKNOD, CAP_NET_RAW, CAP_SETGID, CAP_SETUID, CAP_SETFCAP, CAP_SETPCAP, CAP_NET_BIND_SERVICE, CAP_SYS_CHROOT, CAP_KILL, CAP_AUDIT_WRITE. All other capabilities are dropped. Kubernetes restricted PodSecurityStandard drops all capabilities and allows only NET_BIND_SERVICE to be added back.

The seccomp filter must be installed before execve() so that the filter applies to the container's entrypoint and all its descendants. Docker's default seccomp profile blocks ~44 dangerous syscalls (e.g., kexec_load, reboot, mount). Kubernetes PodSecurityStandards mandate seccomp profiles for restricted workloads.

Seccomp Filter Stacking and Composition:

Nested containers require multiple independent seccomp filters to coexist on a single thread: the OCI runtime installs a broad container policy (filter F1) during container setup, and the container workload may subsequently install its own application-specific filter (filter F2) via prctl(PR_SET_SECCOMP) or seccomp(SECCOMP_SET_MODE_FILTER, ...). UmkaOS implements Linux-compatible stacking semantics so that existing container runtimes (runc, containerd) work without modification.

Stacking rules:

Filters stack: each seccomp(SECCOMP_SET_MODE_FILTER, ...) call appends a new filter to the thread's filter list. All previously installed filters remain active. Filters cannot be removed.
Evaluation order: on each syscall entry, filters are evaluated in reverse installation order — newest filter first, oldest filter last. All installed filters are evaluated; there is no short-circuit on SECCOMP_RET_ALLOW.

Exception: SECCOMP_RET_KILL_PROCESS and SECCOMP_RET_KILL_THREAD cause immediate thread or process termination without evaluating any remaining filters. This matches Linux behavior.

Action priority: when multiple filters return different actions for the same syscall, the highest-severity action wins regardless of evaluation order:

Priority	Action	Effect
1 (highest)	`SECCOMP_RET_KILL_PROCESS`	Terminate entire process
2	`SECCOMP_RET_KILL_THREAD`	Terminate calling thread
3	`SECCOMP_RET_TRAP`	Deliver SIGSYS
4	`SECCOMP_RET_ERRNO`	Return specified errno
5	`SECCOMP_RET_USER_NOTIF`	Notify supervisor via fd
6	`SECCOMP_RET_TRACE`	Notify ptrace tracer
7	`SECCOMP_RET_LOG`	Allow and log
8 (lowest)	`SECCOMP_RET_ALLOW`	Allow syscall

Example: if F1 returns SECCOMP_RET_ALLOW and F2 returns SECCOMP_RET_ERRNO(EPERM), the syscall is blocked with EPERM. A workload-installed filter can make the effective policy strictly more restrictive than the runtime-installed filter, but never less restrictive.

Unknown actions: If a filter returns an action value not in the table above (e.g., a future SECCOMP_RET_* value that UmkaOS does not yet recognize), UmkaOS treats it with SECCOMP_RET_KILL_THREAD-level priority (restrictive default). This matches Linux's signed-comparison semantics where unknown low numeric values get high priority. Unknown actions are never treated as SECCOMP_RET_ALLOW. See seccomp_action_priority() in Section 10.3 for the implementation.

NO_NEW_PRIVS requirement: a thread must have no_new_privs = 1 (set via prctl(PR_SET_NO_NEW_PRIVS, 1)) before installing a seccomp filter unless it holds CAP_SYS_ADMIN. This is identical to Linux. Container runtimes set no_new_privs as part of their standard setup sequence.
Maximum filter count: 512 filters per thread. Linux limits total BPF instruction count (MAX_INSNS_PER_PATH = 32768), not filter count; UmkaOS imposes an explicit filter-count ceiling at 512 (matching Section 10.3). Attempting to install a 513th filter returns E2BIG.
Filter inheritance: child processes created via fork() or clone() inherit the parent's complete filter stack. The inherited filters are immutable in the child — the child may only append further filters, never remove inherited ones.
Memory lifecycle: Each compiled seccomp filter is reference-counted via Arc<SeccompFilter>. On fork(), the child increments the refcount of every filter in its inherited stack. On task exit, the task drops its Arc references to all filters in its stack; when the last Arc reference to a filter drops, the filter's BPF bytecode and compiled representation are freed. The maximum memory per task with 512 stacked filters is bounded: 512 Arc increments on fork, 512 Arc decrements on exit.

Nested container policy: when an OCI runtime installs filter F1 (broad allow-list, blocking dangerous syscalls) and the container workload subsequently installs filter F2 (narrow application allow-list), both filters are active simultaneously. The effective policy is the union of restrictions from both filters: a syscall is allowed only if both F1 and F2 allow it. This composability property is what makes layered container security correct — deeper container nesting cannot relax an outer filter's restrictions.

UmkaOS implementation note: UmkaOS compiles the filter stack into a single BPF program at installation time. When a new filter is added to an existing stack, the kernel combines the compiled representation of the existing stack with the new filter's BPF bytecode and recompiles the result into a single executable program. This single-program approach is semantically identical to sequential per-filter evaluation (the action priority table above is preserved exactly) but eliminates repeated per-filter dispatch overhead at syscall entry. The recompilation occurs once at seccomp(SECCOMP_SET_MODE_FILTER, ...) time, not on each syscall.

LSM Integration: UmkaOS supports pluggable Linux Security Modules (AppArmor, SELinux profiles). Container runtimes can specify an LSM profile via OCI annotations, which UmkaOS applies at execve() time. The integrity measurement framework (Section 9.5, 08-security.md) provides the foundation for policy enforcement. The full LSM framework — hook table, security blob allocation, module registration, and AND-logic stacking — is specified in Section 9.8.

17.1.10 Cross-Node Namespace ID Translation¶

In a distributed UmkaOS cluster (Section 5.1), each node maintains its own independent PID, UID, and mount namespace hierarchies. When a capability or IPC message crosses node boundaries, namespace-scoped identifiers (PIDs, UIDs, GIDs) must be translated.

Protocol: Cross-node operations use cluster-global identifiers rather than translating between per-node namespace IDs:

PID namespace: Each task has a cluster-unique ClusterTaskId = (node_id: u16, task_id: TaskId) — the node id plus the task's global u64 TaskId, which is NEVER reused (Section 8.1). It is deliberately NOT (node_id, local_pid): pid_t numbers are recycled per namespace (pid_allocator_free() returns them to the IDR for reallocation), so a pid-carrying ClusterTaskId would reintroduce across the cluster exactly the kill-after-reuse race the never-reused TaskId model eliminates locally — a cross-node kill() arriving after the target died and its number was recycled would hit an unrelated victim. This is a UmkaOS-owned wire protocol (no external u32 mandate), so the u64-identifier rule applies. Wire form: { node_id: Le16, _pad: [u8; 6], task_id: Le64 } — 16 bytes, padding explicit. Cross-node kill() and waitpid() operate on ClusterTaskId; the receiving node resolves task_id directly via find_task_by_tid() — a dead task yields ESRCH, never a recycled victim — and applies its local namespace visibility rules to the resolved task. Tasks are addressed by kernel identity, not by any namespace's number space; there is no init-namespace PID translation step.
User namespace (UID/GID): Cross-node operations assume a shared UID/GID directory service (LDAP, /etc/passwd synchronisation). The wire protocol carries raw uid_t/gid_t values. The receiving node interprets them in its init user namespace. Non-init user namespace UID mappings are strictly node-local and are NOT translated across nodes. A container's mapped UIDs are meaningful only on the node hosting that container.
Mount namespace: Mount namespaces are strictly node-local. Cross-node filesystem access uses the capability-based VFS service provider protocol (Section 14.1), not mount namespace sharing. A remote file access carries a (node_id, inode_id, fs_id) triple — the receiving node resolves this against its own mount table.
Network namespace: Cross-node network namespace awareness is limited to ClusterTaskId-scoped socket operations. The networking stack on each node operates independently; cross-node traffic uses the RDMA transport layer (Section 5.4), which bypasses per-node network namespaces.

See also: - Section 19.2: eBPF subsystem including seccomp-bpf - Section 9.5 (08-security.md): Runtime Integrity Measurement (IMA) - Section 9.9: Credential model and capability dropping

17.2 Control Groups (Cgroups v2)¶

Linux cgroups v2 provide hierarchical resource allocation and limiting. UmkaOS implements the unified cgroup v2 interface, mapping controller semantics to UmkaOS's native scheduler, memory manager, and I/O subsystems.

Cgroup v1 compatibility shim: Docker (Moby) and older systemd versions (pre-247) require cgroup v1 hierarchy paths. UmkaOS provides a read-mostly v1 compatibility shim that: - Exposes /sys/fs/cgroup/{cpu,memory,pids,blkio,...} mount points - Translates v1 control file reads/writes to v2 equivalents (e.g., memory.limit_in_bytes → memory.max, cpu.shares → cpu.weight) - Supports the 4 most common v1 controllers: cpu, memory, blkio, pids - Returns -ENOSYS for v1-only features with no v2 equivalent (e.g., cpuacct separate hierarchy, net_cls, net_prio) - Multi-hierarchy emulation: each v1 controller appears as a separate mount, but all are backed by the single v2 unified hierarchy

Specification scope: The v1 shim control file format details (exact file paths, value format, error responses) are deferred to Phase 4. The core v2 implementation below is the authoritative resource control mechanism. Until Phase 4, Moby/systemd v1 compatibility cannot be integration-tested — only the semantic translation (which v1 files → which v2 controls) is validated.

Note: Modern Docker (Moby ≥ 20.10), Kubernetes (≥ 1.25), and systemd (≥ 247) all work natively with cgroup v2. The v1 shim is needed only for legacy container runtimes. Systems running current versions of these tools can operate entirely on the v2 interface without the shim.

Phase 3 gate: The cgroup v2 procfs/sysfs detection surface (/sys/fs/cgroup/cgroup.controllers, /proc/self/cgroup in 0::/ format, cgroup2 mount type) is a Phase 3 exit requirement. Without correct v2 detection responses, Docker/runc falls back to cgroup v1 mode. See Section 24.2 for the full checklist.

17.2.1 Core Data Structures¶

17.2.1.1 Cgroup Node¶

The Cgroup struct is the central object in the cgroup v2 hierarchy. Every directory under /sys/fs/cgroup/ corresponds to one Cgroup node. The hierarchy is a tree; the root node is owned by CgroupRoot.

UmkaOS's cgroup design avoids two sources of complexity present in Linux's implementation: - No multi-hierarchy: cgroup v1's per-controller separate hierarchies are gone; the single v2 unified hierarchy is the only model. The v1 shim (see above) re-exposes v2 state through legacy paths at the cgroupfs layer without creating second hierarchies inside the kernel. - No cgroup_subsys indirect call: Linux routes every controller operation through a Linux cgroup_subsys vtable, adding an indirect CALL on every resource charge. UmkaOS gives each built-in controller a dedicated typed field reached through a per-controller RcuPtr<ControllerState> (interior mutability for runtime enable/disable, lock-free reads): a disabled controller is a NULL pointer (a cheap branch, no state), and an enabled one is read with a direct typed atomic access after a single Acquire pointer load — no virtual dispatch on any charge. See the Cgroup "Memory layout note" for why RcuPtr rather than a bare embedded Option.

/// A cgroup node in the cgroup v2 unified hierarchy.
///
/// The hierarchy is a tree rooted at `CgroupRoot.root`. Tasks are assigned
/// to leaf or intermediate cgroups. Resource controllers operate per-cgroup.
///
/// # Memory layout note
/// Controller state is reachable through a per-controller `RcuPtr<T>` (a nullable
/// RCU-owned pointer), not a bare embedded `Option<T>`. The `RcuPtr` is REQUIRED,
/// not a preference: the `cgroup.subtree_control` handler enables/disables
/// controllers on already-shared children at runtime, writing the field through a
/// shared `&Arc<Cgroup>` — which a bare `Option` (writable only through `&mut`)
/// cannot express, while a lock would serialize the hot-path charge readers. A
/// disabled controller is a NULL pointer, so the hot-path check is one `Acquire`
/// load + null branch under an `RcuReadGuard` (disabled = essentially free); an
/// enabled controller costs one data indirection to reach its (typed,
/// direct-atomic) fields — matching Linux's own `cgroup->subsys[]` pointer array,
/// and still with NO Linux `cgroup_subsys` indirect CALL. The writer publishes under the
/// `config_lock` `MutexGuard` (the sealed `WriterProof`) and the previous box is
/// reclaimed after a grace period, so no reader can observe a freed controller.
/// See the "Resource controller state" field group below for the full rationale.
/// Alias for cgroup tree nodes. All cgroup references are `Arc<Cgroup>` —
/// tree ownership flows downward (parent → children); parent pointers are `Weak`.
/// Maximum cgroup nesting depth (stack-safety bound). Linux uses `INT_MAX`,
/// i.e., effectively unlimited; 256 levels are sufficient for all practical
/// deployments including deeply nested container orchestrators.
pub const CGROUP_MAX_DEPTH: usize = 256;

/// Maximum number of cgroups on any root-to-node path, INCLUSIVE of both the
/// root (depth 0) and the deepest node (depth `CGROUP_MAX_DEPTH`). A node at
/// depth `d` has `d + 1` cgroups on its path to the root, so a walk that visits
/// every cgroup from a node up to and including the root can touch up to
/// `CGROUP_MAX_DEPTH + 1` = 257 nodes. Root-inclusive ancestor walks
/// (`pids_precharge_fork`, `rollback_fork`) size their `ArrayVec` to this bound —
/// NOT `CGROUP_MAX_DEPTH`, which is one short and would panic on `push` at the
/// maximum-depth boundary. (Component walks that STOP before the root — e.g.
/// `cgroup_path_from_reader` collecting names between a node and its ns root —
/// use `CGROUP_MAX_DEPTH` instead, since they exclude one endpoint.)
pub const CGROUP_MAX_PATH_NODES: usize = CGROUP_MAX_DEPTH + 1;

pub type CgroupNode = Arc<Cgroup>;

/// Global cgroup registry: `CgroupId` → `Arc<Cgroup>` (integer key → XArray
/// per [Section 3.13](03-concurrency.md#collection-usage-policy); RCU-compatible lock-free reads).
///
/// This is the resolution path for the `CgroupId` values cached in scheduler
/// entities (`EevdfTask.cgroup_id`) — the scheduler's `cgroup_from_id()`
/// helper ([Section 7.1](07-scheduling.md#scheduler--eevdf-algorithm-specification)) reads this
/// XArray from tick context (preemption disabled = implicit RCU read
/// section under non-preemptible RCU).
///
/// **Writers**: `cgroup_mkdir()` inserts after the cgroup is fully
/// initialized (id assigned, controllers allocated). `cgroup_rmdir()` does
/// NOT erase here — it latches `lifecycle = Draining/Dead` and leaves the
/// entry as a resolvable TOMBSTONE so a residual charge that outlives the
/// cgroup (RDMA/perf/hugetlb/in-flight-bio uncharge) can still resolve its
/// anchored counter. The entry is erased — dropping the LAST `Arc<Cgroup>` and
/// so freeing the struct — only when `Cgroup.residual_refs` reaches 0 after
/// `lifecycle == Dead` (drain Phase 6 or the last residual uncharge, whichever
/// observes the drained state; see
/// [Section 17.2](#control-groups--registry-tombstone-and-residual-charge-drain)). Because
/// the registry entry itself holds the strong reference, the tombstone keeps
/// the cgroup alive exactly as long as a residual charge can still target it —
/// Linux's offline-css / `percpu_ref` model. A reader that resolved a cgroup
/// reference under RCU never observes a freed cgroup. Writes are serialized by
/// `HIERARCHY_LOCK` (write mode); the final erase is serialized by the
/// Dekker interlock in
/// [Section 17.2](#control-groups--registry-tombstone-and-residual-charge-drain).
///
/// IDs are never reused (`id` is allocated from a monotonic u64 counter),
/// so a stale cached `CgroupId` resolves to the tombstone (same cgroup, now
/// Dead) until final erase, then to `None` — never to a DIFFERENT cgroup.
pub static CGROUP_REGISTRY: XArray<Arc<Cgroup>> = XArray::new();

/// Stable identifier of a cgroup, equal to `Cgroup.id` and the cgroupfs
/// directory inode number. Allocated from `CgroupRoot.next_id` (monotonic
/// `AtomicU64`, never reused). The single canonical resolution path is
/// `CGROUP_REGISTRY.get(id)`. `u64` (not `NonZeroU64`) because it is the key
/// type of the `XArray` registry and mirrors `Cgroup.id: u64`; 0 is never
/// allocated (the root cgroup is `id == 1`), so a zeroed `MemCgroupStock`
/// slot's `Option<CgroupId>` still distinguishes empty from any real id.
pub type CgroupId = u64;

/// Identifier for a cgroup subsystem that is *registered* at boot rather than
/// carried as a dedicated built-in controller field in `Cgroup`. Indexes
/// `Cgroup.dyn_subsys`. Built-in controllers (`cpu`, `memory`, `io`, `pids`,
/// `cpuset`, `rdma`, `hugetlb`, `misc`, `perf_event`) are NOT here — they are
/// hot-path and stay dedicated typed `RcuPtr<T>` fields.
#[repr(u32)]
pub enum CgroupSubsysId {
    /// ML policy per-cgroup parameter overrides ([Section 23.1](23-ml-policy.md#aiml-policy-framework-closed-loop-kernel-intelligence)).
    MlPolicy = 0,
    /// Accelerator compute/memory limits ([Section 22.5](22-accelerators.md#accelerator-isolation-and-scheduling--cgroup-integration)).
    Accel    = 1,
}

/// Number of registered-subsystem slots per cgroup. Sized to the registered
/// subsystems above with headroom; validated at registration (`register_cgroup_subsys`
/// panics if `id >= MAX_DYN_CGROUP_SUBSYS`). Not a system-wide policy limit —
/// a compile-time slot-array bound like `NUMA_NODES_STACK_CAP`.
pub const MAX_DYN_CGROUP_SUBSYS: usize = 4;

/// Header every registered subsystem-state object embeds as its FIRST field
/// (`#[repr(C)]`, so a `*mut CgroupSubsysStateHeader` from a `dyn_subsys` slot
/// can be up-cast to the concrete `T` after an `id` match). Replaces the
/// earlier subsystem-state `cgroup(&self) -> &Arc<Cgroup>` contract,
/// which was unimplementable: it forced every implementor to OWN a strong
/// `Arc<Cgroup>` back-edge, recreating the exact parent↔child reference cycle
/// the `MemCgroup::owner: Weak<Cgroup>` design exists to prevent. The
/// subsystem stores a `CgroupId` (a plain integer, no strong edge) and
/// resolves to the live `Cgroup` on demand via `CGROUP_REGISTRY` under RCU.
#[repr(C)]
pub struct CgroupSubsysStateHeader {
    /// Owning cgroup's id. Resolve to `Option<&Cgroup>` via
    /// `CGROUP_REGISTRY.get(cgroup_id)` inside an RCU read section — returns
    /// `None` once the cgroup has been unpinned by `rmdir` (the same
    /// stale-id-resolves-to-None discipline the scheduler relies on).
    pub cgroup_id: CgroupId,
    /// Which subsystem this header belongs to (a `CgroupSubsysId` discriminant).
    /// Checked before the `subsys_state::<T>()` up-cast.
    pub subsys_id: u32,
    /// Live reference count for the state object (signed, Linux convention:
    /// negative detects double-free). Dropped to zero by `free_state` at drain.
    pub refcount: AtomicI32,
}
// kernel-internal (never KABI/wire), but `#[repr(C)]` for the offset-0 up-cast.
// Layout: 8 (cgroup_id) + 4 (subsys_id) + 4 (refcount) = 16 bytes.
const_assert!(size_of::<CgroupSubsysStateHeader>() == 16);

/// Callbacks a registered subsystem provides at `register_cgroup_subsys()`
/// time (boot, Phase 5c). All run on warm/cold cgroup lifecycle paths, never
/// the hot path. `alloc_state` returns the heap pointer stored into the owning
/// cgroup's `dyn_subsys` slot; `free_state` is invoked during the cgroup drain
/// (Phase 3.5) after the task list is empty.
pub struct CgroupSubsysOps {
    pub alloc_state: fn(cgroup_id: CgroupId) -> Result<*mut CgroupSubsysStateHeader, KernelError>,
    pub free_state:  fn(*mut CgroupSubsysStateHeader),
    pub activate:    fn(*mut CgroupSubsysStateHeader),
    pub deactivate:  fn(*mut CgroupSubsysStateHeader),
}

/// Boot-time subsystem registry (indexed by `CgroupSubsysId`). Populated by
/// `register_cgroup_subsys(id, ops)` during subsystem init; read-only
/// thereafter, so a plain array of `Option` behind a `OnceCell`-style
/// write-once discipline suffices (no runtime lock).
pub static CGROUP_SUBSYS_OPS: [OnceCell<CgroupSubsysOps>; MAX_DYN_CGROUP_SUBSYS] =
    [const { OnceCell::new() }; MAX_DYN_CGROUP_SUBSYS];

pub struct Cgroup {
    /// Unique cgroup ID (assigned at creation, never reused).
    /// Also used as the inode number of the cgroupfs directory.
    pub id: u64,

    /// Parent cgroup. `None` only for the root cgroup (id == 1).
    /// `Weak` avoids reference cycles: the tree is owned downward
    /// (`CgroupRoot → Arc<Cgroup> → Arc<Cgroup> children`); the
    /// parent pointer is a non-owning back-edge.
    pub parent: Option<Weak<Cgroup>>,

    /// Child cgroups. RCU-protected for lockless read traversal
    /// (`for_each_descendant`, cgroupfs directory listing, recursive
    /// accounting). Writers (mkdir, rmdir) acquire both `hierarchy_lock`
    /// in `CgroupRoot` and `children_lock`, then publish changes via RCU:
    ///   1. Clone the Vec under the SpinLock.
    ///   2. Modify the clone (insert or remove child).
    ///   3. Swap the RcuCell to point to the new Vec.
    ///   4. Old Vec is freed after the RCU grace period.
    /// Readers call `children.read()` under `rcu_read_lock()` — no lock
    /// acquisition, no contention. The Vec is unbounded (K8s may create
    /// hundreds of cgroups under `system.slice`) but acceptable per
    /// collection policy §3.1.13: cgroup creation is cold-path.
    ///
    /// **Performance note**: Clone-and-swap is O(N) per mkdir/rmdir where
    /// N = number of siblings. At 500 siblings, the clone copies 500 * 8 =
    /// 4000 bytes (~1us). For extreme cgroup counts (>10K siblings), XArray
    /// migration would improve scalability, but K8s pod creation (~1-10/sec)
    /// is well within the cold-path budget.
    pub children: RcuCell<Vec<Arc<Cgroup>>>,
    /// SpinLock protecting `children` writes. Only held during structural
    /// modifications (mkdir/rmdir) — never on the read path. Its
    /// `SpinLockGuard` is the `WriterProof` passed to `children.update()`
    /// ([Section 3.4](03-concurrency.md#cumulative-performance-budget--rcu-protected-container-types)) —
    /// a short non-sleeping structural edit is exactly what the SpinLock
    /// implementor of the sealed proof trait exists for.
    pub children_lock: SpinLock<()>,

    /// Depth from the root cgroup (root = 0, root's children = 1, etc.).
    /// Used by `cgroup_lca()` to compute the Lowest Common Ancestor
    /// during task migration. Set at `cgroup_mkdir()` time as
    /// `parent.depth + 1`. Maximum: `CGROUP_MAX_DEPTH` (256).
    pub depth: u32,

    /// Name of this cgroup relative to parent (max 255 bytes, no '/').
    /// Fixed-size inline storage avoids heap allocation for short names
    /// (typical names: "docker", "system.slice", container IDs ≤ 64 bytes).
    pub name: CgroupName,

    /// Tasks directly assigned to this cgroup (not descendants).
    /// Written by task migration; read by cgroupfs `cgroup.procs` output.
    ///
    /// Uses `RwLock<XArray<()>>` keyed by `TaskId` (integer key) for O(1) insert,
    /// remove, and membership test. Per the collection policy, integer-keyed
    /// membership sets use XArray (not HashMap/FxHashSet). The unit value `()`
    /// means this is a pure membership set — only presence/absence matters.
    /// Readers (cgroupfs `cgroup.procs` output) take a read lock; writers
    /// (task migration step 10, `commit_fork()`, `detach_exiting_task()`) take a
    /// write lock.
    ///
    /// **Lock level**: `CGROUP_TASKS_LOCK` (215) in the master lock table
    /// ([Section 3.4](03-concurrency.md#cumulative-performance-budget--lock-ordering)). Same-level
    /// tiebreaker when holding two instances (migration step 10's
    /// source/target pair): ascending `CgroupId` order, released in reverse.
    pub tasks: RwLock<XArray<()>>,

    /// Number of tasks assigned **directly** to this cgroup (NOT descendants).
    /// Written only by the three membership sites — `commit_fork()`,
    /// `detach_exiting_task()` step 4, task migration steps 10-11 — each of which
    /// touches ONLY this local counter (one atomic on the leaf cgroup), never
    /// a to-root walk.
    ///
    /// **Subtree-populated propagation (0↔1 only)**: whole-subtree population
    /// is derived, not stored per-node as a running total. A cgroup's subtree
    /// is populated iff `population > 0 || nr_populated_children > 0`
    /// (`Cgroup::is_populated()`). The only ancestor-ward walk happens when a
    /// cgroup's subtree-populated state *flips* (its `population` goes 0→1 with
    /// no populated children, or 1→0 with none): `cgroup_propagate_populated()`
    /// then adjusts each ancestor's `nr_populated_children` and STOPS at the
    /// first ancestor whose own populated state does not flip. This is the
    /// Linux `cgroup_update_populated()` model (`kernel/cgroup/cgroup.c`,
    /// `torvalds/linux` master): steady-state fork/exit into an
    /// already-populated cgroup touches exactly one cacheline (the leaf's
    /// `population`), never the root's. It eliminates the globally-contended
    /// root-cacheline bounce that a naive to-root `fetch_add`/`fetch_sub` per
    /// fork+exit would cause at 10⁴–10⁶ ops/s on 256 cores — the negative-
    /// overhead requirement ([Section 3.4](03-concurrency.md#cumulative-performance-budget)).
    ///
    /// **Flip-edge claim (race-free without a hot-path lock)**: the task that
    /// causes the `population` 0↔1 *edge* is identified by the RETURN VALUE of
    /// its own `population.fetch_add`/`fetch_sub` — `prev == 0` on an add /
    /// `prev == 1` on a sub is the only direct-membership op that CAN change
    /// `is_populated()` via the `population` term; any other return value is a
    /// steady-state op that returns WITHOUT touching `flip_lock` or any ancestor
    /// (the one-cacheline fast path). Only such an edge task enters the flip
    /// path, takes the leaf's `flip_lock`, and reconciles the leaf's
    /// `subtree_populated` bit: it recomputes `is_populated()` under the lock and
    /// propagates to the parent ONLY if that differs from the published bit (see
    /// `subtree_populated` — a bare `nr_populated_children == 0` test here would
    /// race a concurrent child-subtree edge and double-count). Linux runs the
    /// equivalent Linux decision under `css_set_lock`; UmkaOS confines the
    /// serialization to the cold per-cgroup `flip_lock` so the steady state
    /// stays lock-free. See `flip_lock`.
    ///
    /// Used by `rmdir`'s emptiness gate (`is_populated() == false`), the
    /// "a populated descendant keeps every ancestor un-`rmdir`-able, hence
    /// strongly reachable from the root" `Weak::upgrade()` liveness proofs
    /// (`cgroup_lca`, `pids_precharge_fork`, `commit_fork` — see
    /// `is_populated()`), and "is this cgroup populated?" checks.
    pub population: AtomicU64,

    /// Number of immediate children whose SUBTREE is populated (each counted
    /// once, regardless of how many tasks its subtree holds). Maintained by
    /// `cgroup_propagate_populated()` on 0↔1 subtree-populated transitions of a
    /// child. Together with `population`, defines `is_populated()`. `AtomicU32`
    /// (bounded by the sibling count). Every increment/decrement AND the
    /// paired `is_populated()` re-evaluation happen under THIS cgroup's
    /// `flip_lock` (taken by `cgroup_propagate_populated` for each ancestor it
    /// visits), so two concurrent flip walkers passing through the same ancestor
    /// cannot both mis-read the stop condition — the earlier claim that the
    /// `Acquire`/`Release` on `population` serialized these was WRONG (it orders
    /// one cgroup's own counter, not two independent ancestor walks), and caused
    /// the double-count-→-permanent-rmdir-EBUSY and `fetch_sub`-at-0 u32-wrap
    /// races. The lock is cold (0↔1 edges only). See `flip_lock`.
    pub nr_populated_children: AtomicU32,

    /// Serializes the SUBTREE-populated flip decision for this cgroup — the
    /// read-modify-decide sequence on `population`/`nr_populated_children` that
    /// `commit_fork`, `detach_exiting_task`, migration step 11, and
    /// `cgroup_propagate_populated` (per visited ancestor) perform. Taken ONLY
    /// on the cold 0↔1 transition path: a steady-state fork/exit into an
    /// already-populated cgroup returns on its `fetch_add`/`fetch_sub` return
    /// value (`prev > 0` / `prev > 1`) BEFORE reaching this lock, so the
    /// one-cacheline fast path never contends it. Per-cgroup, so it is NOT the
    /// globally-contended root cacheline the propagation design removed. A
    /// `SpinLock` (the guarded window is atomics only, never sleeps). Held below
    /// `CGROUP_TASKS_LOCK` and above the runqueue locks; never nested with
    /// another cgroup's `flip_lock` (the walk drops each before taking the next).
    pub flip_lock: SpinLock<()>,

    /// PUBLISHED subtree-populated state — the single source of truth the flip
    /// PROPAGATION decision reads, written ONLY under `flip_lock`. `1` iff this
    /// cgroup's subtree held a task as of the last under-lock reconcile.
    ///
    /// **Why a published bit and not a bare counter read.** The propagation
    /// decision ("did MY edge flip this cgroup's subtree-populated state?")
    /// cannot be answered by testing one counter: a direct-membership edge
    /// changes `population` (claimed lock-free by the `fetch_add`/`fetch_sub`
    /// return value), while a child-subtree edge changes `nr_populated_children`
    /// (under `flip_lock`). If the leaf decision tested only
    /// `nr_populated_children == 0`, a fork's `population` 0→1 edge racing a
    /// child's subtree-emptying could BOTH conclude "subtree flipped" for the
    /// same cgroup — double-incrementing the parent's `nr_populated_children`
    /// and pinning it populated forever (permanent `rmdir` EBUSY). Instead,
    /// every edge task (direct-membership OR child-walker), under `flip_lock`,
    /// recomputes `is_populated()` (which reads BOTH counters) and compares it
    /// to this bit: propagate the delta to the parent ONLY when the two differ,
    /// then republish. Because the recompute-compare-republish runs entirely
    /// under `flip_lock`, two racing edges at the same cgroup each observe the
    /// other's committed counter change and the shared bit, so exactly one edge
    /// drives each true 0↔1 subtree flip. Consumers of "is this subtree
    /// populated?" (rmdir gate, `Weak::upgrade()` liveness) still call
    /// `is_populated()`; this bit lags the counters by at most one under-lock
    /// reconcile and exists only to make the propagation decision race-free.
    /// `AtomicU8` (0/1) — a repr-adjacent flag defended against a torn read.
    /// Initialized `0` (a fresh cgroup is unpopulated).
    pub subtree_populated: AtomicU8, // 0 = unpopulated, 1 = populated

    /// In-flight `CLONE_INTO_CGROUP` forks that resolved THIS cgroup as their
    /// target but have not yet completed `commit_fork()`. Incremented
    /// (Acquire) by `precharge_fork` when the target is this cgroup,
    /// decremented (Release) by `commit_fork`/`rollback_fork`. Gates
    /// `cgroup_rmdir` step 3a: a nonzero count blocks destruction (EBUSY),
    /// closing the window where an open cgroup fd is used to `clone3` into a
    /// cgroup that `rmdir` is tearing down. Separate from `population` because
    /// the child is not yet a member (population increment happens in
    /// post_fork). u64: bounded by concurrent forks, never wraps.
    pub fork_pins: AtomicU64,

    // ── Resource controller state ────────────────────────────────────────
    // Each field is NULL (`RcuPtr::null()`) when the controller is disabled for
    // this cgroup. Controller state is present only when the controller is listed
    // in the parent's `subtree_control` mask (or, for the root cgroup, in
    // `cgroup.controllers`).
    //
    // **Why `RcuPtr<T>`, not `Option<T>`**: the `cgroup.subtree_control` write
    // handler ENABLES and DISABLES controllers on EXISTING, already-shared children
    // at runtime — `child_controller_alloc`/`child_controller_free` mutate these
    // fields through a shared `&Arc<Cgroup>` (the child is reachable to concurrent
    // RCU readers via the published `children` list). A bare `Option<T>` cannot be
    // written through `&Arc<Cgroup>` (no `&mut`), and a lock would put the hot-path
    // charge/tick/bio-submit readers behind a critical section. `RcuPtr<T>` gives
    // interior mutability with lock-free reads: the hot paths read the controller
    // under an `RcuReadGuard` (`field.read(&guard) -> Option<&T>`, one `Acquire`
    // pointer load + null branch — disabled = NULL, essentially free); the writer
    // publishes under the `config_lock` `MutexGuard` as the sealed `WriterProof`
    // ([Section 3.4](03-concurrency.md#cumulative-performance-budget--rcuptrt-nullable-single-owner-rcu-pointer)),
    // and the previous box is reclaimed after a grace period so no reader observes a
    // freed controller. Merit: lock-free reads + a typed writer proof. This is the
    // SAME house pattern the embedded `CpuController.cbs: RcuPtr<CbsGroupConfig>`
    // field already uses. Reading a controller's own fields (`weight`, `usage`, …)
    // stays a direct typed atomic access after the one pointer load — there is still
    // NO `cgroup_subsys` indirect CALL; the single data indirection matches Linux's
    // own `cgroup->subsys[]` pointer array.

    /// CPU bandwidth controller (`cpu.weight`, `cpu.max`, `cpu.guarantee`).
    pub cpu: RcuPtr<CpuController>,

    /// Memory controller (`memory.max`, `memory.high`, `memory.current`, etc.).
    /// `Arc`-wrapped inside the `RcuPtr` (`RcuPtr<Arc<MemCgroup>>`) so that
    /// `MemCgroup::owner: Weak<Cgroup>` is a genuine back-edge (strong
    /// `Cgroup → MemCgroup`, weak `MemCgroup → Cgroup`, no cycle) and so external
    /// long-lived holders — `OomContext.memcg`, `HIBERNATE_CANDIDATES`
    /// ([Section 4.5](04-memory.md#oom-killer)) — can keep independent strong clones that outlive any
    /// single RCU read section: a reader loads `&Arc<MemCgroup>` under the guard,
    /// `Arc::clone`s it (one refcount bump), and drops the guard.
    /// `Cgroup::memcg(&guard)` performs exactly that guarded load-and-borrow.
    pub memory: RcuPtr<Arc<MemCgroup>>,

    /// Block I/O controller (`io.max`, `io.weight`).
    pub io: RcuPtr<IoController>,

    /// PID controller (`pids.max`, `pids.current`).
    pub pids: RcuPtr<PidsController>,

    /// CPU affinity controller (`cpuset.cpus`, `cpuset.mems`, partition mode).
    pub cpuset: RcuPtr<CpusetController>,

    /// RDMA/InfiniBand resource controller (`rdma.max`).
    pub rdma: RcuPtr<RdmaController>,

    /// Huge page controller (`hugetlb.<size>.max`).
    pub hugetlb: RcuPtr<HugetlbController>,

    /// Miscellaneous resource controller (`misc.max`; e.g., SGX EPC pages).
    pub misc: RcuPtr<MiscController>,

    /// perf_event cgroup controller. Limits per-cgroup PMU resource usage
    /// to prevent container perf_event exhaustion.
    pub perf_event: RcuPtr<PerfEventController>,

    // ── Registered (non-built-in) subsystem state ────────────────────────

    /// Per-cgroup state for subsystems registered at boot rather than embedded
    /// as a built-in controller field above — `ml_policy` ([Section 23.1](23-ml-policy.md#aiml-policy-framework-closed-loop-kernel-intelligence))
    /// and `accel` ([Section 22.5](22-accelerators.md#accelerator-isolation-and-scheduling--cgroup-integration)).
    /// One `AtomicPtr` slot per `CgroupSubsysId`; null = not attached to this
    /// cgroup. The subsystem's `alloc_state` publishes its state pointer here
    /// (lazily, on first use), `free_state` clears it during the drain (Phase 3.5).
    ///
    /// **Why a slot array and not more built-in fields**: the built-in
    /// controllers are hot-path (memory charge, scheduler tick, bio submit) and
    /// get dedicated TYPED `RcuPtr<T>` fields — a direct typed read after one
    /// `Acquire` pointer load, no id-match and no `dyn` up-cast. Registered
    /// subsystems are warm/cold ONLY (ML override decay every 2 s, accel limit
    /// checks on allocation), so the extra id-match + `*mut CgroupSubsysStateHeader`
    /// up-cast per walk step is acceptable and buys dynamic registration without a
    /// typed field per rarely-used subsystem. Both are single relaxed pointer loads;
    /// the built-in path avoids only the id-match/up-cast — neither uses an indirect
    /// call.
    /// `SubsysDescendants` (via `descendants()`)/`subsys_state` read these slots under RCU.
    pub dyn_subsys: [AtomicPtr<CgroupSubsysStateHeader>; MAX_DYN_CGROUP_SUBSYS],

    // ── Network bandwidth note ───────────────────────────────────────────
    // UmkaOS (like Linux cgroup v2) has NO dedicated network bandwidth
    // controller. Network bandwidth limiting is achieved through
    // BPF_PROG_TYPE_CGROUP_SKB programs attached via `BPF_CGROUP_INET_EGRESS`
    // / `BPF_CGROUP_INET_INGRESS` hooks combined with TC qdiscs
    // ([Section 16.21](16-networking.md#traffic-control-and-queue-disciplines)). This is the standard
    // approach used by Cilium, systemd, and modern container runtimes.
    // The v1 `net_cls` and `net_prio` controllers are NOT implemented.

    // ── Hierarchy control ────────────────────────────────────────────────

    /// Which controllers are enabled for this cgroup's children (a
    /// `ControllerMask` bit set, stored as its raw `u32`). Written by the
    /// `cgroup.subtree_control` write handler
    /// ([Section 17.2](#control-groups--cgroupsubtreecontrol-write-handler)); read on
    /// every child `mkdir` and by the no-internal-process check.
    ///
    /// `AtomicU32` (not a bare `ControllerMask`) because the write handler
    /// mutates it through a shared `&Cgroup` (`Arc<Cgroup>`): a bare field
    /// could not be written without `&mut`. Load with
    /// `ControllerMask::from_bits(self.subtree_control.load(Acquire))`, publish
    /// with `.store(mask.bits(), Release)` under `config_lock` (the child
    /// controller-state alloc/free that a change implies runs under
    /// `hierarchy_lock` write — see the handler). The mask read is O(1); the
    /// atomicity makes a concurrent child `mkdir` observe either the old or the
    /// new complete mask, never a torn value.
    pub subtree_control: AtomicU32,

    // ── Freeze state ─────────────────────────────────────────────────────

    /// Per-cgroup freeze request: set to `true` when userspace writes `1` to
    /// `cgroup.freeze`; cleared when userspace writes `0`.
    /// See Section 17.2.8 for the freeze/thaw protocol.
    pub freeze: AtomicBool,

    /// Effective freeze state: `true` iff `self.freeze || parent.e_freeze`.
    /// Computed by propagating downward on freeze/thaw writes:
    ///   `child.e_freeze = child.freeze || parent.e_freeze`
    /// A task remains frozen until ALL ancestor cgroups are thawed AND its own
    /// `freeze` is cleared. This two-boolean model matches
    /// Linux `cgroup_freezer_state` (`bool freeze` + `bool e_freeze` in
    /// `include/linux/cgroup-defs.h`).
    pub e_freeze: AtomicBool,

    /// Lifecycle state for zero-residual destruction. See
    /// [Section 17.2](#control-groups--cgroup-zero-residual-destruction).
    pub lifecycle: AtomicU8,

    /// Count of charges that can OUTLIVE `rmdir` — resources whose lifetime is
    /// NOT bounded by this cgroup's (already-empty at rmdir) task set: RDMA
    /// verbs objects (fds passed to other processes), perf_event fds (held by
    /// tasks in other cgroups), in-flight bios (async completion after the
    /// charging task exits), and huge folios charged here (mapped by processes
    /// in other cgroups). Each such charge does `residual_refs.fetch_add(1)`;
    /// each uncharge `fetch_sub(1)`. For the HIERARCHICAL residual controllers
    /// (rdma, perf_event, misc) whose counter is decremented at every level of
    /// an anchor→root walk, the charge bumps this at EVERY charged level (not
    /// just the anchor) so each level stays a resolvable tombstone until its own
    /// decrement — otherwise a freed intermediate ancestor breaks the walk. This
    /// is the keep-alive that lets the
    /// `CGROUP_REGISTRY` entry remain a resolvable TOMBSTONE after `rmdir` so a
    /// late uncharge can still find its anchored counter — the ONE mechanism the
    /// four residual controllers share (see
    /// [Section 17.2](#control-groups--registry-tombstone-and-residual-charge-drain)). The
    /// registry entry (which HOLDS the last `Arc<Cgroup>`) is erased — freeing
    /// the cgroup — only when this reaches 0 AFTER `lifecycle == Dead`, mirroring
    /// Linux's `percpu_ref` on an offline css. `AtomicU64`: a refcount bounded by
    /// concurrent kernel objects (exempt from the monotonic-counter policy), u64
    /// so it never wraps under 50-year churn.
    pub residual_refs: AtomicU64,

    // ── Generation counter for walk-free limit propagation ───────────────

    /// Incremented whenever any resource limit in this cgroup or any
    /// ancestor changes. Each task caches the generation value at the
    /// time its limits were last computed. On the next resource charge,
    /// the task compares its cached generation against this field. On
    /// mismatch, the task re-walks from its cgroup to the root to
    /// recompute its effective limits, then updates the cache.
    ///
    /// This makes limit changes O(1) to publish (one atomic increment)
    /// and amortizes the re-walk cost to the next resource operation on
    /// each task — no per-tick accounting, no broadcast, no lock convoy.
    pub generation: AtomicU64,

    // ── cgroupfs integration ──────────────────────────────────────────────

    /// Inode for this cgroup's directory in the cgroupfs pseudo-filesystem.
    /// Empty before the cgroupfs is mounted (the root cgroup exists before
    /// `mount("cgroup2", …)`; non-root cgroups get their inode inside
    /// `cgroup_mkdir()` step 8). `OnceCell` (not `Option<Arc<Inode>>`)
    /// because the inode is published through a shared `&Cgroup` AFTER the
    /// node itself is reachable to RCU readers — a plain `Option` could not be
    /// written without `&mut`. Write-once (`set()` at creation/mount),
    /// read-many (`get()`); the backing `Arc<Inode>` is released when the
    /// `Cgroup` is dropped at the end of the zero-residual drain. The cgroupfs
    /// path helper walks the `parent`/`name` chain, not the inode, so a
    /// not-yet-mounted cgroup still has a well-defined path.
    pub inode: OnceCell<Arc<Inode>>,

    // ── Config-write serialization ───────────────────────────────────────

    /// Serializes cgroupfs write-handler mutations of the controller-config
    /// state that is NOT individually atomic: the `cpu.core_type`/`cbs`
    /// installation, the `cpuset` masks and partition mode, the `subtree_control`
    /// mask store (paired with the structural child-controller alloc/free under
    /// `hierarchy_lock`), and the `RcuCell`/`RcuPtr` writer proof token for the
    /// per-controller RCU fields (`CpuController.cbs`, `CpusetController.*`).
    /// A sleeping `Mutex` (ordering below `THREADGROUP_RWSEM`): every writer is
    /// a cold/warm process-context cgroupfs write. **Never taken on the hot
    /// path** — resource charges use per-field atomics
    /// (`MemCgroup::usage`, `PidsController::current`, `CpuController::weight`),
    /// which is why config mutation and charging never contend. Its
    /// `MutexGuard` is the `WriterProof` that the per-field
    /// `RcuCell::update`/`RcuPtr` writers require
    /// ([Section 3.4](03-concurrency.md#cumulative-performance-budget--rcu-protected-container-types)).
    /// Distinct from `children_lock` (structural child-list edits) and
    /// `hierarchy_lock` (tree-wide mkdir/rmdir/migration serialization).
    pub config_lock: Mutex<()>,

    // ── BPF integration ──────────────────────────────────────────────────

    /// Attached BPF programs for this cgroup (ingress/egress/device/sysctl) —
    /// the WRITE-side authoritative list this cgroup's own attachments live in.
    /// Max 64 programs per cgroup (matching Linux BPF_CGROUP_MAX_PROGS).
    pub bpf_progs: SpinLock<ArrayVec<BpfCgroupLink, 64>>,

    /// Per-attach-type EFFECTIVE program lists, published for lock-free read on
    /// the packet/socket-op hot path (`cgroup_bpf_run`). This is the backing
    /// field the "published for lock-free read" effective set requires: an
    /// `Arc<[Arc<BpfProg>]>` per cgroup-BPF attach type, so the hot path does an
    /// RCU read of one slot and iterates a flat slice — NO per-packet ancestor
    /// walk. Recomputed (Linux `compute_effective_progs`) under `bpf_progs.lock()`
    /// on every attach/detach/rmdir and PUBLISHED here via `RcuCell`; because an
    /// attach on an ANCESTOR changes every descendant's effective list, the
    /// recompute walks the whole SUBTREE (self + descendants) and republishes
    /// each node's slot. Indexed by attach type (`NR_CGROUP_BPF_ATTACH_TYPES`
    /// bounded slots); a slot is `Arc<[]>` (empty) when no program of that type
    /// is effective here. Cold writer (attach/detach), hot lock-free reader.
    pub bpf_effective:
        [RcuCell<Arc<[Arc<BpfProg>]>>; NR_CGROUP_BPF_ATTACH_TYPES],
}

/// Fixed-size inline cgroup name (max 255 bytes, not NUL-terminated).
/// Avoids heap allocation for the common case (names ≤ 255 bytes).
pub struct CgroupName {
    /// Number of valid bytes in `data`.
    len: u8,
    /// Raw UTF-8 bytes. Characters '/' and '\0' are rejected at creation.
    data: [u8; 255],
}

impl CgroupName {
    /// Validate and construct a cgroup name. The `len: u8` + `[u8; 255]` buffer
    /// CANNOT overflow: a name longer than 255 bytes is REJECTED with EINVAL
    /// rather than truncated. Rejects '/' and '\0'. This is the constructor
    /// `cgroup_mkdir` step 1 uses — NOT `ArrayString::from` (the struct is
    /// `CgroupName`, not `ArrayString`, and `from` would either truncate or
    /// panic on overflow).
    pub fn new(s: &str) -> Result<CgroupName, Errno> {
        let bytes = s.as_bytes();
        if bytes.len() > 255 { return Err(Errno::EINVAL); }        // no overflow
        if bytes.iter().any(|&b| b == b'/' || b == 0) { return Err(Errno::EINVAL); }
        let mut data = [0u8; 255];
        data[..bytes.len()].copy_from_slice(bytes);
        Ok(CgroupName { len: bytes.len() as u8, data })
    }

    /// The root cgroup's name: empty (the root has no relative component).
    pub const fn root() -> CgroupName { CgroupName { len: 0, data: [0u8; 255] } }

    /// Borrow the name as a `&str` (the valid `len` prefix).
    pub fn as_str(&self) -> &str {
        // SAFETY: `new()` only stores valid UTF-8 (from a `&str`); `root()` is empty.
        unsafe { core::str::from_utf8_unchecked(&self.data[..self.len as usize]) }
    }
}

/// Bitmask of resource controllers. One bit per controller type.
/// Used for `subtree_control` (enabled-for-children) and for
/// `cgroup.controllers` (available on the system).
#[derive(Clone, Copy, Default)]
pub struct ControllerMask(pub u32);

impl ControllerMask {
    pub const CPU:     u32 = 1 << 0;
    pub const MEMORY:  u32 = 1 << 1;
    pub const IO:      u32 = 1 << 2;
    pub const PIDS:    u32 = 1 << 3;
    pub const CPUSET:  u32 = 1 << 4;
    pub const RDMA:    u32 = 1 << 5;
    pub const HUGETLB: u32 = 1 << 6;
    pub const MISC:    u32 = 1 << 7;
    pub const PERF_EVENT: u32 = 1 << 8;
    /// Accelerator (GPU/NPU/CXL) compute+memory controller. Backed by a
    /// registered subsystem (`dyn_subsys[CgroupSubsysId::Accel]`), not an
    /// embedded field — see [Section 22.5](22-accelerators.md#accelerator-isolation-and-scheduling--cgroup-integration).
    pub const ACCEL: u32 = 1 << 9;

    /// Reconstruct a mask from its raw bits (the storage form of
    /// `Cgroup.subtree_control: AtomicU32`).
    pub const fn from_bits(bits: u32) -> ControllerMask { ControllerMask(bits) }

    /// Raw bits for atomic storage / `cgroup.subtree_control` publication.
    pub const fn bits(self) -> u32 { self.0 }

    /// Returns `true` if the given controller bit is set.
    pub fn has(self, bit: u32) -> bool { self.0 & bit != 0 }

    /// Returns the union of two masks (used by the `+controller` token path of
    /// the `cgroup.subtree_control` write handler's publish step).
    pub fn union(self, other: ControllerMask) -> ControllerMask {
        ControllerMask(self.0 | other.0)
    }

    /// Returns `self` with `bit` cleared (the `-controller` case of the
    /// `cgroup.subtree_control` write handler — the removal operation the v2
    /// ABI's `-cpu`/`-memory` tokens require).
    pub fn without(self, bit: u32) -> ControllerMask { ControllerMask(self.0 & !bit) }
}

/// The controllers advertised in the ROOT cgroup's `cgroup.controllers` — the
/// availability set for the root's `cgroup.subtree_control`. Every built-in
/// controller plus `accel` (backed by a registered subsystem). This is the
/// canonical source for the `cgroup.controllers` file contents; the ASCII tree
/// below lists the same set.
pub const ROOT_AVAILABLE_CONTROLLERS: ControllerMask = ControllerMask(
    ControllerMask::CPU | ControllerMask::MEMORY | ControllerMask::IO
    | ControllerMask::PIDS | ControllerMask::CPUSET | ControllerMask::RDMA
    | ControllerMask::HUGETLB | ControllerMask::MISC | ControllerMask::PERF_EVENT
    | ControllerMask::ACCEL,
);

/// Iterate the set controller bits of `mask`, yielding each as a single-bit
/// `u32` mask — the form `ControllerMask::has`, `child_controller_alloc`, and
/// `child_controller_free` consume. Bits are visited low→high so enable/disable
/// application order is deterministic across nodes.
fn iter_bits(mask: ControllerMask) -> impl Iterator<Item = u32> {
    let bits = mask.bits();
    (0..u32::BITS).map(|i| 1u32 << i).filter(move |b| bits & b != 0)
}

/// Parse a `cgroup.subtree_control` write buffer (e.g. `"+cpu -memory"`) into an
/// `(enable, disable)` mask pair. Tokens are ASCII-whitespace-separated; each is
/// `+<controller>` (enable) or `-<controller>` (disable). An unknown controller
/// name, a token missing its `+`/`-` sign, or an empty controller name yields
/// `EINVAL` with NO partial result — the caller applies the pair all-or-nothing.
/// A controller named twice on a side, or on both sides, is merged by mask union
/// (idempotent). Mirrors Linux `cgroup_subtree_control_write()`
/// (`kernel/cgroup/cgroup.c`), which tokenizes the buffer and rejects unknown
/// controller names with `-EINVAL`.
fn parse_subtree_control_tokens(
    buf: &[u8],
) -> Result<(ControllerMask, ControllerMask), Errno> {
    let s = core::str::from_utf8(buf).map_err(|_| Errno::EINVAL)?;
    let mut enable = ControllerMask(0);
    let mut disable = ControllerMask(0);
    for tok in s.split_ascii_whitespace() {
        // `split_ascii_whitespace` never yields an empty token, so `bytes[0]`
        // (the sign) and `bytes[1..]` (the name, possibly empty) are in bounds.
        let bytes = tok.as_bytes();
        let name = core::str::from_utf8(&bytes[1..]).map_err(|_| Errno::EINVAL)?;
        let bit = match name {
            "cpu" => ControllerMask::CPU,
            "memory" => ControllerMask::MEMORY,
            "io" => ControllerMask::IO,
            "pids" => ControllerMask::PIDS,
            "cpuset" => ControllerMask::CPUSET,
            "rdma" => ControllerMask::RDMA,
            "hugetlb" => ControllerMask::HUGETLB,
            "misc" => ControllerMask::MISC,
            "perf_event" => ControllerMask::PERF_EVENT,
            "accel" => ControllerMask::ACCEL,
            _ => return Err(Errno::EINVAL), // unknown/empty controller name
        };
        match bytes[0] {
            b'+' => enable = ControllerMask(enable.bits() | bit),
            b'-' => disable = ControllerMask(disable.bits() | bit),
            _ => return Err(Errno::EINVAL), // token missing its +/- sign
        }
    }
    Ok((enable, disable))
}

/// Per-task cgroup migration state. Stored in the task struct as an
/// `AtomicU8` to allow lock-free reads during `cgroup.procs` enumeration.
/// The migration protocol uses this to ensure a task is always visible in
/// exactly one cgroup and that bandwidth enforcement is suspended while
/// the task's scheduler-tree membership is in transit.
///
/// There are NO dedicated transition functions — every transition is an
/// inline atomic in the protocols that own it:
/// - `None(0) → Migrating(1)`: migration step 3's
///   `compare_exchange(0, 1, Acquire, Relaxed)` (the CAS trylock), and
///   `detach_exiting_task()` step 0a's TERMINAL claim of the same CAS (never
///   released — an exited task is permanently non-migratable).
/// - `Migrating(1) → Complete(2)`: migration step 10's store, inside the
///   level-215 task-list critical section (membership has moved).
/// - `Complete(2) → None(0)`: migration step 16's store, after the
///   step-15 runqueue re-enqueue (bandwidth enforcement resumes).
/// - Any state `→ None(0)`: the rollback paths (steps 3/4/7/9 failure).
/// The `cgroup_migration_state: AtomicU8` field is stored in the `Task` struct
/// ([Section 8.1](08-process.md#process-and-task-management--process-identity-model)).
#[repr(u8)]
pub enum CgroupMigrationState {
    /// Task is a normal member of its cgroup (steady state).
    None      = 0,
    /// Task is being migrated: still in the source cgroup's task list
    /// but its `task.cgroup` pointer may already point to the target
    /// (steps 3-9). Readers of `cgroup.procs` include MIGRATING tasks
    /// in their source cgroup for consistency. Also the terminal value
    /// installed by `detach_exiting_task()` step 0a.
    Migrating = 1,
    /// Membership has moved (task is in the TARGET cgroup's task list,
    /// step 10) but the step-15 runqueue re-enqueue is still pending —
    /// the scheduler-tree link (`EevdfTask.cgroup_id`) may still name
    /// the source. The scheduler skips cgroup bandwidth enforcement
    /// (cpu.max, CBS) while in this state; charges issued through the
    /// stale `cgroup_id` in this bounded (µs) window land in the source
    /// and are benign. Transitions to None(0) at step 16.
    Complete  = 2,
}

/// Iterate over all descendants of a cgroup in pre-order (parent before children).
///
/// Callback-based walk used by the memory-charge reparent and lifecycle-scoped
/// passes. (The registered-subsystem iterator `SubsysDescendants` (via `descendants()`) is a
/// separate PULL iterator with its own DFS stack — an `FnMut`-taking function
/// cannot be pumped as an `Iterator`.) The traversal holds an RCU read-side
/// reference, so the caller must be in an RCU read-side critical section.
/// Structural modifications (mkdir/rmdir) are blocked by hierarchy_lock but do
/// not block this iterator — concurrent rmdir leaves the cgroup visible until
/// the RCU grace period completes.
///
/// Worst-case complexity: O(N) where N = number of descendants.
/// Cgroup trees in practice are shallow (depth ≤ 8) and narrow (breadth
/// ≤ hundreds), so this is bounded by the total cgroup count.
impl Cgroup {
    /// Borrow this cgroup's memory-controller state, or `None` if the memory
    /// controller is not enabled on this cgroup. Bridges the generic `Cgroup`
    /// to its per-subsystem `MemCgroup` (e.g. for the OOM scorer's
    /// `memcg(&guard).and_then(|mc| mc.fma_stats())` walk). `memory` is an
    /// `RcuPtr<Arc<MemCgroup>>`, so the read takes the caller's `RcuReadGuard`
    /// and the returned reference is borrowed for the guard's lifetime; to keep
    /// the `MemCgroup` past the RCU section, `Arc::clone` it under the guard (as
    /// `OomContext.memcg` does) rather than retaining the borrow.
    pub fn memcg<'g>(&'g self, guard: &'g RcuReadGuard) -> Option<&'g MemCgroup> {
        self.memory.read(guard).map(|arc| &**arc)
    }

    /// Walk all descendants in pre-order. The callback receives each
    /// descendant cgroup (excluding `self`). Returns early if `f` returns
    /// `ControlFlow::Break`.
    ///
    /// Acquires and holds its OWN RCU read-side critical section for the whole
    /// walk — the `children` snapshots stored on the DFS stack are bound to that
    /// guard's lifetime — so callers need not already be in one; nesting inside a
    /// caller's existing RCU section is also legal. The callback runs UNDER this
    /// guard, so it must not sleep, allocate, or take a sleeping lock (e.g. the
    /// per-cgroup `tasks` `RwLock` at `CGROUP_TASKS_LOCK(215)`); a caller that
    /// needs sleeping per-descendant work `Arc::clone`s each descendant out
    /// inside the callback and does that work after the walk returns (this
    /// file's snapshot-then-drop idiom).
    /// **Lifecycle safety**: Callers that must not operate on cgroups
    /// undergoing destruction MUST check `cg.lifecycle.load(Acquire) ==
    /// CgroupLifecycle::Active as u8` for each visited descendant
    /// (`lifecycle` is `AtomicU8` holding the `#[repr(u8)]` discriminant —
    /// the `as u8` cast is required; the enum has no `PartialEq<u8>`).
    /// Concurrent
    /// `cgroup_rmdir()` leaves the cgroup visible in the tree until
    /// the RCU grace period completes, so the iterator may yield cgroups
    /// that are mid-teardown.
    pub fn for_each_descendant<F>(&self, f: F)
    where
        F: FnMut(&Arc<Cgroup>) -> core::ops::ControlFlow<()>,
    {
        // Implementation: iterative pre-order DFS using sibling-then-child
        // traversal with O(depth) stack space. The stack stores the current
        // position at each depth level, NOT all children at any level. This
        // avoids the breadth explosion problem: K8s `kubepods.slice` may have
        // 500+ child cgroups at one level, but the tree is at most
        // `CGROUP_MAX_DEPTH` (256) levels deep.
        //
        // No heap allocation — this runs inside an RCU read-side critical
        // section where sleeping (and therefore demand-paging a heap
        // allocation) is forbidden. Reads `children` via `RcuCell::read()`
        // under `rcu_read_lock()` — no lock acquisition. The RcuCell
        // guarantees a consistent snapshot; structural modifications
        // (mkdir/rmdir) publish a new Vec via RCU, so the old Vec remains
        // valid for the read-side grace period.
        //
        // `CGROUP_MAX_DEPTH` bound is enforced at `cgroup_mkdir()` time
        // (step 3a rejects `parent.depth + 1 > CGROUP_MAX_DEPTH` with EAGAIN),
        // so the stack cannot overflow during a well-formed traversal.
        //
        // The RCU read guard is acquired ONCE at the top and outlives every
        // `&Vec` reference stored on the stack, so the snapshots stay valid for
        // the whole walk. `children.read(&guard)` is the canonical
        // `RcuCell::read` API
        // ([Section 3.4](03-concurrency.md#cumulative-performance-budget--rcu-protected-container-types)): it
        // takes `&RcuReadGuard` and returns `&Vec<Arc<Cgroup>>` bound to the
        // guard's lifetime — there is no separate `RcuRef` type. That guard-tied
        // lifetime also decouples each derived `&Vec`/`&Arc` from the transient
        // `stack.last_mut()` borrow, so `stack.push()` below is legal once the
        // derived references are taken — the same borrow structure the
        // `SubsysDescendants` pull iterator above relies on.
        let mut f = f;
        let guard = rcu_read_lock(); // held until function return
        // Each entry: (children snapshot at this level, next sibling index).
        let mut stack: ArrayVec<(&Vec<Arc<Cgroup>>, usize), CGROUP_MAX_DEPTH> =
            ArrayVec::new();
        let root_children: &Vec<Arc<Cgroup>> = self.children.read(&guard);
        if root_children.is_empty() {
            return;
        }
        stack.push((root_children, 0));
        while let Some((children, idx)) = stack.last_mut() {
            if *idx >= children.len() {
                stack.pop(); // exhausted this level, backtrack
                continue;
            }
            let cg: &Arc<Cgroup> = &children[*idx];
            *idx += 1; // advance to next sibling for the next iteration
            if let core::ops::ControlFlow::Break(()) = f(cg) {
                return;
            }
            let grandchildren: &Vec<Arc<Cgroup>> = cg.children.read(&guard);
            if !grandchildren.is_empty() {
                stack.push((grandchildren, 0)); // descend
            }
        }
    }

    /// True iff this cgroup's SUBTREE holds at least one task — the derived
    /// predicate that replaces a stored to-root running total. `rmdir`'s
    /// emptiness gate is `!is_populated()`, and the `Weak::upgrade()` liveness
    /// proofs throughout this section rest on it: while any descendant is
    /// populated, `is_populated()` is true on every ancestor (maintained by
    /// `cgroup_propagate_populated`), so no ancestor can pass its own `rmdir`
    /// gate — every ancestor therefore remains strongly reachable from
    /// `CgroupRoot.root` and its `Weak` parent-edge upgrades.
    pub fn is_populated(&self) -> bool {
        self.population.load(Ordering::Acquire) != 0
            || self.nr_populated_children.load(Ordering::Acquire) != 0
    }

    /// Borrow this cgroup's state for a registered subsystem `S`, or `None` if
    /// `S` is not attached to this cgroup. Reads the `dyn_subsys` slot for
    /// `S::SUBSYS_ID` (a relaxed `AtomicPtr` load) and, on a non-null pointer
    /// whose header `subsys_id` matches, up-casts to `&S` (the header is `S`'s
    /// first `#[repr(C)]` field). Must be called inside an RCU read section:
    /// the pointer is cleared and the object freed by `free_state` only after a
    /// grace period ([Section 17.2](#control-groups--drain-protocol), Phase 3.5).
    pub fn subsys_state<S: CgroupSubsys>(&self) -> Option<&S> {
        let p = self.dyn_subsys[S::SUBSYS_ID as usize].load(Ordering::Acquire);
        if p.is_null() {
            return None;
        }
        // SAFETY: non-null `dyn_subsys` slots are published by `alloc_state` and
        // point at a `#[repr(C)]` `S` whose first field is the header; the
        // pointer stays valid for this RCU read section (freed only after a
        // grace period). The `subsys_id` match rules out a slot/type mismatch.
        let header = unsafe { &*p };
        debug_assert_eq!(header.subsys_id, S::SUBSYS_ID as u32);
        Some(unsafe { &*(p as *const S) })
    }
}

/// Reconcile `cg`'s OWN published `subtree_populated` bit after a
/// DIRECT-MEMBERSHIP edge (a `population` 0→1 / 1→0 already applied by the
/// caller and claimed by its `fetch_add`/`fetch_sub` return value). Under
/// `cg.flip_lock`, recomputes `is_populated()` and compares it to the published
/// bit; on a genuine flip, republishes and drives ancestor propagation. The
/// under-lock `is_populated()`-vs-published compare — NOT a bare
/// `nr_populated_children == 0` read — is what prevents a concurrent
/// child-subtree edge (which fills/empties a child between the caller's
/// `population` write and this lock) from being mistaken for a subtree flip:
/// both edges serialize on `flip_lock` and observe the same committed counters
/// and the shared bit, so exactly one drives each true 0↔1 subtree flip. Called
/// by `commit_fork`, `detach_exiting_task` step 4, and migration step 11.
fn cgroup_reconcile_self_populated(cg: &Arc<Cgroup>) {
    let now;
    {
        let _flip = cg.flip_lock.lock();
        now = cg.is_populated();
        if now == (cg.subtree_populated.load(Ordering::Acquire) != 0) {
            // Published bit already correct — a child (or another direct member)
            // still holds the subtree in the same state. NOT a subtree flip.
            return;
        }
        cg.subtree_populated.store(now as u8, Ordering::Release);
    }
    // Leaf `flip_lock` dropped before taking the parent's (never nest two
    // flip_locks). Propagate the delta root-ward.
    cgroup_propagate_populated(cg, now);
}

/// Ancestor-ward propagation of a SUBTREE-populated flip. Called when `cg`'s OWN
/// published `subtree_populated` bit has just flipped to `now_populated`
/// (decided under `cg.flip_lock`). Walks parents: at each ancestor, under that
/// ancestor's `flip_lock`, adjusts `nr_populated_children`, then reconciles the
/// ancestor's OWN published bit against its live `is_populated()`, and STOPS at
/// the first ancestor whose published bit does not flip — so a fork/exit into an
/// already-populated subtree performs ZERO ancestor writes. Propagation cost is
/// proportional to the number of ancestors whose bit actually flips, never to
/// subtree depth.
///
/// `cg` is the cgroup whose subtree bit just flipped; `now_populated` is its
/// new state (also the direction of the `nr_populated_children` edge at each
/// ancestor — a child that became populated only ever drives its parent's
/// bit false→true, and vice-versa, so a flipped ancestor's new state always
/// equals `now_populated`). Ancestor liveness for `upgrade()`: `cg` (and thus
/// every ancestor up to the root) is pinned by the caller's context — the
/// flipping task's own cgroup Arc during fork/exit, or the migration snapshot
/// Arcs.
fn cgroup_propagate_populated(cg: &Arc<Cgroup>, now_populated: bool) {
    let mut cursor = match cg.parent {
        Some(ref p) => p.upgrade()
            .expect("populated subtree keeps every ancestor un-rmdir-able, hence alive"),
        None => return, // root has no parent to notify
    };
    loop {
        // Serialize THIS ancestor's counter update + published-bit reconcile
        // against any other edge (direct-membership OR another child-walker)
        // passing through it (per-cgroup `flip_lock`, cold — 0↔1 edges only).
        // The lock is taken and dropped per level; it is NEVER held while
        // taking the next ancestor's `flip_lock`, so no ordering hazard.
        let flipped;
        {
            let _flip = cursor.flip_lock.lock();
            if now_populated {
                cursor.nr_populated_children.fetch_add(1, Ordering::Release);
            } else {
                cursor.nr_populated_children.fetch_sub(1, Ordering::Release);
            }
            // Reconcile THIS ancestor's PUBLISHED bit against its live state.
            // Using is_populated()-vs-published (which reads BOTH counters), not
            // before/after of just this op, is what keeps a direct-membership
            // edge and this child edge coherent when they race here: both see
            // the same committed counters and the shared bit under this lock.
            let nowp = cursor.is_populated();
            if nowp != (cursor.subtree_populated.load(Ordering::Acquire) != 0) {
                cursor.subtree_populated.store(nowp as u8, Ordering::Release);
                flipped = true;
            } else {
                flipped = false;
            }
        }
        // Stop as soon as this ancestor's own published bit did NOT flip:
        // everything above it already reflects the correct state.
        if !flipped {
            break;
        }
        cursor = match cursor.parent {
            Some(ref p) => p.upgrade()
                .expect("populated subtree keeps every ancestor un-rmdir-able, hence alive"),
            None => break,
        };
    }
}

/// Trait implemented by each registered subsystem's per-cgroup state object
/// (`MlPolicyCss`, the accel css). Associates the concrete type with its
/// `CgroupSubsysId` slot and gives access to its embedded header — it does NOT
/// require the implementor to own an `Arc<Cgroup>` (the old
/// `fn cgroup(&self) -> &Arc<Cgroup>` contract, which forced a cycle-inducing
/// strong back-edge). The owning cgroup is resolved on demand from the
/// header's `cgroup_id` via `CGROUP_REGISTRY`.
///
/// # Safety
/// The implementor MUST be `#[repr(C)]` with `CgroupSubsysStateHeader` as its
/// first field, so `subsys_state::<Self>()` may up-cast a header pointer to
/// `&Self`.
pub unsafe trait CgroupSubsys: Sized {
    /// Which `dyn_subsys` slot this subsystem occupies.
    const SUBSYS_ID: CgroupSubsysId;
    /// Borrow the embedded header (the type's first field).
    fn header(&self) -> &CgroupSubsysStateHeader;
    /// Resolve the owning cgroup under RCU, or `None` if it has been unpinned
    /// by `rmdir`. Default impl via `CGROUP_REGISTRY.get()` — the same
    /// RCU-read XArray lookup the scheduler's `cgroup_from_id()` uses
    /// ([Section 7.1](07-scheduling.md#scheduler--eevdf-algorithm-specification)); the `guard` binds the
    /// returned reference to the read section.
    fn cgroup<'g>(&self, guard: &'g RcuReadGuard) -> Option<&'g Cgroup> {
        let _ = guard;
        CGROUP_REGISTRY.get(self.header().cgroup_id)
    }
}

/// Pre-order, RCU-protected iterator over the per-cgroup state of subsystem `S`
/// for every descendant of a root cgroup that HAS `S` attached (descendants
/// where the `dyn_subsys[S]` slot is null are skipped). Yields `&'g S` bound to
/// the held RCU guard.
///
/// This is a real pull iterator with an explicit O(depth) DFS stack — NOT a
/// wrapper around the callback-based `for_each_descendant` (a closure-taking
/// function cannot be stored and pumped as an iterator). The stack stores
/// `(&children_snapshot, next_index)` cursors; `next()` advances the DFS and
/// returns the next descendant's `S` state.
pub struct SubsysDescendants<'g, S: CgroupSubsys> {
    /// The RCU read guard whose lifetime bounds every yielded `&S`.
    guard: &'g RcuReadGuard,
    /// DFS cursor stack: (children snapshot at this level, next sibling index).
    stack: ArrayVec<(&'g Vec<Arc<Cgroup>>, usize), CGROUP_MAX_DEPTH>,
    _marker: PhantomData<fn() -> &'g S>,
}

impl<'g, S: CgroupSubsys> Iterator for SubsysDescendants<'g, S> {
    type Item = &'g S;
    fn next(&mut self) -> Option<&'g S> {
        while let Some((children, idx)) = self.stack.last_mut() {
            if *idx >= children.len() {
                self.stack.pop();
                continue;
            }
            let cg: &'g Arc<Cgroup> = &children[*idx];
            *idx += 1;
            let grandchildren: &'g Vec<Arc<Cgroup>> = cg.children.read(self.guard);
            if !grandchildren.is_empty() {
                self.stack.push((grandchildren, 0));
            }
            // Yield this descendant's S state if attached (else keep walking).
            if let Some(state) = cg.subsys_state::<S>() {
                return Some(state);
            }
        }
        None
    }
}

/// Pre-order RCU walk over all descendants of `root_cgroup` yielding the
/// subsystem-`S` state of each descendant that has `S` attached. The caller
/// supplies the RCU read guard (it must outlive the iterator). Consumers:
/// ML-policy override decay ([Section 23.1](23-ml-policy.md#aiml-policy-framework-closed-loop-kernel-intelligence))
/// iterates `descendants::<MlPolicyCss>(root, &guard)` and reads
/// each `MlPolicyCss` directly.
pub fn descendants<'g, S: CgroupSubsys>(
    root_cgroup: &Cgroup,
    guard: &'g RcuReadGuard,
) -> SubsysDescendants<'g, S> {
    let mut stack = ArrayVec::new();
    let root_children = root_cgroup.children.read(guard);
    if !root_children.is_empty() {
        stack.push((root_children, 0usize));
    }
    SubsysDescendants { guard, stack, _marker: PhantomData }
}

17.2.1.2 CPU Controller State¶

/// CPU controller state, present when the `cpu` controller is enabled
/// for this cgroup (listed in parent's `subtree_control`).
///
/// Maps to `cpu.weight`, `cpu.max`, `cpu.guarantee`, and `cpu.stat`
/// cgroupfs files. See Section 17.2.3 for the integration with UmkaOS's
/// EEVDF scheduler and CBS bandwidth enforcement.
pub struct CpuController {
    /// `cpu.weight`: relative CPU share among siblings (1..=10000, default 100).
    /// Used directly as the EEVDF task-group weight.
    pub weight: AtomicU32,

    /// `cpu.max` quota: microseconds of CPU time allowed per `period_us`.
    /// `u64::MAX` means unlimited (no throttling — the default). Uses a sentinel
    /// value instead of `Option` to avoid branching overhead on the hot path
    /// (every scheduler tick checks this field). Matches the representation in
    /// `CpuBandwidthThrottle.quota_us`.
    pub max_us: AtomicU64,

    /// `cpu.max` period in microseconds (default 100,000 = 100 ms).
    /// Always set even when `max_us` is `u64::MAX` (holds the configured period
    /// for when a quota is later added).
    pub period_us: AtomicU64,

    /// `cpu.max` bandwidth throttle state (quota, period, runtime pool, stats).
    /// This is the single source of truth for all bandwidth throttling accounting
    /// — `cpu.stat` reads for nr_periods, nr_throttled, and throttled_time are
    /// served from this struct. See [Section 7.6](07-scheduling.md#cpu-bandwidth-guarantees--cpumax-ceiling-enforcement-bandwidth-throttling).
    pub bandwidth: CpuBandwidthThrottle,

    /// CBS (Constant Bandwidth Server) configuration for `cpu.guarantee`
    /// and `cpu.max` enforcement. Per-CPU servers (`CbsCpuServer`) are
    /// allocated lazily on each CPU's runqueue; this struct holds the
    /// cgroup-wide parameters they read at replenishment time.
    /// See [Section 7.6](07-scheduling.md#cpu-bandwidth-guarantees) for the per-CPU CBS model.
    ///
    /// `RcuPtr` (the nullable RCU sibling of `RcuCell`,
    /// [Section 3.4](03-concurrency.md#cumulative-performance-budget--rcuptrt-nullable-single-owner-rcu-pointer)):
    /// null until the first `cpu.guarantee` write installs a config, non-null
    /// thereafter. `RcuPtr` gives the exact `Option<&CbsGroupConfig>` read
    /// semantics of the old plain `Option`, but with interior mutability — the
    /// config is installed through a shared `&Cgroup` (write side takes the
    /// `config_lock` `MutexGuard` as its writer proof) while the per-CPU
    /// replenishment path reads it lock-free (warm path). A plain
    /// `Option<CbsGroupConfig>` could not be written through `Arc<Cgroup>`.
    pub cbs: RcuPtr<CbsGroupConfig>,

    // ── Heterogeneous CPU scheduling (big.LITTLE / P-core/E-core) ──────

    /// `cpu.core_type`: the set of CPU core types tasks in this cgroup may run
    /// on, stored as a bitmask over `CoreType` discriminants
    /// ([Section 7.2](07-scheduling.md#heterogeneous-cpu-scheduling)) — bit `1 << (CoreType::X as u32)`.
    /// `0` OR all-supported-bits-set means "all" (no preference — the default).
    /// The multi-value file grammar ("all", "performance", "efficiency",
    /// "performance mid", …) requires a MASK, not a single value: a single
    /// `Option<CpuCoreType>` could not represent "performance mid". This is why
    /// the earlier local 2-variant `CpuCoreType` enum is removed — control-groups
    /// does NOT redefine the core-type taxonomy; it references the scheduler's
    /// authoritative 4-variant `CoreType` (Performance/Efficiency/Mid/Symmetric)
    /// and the boot capacity→`CoreType` classification rule (the doc-comment
    /// classification block under [Section 7.2](07-scheduling.md#heterogeneous-cpu-scheduling--cpu-capacity-model)).
    /// `AtomicU32`:
    /// written through `&Cgroup` by the `cpu.core_type` handler under
    /// `config_lock`, read on the placement warm path.
    pub allowed_core_types: AtomicU32,

    /// `cpu.capacity.min`: minimum CPU capacity required (0..=1024).
    /// The scheduler will not place tasks from this cgroup on CPUs whose
    /// normalized capacity is below this value. ARM defines CPU capacity as
    /// a DMIPS/MHz-normalized value (0-1024) where 1024 = the most capable
    /// core in the system. Default: 0 (no minimum — tasks may run on any core).
    /// Used by Android-style EAS (Energy Aware Scheduling) and heterogeneous
    /// server workloads (e.g., "this cgroup needs big cores").
    /// **UmkaOS extension**: This knob is NOT present in upstream Linux cgroups v2.
    /// Linux exposes capacity via `sched_setattr(SCHED_FLAG_UTIL_CLAMP_MIN)` per-task
    /// only. UmkaOS elevates it to a per-cgroup knob for container-level capacity
    /// pinning. Tools that do not recognize `cpu.capacity.min` will ignore it.
    pub capacity_min: AtomicU32,

    /// `cpu.capacity.max`: maximum CPU capacity allowed (0..=1024).
    /// The scheduler will not place tasks from this cgroup on CPUs whose
    /// normalized capacity exceeds this value. Default: 1024 (no maximum —
    /// tasks may run on the most capable cores). Setting this below the
    /// system's maximum capacity constrains the cgroup to efficiency cores,
    /// useful for background/batch workloads that should not contend with
    /// latency-sensitive workloads for high-performance cores.
    /// **UmkaOS extension**: Same as `capacity_min` — not in upstream Linux
    /// cgroups v2. Linux equivalent is per-task `SCHED_FLAG_UTIL_CLAMP_MAX`.
    pub capacity_max: AtomicU32,

    // ── Latency tuning ──────────────────────────────────────────────────

    /// `cpu.latency_nice`: per-cgroup latency-nice hint (-20 to +19, default 0).
    ///
    /// **UmkaOS-original extension** — NOT a Linux feature. `latency_nice` was
    /// proposed on LKML (Vincent Guittot / Parth Shah, 2022-2024) but never
    /// merged into `torvalds/linux` mainline. As of Linux 6.17+, there is no
    /// `latency_nice` field in `struct sched_attr`, no `SCHED_FLAG_LATENCY_NICE`
    /// bit, and no per-cgroup `cpu.latency_nice` knob. Applications and cgroup
    /// configurations using this feature are UmkaOS-only.
    ///
    /// Shifts the EEVDF eligibility window for all tasks in this cgroup:
    ///   - Negative values (e.g., -20): earlier eligibility → lower scheduling
    ///     latency (latency-sensitive workloads: databases, interactive UIs).
    ///   - Positive values (e.g., +19): later eligibility → higher throughput
    ///     but increased scheduling latency (batch, background workloads).
    ///   - Zero: no adjustment (default EEVDF behavior).
    ///
    /// Written via the `cpu.latency_nice` cgroupfs file. When a task's
    /// per-task `latency_nice` (set via `sched_setattr(2)`) differs from the
    /// cgroup value, the more latency-sensitive (lower) value wins — the
    /// effective latency-nice is `min(task.latency_nice, cgroup.latency_nice)`.
    ///
    /// The value is propagated to the scheduler via the same two-phase
    /// mechanism as `cpu.weight`: atomic store here, then per-CPU IPI to
    /// update all `SchedEntity` instances in the cgroup's task list.
    ///
    /// Cross-reference: [Section 7.1](07-scheduling.md#scheduler) for EEVDF latency-nice integration
    /// and the `LATENCY_NICE_TO_WEIGHT` table.
    pub latency_nice: AtomicI32,

    // ── Accumulated statistics (read via `cpu.stat`) ─────────────────────
    // STORED IN NANOSECONDS (the scheduler's native `rq.clock_task` delta unit),
    // rendered to the `*_usec` cgroupfs fields by dividing by 1000 at READ time.
    // This is Linux's model (`sum_exec_runtime` is ns; `cpu.stat` divides by
    // `NSEC_PER_USEC` when formatting) and avoids the persistent per-tick
    // truncation bias of charging `delta_exec / 1000` (up to 999 ns lost every
    // tick — ~250 µs/s at HZ=250 on a busy CPU). The `#cpu.stat` read handler
    // performs the ns→µs conversion (see the cgroupfs pressure/stat renderer and
    // the v1 `cpuacct.usage` row below, which already records "ns→µs at read").
    /// `cpu.stat usage_usec` backing: total CPU time consumed (NANOSECONDS),
    /// user + system. Monotonically increasing. Rendered `/1000` as `usage_usec`.
    pub usage_ns: AtomicU64,
    /// `cpu.stat user_usec` backing: CPU time in user mode (NANOSECONDS).
    pub user_ns: AtomicU64,
    /// `cpu.stat system_usec` backing: CPU time in kernel mode (NANOSECONDS).
    pub system_ns: AtomicU64,
    // NOTE: Bandwidth throttling stats (nr_periods, nr_throttled, throttled_time_us)
    // are maintained in `CpuBandwidthThrottle` — the single source of truth for
    // all cpu.max bandwidth accounting. See [Section 7.6](07-scheduling.md#cpu-bandwidth-guarantees--cpumax-ceiling-enforcement-bandwidth-throttling).
    // `cpu.stat` reads are served by reading from `bandwidth.nr_periods` etc.
    //
    // WRITER (all three counters): the scheduler tick accounting path
    // `account_cgroup_exec_time(task, delta_exec_ns, mode)` — invoked from
    // `update_curr()` on the currently-running task's cgroup on every tick and
    // at every dequeue. It adds the raw `rq.clock_task` delta (NANOSECONDS) to
    // `usage_ns` and, by the sampled execution `mode` (user vs kernel at the tick
    // boundary, the same sampling Linux `cpustat` uses), to `user_ns` or
    // `system_ns`. Relaxed ordering (single logical writer per tick per task;
    // `cpu.stat` reads tolerate the last-tick skew). This is the ONLY writer —
    // there is no separate per-syscall charge, matching the `usage`/`weight`
    // single-writer discipline elsewhere in this struct.
    //
    // **Cross-file handoff (recorded)**: `account_cgroup_exec_time` is DEFINED on
    // the scheduler side, called from `update_curr()`'s step 3c — see
    // [Section 7.1](07-scheduling.md#scheduler--eevdf-algorithm-specification). (Like the RDMA/hugetlb/misc/
    // perf_event/taskstats additions, the writer lives in its owning subsystem;
    // this note records the handoff so the counters are not phantom-written.)
}

Per-CPU CBS Budget Tracking

CpuController.cbs stores the cgroup-wide CBS configuration (RcuPtr<CbsGroupConfig>). A null pointer means no CBS guarantee has been configured for this cgroup (the default); the first cpu.guarantee write installs the config through RcuPtr under config_lock. Actual budget enforcement is per-CPU via CbsCpuServer structures, one per cgroup per CPU that has runnable tasks. The per-CPU model eliminates global pool lock contention — all budget operations are per-CPU atomics or CAS on sibling CPUs. See Section 7.6 for the complete per-CPU CBS design including:

CbsGroupConfig (cgroup-wide parameters: quota, period, burst, total_weight)
CbsCpuServer (per-CPU: budget, deadline, throttled state, local_weight)
Proportional share replenishment (no global timer, per-CPU timers)
Atomic steal protocol (exhaust → steal from NUMA-local siblings first)
Task migration handling (weight transfer, lazy proportional rebalance)

Throttling mechanics: when CbsCpuServer.throttled is set, the scheduler's pick_next_task() skips tasks in the throttled cgroup. Tasks already running when the budget expires are preempted at the next scheduler tick. On per-CPU timer replenishment, throttled servers are un-throttled and their tasks' OnRqState transitions from CbsThrottled back to Queued, re-entering the EEVDF runqueue.

17.2.1.3 Memory Controller State¶

/// Memory controller state, present when the `memory` controller is enabled.
///
/// Maps to `memory.current`, `memory.high`, `memory.max`, `memory.swap.max`,
/// `memory.oom.group`, and `memory.events` cgroupfs files.
/// See Section 17.2.4 for the integration with the physical memory allocator.
pub struct MemCgroup {
    /// `memory.current`: total bytes of memory charged to this cgroup.
    /// Updated on every page charge/uncharge (one atomic add per page fault
    /// or page table manipulation). Monotonically tracks live usage.
    pub usage: AtomicU64,

    /// `memory.high`: soft limit in bytes. When `usage` exceeds this, the
    /// cgroup's tasks are throttled (sleeping in the allocator path) and
    /// reclaim is prioritized for pages belonging to this cgroup.
    /// `u64::MAX` means unlimited (default).
    pub high: AtomicU64,

    /// `memory.max`: hard limit in bytes. When `usage` would exceed this,
    /// the per-cgroup OOM killer is invoked before the allocation completes.
    /// `u64::MAX` means unlimited (default).
    pub max: AtomicU64,

    /// `memory.swap.max`: swap usage hard limit in **bytes**.
    /// `u64::MAX` means unlimited (default).
    /// Controls how much of this cgroup's memory may be swapped out.
    ///
    /// **Authoritative swap-accounting home**: `MemCgroup` (reached as
    /// `cgroup.memory`) is the single source of truth for cgroup swap
    /// accounting, and its unit is BYTES — consistent with every other
    /// `MemCgroup` limit (`max`, `high`, `min`, `low`) and with the
    /// `memory.swap.max`/`memory.swap.current` ABI (both byte-valued). The
    /// swap subsystem's `CgroupSwapState` (pages, reached as
    /// `cgroup.mem_controller.swap_state`) is the divergent duplicate; it must
    /// be reconciled to read/write `MemCgroup.swap_usage`/`swap_max` in bytes
    /// (deferred handoff recorded — see the fix report). Both a byte limit and
    /// a byte usage counter live HERE so the limit is enforceable from this
    /// struct alone.
    pub swap_max: AtomicU64,

    /// `memory.swap.current`: current swap usage in **bytes** (the read-only
    /// counter the `memory.swap.max` limit is checked against). Charged when a
    /// page belonging to this cgroup is written to swap, uncharged when it is
    /// read back in or the swap slot is freed. This is the backing field the
    /// swap-out admission check (`swap_max` comparison) reads; without it
    /// `swap_max` would be unenforceable. Bytes, not pages — see `swap_max`.
    pub swap_usage: AtomicU64,

    /// `memory.min`: absolute minimum memory guarantee (bytes). Memory below
    /// this threshold is NEVER reclaimed, even under global OOM pressure.
    /// This provides a hard guarantee for critical workloads. The page scanner
    /// unconditionally skips pages belonging to cgroups whose `usage` is at or
    /// below `memory_min`. Default: 0 (no guarantee).
    ///
    /// Effective value propagation: a cgroup's effective `memory.min` is
    /// `min(memory_min, parent.effective_memory_min * memory_min / siblings_sum)`.
    /// This ensures children cannot collectively claim more protection than
    /// the parent offers.
    pub memory_min: AtomicU64,

    /// `memory.low`: best-effort memory protection (bytes). Memory below this
    /// threshold is protected from reclaim unless there is no other reclaimable
    /// memory in the system. Provides softer protection than `memory_min`:
    /// the page scanner deprioritizes pages in cgroups under their `memory.low`
    /// threshold, but will still reclaim them as a last resort before invoking
    /// the OOM killer. Default: 0 (no protection).
    ///
    /// Effective value propagation: same formula as `memory_min`. The reclaim
    /// path computes `effective_low` by walking the cgroup hierarchy from the
    /// target cgroup to the root, distributing the parent's protection budget
    /// proportionally among siblings based on their configured `memory.low`
    /// values.
    pub memory_low: AtomicU64,

    /// `memory.oom.group`: when `true`, the OOM killer kills **all tasks**
    /// in the cgroup rather than selecting a single victim. Useful for
    /// atomically terminating a container that has overrun its memory budget.
    pub oom_group: AtomicBool,

    /// Back-pointer to the owning `Cgroup`. `Weak` — the `Cgroup` owns its
    /// `MemCgroup` controller (strong), so a strong back-edge would be a
    /// cycle. Set once at cgroup creation, immutable thereafter.
    /// Consumers: the OOM killer ([Section 4.5](04-memory.md#oom-killer)) upgrades this to
    /// anchor its allocation-free parent-chain walks — subtree membership
    /// (`cgroup_chain_contains_memcg()`, testing a task's ancestor chain
    /// against this memcg), hierarchical `memory.events` increments
    /// (`memcg_event_hierarchy()`), and Step 0 hibernation scoping. No
    /// lock is taken for these walks (populated-ancestor liveness
    /// invariant, see [Section 17.2](#control-groups--cgroup-zero-residual-destruction)).
    /// Upgrade returns `None` only during cgroup teardown, when no tasks
    /// remain to scan — callers treat `None` as an empty set / no-op.
    pub owner: Weak<Cgroup>,

    /// Hibernation state for the OOM Step 0 background-cgroup path
    /// (`CgroupHibernate` is defined in
    /// [Section 4.5](04-memory.md#oom-killer--process-memory-hibernation)). Embeds the freeze
    /// state machine, `hibernate_priority`, and reclaim bookkeeping.
    ///
    /// **Registration contract** (consumed by `HIBERNATE_CANDIDATES` in
    /// [Section 4.5](04-memory.md#oom-killer)): the cgroupfs `memory.hibernate_priority` write
    /// handler pushes this cgroup's `Arc<MemCgroup>` into
    /// `HIBERNATE_CANDIDATES` on a 0→nonzero transition and removes it on
    /// a nonzero→0 write. Cgroup destruction (rmdir) also removes it —
    /// ordered BEFORE the controller teardown so Step 0 never selects a
    /// dying cgroup.
    pub hibernate: CgroupHibernate,

    /// `memory.events` counters. These seven counters map to the fields
    /// in the `memory.events` cgroupfs file (Linux cgroup v2 interface).

    /// `memory.events.low`: number of times the cgroup was reclaimed below
    /// `memory.low` (i.e., the cgroup's guaranteed minimum was breached).
    pub events_low: AtomicU64,
    /// `memory.events.high`: number of times `usage` exceeded `memory.high`,
    /// triggering throttling and priority reclaim.
    pub events_high: AtomicU64,
    /// `memory.events.max`: number of times `usage` hit `memory.max` and
    /// allocation attempts were stalled or failed.
    pub events_max: AtomicU64,
    /// `memory.events.oom`: number of times the OOM killer was invoked
    /// for this cgroup OR any descendant (regardless of whether a kill
    /// actually happened). HIERARCHICAL: the OOM killer increments the
    /// triggering memcg and every ancestor via `memcg_event_hierarchy()`
    /// ([Section 4.5](04-memory.md#oom-killer), kill-sequence step 6a).
    pub events_oom: AtomicU64,
    /// `memory.events.oom_kill`: number of tasks killed by the OOM killer
    /// in this cgroup or any descendant. Incremented once PER VICTIM
    /// (group kills count every member — Linux `MEMCG_OOM_KILL` is raised
    /// per process in Linux `__oom_kill_process()`), starting at the victim's
    /// own memcg and walking every ancestor (hierarchical, step 6a).
    pub events_oom_kill: AtomicU64,
    /// `memory.events.oom_group_kill`: number of times this cgroup (or a
    /// descendant) was killed as a group due to `memory.oom.group = 1`.
    /// Once per group event, not per task; hierarchical from the
    /// group-kill root (step 6a). Distinct from `events_oom_kill` which
    /// counts individual process kills.
    pub events_oom_group_kill: AtomicU64,
    /// `memory.events.sock_throttled`: number of times network sockets
    /// associated with this cgroup are throttled due to memory pressure.
    pub sock_throttled: AtomicU64,

    /// Per-`(memcg, node)` MGLRU generation state for pages charged to this
    /// cgroup — one `CgroupLru` PER NUMA node, because reclaim
    /// operates independently per node. This is the SINGLE canonical
    /// linkage and struct: the `CgroupLru` type and this `lru_gen` field name
    /// are defined once in [Section 4.4](04-memory.md#page-cache--generational-lru-page-reclaim)
    /// (`max_seq`/`min_seq`/`timestamps`/per-gen folio lists). The earlier
    /// single-instance `lru: SpinLock<CgroupLru>` form here — with a divergent
    /// `generations`/`oldest_gen`/`writeback` body — is removed; it could not
    /// serve per-node reclaim and duplicated the symbol. Intra-node
    /// synchronization is internal to `CgroupLru` (atomic seqs + intrusive
    /// `Page.lru` lists), so no outer `SpinLock` wrapper is needed and none of
    /// the reclaim paths sleep. The reclaim path consults the node's
    /// generations when `memory.high` is exceeded or under global pressure;
    /// the oldest generation is the primary reclaim target.
    pub lru_gen: ArrayVec<CgroupLru, NUMA_NODES_STACK_CAP>,

    /// FMA (Fault-Management Architecture) memory-pressure statistics for this
    /// memory cgroup, consulted by the internal OOM scorer
    /// ([Section 4.5](04-memory.md#oom-killer)). Lazily initialized on the first per-node reclaim
    /// reclaim pass (not at cgroup creation), so cgroups that never hit pressure
    /// pay nothing. Never deallocated once set (bounded size: two `AtomicU64`).
    /// `OnceCell` yields `Option<&FmaCgroupStats>` via `get()`.
    pub fma: OnceCell<FmaCgroupStats>,
}

impl MemCgroup {
    /// FMA pressure statistics for this memory cgroup, or `None` if the cgroup
    /// has never triggered reclaim (the `FmaCgroupStats` is lazily allocated on
    /// the first per-node reclaim pass).
    ///
    /// The returned reference is borrowed from `&self`. Callers reach it via
    /// `Task.cgroup.load()` (`ArcSwapGuard<Cgroup>`) → `Cgroup::memcg(&rcu_guard)`
    /// (`Option<&MemCgroup>`, an `RcuPtr<Arc<MemCgroup>>` read) → `fma_stats()`; the
    /// returned lifetime is bound to BOTH the `ArcSwap` guard (keeps the `Cgroup`
    /// alive) and the `RcuReadGuard` (keeps the `MemCgroup` borrow valid), so both
    /// must outlive the reference — or `Arc::clone` the `MemCgroup` under the guards.
    pub fn fma_stats(&self) -> Option<&FmaCgroupStats> {
        self.fma.get()
    }
}

17.2.1.3.1 Per-CPU Memory Charge Batching (`MemCgroupStock`)¶

On systems with 128+ CPUs, every page allocation contends on the memory controller's global usage: AtomicU64 counter. To eliminate this hot-path atomic contention, UmkaOS uses a per-CPU charge cache (MemCgroupStock) held in the CpuLocalBlock:

/// Per-CPU charge cache for the memory controller.
/// Amortizes global AtomicU64 contention on `MemCgroup::usage`.
/// Stored in the `CpuLocalBlock` (see [Section 3.2](03-concurrency.md#cpulocal-register-based-per-cpu-fast-path)).
// kernel-internal, not KABI — per-CPU charge cache, never crosses a boundary.
#[repr(C)]
pub struct MemCgroupStock {
    /// Pre-charged bytes available for allocation without touching the global counter.
    cached_charge: u64,
    /// The cgroup this stock is cached for. `None` = stock is empty.
    /// Note: kernel-internal struct (per-CPU, never crosses KABI or wire
    /// boundary). `Option<CgroupId>` layout is stable within a single
    /// compilation. Consider `NonZeroU64` for `CgroupId` to get niche
    /// optimization (`Option<CgroupId>` = 8 bytes, zero = None).
    cached_cgroup: Option<CgroupId>,
}
// MemCgroupStock: cached_charge u64(8) + cached_cgroup Option<CgroupId>(16;
// CgroupId = plain u64, so Option has no niche) = 24. Kernel-internal per-CPU cache.
const_assert!(core::mem::size_of::<MemCgroupStock>() == 24);

Operation:

Allocator entry points: the page-level callers of this protocol are cgroup_charge_page / cgroup_charge_page_root / cgroup_uncharge_page (Section 4.2). Those are thin wrappers that add page framing — cgroup resolution and the page.mem_cgroup ownership tag — over MemCgroup::charge / MemCgroup::uncharge. The stock and IrqDisabledGuard protocol specified below is owned HERE and is never restated allocator-side.

Charge (hot path): MemCgroup::charge(cg, size) acquires an IrqDisabledGuard (via CpuLocal::irq_save()) then checks the local CPU's MemCgroupStock. If cached_cgroup == cg and cached_charge >= size, deduct locally — zero atomics, zero contention. The guard MUST disable IRQs (not merely preemption) for the entire read-check-deduct sequence, matching Linux's Linux local_irq_save() / local_irq_restore() in consume_stock(). A bare PreemptGuard is INSUFFICIENT: the stock drains (item 3) are IPI-driven (smp_call_function_all), and preempt-disable does not mask an incoming IPI, so a drain handler could fire on THIS CPU BETWEEN the cached_charge >= size check and the deduct — it would usage.fetch_sub(cached_charge) and clear the stock, after which the interrupted consume writes back its stale cached_charge - size, resurrecting a stock whose pre-charge was already returned to the global counter (charge consumed with no global accounting; after rmdir, a stock naming a dead CgroupId). IRQ-disable additionally protects the NON-atomic cached_charge: u64 / cached_cgroup fields on 32-bit arches (ARMv7, PPC32), where a u64 store is two word-stores and a mid-store drain IPI would otherwise read a torn value on the same CPU. Task migration to a different CPU mid-sequence is excluded as a side effect; the deferred drain IPI simply runs once the window closes.
Refill: When the stock is empty or for a different cgroup, perform a single usage.fetch_add(STOCK_SIZE) on the global counter, caching STOCK_SIZE bytes locally. Default STOCK_SIZE = 32 * PAGE_SIZE (128 KiB on 4K pages; 2 MiB on AArch64/PPC64 with 64K pages). The proportional scaling is intentional: larger pages mean larger minimum allocation granularity, so the batching stock scales proportionally to avoid excessive global counter traffic.
Drain triggers: The per-CPU stock is drained (returned to the global counter) on:
CPU offline (cpu_dead notifier)
Task migration to a different cgroup (the old cgroup's stock is flushed)
memory.high breach detection (all CPUs' stocks for that cgroup are drained via IPI to get an accurate reading)
memory.max limit check (drain before comparing against limit)
rmdir / Active → Draining transition (cgroup_rmdir step 5): every CPU's stock cached for the dying cgroup is flushed via smp_call_function_all (the SAME IPI mechanism as memory.high). The flush RETURNS the unconsumed pre-charge — usage.fetch_sub(cached_charge) — and clears the stock. Direction: refill (item 2) PRE-CHARGES via usage.fetch_add(STOCK_SIZE), so an unconsumed remote stock leaves usage ABOVE true usage; the flush must SUBTRACT the unconsumed remainder to bring usage down (Linux drain_stock() → page_counter_uncharge()). Without THIS trigger the four above never fire at destruction time, so a stock pre-charged on a remote CPU leaves usage inflated and the drain's Phase 5 (usage == 0) is unreachable — every rmdir would stall the full CGROUP_MEM_DRAIN_TIMEOUT_MS and force-reparent. This trigger makes zero-residual Phase 5 reachable.
Accuracy: because refill pre-charges, the global usage counter may be up to nr_cpus * STOCK_SIZE ABOVE the true usage (the unconsumed pre-charge held in remote stocks). Limit enforcement (memory.max) drains all stocks (fetch_sub each unconsumed cached_charge) before rejecting an allocation, bringing usage down to the exact true value so limits are respected exactly. Soft limits (memory.high) tolerate the imprecision — the reclaim signal may fire slightly late, which is acceptable.
Post-Dead stock safety: after the rmdir flush (trigger 5) the dying cgroup's cached_cgroup slots are cleared, so no stray stock names the dead CgroupId. Should a race leave one (a refill that lost to the flush), its eventual drain resolves CGROUP_REGISTRY.get(cached_cgroup); a None result (the cgroup was unregistered at rmdir step 4b) means the charge has no live home — the bytes are simply discarded (they were already force-reparented or reconciled by the drain), NOT fetch_add'd into a freed counter. The uncharge (page-free) path is unaffected: per Section 4.2, MemCgroup::uncharge decrements the GLOBAL usage counter directly (never the per-CPU stock), so page frees during and after drain never double-count.

17.2.1.4 PID Controller State¶

/// Monotonic, globally-unique serial stamped into every `PidsController` at
/// allocation (`child_controller_alloc(_, PIDS)`). It is the box IDENTITY the
/// fork charge anchors on: a `-pids` (disable) frees the controller box via
/// `RcuPtr::update(None)` and a later `+pids` (re-enable) allocates a FRESH box
/// on the same cgroup, so the cgroup identity alone cannot tell "same box" from
/// "swapped box". A `u64` serial disambiguates unambiguously and is ABA-safe
/// (a reused heap address gets a new serial), which `*const PidsController`
/// pointer identity would not be. `AtomicU64`, cold writer (controller
/// enable only) — never wraps in the operational lifetime.
pub static PIDS_CONTROLLER_SERIAL: AtomicU64 = AtomicU64::new(1); // 0 reserved

/// PID controller state, present when the `pids` controller is enabled.
///
/// Maps to `pids.current`, `pids.max`, and `pids.events` cgroupfs files.
/// See Section 17.2.6 for fork-bomb prevention semantics.
pub struct PidsController {
    /// Box-identity serial from `PIDS_CONTROLLER_SERIAL.fetch_add(1)` at
    /// allocation. The fork charge records this per incremented ancestor so its
    /// paired uncharge (`rollback_fork`) and the `commit_fork`
    /// reconcile act on the EXACT box that was charged, never a re-resolved one
    /// that a mid-fork disable+enable may have swapped in. See
    /// [Section 17.2](#control-groups--cgroup-fork-hooks) `CgroupForkCharge`.
    pub id: u64,

    /// `pids.current`: number of tasks (threads + processes) currently in
    /// this cgroup subtree. Incremented by fork/clone, decremented by exit.
    pub current: AtomicU64,

    /// `pids.max`: maximum tasks allowed in this cgroup subtree.
    /// `u64::MAX` means unlimited (default). `fork()`/`clone()` checks
    /// `current < max` before allocating a new task; returns `EAGAIN` on failure.
    pub max: AtomicU64,

    /// `pids.events max`: number of fork/clone calls that were rejected
    /// because `current` reached `max`. Monotonically increasing.
    pub events_max: AtomicU64,
}

17.2.1.5 I/O Controller State¶

/// I/O controller state, present when the `io` controller is enabled.
///
/// Maps to `io.max`, `io.weight`, `io.stat`, and `io.pressure` cgroupfs files.
/// See [Section 15.18](15-storage.md#io-priority-and-scheduling) for integration with the block I/O scheduler.
pub struct IoController {
    /// `io.weight`: relative I/O weight among siblings (1..=10000, default 100).
    /// Used by the I/O scheduler to compute per-cgroup I/O bandwidth shares.
    /// Higher weight = larger share of available I/O bandwidth relative to siblings.
    /// At I/O dispatch time, the block layer's I/O scheduler reads the
    /// cgroup's `io.weight` via `cgroup_io_weight()` to compute proportional
    /// share. Weight is NOT applied at `BlockDriverOps::submit()` time — tagging at
    /// submission would not account for request merging and reordering.
    ///
    /// **Block-layer consumer**: The KABI I/O scheduler dispatches bios with
    /// a priority derived from `io.weight`. At dispatch time, the scheduler
    /// calls `cgroup_io_weight(bio.task.cgroup)` and maps the weight to
    /// a proportional share of the device's dispatch budget. The mapping is:
    /// `dispatch_share = cgroup_weight / sum(sibling_weights)`. Bios from
    /// higher-weight cgroups are dequeued proportionally more often during each
    /// dispatch round. This is implemented in the block layer's dispatch loop
    /// ([Section 15.2](15-storage.md#block-io-and-volume-management--bio-submission-entry-point)),
    /// which evaluates cgroup weight when selecting the next request to issue
    /// to the device.
    pub weight: AtomicU32,

    /// `io.latency` target in microseconds per device. When the cgroup's average
    /// `io.latency`: per-device latency targets + EMA state. `io.latency` is
    /// PER-DEVICE (`MAJ:MIN target=<usec>`), so a single scalar cannot hold it —
    /// this is an `XArray<IoDeviceLatency>` keyed by `dev_t`, each entry holding
    /// its own target and the "per-device exponential moving average" the
    /// enforcement text below requires. An absent entry = no target for that
    /// device (default). Warm path (updated on bio completion).
    ///
    /// **Enforcement mechanism**: On each bio completion, the block layer updates
    /// that device's EMA of completion latency:
    /// `ema = ema * 7/8 + actual_latency * 1/8`. When `ema > target_us`, the
    /// cgroup is "latency-protected" on that device and the I/O scheduler
    /// applies backpressure to sibling cgroups: their dispatch budget is reduced
    /// to `max(1, normal_budget * target_us / sibling_ema)`. The 8-sample EMA
    /// gives hysteresis; protected status clears when `ema <= target_us` for 8
    /// consecutive samples (`protected_streak`).
    pub latency: XArray<IoDeviceLatency>,

    /// `io.stat`: per-device I/O accounting counters. `io.stat` is per-device
    /// (`MAJ:MIN rbytes=N wbytes=N rios=N wios=N dbytes=N dios=N`,
    /// [Section 15.18](15-storage.md#io-priority-and-scheduling)), so counters live PER DEVICE, not as
    /// controller-wide scalars. `XArray<IoDeviceStat>` keyed by `dev_t`; the
    /// block layer's completion path increments the matching entry (creating it
    /// on first I/O to a device). Read to render `io.stat`.
    pub stat: XArray<IoDeviceStat>,

    /// Per-device rate limits (write-side authoritative copy).
    /// Configuration writes (`io.max` writes from cgroupfs) acquire the
    /// Mutex, modify the Vec, then publish a new immutable snapshot to
    /// `devices_snapshot`. Only the cgroupfs write path touches this.
    /// Bounded by unique `(major, minor)` device pairs. Cap: 1024 per cgroup
    /// (enforced by cgroupfs write handler; returns ENOSPC if exceeded).
    /// Typical servers have 10-50 block devices; 1024 provides ample headroom.
    pub devices_config: Mutex<Vec<IoDeviceLimits>>,

    /// Per-device rate limits (read-side snapshot for hot path). Immutable
    /// STATIC limits only. XArray keyed by `dev_t` (u64 integer key) — O(1)
    /// lookup per bio. Published via RCU clone-and-swap when `devices_config`
    /// changes. `cgroup_io_throttle()` reads this under `rcu_read_lock()` — no
    /// Mutex contention, no heap allocation on the per-I/O path.
    ///
    /// **`cgroup_io_throttle()` lookup path** (called from the block layer
    /// bio submission path):
    /// ```
    /// // CGROUP_REGISTRY is the ONE canonical id→cgroup registry (NOT the old
    /// // `CGROUP_TABLE`); the controller field is `cgroup.io` (NOT
    /// // `cgroup.subsystems.io`).
    /// let guard = rcu_read_lock();
    /// let cgroup = CGROUP_REGISTRY.get(bio.cgroup_id)?;   // None → unthrottled
    /// let io_ctl = cgroup.io.read(&guard)?;               // None → unthrottled (RcuPtr)
    /// io_ctl.throttle(bio, &guard);
    /// ```
    /// Inside `io_ctl.throttle(bio, guard)`: reads `devices_snapshot` via RCU
    /// for the STATIC limits, then consults the MUTABLE `throttle` state
    /// (below) for `bio.dev`, applies rate-limit checks (rbps/wbps/riops/wiops),
    /// and sleeps on the throttle state's wait queue if the token bucket is
    /// empty. The RCU read lock is dropped before sleeping.
    pub devices_snapshot: RcuCell<XArray<IoDeviceLimits>>,

    /// Per-(cgroup, device) MUTABLE token-bucket enforcement state — the home
    /// the block layer's `cgroup_io_throttle` needs but that neither
    /// `IoDeviceLimits` (static) nor `devices_snapshot` (RCU clone-and-swap,
    /// republished wholesale on every `io.max` write) can hold: in-struct
    /// counters there would reset mid-flight on republication. `IoThrottleState`
    /// is defined in [Section 15.2](15-storage.md#block-io-and-volume-management--cgroup-io-throttling)
    /// (token balance, last-refill timestamp, waiter queue; 1 ms refill tick).
    /// `XArray<Arc<IoThrottleState>>` keyed by `dev_t`, created lazily when
    /// `io.max` is first set for a device. Values are `Arc`-wrapped so a
    /// bio-submission lookup can clone out a counted reference that outlives the
    /// `rcu_read_lock()` guard (the `Throttle` ticket may sleep in
    /// `wait_for_token()` after the guard is dropped). This is the field
    /// block-io's "looks up the throttle state from the cgroup's `IoController`"
    /// clause resolves to.
    pub throttle: XArray<Arc<IoThrottleState>>,

    /// PSI (Pressure Stall Information) for this cgroup's I/O subsystem.
    /// Exposed as `io.pressure`. Tracks the fraction of time tasks in this
    /// cgroup are stalled waiting for I/O completions.
    pub psi: PsiState,

    /// In-flight bio counter for this cgroup — the O(1) predicate the
    /// zero-residual drain (Phase 4) polls. Incremented by the block layer when
    /// a bio is enrolled in its device's `BlkInflightTable`, decremented when
    /// removed at completion — the SAME two hooks that stamp `bio.cgroup_id`
    /// ([Section 15.2](15-storage.md#block-io-and-volume-management--per-device-in-flight-bio-table)). One
    /// extra atomic per bio on paths that already touch per-cgroup io throttle
    /// state; lets `cgroup_drain_residual` check "no in-flight bios for this
    /// cgroup" without scanning every device's sharded inflight table (which is
    /// racy — `capture_all()` is crash-path-only — and O(devices × shards)).
    ///
    /// **Reachable decrement after rmdir (tombstone)**: a bio can still be in
    /// flight when its cgroup is `rmdir`'d, and the completion hook resolves the
    /// cgroup from `bio.cgroup_id` via `CGROUP_REGISTRY`. rmdir step 4b leaves the
    /// registry entry as a TOMBSTONE (it does NOT erase while residuals remain),
    /// so the post-rmdir completion still resolves the (Dead) cgroup and
    /// decrements this counter to 0 — without the tombstone the lookup would fail,
    /// this counter would never reach 0, and every rmdir with in-flight I/O would
    /// eat the full drain timeout. Each enrolled bio also bumps
    /// `Cgroup.residual_refs` (dropped at completion), so the cgroup is freed by
    /// the last completion if the drain kworker already finished. See
    /// [Section 17.2](#control-groups--registry-tombstone-and-residual-charge-drain).
    /// **Cross-file handoff**: the block layer owns the inc/dec sites (and the
    /// paired `residual_refs` inc/dec).
    pub inflight_bios: AtomicU64,
}

/// Per-device `io.latency` state (one entry per device in `IoController.latency`).
pub struct IoDeviceLatency {
    /// `io.latency` target for this device, microseconds. 0 = no target.
    pub target_us: AtomicU64,
    /// Exponential moving average of completion latency (µs), 8-sample window.
    pub ema_us:    AtomicU64,
    /// Consecutive samples with `ema <= target_us`; protected status clears at 8.
    pub protected_streak: AtomicU32,
}

/// Per-device `io.stat` counters (one entry per device in `IoController.stat`).
/// Field names match the Linux `io.stat` line tokens.
pub struct IoDeviceStat {
    pub rbytes: AtomicU64,
    pub wbytes: AtomicU64,
    pub rios:   AtomicU64,
    pub wios:   AtomicU64,
    pub dbytes: AtomicU64, // discarded bytes
    pub dios:   AtomicU64, // discard operations
}

/// Block device identifier encoded as (major, minor) pair.
/// Linux ABI: `dev_t` is a 64-bit value with major in bits 8-19 and 32-63,
/// minor in bits 0-7 and 20-31 (glibc encoding). This struct stores the
/// decoded components; `from_dev_t` / `to_dev_t` handle the bit packing.
#[repr(C)]
pub struct DeviceNumber {
    /// Major device number (identifies the driver).
    pub major: u32,
    /// Minor device number (identifies the device instance within the driver).
    pub minor: u32,
}
// kernel-internal, not KABI. Layout: 4 + 4 = 8 bytes.
const_assert!(size_of::<DeviceNumber>() == 8);

impl DeviceNumber {
    /// Decode a Linux `dev_t` (glibc encoding) into (major, minor).
    pub fn from_dev_t(dev: u64) -> Self {
        Self {
            major: ((dev >> 8) & 0xFFF | (dev >> 32) & !0xFFF) as u32,
            minor: (dev & 0xFF | (dev >> 12) & !0xFF) as u32,
        }
    }

    /// Encode as Linux `dev_t` (glibc encoding).
    pub fn to_dev_t(&self) -> u64 {
        let maj = self.major as u64;
        let min = self.minor as u64;
        ((maj & 0xFFF) << 8) | ((maj & !0xFFF) << 32)
            | (min & 0xFF) | ((min & !0xFF) << 12)
    }
}

/// Per-device I/O limits for one block device within a cgroup.
/// All limit fields use `u64::MAX` as the "unlimited" sentinel, consistent with
/// `MemCgroup.max`, `MemCgroup.high`, `PidsController.max`, and `swap_max`.
/// This avoids the 32-byte discriminant overhead of `Option<u64>` (no niche
/// optimization for `u64`) and eliminates the per-field branch on the bio
/// submission hot path. Read under RCU via `devices_snapshot`.
pub struct IoDeviceLimits {
    /// Block device identified by (major, minor) numbers.
    pub dev: DeviceNumber,
    /// Read bandwidth limit in bytes per second. `u64::MAX` = unlimited.
    pub rbps: u64,
    /// Write bandwidth limit in bytes per second. `u64::MAX` = unlimited.
    pub wbps: u64,
    /// Read I/O operations per second limit. `u64::MAX` = unlimited.
    pub riops: u64,
    /// Write I/O operations per second limit. `u64::MAX` = unlimited.
    pub wiops: u64,
}

17.2.1.6 Additional Controller State Structs¶

The following structs back the rdma, hugetlb, misc, cpuset, and shared PSI/LRU fields referenced in Cgroup above. They are defined here rather than inline so that the Cgroup struct definition in Section 17.2.1.1 remains readable.

/// RDMA cgroup controller. Limits RDMA/InfiniBand resource usage per cgroup.
/// Controls: MR (memory regions), MW (memory windows), PD (protection domains),
/// AH (address handles), QP (queue pairs), SRQ (shared receive queues).
/// Mirrors Linux's `rdma` cgroup subsystem (kernel 4.11+).
///
/// **Charge integration (not a no-op limit), HIERARCHICAL**: the RDMA verbs
/// subsystem calls `rdma_charge(cg_id, dev_index, kind)` on every resource
/// allocation (Linux `ib_create_qp`, `ib_reg_mr`, …) and `rdma_uncharge(cg_id,
/// dev_index, kind)` on release. Both WALK the anchored cgroup → root, mirroring
/// Linux `rdmacg_try_charge` (`for (p = cg; p; p = parent_rdmacg(p))`,
/// `kernel/cgroup/rdma.c`, `torvalds/linux` master, web-verified) and this
/// file's own `PerfEventController` step 2/3 root-ward walk: `rdma_charge`
/// `fetch_add`s the matching `RdmaDeviceUsage` counter at each level, and if any
/// level exceeds its `RdmaDeviceLimit` it rolls back the increments done so far
/// on THIS walk and returns `EAGAIN` — so a parent's `rdma.max` constrains all
/// descendants. `rdma_uncharge` `fetch_sub`s the SAME chain. The `cg_id` is the
/// resource's CHARGE ANCHOR — see `RdmaResourceAnchor` — NOT the destroying
/// task's current cgroup (which drifts on migrate/exit).
///
/// **Outlives-rmdir (residual, tombstone)**: RDMA resources outlive the charging
/// task (fds are passed out), so the anchored cgroup can be `rmdir`'d first. Each
/// `rdma_charge` bumps `Cgroup.residual_refs` at EVERY level of the charge walk
/// (each `rdma_uncharge` drops it per level) — pinning the WHOLE charged chain,
/// not just the anchor, so the root-ward uncharge walk's `parent.upgrade()`
/// reaches every level even when an INTERMEDIATE ancestor emptied and was
/// `rmdir`'d (anchor-only pinning would let a freed intermediate strand the
/// per-level counter and break the walk — the same shape fixed on
/// `PerfEventController`). `rmdir` leaves the registry entry as a TOMBSTONE, so a
/// post-rmdir `rdma_uncharge` resolves its anchor via `CGROUP_REGISTRY.get(cg_id)`
/// and decrements the (Dead) cgroup's counter chain — the last one frees the cgroup.
/// See [Section 17.2](#control-groups--registry-tombstone-and-residual-charge-drain). Wiring
/// the hooks into [Section 13.6](13-device-classes.md#rdma-subsystem) and
/// [Section 22.7](22-accelerators.md#accelerator-networking-rdma-and-linux-gpu-compatibility) is a deferred
/// cross-file handoff (recorded in the fix report by symbol).
pub struct RdmaController {
    /// Per-device RDMA resource limits. Key: RDMA device index (u32 integer key).
    /// XArray: O(1) lookup by device index, RCU-compatible reads.
    pub limits: XArray<RdmaDeviceLimit>,
    /// Current RDMA resource usage. Key: RDMA device index (u32 integer key).
    pub usage: XArray<RdmaDeviceUsage>,
}

/// Every long-lived RDMA resource (QP/MR/MW/PD/AH/SRQ) stores this anchor so
/// its uncharge credits the SAME cgroup it was charged to. `CgroupId` (an
/// integer, resolved via `CGROUP_REGISTRY`), not `Arc<Cgroup>`: it must survive
/// the charging task migrating or exiting between create and destroy —
/// `detach_exiting_task()` step 5 repoints `task.cgroup` at the root, so uncharging via
/// the destroying task's current cgroup would drift `RdmaDeviceUsage`
/// permanently. Deferred handoff: [Section 13.6](13-device-classes.md#rdma-subsystem) embeds this in each
/// resource struct. Same anchor shape as `PerfEventController`'s per-event
/// charge site.
#[repr(C)]
pub struct RdmaResourceAnchor {
    pub charged_cgroup: CgroupId,
    pub dev_index:      u32,
    /// Explicit trailing padding (rule 11: no implicit holes).
    pub _pad:           [u8; 4],
}
// kernel-internal anchor embedded in RDMA resource structs.
// Layout: 8 (charged_cgroup) + 4 (dev_index) + 4 (pad) = 16 bytes.
const_assert!(size_of::<RdmaResourceAnchor>() == 16);

/// Per-device RDMA resource limits for one cgroup.
/// Sentinel `u32::MAX` = unlimited (Linux `rdma` uses "max"), for every field.
/// `mkdir` default (and any un-set field): `u32::MAX` (unlimited).
///
/// Write format (`rdma.max`, Linux-compatible): `<device> hca_handle=<N>
/// hca_object=<N>`. Per the verified Linux master `kernel/cgroup/rdma.c` the two
/// resources Linux's rdma controller limits are:
///   - `hca_handle` = number of HCA HANDLES, i.e. uverbs device CONTEXTS
///     (`ib_ucontext`) — NOT queue pairs. Backed by `max_hca_handle`.
///   - `hca_object` = number of ALL verbs OBJECTS aggregated (QP + CQ + MR +
///     MW + PD + AH + SRQ) — NOT just memory regions. Backed by `max_hca_object`;
///     the matching usage is the SUM of the per-kind object counters.
/// (The earlier `hca_handle→max_qp` / `hca_object→max_mr` mapping was wrong: it
/// would cap QPs when an admin wrote `hca_handle=` and MRs on `hca_object=`,
/// silently diverging from every Linux `rdma.max` script.)
/// The per-kind `max_mr`/`max_mw`/… fields are a UmkaOS EXTENSION accepting
/// `<kind>=<N>` tokens for finer control; they are NOT part of the Linux ABI.
pub struct RdmaDeviceLimit {
    /// Max HCA handles (uverbs contexts) — Linux `hca_handle`. `u32::MAX` = unlimited.
    pub max_hca_handle: u32,
    /// Max HCA objects (aggregate over ALL verbs object kinds) — Linux
    /// `hca_object`. Checked against the SUM of the per-kind usages. `u32::MAX` = unlimited.
    pub max_hca_object: u32,
    /// Max memory regions (UmkaOS extension token `mr=`). `u32::MAX` = unlimited.
    pub max_mr:  u32,
    /// Max memory windows (extension `mw=`). `u32::MAX` = unlimited.
    pub max_mw:  u32,
    /// Max protection domains (extension `pd=`). `u32::MAX` = unlimited.
    pub max_pd:  u32,
    /// Max address handles (extension `ah=`). `u32::MAX` = unlimited.
    pub max_ah:  u32,
    /// Max queue pairs (extension `qp=`). `u32::MAX` = unlimited.
    pub max_qp:  u32,
    /// Max shared receive queues (extension `srq=`). `u32::MAX` = unlimited.
    pub max_srq: u32,
}

/// Current RDMA resource usage for one cgroup on one device.
/// Incremented by `rdma_charge`, decremented by `rdma_uncharge`. The Linux
/// `hca_object` usage is `mr + mw + pd + ah + qp + srq` (summed on demand for
/// the `max_hca_object` check and the `rdma.current` render); `hca_handle` usage
/// is its own counter (uverbs contexts).
pub struct RdmaDeviceUsage {
    /// HCA handles = open uverbs contexts (Linux `hca_handle`).
    pub hca_handle: AtomicU32,
    pub mr:  AtomicU32,
    pub mw:  AtomicU32,
    pub pd:  AtomicU32,
    pub ah:  AtomicU32,
    pub qp:  AtomicU32,
    pub srq: AtomicU32,
}

/// Huge-page cgroup controller. Limits huge page usage per cgroup per page size.
/// Maps to Linux's `hugetlb` cgroup subsystem.
///
/// **Charge integration (not a no-op limit)**: `hugetlbfs` page reservation and
/// fault-in ([Section 14.18](14-vfs.md#pseudo-filesystems), [Section 4.8](04-memory.md#virtual-memory-manager)) call
/// `hugetlb_charge(cg_id, size, bytes)` on reserve/allocate and
/// `hugetlb_uncharge(cg_id, size, bytes)` on free/unreserve, keyed by page
/// size. `cg_id` is the folio's charge anchor (stored on the huge folio, same
/// anchor discipline as RDMA/memory).
///
/// **Outlives-rmdir (residual, tombstone — NOT a reparent)**: a huge folio can
/// be mapped by a process in another cgroup, so it outlives this cgroup's
/// `rmdir`. UmkaOS keeps no per-cgroup huge-folio list, so there is no
/// enumeration by which to reparent, and moving the `usage` counter to the
/// parent while the folios' stored anchor still points HERE would double-subtract
/// at their eventual uncharge. Instead each `hugetlb_charge` bumps
/// `Cgroup.residual_refs` (each uncharge drops it); the folios keep their anchor;
/// a post-rmdir `hugetlb_uncharge` resolves the registry TOMBSTONE and
/// `fetch_sub`s this (Dead) cgroup's `usage[size]`; the cgroup frees when the
/// last such folio is uncharged. See
/// [Section 17.2](#control-groups--registry-tombstone-and-residual-charge-drain) (this is the
/// "lazy uncharge-time redirect" mechanism; drain Phase 2 does no counter work).
/// Wiring the hooks into hugetlbfs is a deferred cross-file handoff (recorded in
/// the fix report).
pub struct HugetlbController {
    /// Maximum huge-page bytes allowed per page size. Value `u64::MAX` =
    /// unlimited (`mkdir` default). Key = `HugePageSize` (bytes), value = limit.
    /// `XArray<K, V>` two-parameter form is the documented corpus convention for
    /// integer-keyed maps with a non-`Arc` value ([Section 3.13](03-concurrency.md#collection-usage-policy);
    /// same form as `TaskId`/`CapId`/`DomainId`-keyed XArrays).
    pub limits: XArray<HugePageSize, u64>,
    /// Current huge-page bytes in use per page size. Same two-parameter
    /// `XArray<HugePageSize, AtomicU64>` spelling as `limits` (no bare
    /// single-parameter form — consistent within this controller).
    pub usage:  XArray<HugePageSize, AtomicU64>,
}

/// Huge page size in bytes. The `hugetlb.<size>.max` file name maps to a byte
/// key via the human-readable size token: `hugetlb.2MB.max` → `2 * 1024 * 1024`,
/// `hugetlb.1GB.max` → `1024 * 1024 * 1024`, `hugetlb.16MB.max` (PPC64) →
/// `16 * 1024 * 1024`, `hugetlb.512MB.max` (ARM64 with 64K base) →
/// `512 * 1024 * 1024`, `hugetlb.64KB.max` (ARM64 contiguous) → `64 * 1024`.
/// The set of valid keys is exactly the huge-page sizes the boot-time page
/// tables enumerate for the architecture ([Section 4.8](04-memory.md#virtual-memory-manager)); a
/// write to a `<size>` the hardware does not support returns `EINVAL`.
pub type HugePageSize = u64;

/// System-wide miscellaneous-resource capacities, keyed by resource name (the
/// provider-set `total` for the WHOLE machine). This is a GLOBAL registry, not
/// per-cgroup state: capacity is a property of the host, exactly as Linux keeps
/// it — Linux `static u64 misc_res_capacity[MISC_CG_RES_TYPES]` in
/// `kernel/cgroup/misc.c` (`torvalds/linux` master, web-verified). Storing it
/// per-cgroup (the earlier `MiscController.capacities` field) was wrong: every
/// cgroup would carry a duplicate, and `misc_register_capacity` would have no
/// single instance to write. A name absent here (capacity 0) cannot be charged.
/// Cold path: `misc_register_capacity` writes at provider init; the charge path
/// takes a read lock. `RwLock<BTreeMap<Box<str>, u64>>` — string keys (a
/// non-integer ordered map is the sanctioned `BTreeMap` case), bounded (<10
/// kinds system-wide).
pub static MISC_CAPACITIES: RwLock<BTreeMap<Box<str>, u64>> =
    RwLock::new(BTreeMap::new());

/// Register a system-wide capacity for a misc resource kind (the
/// Linux `misc_cg_set_capacity()` contract). Cold path, provider init. A `total` of 0
/// un-registers the kind (all subsequent `misc.max` writes for it return
/// `EINVAL`).
pub fn misc_register_capacity(name: &str, total: u64) {
    let mut caps = MISC_CAPACITIES.write();
    if total == 0 { caps.remove(name); } else { caps.insert(name.into(), total); }
}

/// Miscellaneous cgroup controller (Linux 5.13+). Provides per-resource usage
/// limits for resources that do not fit into other controllers (e.g., SGX EPC
/// pages, UHID entries).
///
/// **Provider registration (not a no-op limit)**: a resource kind is usable
/// only after its provider registers a SYSTEM-WIDE capacity via
/// `misc_register_capacity(name, total)` into the global `MISC_CAPACITIES`
/// (Linux `misc_cg_set_capacity()`); a kind with no registered capacity rejects
/// all `misc.max` writes with `EINVAL` and never appears in `misc.current`. The
/// provider then calls `misc_charge(cg_id, name, amount)` on acquire and
/// `misc_uncharge(cg_id, name, amount)` on release, anchored on `cg_id`. SGX EPC
/// ([Section 9.7](09-security.md#confidential-computing)) and any device-class provider
/// wire these hooks — a deferred cross-file handoff (recorded in the fix report).
pub struct MiscController {
    /// Per-cgroup per-resource limit + live usage. Key: resource name (e.g.
    /// "sgx_epc"). Warm path: charge/uncharge on device open/close.
    ///
    /// **Interior mutability**: `MiscController` is reached only through
    /// `Arc<Cgroup>`, so entry CREATION (first `misc.max` write or first charge
    /// of a new kind) must mutate this map through a shared reference — a plain
    /// `BTreeMap` field is un-compilable there (the same rule `MiscResource.max`
    /// states one struct below). It is therefore a `SpinLock<BTreeMap<Box<str>,
    /// Arc<MiscResource>>>`: the lock guards entry insertion (under `config_lock`
    /// for `misc.max`, or briefly on the charge path's get-or-create); the value
    /// is `Arc<MiscResource>` so a charger clones the Arc under the lock, RELEASES
    /// the lock, and does its `fetch_add`/`fetch_sub` on the Arc's atomics (a
    /// bare `&MiscResource` could dangle across a concurrent `BTreeMap` insert
    /// that splits nodes). Entries are never removed, so a held Arc stays valid.
    /// `BTreeMap`: non-integer (string) ordered keys, bounded (<10 kinds).
    pub resources: SpinLock<BTreeMap<Box<str>, Arc<MiscResource>>>,
}

/// One named miscellaneous resource tracked by a cgroup's `MiscController`.
///
/// **Hierarchical charge** (Linux `misc_cg_try_charge` walks
/// `for (i = cg; i; i = parent_misc(i))`, web-verified): `misc_charge(cg_id,
/// name, amount)` resolves the anchored cgroup and walks it → root; at each
/// level it `fetch_add(amount)`s that level's `MiscResource.usage`, then checks
/// the result against that level's `max` AND the global
/// `MISC_CAPACITIES[name]`. If any level fails, it rolls back the `fetch_add`s
/// done so far on THIS walk (add-then-check, the overshoot-safe pattern
/// `PerfEventController` uses) and returns `EBUSY`. `misc_uncharge` walks the
/// SAME anchored chain, `fetch_sub`(amount) per level. The anchor is the
/// resource's stored `CgroupId`; a charge that outlives `rmdir` resolves the
/// registry TOMBSTONE (see [Section 17.2](#control-groups--registry-tombstone-and-residual-charge-drain)),
/// and each residual misc charge bumps `Cgroup.residual_refs` at EVERY level of
/// the charge walk (each uncharge drops it per level) — pinning the whole
/// charged chain so the root-ward uncharge walk survives an intermediate
/// ancestor's `rmdir`, the same shape as `PerfEventController`/`RdmaController`.
pub struct MiscResource {
    /// Maximum units allowed at THIS level. `u64::MAX` = unlimited (`mkdir`
    /// default). Capped at the provider-registered capacity. `AtomicU64`:
    /// `misc.max` is a writable cgroupfs file written through a shared
    /// `&MiscController` (inside `Option` in the shared `Cgroup`), so it needs
    /// interior mutability like every sibling limit field — a plain `u64` could
    /// not be written through `Arc<Cgroup>`. Written by the `misc.max` handler
    /// under `config_lock`.
    pub max:   AtomicU64,
    /// Current units in use at THIS level (subtree total, via the hierarchical
    /// charge walk above).
    pub usage: AtomicU64,
}

/// Cpuset cgroup controller. Pins tasks in a cgroup to specific CPUs and NUMA nodes.
/// Maps to Linux's `cpuset` subsystem (cgroup v2: `cpuset.cpus`, `cpuset.mems`).
///
/// **Migration attach hook** (Linux `cpuset_can_attach()`/`cpuset_attach()`
/// equivalent, `kernel/cgroup/cpuset.c`): enforced by the task-migration
/// protocol — admission at step 4e (ENOSPC when the target's effective CPU
/// set is empty) and application at step 15 (`Task.cpu_affinity`
/// replacement; a QUEUED task on an excluded CPU moves immediately, a
/// RUNNING one moves at its next context-switch boundary via the
/// deferred-placement rule). See
/// [Section 17.2](#control-groups--task-migration-cgroupprocs-write). A task never
/// CONTINUES running on a CPU outside its cgroup's `allowed_cpus` beyond
/// the bounded resched-IPI window.
pub struct CpusetController {
    /// CPUs this cgroup's tasks are allowed to run on. Empty = inherit from
    /// parent. Written by the `cpuset.cpus` write handler
    /// ([Section 17.2](#control-groups--cpuset-write-path-and-hotplug)) through a shared
    /// `&Cgroup`; read on the scheduler placement warm path. `RcuCell` (writer
    /// proof = `config_lock` `MutexGuard`) gives lock-free reads of a coherent
    /// mask snapshot; a bare `CpuMask` could not be written through
    /// `Arc<Cgroup>`, and a torn read could place a task on a just-removed CPU.
    pub allowed_cpus:  RcuCell<CpuMask>,
    /// NUMA memory nodes this cgroup's tasks may allocate from. Empty = any
    /// node. Same `RcuCell` mutability story as `allowed_cpus` (written by
    /// `cpuset.mems`, read by the allocator's `task_allowed_mems()`).
    pub allowed_mems:  RcuCell<NodeMask>,
    /// `cpuset.cpus.partition` mode. Backing storage for the tri-state
    /// partition file (`member`/`root`/`isolated`) — a distinct concept from
    /// `cpu_exclusive` (a bool), which is why it needs its own field. Stored as
    /// an `AtomicU8` holding a `CpusetPartition` discriminant, written by the
    /// `cpuset.cpus.partition` handler under `config_lock`.
    pub partition:     AtomicU8,
    /// If true, enforce CPU affinity even during load balancing (exclusive
    /// cpuset). `AtomicBool`: written by `cpuset.cpus` handler, read on the
    /// load-balance warm path.
    pub cpu_exclusive: AtomicBool,
    /// If true, enforce NUMA node affinity for memory allocation. `AtomicBool`,
    /// same rationale as `cpu_exclusive`.
    pub mem_exclusive: AtomicBool,
    /// If true, allow migration of tasks off their cpuset during hotplug events.
    ///
    /// **Scope: hotplug ONLY.** UmkaOS does NOT implement migration-time
    /// physical page migration (the cgroup-v1 `cpuset.memory_migrate`
    /// semantics): moving a task into a cpuset with different
    /// `allowed_mems` leaves already-allocated pages where they are —
    /// only FUTURE allocations follow the new mask via the lazy
    /// `task_allowed_mems()` resolution ([Section 4.11](04-memory.md#numa-topology-and-policy)).
    /// This is v2-faithful: cgroup v2 cpuset has no `memory_migrate`
    /// file, and Linux v2 behaves the same way. Workloads that need
    /// page placement to follow the task use `migrate_pages(2)` /
    /// `move_pages(2)` explicitly.
    /// `AtomicBool`: config-written, read on the hotplug path.
    pub mem_migrate:   AtomicBool,
}

/// `cpuset.cpus.partition` mode (the tri-state backing `CpusetController.partition`).
#[repr(u8)]
pub enum CpusetPartition {
    /// Normal cgroup — CPUs are shared with siblings (default).
    Member   = 0,
    /// Partition root — owns an exclusive slice of CPUs for its subtree.
    Root     = 1,
    /// Isolated partition — exclusive CPUs with load balancing disabled
    /// (equivalent to `isolcpus`, for latency-critical workloads).
    Isolated = 2,
}

/// NUMA node affinity mask. Bit N = NUMA node N is allowed.
///
/// Supports up to 1024 NUMA nodes (`[u64; 16]` = 128 bytes), matching Linux's
/// `MAX_NUMNODES` configuration default. For systems with fewer nodes, only the
/// first `ceil(nr_nodes / 64)` words are meaningful; the remainder are zero.
///
/// 1024 nodes is sufficient for the largest production systems (SGI UV3000: 256
/// nodes; HPE Superdome Flex: 32 nodes). `NodeMask` is used in cgroup cpuset
/// configuration (cold path), so the 128-byte size is acceptable — it is never
/// allocated per-page or per-task on the hot path.
pub struct NodeMask {
    pub bits: [u64; 16],
}

impl NodeMask {
    /// True iff any node is allowed by BOTH masks. Used by the OOM
    /// killer's MPOL_BIND candidate filter (`oom_task_in_nodemask()`,
    /// [Section 4.5](04-memory.md#oom-killer)) and by cpuset/mempolicy admission checks.
    /// Allocation-free: 16 word-ANDs.
    pub fn intersects(&self, other: &NodeMask) -> bool {
        self.bits.iter().zip(other.bits.iter()).any(|(a, b)| a & b != 0)
    }
}

/// The three PSI-tracked resources. Indexes the per-CPU stall-time arrays in
/// `PsiCpuState`. One `PsiState` is exposed per resource (`cpu.pressure`,
/// `memory.pressure`, `io.pressure`).
#[repr(u32)]
pub enum PsiResource { Cpu = 0, Memory = 1, Io = 2 }
pub const NR_PSI_RESOURCES: usize = 3;

/// Per-CPU PSI accumulator — the measurement substrate the aggregate EMAs are
/// folded from. One instance per CPU per cgroup (`PsiState.per_cpu`). This is
/// the UmkaOS equivalent of Linux `struct psi_group_cpu` (`kernel/sched/psi.c`).
///
/// **Stall-state derivation** (which task states count toward each resource):
/// on every task state change the scheduler updates the running CPU's counters:
/// - a task is *productive* if `Running` or runnable-and-not-stalled;
/// - `Memory` stall: task blocked in reclaim / refault / swap-in wait
///   (`OnRqState::Off` with `PF_MEMSTALL` set, or throttled on `memory.high`);
/// - `Io` stall: task in uninterruptible block-I/O wait (`iowait`);
/// - `Cpu` stall: task `Queued` (runnable) but NOT running (CPU contention).
/// SOME[res] holds while ≥1 non-productive task on this CPU is stalled on `res`;
/// FULL[res] holds while ALL non-idle tasks on this CPU are stalled on `res`
/// (no task is making progress). These are the exact Linux `PSI_MEM_SOME` /
/// Linux `PSI_MEM_FULL` (etc.) conditions.
// kernel-internal, not KABI — per-CPU accumulator, never crosses a boundary.
pub struct PsiCpuState {
    /// Number of tasks on this CPU currently in each stall class (indexed by a
    /// packed (resource, some/full) enumeration). Updated lock-free by the
    /// owning CPU under its `rq.lock` (single-writer per CPU).
    pub tasks: [AtomicU32; NR_PSI_RESOURCES * 2],
    /// Accumulated nanoseconds this CPU spent with SOME[res] active, per
    /// resource. The owning CPU adds `now - state_start` at each state change.
    pub some_time_ns: [AtomicU64; NR_PSI_RESOURCES],
    /// Accumulated nanoseconds with FULL[res] active, per resource.
    pub full_time_ns: [AtomicU64; NR_PSI_RESOURCES],
    /// Timestamp of this CPU's last state change (ns since boot).
    pub last_change_ns: AtomicU64,
}

/// Per-`PsiState` lock serializing the rstat flush + PSI EMA aggregation
/// against the cgroupfs pressure-file read path. One instance lives in each
/// `PsiState` (`PsiState.rstat_lock`), so a cgroup with `cpu.pressure`,
/// `memory.pressure`, and `io.pressure` has three independent locks — the type
/// is an alias, the storage is the per-`PsiState` field. Held by the 2 s
/// aggregation callback (writer of `some_avg`/`full_avg`/`*_total` and the
/// triggers) and by the matching pressure-file reader. A `SpinLock` (the
/// aggregation runs in timer-callback context and must not sleep). Lock level:
/// `CGROUP_RSTAT_LOCK`, above the runqueue locks and below the tasks lock —
/// the per-CPU folding takes each CPU's `rq.lock` briefly to snapshot, then
/// releases before taking this lock (never nested the other way). The master
/// lock-ordering table row is a deferred handoff (see fix report).
///
/// **Payload = the registered `triggers` list.** The lock CARRIES the
/// `ArrayVec<PsiTrigger, 32>` as its guarded data, so trigger registration and
/// removal mutate it through `&PsiState` via `rstat_lock.lock()` (interior
/// mutability — `PsiState` is reachable only through `Arc<Cgroup>`, so a plain
/// `ArrayVec` field could not be mutated). The atomic `some_avg`/`full_avg`/
/// `*_total` EMAs stay separate atomic fields (read lock-free-ish on the
/// pressure-file path for defense-in-depth) but are WRITTEN by the aggregation
/// callback while it holds this same lock, so a reader that takes the lock sees
/// the triggers and the EMAs/totals consistently — the ONE lock the finding
/// requires, not a second independent container lock.
///
/// NOTE: this resolves the scheduler's "per-CPU accumulation is lock-free"
/// claim — the PER-CPU writes ARE lock-free (single-writer under `rq.lock`);
/// `CgroupRstatLock` guards only the periodic AGGREGATION, the triggers, and
/// the reads, which are not on any hot path.
pub type CgroupRstatLock = SpinLock<ArrayVec<PsiTrigger, 32>>;

/// A userspace PSI pressure trigger (`poll()`/`epoll` on a pressure file with
/// a `"some <threshold_us> <window_us>"` write). This is what systemd-oomd and
/// Meta's oomd consume to react to memory pressure. When the accumulated stall
/// in any `window_ns`-sized sliding window exceeds `threshold_ns`, the trigger
/// fires (the pressure file becomes readable / `EPOLLPRI`).
pub struct PsiTrigger {
    /// SOME (0) vs FULL (1) stall class this trigger watches.
    pub full: u8, // 0 = some, 1 = full
    /// Sliding window length in nanoseconds (Linux allows 500 ms–10 s).
    pub window_ns: u64,
    /// Stall threshold within the window (ns) that fires the trigger.
    pub threshold_ns: u64,
    /// Accumulated stall observed in the current window (ns).
    pub scaled_ns: AtomicU64,
    /// Wait queue woken (EPOLLPRI) when the trigger fires.
    pub waiters: WaitQueueHead,
}

/// Pressure Stall Information (PSI) state for one resource (CPU, memory, or I/O).
/// Exposed via /sys/fs/cgroup/<cgroup>/cpu.pressure, memory.pressure, io.pressure.
/// Two-level aggregate: per-CPU stall accumulators live in the `PsiCpuState`
/// substrate; the folded totals and EMAs live here.
pub struct PsiState {
    /// Which resource this state tracks (indexes `PsiCpuState`'s per-resource
    /// arrays).
    pub resource: PsiResource,
    /// Per-CPU accumulator substrate. The 2 s aggregation runs on ONE CPU and
    /// folds EVERY CPU's `some_time_ns[resource]`/`full_time_ns[resource]` into
    /// the totals and EMAs below — a CROSS-CPU read.
    ///
    /// **Not `PerCpu<T>`**: `PerCpu<T>::get()` requires the OWNING CPU's
    /// `PreemptGuard` (Ch 3), so it cannot serve the aggregation's remote reads
    /// (the SAME rule the taskstats `ListenerList` substrate cites for its
    /// `Box<[ListenerList]>`). This is therefore a CPU-indexed `Box<[T]>`
    /// (`[cpu]` indexing, boot-sized to `num_possible_cpus()` — a CPU that later
    /// offlines with nonzero accumulators still contributes). Writers are the
    /// owning CPU under its `rq.lock` (single-writer per slot, so the per-slot
    /// `AtomicU32`/`AtomicU64` fields need only Relaxed/Release); the fold reader
    /// snapshots each slot under that CPU's `rq.lock` briefly, then releases it
    /// before taking `rstat_lock`.
    pub per_cpu: Box<[PsiCpuState]>,
    /// Exponentially-weighted moving average of stall time, in units of 0.01%.
    /// Index 0 = 10-second window, 1 = 60-second window, 2 = 300-second window.
    /// `some_avg`: at least one task stalled (partial stall).
    /// Written by the 2 s aggregation callback under this `PsiState`'s
    /// `rstat_lock`; read on the cgroupfs pressure-file path under the same
    /// lock. `AtomicU32` for defense-in-depth so a missed lock does not produce
    /// UB on non-x86.
    pub some_avg:  [AtomicU32; 3],
    /// `full_avg`: all tasks stalled (full stall).
    /// Same locking and atomicity rationale as `some_avg`.
    pub full_avg:  [AtomicU32; 3],
    /// Cumulative stall time in microseconds since cgroup creation (the fold
    /// target). Read for the `total=` field of the pressure file.
    pub some_total: AtomicU64,
    pub full_total: AtomicU64,
    /// Timestamp of the last PSI aggregation (nanoseconds since boot).
    pub last_update_ns: AtomicU64,
    /// The actual per-`PsiState` instance of `CgroupRstatLock` (the original
    /// finding was "defined nowhere" — the type alias alone is not a home). Its
    /// PAYLOAD is this `PsiState`'s registered userspace triggers (poll/epoll)
    /// list, `ArrayVec<PsiTrigger, 32>` — bounded, cold path (registration is
    /// rare); a new trigger is checked against the window at each aggregation.
    /// Carrying the triggers as the lock payload gives trigger
    /// registration/removal an interior-mutability access path through
    /// `&PsiState` (a plain `ArrayVec` field could not be mutated, since
    /// `PsiState` is reached only via `Arc<Cgroup>`). Held by the 2 s
    /// aggregation callback across the fold + EMA update + the trigger scan, and
    /// by the `cpu.pressure`/`memory.pressure`/`io.pressure` reader. The per-CPU
    /// snapshot briefly takes each CPU's `rq.lock` and RELEASES it before this
    /// lock is taken (never nested rq→rstat the other way). Because holding it
    /// covers the triggers AND (by the write-under-lock discipline)
    /// `some_avg`/`full_avg`/the totals, all are consistent to any reader — the
    /// ONE lock, not a second independent container lock.
    pub rstat_lock: CgroupRstatLock,
}

// NOTE: `CgroupLru` (the per-`(memcg, node)` MGLRU generation state linked
// from `MemCgroup.lru_gen`) is defined ONCE, canonically, in
// [Section 4.4](04-memory.md#page-cache--generational-lru-page-reclaim) (`max_seq`/`min_seq`/
// `timestamps`/per-generation folio lists). It is NOT redefined here: the
// earlier duplicate `CgroupLru` in this file (with `generations`/`oldest_gen`/
// `youngest_gen`/`writeback` fields and a single-instance `SpinLock` owner)
// diverged from the page-cache struct in both body and linkage and is removed.
// Reclaim and the rmdir Phase-1 reparent walk operate on the per-node
// `MemCgroup.lru_gen[node]` instances.

17.2.1.7 Hierarchy Root¶

/// Root of the cgroup v2 unified hierarchy. One instance per system.
///
/// UmkaOS has a single `CgroupRoot` (no per-controller separate hierarchies —
/// those were the v1 design that UmkaOS eliminates). The root cgroup has `id == 1`
/// and no parent.
pub struct CgroupRoot {
    /// The root cgroup node. All other cgroups are reachable from here via
    /// `children` links. `Arc` because `CgroupNamespace` instances hold
    /// per-namespace root references into this tree (at arbitrary subtree nodes).
    pub root: Arc<Cgroup>,

    /// Lock protecting hierarchy structure changes (mkdir, rmdir, task migration).
    /// Held in **write mode** during cgroup creation and destruction (mkdir/rmdir).
    /// Held in **read mode** during task migration (steps 2-14 of the migration
    /// protocol; concurrent migrations allowed). **Not** held during resource
    /// charging (those operations use per-cgroup atomics).
    ///
    /// `RwLock` (level 210 in the lock ordering table): concurrent hierarchy
    /// traversals (cgroupfs readdir, population-count propagation) and task
    /// migrations hold read locks; mkdir/rmdir hold write locks.
    pub hierarchy_lock: RwLock<()>,

    // NOTE: there is NO per-root `id_map` — the SINGLE canonical
    // `CgroupId → Arc<Cgroup>` registry is the global `CGROUP_REGISTRY`
    // static (declared at the top of this section). The earlier `id_map`
    // field here was a THIRD divergent registry (alongside `CGROUP_REGISTRY`
    // and the io pseudocode's `CGROUP_TABLE`) that `cgroup_mkdir()` populated
    // while the scheduler read `CGROUP_REGISTRY` — so scheduler cgroup
    // resolution always returned `None`. It is removed. cgroupfs inode→cgroup
    // resolution and the `CLONE_NEWCGROUP` anchor lookup both go through
    // `CGROUP_REGISTRY.get(id)`. Merit of a global static over a
    // `CgroupRoot` field: the scheduler's `cgroup_from_id()` reads it from
    // tick context (preemption-off = implicit RCU) with no `cgroup_root()`
    // deref, and it is the integer-keyed `XArray` the collection policy
    // mandates.

    /// Monotonically increasing ID counter. Assigned at cgroup creation;
    /// never reused (even after cgroup destruction). `AtomicU64` allows
    /// lock-free ID allocation at mkdir time.
    pub next_id: AtomicU64,

    /// Number of task migrations currently in flight (gauge, not monotonic —
    /// bounded by concurrent `cgroup.procs`/`cgroup.threads` writers, so u64
    /// never approaches wrap). Incremented once per migration operation after
    /// step 3's per-thread CAS acquisition succeeds (`fetch_add(1, Acquire)`);
    /// decremented (`fetch_sub(1, Release)`) at step 16 completion AND on
    /// every rollback exit path (step 4 constraint failure after the CAS,
    /// step 7 ENOMEM rollback, step 9 ENOSPC rollback).
    ///
    /// **Consumer**: live-evolution Phase A' quiescence for cgroup-subsystem
    /// components — after acquiring `hierarchy_lock` in write mode (which
    /// drains Phase-1 migration sections), the evolution orchestrator waits
    /// for this counter to reach zero, which drains Phase-2 sections (the
    /// post-`hierarchy_lock` RQ_LOCK dequeue/enqueue window, bounded to
    /// microseconds — Phase 2 never sleeps). See
    /// [Section 17.2](#control-groups--interaction-with-live-evolution).
    pub migrations_in_flight: AtomicU64,

    /// VFS mount point for the cgroupfs pseudo-filesystem. Empty until
    /// `mount("cgroup2", "/sys/fs/cgroup", "cgroup2", 0, NULL)` runs; the
    /// returned `Mount` (Section 14.3) is `set()` once at that point.
    /// `OnceCell` (not `Option<Mount>`): `CGROUP_ROOT` is a `BootOnceCell<CgroupRoot>`
    /// populated at boot BEFORE cgroupfs is mounted, so the mount path must
    /// write this field through `&'static CgroupRoot` — a plain `Option` could
    /// not be written post-init. Write-once (`set()` at mount), read-many
    /// (`get()`); readers need no lock. cgroup2 is a singleton pseudo-fs that is
    /// never unmounted (kernfs-style), so `OnceCell` (no reset) is exact — an
    /// unmount attempt returns `EBUSY`.
    pub mount: OnceCell<Mount>,
}

/// Global singleton root of the cgroup v2 hierarchy. Populated once during
/// cgroup subsystem boot init — before `/sys/fs/cgroup` is mounted and
/// before any `cgroup_mkdir()` call can occur (PID 1's initial cgroup
/// assignment happens after this point). `BootOnceCell` matches the
/// write-once/read-many lifecycle: no writer contention after init, no
/// locking overhead on the read path.
///
/// This is the named accessor for `root` as referenced throughout this
/// section's pseudocode (`root.hierarchy_lock`, `root.migrations_in_flight`,
/// `root.next_id`) — those steps treat `root` as ambiently reachable;
/// `cgroup_root()` is the concrete, compiler-checkable form of that assumption.
static CGROUP_ROOT: BootOnceCell<CgroupRoot> = BootOnceCell::new();

/// Returns the global cgroup hierarchy root.
///
/// # Panics
/// Panics if called before cgroup subsystem boot init has populated
/// `CGROUP_ROOT`. No code path that can reach a `Cgroup` (task cgroup
/// assignment, `cgroup_mkdir`, OOM subtree iteration) runs before that
/// init completes.
pub fn cgroup_root() -> &'static CgroupRoot {
    CGROUP_ROOT.get().expect("cgroup_root() called before cgroup subsystem init")
}

/// Cgroup subsystem boot init. Constructs the root cgroup and publishes
/// `CGROUP_ROOT` and the root's `CGROUP_REGISTRY` entry BEFORE `/sys/fs/cgroup`
/// is mounted and before PID 1's initial cgroup assignment. The root cannot be
/// built by `cgroup_mkdir()` (that requires a parent + `hierarchy_lock`, both
/// of which live inside the `CgroupRoot` being constructed) — hence this
/// dedicated bootstrap. Called once from the early-boot init sequence
/// ([Section 2.3](02-boot-hardware.md#boot-init-cross-arch)), after the slab allocator and XArray are
/// available and before the scheduler starts (the scheduler's
/// `cgroup_from_id()` reads `CGROUP_REGISTRY`, so the root must be registered
/// first).
pub fn cgroup_subsys_init() {
    // 1. Build the root Cgroup:
    //    - id = 1 (root sentinel; `next_id` starts the counter at 2).
    //    - parent = None; depth = 0.
    //    - name = CgroupName::root() (empty — the root has no relative name).
    //    - lifecycle = Active; population = 0; nr_populated_children = 0;
    //      subtree_populated = 0 (unpopulated).
    //    - ALL built-in controllers instantiated as a non-null RcuPtr::new(state)
    //      with defaults (the root always has every available controller present,
    //      unlike children whose controllers follow the parent's subtree_control):
    //      cpu/memory/io/pids/cpuset/rdma/hugetlb/misc/perf_event.
    //    - subtree_control = AtomicU32::new(0) (nothing delegated to children
    //      until userspace writes cgroup.subtree_control).
    //    - inode = OnceCell::new() (filled at cgroupfs mount, step below).
    //    - config_lock = Mutex::new(()); children = RcuCell::new(Vec::new())?;
    //      dyn_subsys = [null; MAX_DYN_CGROUP_SUBSYS].
    //    (The `Cgroup { id: 1, parent: None, depth: 0, name: CgroupName::root(),
    //     .. }` literal is built inline from the field list above.)
    let root: Arc<Cgroup> = Arc::new(/* root Cgroup literal, fields above */);
    // 2. Register the root in the ONE canonical registry — the scheduler and
    //    every id→cgroup consumer resolve through it.
    CGROUP_REGISTRY.store(root.id, Arc::clone(&root));
    // 3. Publish the CgroupRoot (hierarchy_lock, next_id=2, migrations
    //    in-flight=0, mount = OnceCell::new()).
    CGROUP_ROOT.set(CgroupRoot {
        root,
        hierarchy_lock: RwLock::new(()),
        next_id: AtomicU64::new(2),
        migrations_in_flight: AtomicU64::new(0),
        mount: OnceCell::new(),
    }).ok().expect("cgroup_subsys_init called twice");
    // 4. `cgroup.controllers` on the root advertises every built-in controller
    //    plus `accel` (backed by a registered subsystem). Registered
    //    subsystems (ml_policy, accel) call register_cgroup_subsys() slightly
    //    later in Phase 5c; their dyn_subsys slots stay null until first use.
    // The cgroupfs mount (root.inode.set(...)) happens later when userspace
    // mounts cgroup2 — NOT here.
}

17.2.1.8 Task Migration (cgroup.procs write)¶

Writing a PID to cgroup.procs atomically moves all threads in the thread group to the target cgroup. This is required by the cgroup v2 ABI — Docker, systemd, and Kubernetes expect that writing a TGID moves all threads. Writing to cgroup.threads (if threaded mode is enabled) moves a single thread.

The migration protocol is O(depth × threads) where depth is the cgroup tree height from source/target to their Lowest Common Ancestor (LCA) and threads is the thread group size. Limit recomputation is deferred lazily via generation counters (not done during migration itself).

Threadgroup atomicity: To prevent concurrent fork() from creating new threads during migration (which would escape the target cgroup), the migrating task acquires process.threadgroup_rwsem (a SLEEPING read-write semaphore on the Process struct — sleeping-lock ordering table entry 3a, Section 3.4) in write mode BEFORE step 2. This serializes with fork(), which acquires it in read mode at create_task step 1 and holds it until the child is fully linked (step 18b), AND with detach_exiting_task(), which acquires it in read mode for its whole charge-release sequence (see the Exit path below) — the two-sided serialization that makes exit-vs-migration accounting exact. All-or-nothing semantics: if migration fails for any thread, ALL threads are rolled back to their source cgroups.

In the common case (migration within the same subtree, depth ≤ 4), the LCA walk touches ≤ 8 nodes per thread (~400-800 ns). Worst case (cross-subtree at depth 256, 1000 threads): ~25-50 ms, bounded by the maximum nesting depth × thread count.

Migration steps for: write(fd_cgroup_procs, pid_str):

1. Resolve PID to TaskId using the writer's PID namespace. Obtain
   the `Process` struct to access the thread group. Per-thread source
   cgroups and LCAs are resolved at step 1c (threads of one process may
   legally live in DIFFERENT cgroups of one threaded domain after
   `cgroup.threads` moves — a single source/LCA pair would silently
   corrupt membership and counters for such threads). The LCA helper:
   /// Compute the Lowest Common Ancestor of two cgroups.
   /// Requires both cgroups to have a `depth` field (distance from root).
   ///
   /// Both arguments are `Arc<Cgroup>` because the caller (step 1 of the migration
   /// protocol) already holds `Arc<Cgroup>` references from `ArcSwap::load()`.
   /// Taking `Arc` directly avoids the need for a hypothetical `self_ref()` method
   /// (which would require an embedded `Weak<Self>` and is unnecessary here).
   ///
   /// NOTE: `cg.parent` is `Option<Weak<Cgroup>>`. Each `.unwrap()` upgrades the
   /// `Weak` to `Arc` (elided here for readability). The upgrade cannot fail because
   /// `population > 0` on every ancestor guarantees liveness while descendants exist.
   fn cgroup_lca(a: Arc<Cgroup>, b: Arc<Cgroup>) -> Arc<Cgroup> {
       let mut wa: Arc<Cgroup> = a;
       let mut wb: Arc<Cgroup> = b;
       // Equalize depths by walking the deeper one up.
       while wa.depth > wb.depth {
           wa = wa.parent.as_ref()
               .expect("non-root cgroup has no parent")
               .upgrade()
               .expect("ancestor population > 0 guarantees liveness -- see hierarchy_lock");
       }
       while wb.depth > wa.depth {
           wb = wb.parent.as_ref()
               .expect("non-root cgroup has no parent")
               .upgrade()
               .expect("ancestor population > 0 guarantees liveness -- see hierarchy_lock");
       }
       // Walk both up until they meet.
       while !Arc::ptr_eq(&wa, &wb) {
           wa = wa.parent.as_ref()
               .expect("non-root cgroup has no parent")
               .upgrade()
               .expect("ancestor population > 0 guarantees liveness -- see hierarchy_lock");
           wb = wb.parent.as_ref()
               .expect("non-root cgroup has no parent")
               .upgrade()
               .expect("ancestor population > 0 guarantees liveness -- see hierarchy_lock");
       }
       wa
   }

1b. **Attach permission check** (before any lock is taken, so denial is
    cheap and never blocks a concurrent migration). This is ABI-visible
    cgroup v2 delegation behavior — systemd user sessions and rootless
    container runtimes are built on it. All three checks evaluate the
    credentials the `cgroup.procs` file was OPENED with (`file.f_cred`,
    not the credentials of the process performing the write — cgroup v2 "delegation containment": a
    privileged opener that passes the fd to an unprivileged process
    deliberately delegates its authority; Linux
    Linux `kernel/cgroup/cgroup.c cgroup_attach_permissions()` uses
    `of->file->f_cred` the same way):
    a. **Destination write access**: the opener credentials must have
       write permission on the TARGET cgroup's `cgroup.procs` file
       (standard inode permission check on the cgroupfs node).
       Denial → `EACCES`.
    b. **Common-ancestor rule (the delegation boundary)**: resolve
       `perm_source` = the thread-group leader's cgroup at this instant
       (`leader.cgroup.load_full()`; for a group inside a threaded
       subtree this is a member of the same threaded domain — the
       delegation boundary is identical for every thread, so the leader
       suffices). The opener credentials must have write permission on
       the `cgroup.procs` of `cgroup_lca(perm_source, target)`. A
       delegated subtree manager can therefore move processes freely
       WITHIN its delegated subtree but cannot pull processes in from
       (or push them out to) cgroups above its delegation point.
       Denial → `EACCES`.
    c. **Cgroup-namespace containment**: both `perm_source` and `target`
       must be descendants of (or equal to) the writer's cgroup
       namespace root. A cgroup outside the writer's cgroupns view does
       not exist for it → `ENOENT`.
    The check is deliberately unlocked: a concurrent migration may
    change the group's cgroup between this check and step 1c's snapshot,
    but any such migration was itself performed by a writer that passed
    the same delegation check — the boundary cannot be widened by the
    race (Linux has the same property: the permission check and the
    attach are not atomic with respect to other attachers).
    The `cgroup.threads` variant applies the same three checks with
    `perm_source` = the target THREAD's cgroup, in addition to its
    threaded-domain preconditions.

1a. Acquire `process.threadgroup_rwsem` in write mode (sleeping rwsem,
    ordering entry 3a — no SpinLock is held here, and ordered SpinLocks
    may legally be taken under it during steps 2-12).
    This prevents concurrent `fork()` from creating new threads during
    migration (`fork()` acquires it in read mode at create_task step 1 and
    releases after the child is fully linked, step 18b) AND serializes
    with `detach_exiting_task()` (which acquires it in read mode — see the Exit
    path below): while this writer is held, no thread of this process
    can release its cgroup charges, and any exit that already ran is
    visible via `PF_EXITING` (step 1c's filter).

1c. **Thread-group snapshot** (the ONLY iteration of the live thread
    list in the whole protocol). Acquire `PROCESS_LOCK(25)` — legal:
    no spin-class lock is held yet (25 must be taken BEFORE step 2's
    level-210 lock; taking it later would be a descending acquisition).
    Holding 25 satisfies the `ThreadGroup.tasks` dual-lock reader
    contract (mutators hold 25 AND `SIGLOCK(40)`; readers hold either —
    [Section 3.4](03-concurrency.md#cumulative-performance-budget--lock-ordering) rows 25/40).
    For each thread in `process.thread_group.tasks`:
      - **Exit filter**: if `thread.flags & PF_EXITING != 0`, SKIP the
        thread — an exiting thread's own `detach_exiting_task()` (serialized
        behind this rwsem) settles its charges against its current
        cgroup, so the migration neither moves nor re-charges it.
        Consistency of
        both interleavings is guaranteed by the rwsem: a skipped thread
        that has NOT yet reached `detach_exiting_task()` keeps `task.cgroup` =
        its current cgroup (we never swap it) and its later
        `detach_exiting_task()` — blocked on the rwsem until step 14 — releases
        its charges from that cgroup; a skipped thread that already
        completed `detach_exiting_task()` holds no charges and is in no task
        set (`task.cgroup` already points at the root cgroup).
      - Otherwise push `(Arc::clone(thread), source_i =
        thread.cgroup.load_full())` into the snapshot.
    Release `PROCESS_LOCK(25)`. Then compute `lca_i =
    cgroup_lca(Arc::clone(&source_i), Arc::clone(&target))` per entry.
    Snapshot storage: `Vec<MigrationEntry>` where `MigrationEntry =
    (Arc<Task>, Arc<Cgroup> /* source_i */, Arc<Cgroup> /* lca_i */)` —
    warm path, bounded by the thread-group size (documented bound per
    the collection policy; the strong `Arc<Task>` clones pin every
    entry's Task struct for the remainder of the protocol).

    **Snapshot stability (why steps 3-16 may iterate it instead of the
    live list)**: new threads cannot appear (fork blocked by the rwsem
    writer); no snapshotted thread can run `detach_exiting_task()` before step
    14 (rwsem read blocks), and `reap_task()` — which unlinks from
    `thread_group.tasks` under 25+40 — is only reachable AFTER
    `detach_exiting_task()`, so no snapshot entry can be unlinked or freed
    during Phase 1. After step 14 releases the rwsem, a snapshot thread
    MAY exit before step 15 reaches it: the held `Arc<Task>` keeps the
    struct alive, `lock_task_rq()` revalidates the CPU, and a dead
    thread is observed as `OnRqState::Off` (its final `schedule()`
    dequeued it with `DEQUEUE_SPECIAL`) — step 15's `Off` arm is a
    no-op, and step 16's state reset on a dead task's struct is
    harmless. In the common case (no threaded subtree), every
    `source_i` is the same cgroup and every `lca_i` is identical — the
    per-thread resolution collapses to one O(depth) walk's worth of
    distinct values and the protocol below degenerates to the
    single-source form.

2. Acquire `root.hierarchy_lock` (RwLock, read mode). This serializes with
   concurrent `cgroup_mkdir()`/`cgroup_rmdir()` (which hold write mode) but
   allows concurrent task migrations (both hold read mode). The lock ordering
   table ([Section 3.4](03-concurrency.md#cumulative-performance-budget)) documents this at level 210.
   Released at step 13 (before step 15's RQ_LOCK acquisition). **Lock
   ordering**: `HIERARCHY_LOCK` (level 210) is ABOVE `RQ_LOCK` (level 50) in
   the hierarchy, so they must NEVER be held simultaneously. The migration
   protocol enforces this via a two-phase structure: Phase 1 (steps 2-13)
   holds `HIERARCHY_LOCK` for cgroup tree operations; Phase 2 (steps 15-16)
   acquires `RQ_LOCK` for runqueue dequeue/enqueue. The lock is released
   between phases (step 13), and the sleeping rwsem strictly after it
   (step 14) so that the rwsem release's waiter wake-up chain
   (`waiters_lock` → `scheduler::unblock` → `RQ_LOCK(50)`) never runs
   under a held level-210 lock.

3. For EACH snapshot entry (step 1c — never the live thread list),
   mark the task as migrating via CAS (two-phase task list protocol):
   `cgroup_migration_state.compare_exchange(0, 1, Acquire, Relaxed)`.
   If any thread's CAS fails (already migrating — e.g., a
   `cgroup.threads` move still in its Phase 2), reset the threads whose
   CAS already succeeded (`store(0, Release)` on each), then release the
   acquired locks in this MANDATORY order: `hierarchy_lock` (level 210)
   FIRST, `threadgroup_rwsem` second (the rwsem release wakes blocked
   sleepers via `RQ_LOCK(50)` — it must run with no spin-class lock
   held), and return `EAGAIN` (`migrations_in_flight` was not yet
   incremented — the increment below happens only after ALL CASes succeed). This is the CAS-based trylock documented
   in the `CgroupMigrationState` enum — `CAS(None=0, Migrating=1)` to
   acquire, `store(None=0, Release)` to release. The CAS atomically marks
   the task as in-transit; no separate "mark migrating" step is needed
   beyond this CAS. The per-cgroup `tasks` RwLock is NOT held here; it is
   acquired later in the task list move step. Each task remains in
   its `source_i.tasks` but is marked as in-transit. Readers of `cgroup.procs`
   include MIGRATING tasks in their source cgroup for consistency,
   ensuring the task is always visible in exactly one cgroup.
   After ALL threads' CASes succeed, increment
   `root.migrations_in_flight.fetch_add(1, Acquire)` (once per migration
   operation) — before step 5's `task.cgroup` swap first publishes the
   membership change. The in-flight gauge is paired: every exit path from
   here on — success (step 16) or rollback (steps 4, 7, 9 failures) — must
   pair it with `fetch_sub(1, Release)`, and every rollback release path uses
   the same 210-before-rwsem release order stated above.

4. Check the target cgroup for constraints. On any failure,
   reset each snapshot thread's `cgroup_migration_state` to
   `CgroupMigrationState::None` (Release), decrement
   `root.migrations_in_flight`, release the locks (`hierarchy_lock`
   first, then `threadgroup_rwsem` — the step-3 release-order rule), and
   return the error (nothing else has been mutated yet — no further
   rollback needed):
   a. **Lifecycle check**: if
      `target.lifecycle.load(Acquire) != CgroupLifecycle::Active as u8`,
      return `ENODEV` (the `as u8` cast converts the `#[repr(u8)]`
      discriminant for comparison against the raw `AtomicU8` value).
      This rejects migration into a `Draining` or `Dead` cgroup.
      REQUIRED for the zero-residual drain invariant: `rmdir` may have
      transitioned the target to `Draining` (under `hierarchy_lock` write)
      *before* this migration acquired its read lock — the write/read
      acquisitions are serial, so the RwLock alone does not exclude this
      interleaving. Without this check, a migration could land tasks in a
      cgroup whose drain phase already assumed `population == 0`. The
      `Acquire` pairs with the `Release` store of the `Active → Draining`
      transition: observing `Active` guarantees the target's controllers
      are still in their valid pre-drain state.
   b. **No-internal-processes rule (cgroup v2)**: If the target cgroup has
      children with controllers enabled (`target.subtree_control != 0`),
      processes cannot be placed directly in it. Return `EBUSY`. This
      prevents the "internal node with both tasks and child cgroups"
      state that breaks resource distribution guarantees.
   c. `pids.max` is deliberately NOT checked here. Migration into a
      cgroup never fails on the pids limit — the limit gates `fork()`
      only. This is ABI-mandated cgroup v2 behavior: Linux
      Linux `kernel/cgroup/pids.c pids_can_attach()` charges unconditionally
      via Linux `pids_charge()` ("does not follow the pid limit set. It
      cannot fail"), and the cgroup-v2 admin guide states a `pids.max`
      limit "does not prevent tasks from being migrated into the
      cgroup". The unconditional charge happens at step 8; a
      post-migration `pids.current > pids.max` overshoot only blocks
      NEW forks in the target subtree.
   d. If target has a `MemCgroup`: verify the task's current RSS would not
      immediately exceed memory.max in the target. If over-limit, return ENOMEM.
   e. **Cpuset admission**: if the target (or nearest ancestor with a
      configured cpuset) has a `CpusetController`, compute
      `target_effective = target.cpuset.allowed_cpus ∩ online_cpus`. If
      `target_effective` is empty, return `ENOSPC`. Matches Linux
      Linux `kernel/cgroup/cpuset.c cpuset_can_attach()`, which returns `-ENOSPC`
      when Linux `cs->effective_cpus` is empty. The affinity *application* (and
      any CPU move) happens in step 15 under RQ_LOCK — this step only
      guarantees the migration cannot fail after mutation begins.

5. For EACH snapshot entry, update the task's cgroup pointer:
     task.cgroup.swap(Arc::clone(&target));
   The `ArcSwap::swap()` atomically replaces the cgroup reference.
   `task.cgroup` is of type `ArcSwap<Cgroup>` (not bare `Arc`), enabling
   atomic replacement with concurrent readers via RCU-like semantics.
   The swap pairs with `ArcSwap::load()` in the resource-charge path,
   ensuring that subsequent charges from this task are credited to the target.

   **Scheduler tree membership is NOT affected by this swap.** The
   scheduler resolves the tree a task is physically linked into
   (`GroupEntity.child_rq`, `CbsCpuServer.tree`,
   `CbsCpuServer.throttled_tasks`) via
   `EevdfTask.cgroup_id` — a cached key refreshed only inside
   `enqueue_task()` under `rq.lock` — never via `task.cgroup`
   ([Section 7.1](07-scheduling.md#scheduler--eevdf-algorithm-specification), `EevdfTask.cgroup_id`
   invariant). Between this swap and step 15's dequeue/enqueue,
   `cgroup_id` still names the SOURCE cgroup, so step 15's dequeue
   removes the task from the tree it is actually in. Resolving the
   dequeue through the already-swapped `task.cgroup` would target the
   TARGET cgroup's tree — where the task is not linked — corrupting both
   trees (use-after-swap). This is also why the swap must NOT be deferred
   to step 15: `process.threadgroup_rwsem` (sleeping rwsem, write mode) is
   released at step 14 — before the per-thread RQ_LOCK phase — and a
   `fork()` in that window must observe the TARGET pointer so the child is
   created in the target cgroup (all-threads-in-target atomicity).

6. **Domain-controller source (steps 6-7 only)**: memory and io are
   DOMAIN controllers — they charge at the process's domain cgroup, and
   threaded child cgroups never carry them. All threads of one process
   therefore share exactly ONE memory-charge source regardless of how
   `cgroup.threads` distributed them: `domain_source` = the threaded-
   domain root of any `source_i` when the sources are threaded cgroups,
   else the (single) `source_i` itself. Steps 6-7 use `domain_source`
   with `lca_dom = cgroup_lca(domain_source, target)`; the per-thread
   `(source_i, lca_i)` pairs apply only to the threaded-controller and
   membership steps (8-11).

   Drain `MemCgroupStock` on the current CPU if it is cached for
   `domain_source`. Take an `IrqDisabledGuard` (`CpuLocal::irq_save()`) over the
   whole read-and-clear window — the SAME discipline the charge/consume path uses
   ([Section 17.2](#control-groups--per-cpu-memory-charge-batching-memcgroupstock)): the stock
   drains are IPI-driven (`smp_call_function_all`), and a bare `PreemptGuard`
   does NOT mask an incoming IPI, so without IRQ-disable a concurrent
   `memory.high`/`rmdir` drain IPI could fire on THIS CPU between reading
   `cached_charge` here and the `fetch_sub` + clear below, double-subtracting the
   same cached pre-charge (the IPI drains it, then this process-context path
   drains it again — one pre-charge returned to the global `usage` counter
   twice). Under the guard: return the unconsumed pre-charge via
   `domain_source.memory.usage.fetch_sub(cached_charge)` and clear the stock
   (refill pre-charges via `fetch_add`, so draining subtracts — see the
   `MemCgroupStock` drain-direction rule). IRQ-disable additionally protects the
   non-atomic `cached_charge: u64` field against a mid-store drain IPI on 32-bit
   arches (ARMv7, PPC32). This brings `usage` down to true RSS before the bulk
   charge transfer in step 7. Must happen AFTER step 5 (cgroup pointer swap) so
   that subsequent per-page charges land in the target, and BEFORE step 7 so that
   the usage counter is fully drained before the bulk transfer reads it.

7. Transfer memory charges from `domain_source` to target via
   `migrate_task_charge(leader, domain_source, target, lca_dom)` (RSS is
   process-wide — one transfer for the whole group, not per thread):
   /// Transfer memory charges from source to target cgroup during
   /// task migration. Charges are moved as bulk counter adjustments
   /// (not per-page transfers) for efficiency.
   ///
   /// # Arguments
   /// - `task`: The task being migrated (provides RSS via process.mm).
   /// - `source`: Source cgroup (charges subtracted).
   /// - `target`: Target cgroup (charges added).
   /// - `lca`: Lowest Common Ancestor (charges above LCA are unchanged).
   ///
   /// # Error handling
   /// Returns `Err(ENOMEM)` if adding the charges to the target OR to ANY
   /// ancestor on the target→LCA path would exceed that node's `memory.max`.
   /// The admission check is done in full BEFORE any counter is mutated, so on
   /// `Err` NOTHING has been transferred — there is no partial-transfer state
   /// to roll back inside this function. (Prefix-undo of the memory transfer,
   /// if a LATER migration step fails, lives in the caller's "OOM during
   /// migration" block, which reverses exactly the walks below.) Also transfers
   /// the process's SWAP charge (`swap_usage`) along the same paths — swap is
   /// hierarchical like RSS and must not be stranded on the source.
   fn migrate_task_charge(
       task: &Task,
       source: Arc<Cgroup>,
       target: Arc<Cgroup>,
       lca: Arc<Cgroup>,
   ) -> Result<(), Errno> {
       // RSS is shared among threads via Process.mm — get the process-level
       // RSS, not per-task (threads share address space).
       // ArcSwap::load() returns an ArcSwapGuard that extends the Arc
       // lifetime — the MmStruct cannot be freed while `mm` is live, so
       // there is no mm-lifetime TOCTOU between this load and the walks.
       // Single load to avoid TOCTOU: rss_pages and rss_bytes must be
       // consistent. A concurrent page fault between two separate loads
       // could cause a one-page divergence.
       //
       // Load-to-walk drift: a concurrent page fault or munmap BETWEEN
       // this rss_pages load and the completion of the two walks below
       // changes the process's true RSS while the transfer uses the
       // snapshot, so source.memory.usage may drift from true RSS by up
       // to one page per concurrent operation. The drift is bounded (no
       // runaway — proportional to fault/unmap rate x migration duration,
       // ~ms), benign (the per-cgroup OOM killer reconciles at its next
       // evaluation), and matches Linux's charge-migration model
       // (mm/memcontrol.c per-page charge/uncharge has the same
       // property). Do not "fix" this with an mm-wide freeze — stalling
       // all page faults of the migrating process for the O(depth) walk
       // is strictly worse.
       let mm = task.process.mm.load();
       let rss_pages = mm.rss_pages.load(Relaxed);
       let rss_bytes = rss_pages as u64 * PAGE_SIZE as u64;
       let swap_bytes = mm.swap_pages.load(Relaxed) as u64 * PAGE_SIZE as u64;

       // ── Admission: pre-check the ENTIRE target→LCA add path ──────────────
       // Not just `target` itself: the add walk fetch_adds every ancestor up to
       // the LCA, so an intermediate ancestor with a tighter memory.max would
       // be pushed over its HARD limit if we only checked the leaf. Check every
       // node on the path BEFORE mutating anything, so a rejection leaves all
       // counters untouched. Check-then-charge race is still benign for the
       // final commit (bounded overshoot, per-cgroup OOM reconciles — same as
       // Linux mem_cgroup_charge); the point of the full pre-check is to refuse
       // a migration whose STEADY-STATE result would exceed an ancestor's max.
       // `memory` is `RcuPtr<Arc<MemCgroup>>`; one RCU guard spans all three
       // non-sleeping counter walks. The migration already holds hierarchy_lock +
       // threadgroup_rwsem (source/target pinned), so no controller is freed here;
       // the guard only satisfies the `RcuPtr::read` API. `&Arc<MemCgroup>` derefs
       // through to the counter fields, so no explicit deref is needed.
       let guard = rcu_read_lock();
       let mut cg_arc = Arc::clone(&target);
       while !Arc::ptr_eq(&cg_arc, &lca) {
           if let Some(mem) = cg_arc.memory.read(&guard) {
               if mem.usage.load(Relaxed) + rss_bytes > mem.max.load(Relaxed) {
                   return Err(Errno::ENOMEM); // nothing mutated yet
               }
           }
           cg_arc = cg_arc.parent.as_ref().unwrap().upgrade().unwrap();
       }

       // ── Commit: subtract from source→LCA, add to target→LCA ──────────────
       // Use Arc<Cgroup> (not &Cgroup) to keep the parent alive across loop
       // iterations. Both RSS and swap move together — swap is hierarchical and
       // must leave the source (finding: swap charge stranded on the source).
       let mut cg_arc = Arc::clone(&source);
       while !Arc::ptr_eq(&cg_arc, &lca) {
           if let Some(mem) = cg_arc.memory.read(&guard) {
               mem.usage.fetch_sub(rss_bytes, Relaxed);
               mem.swap_usage.fetch_sub(swap_bytes, Relaxed);
               // Anon vs file-backed distinction: both are transferred
               // as bulk counter adjustments. Per-page type tracking is
               // maintained by the existing charge_type counters.
           }
           cg_arc = cg_arc.parent.as_ref().unwrap().upgrade().unwrap();
       }
       let mut cg_arc = Arc::clone(&target);
       while !Arc::ptr_eq(&cg_arc, &lca) {
           if let Some(mem) = cg_arc.memory.read(&guard) {
               mem.usage.fetch_add(rss_bytes, Relaxed);
               mem.swap_usage.fetch_add(swap_bytes, Relaxed);
           }
           cg_arc = cg_arc.parent.as_ref().unwrap().upgrade().unwrap();
       }

       Ok(())
   }
   This ordering (pointer update before charge transfer) ensures correctness:
   new allocations after step 5 (cgroup pointer swap) are charged to the target,
   and freed memory from allocations made before migration is correctly unaccounted
   from the source (which still holds the charge until this step transfers it).

   **Bulk vs per-page model**: The bulk RSS charge transfer (fetch_sub/fetch_add
   on `mem.usage`) diverges from Linux's per-page `mem_cgroup_charge()`/
   Linux `mem_cgroup_uncharge()` model. This is an intentional UmkaOS simplification
   that avoids iterating every mapped page during migration. The trade-off:
   the `memory.max` check above uses the full RSS as a single quantum, which
   means a migration can temporarily overshoot `memory.max` by up to the
   entire process RSS (not just one page). This is acceptable because:
   (1) the per-cgroup OOM killer handles overshoots, (2) the overshoot is
   bounded by `process.mm.rss_pages` (not unbounded), and (3) the bulk
   transfer is O(1) vs O(RSS_pages) for the per-page model.

**OOM during migration (steps 5-7)**: If `migrate_task_charge()` (step 7)
returns `ENOMEM`, it did so at its pre-mutation admission check — NOTHING was
transferred, so there is no step-7 state to undo; the migration unwinds steps
6→5→3 only (below). If instead a LATER step (8/9) fails AFTER a successful
`migrate_task_charge`, the kernel reverses the FULL transfer in step order:
- **Step 7 undo** (only when step 7 SUCCEEDED and a later step failed): reverse
  the complete RSS AND swap transfer — add `usage`/`swap_usage` back along the
  `domain_source`→`lca_dom` walk, subtract along the target→`lca_dom` walk (the
  whole transfer, not a prefix — the transfer is all-or-nothing).
- **Step 6 — no undo**: the `MemCgroupStock` drain only returned per-CPU
  unconsumed pre-charge FROM `domain_source.memory.usage` (a `fetch_sub`), which
  is correct regardless of migration outcome. Deliberately NOT reversed.
- **Step 5 undo**: restore each snapshot thread's cgroup pointer via
  `task.cgroup.swap(Arc::clone(&source_i))` (each thread back to ITS
  captured source).
- **Step 3 undo**: reset each thread's `cgroup_migration_state` to
  `CgroupMigrationState::None` (Release) and decrement
  `root.migrations_in_flight` (`fetch_sub(1, Release)`).
Lock release then follows the step-3 order: `hierarchy_lock` first,
`threadgroup_rwsem` second.
Step 8 (PID counts) has NOT yet executed at step-7 failure time and is NOT
rolled back. The `cgroup.procs` write returns `-ENOMEM`. Every task remains
in its source cgroup with all its original charges intact. No partial
migration is ever visible to userspace.

8. Charge PID controllers, per snapshot entry (pids is a THREADED
   controller — each thread is charged at its own cgroup, so the walk
   must use the per-thread pair captured at step 1c):
   - Decrement `PidsController::current` on `source_i` and all ancestors
     up to `lca_i`.
   - Increment `PidsController::current` on target and all ancestors up
     to `lca_i`.
   Unconditional — never fails (see step 4c: the pids limit gates fork,
   not migration). In the common single-source case this collapses to
   one ±`snapshot.len()` adjustment along one source→LCA→target path.

9. CBS weight adjustment for source and target cgroups
   ([Section 7.6](07-scheduling.md#cpu-bandwidth-guarantees--cpumax-ceiling-enforcement-bandwidth-throttling)):
   Because step 1 specifies that writing a PID to `cgroup.procs` moves ALL
   threads in the thread group, the weight adjustment MUST sum across all
   threads, not just the leader — and, like step 8, per-thread against
   each thread's own `source_i` (cpu is a threaded controller). The
   scheduling weight is read through the sanctioned accessor
   `Task::sched_weight()` ([Section 7.1](07-scheduling.md#scheduler--eevdf-algorithm-specification)
   — `EevdfTask.weight` is private to the scheduler):
   // Iterate the step-1c snapshot, NOT the live thread list. `cpu` is
   // `RcuPtr<CpuController>` and `cbs` is `RcuPtr<CbsGroupConfig>`, so both reads
   // need a LIVE RCU read section: `hierarchy_lock` (held) does NOT serialize
   // against the `cbs`/controller writer (its `WriterProof` is the `config_lock`
   // guard), so a bare read would race a concurrent install/teardown. The section
   // holds only atomic `fetch_add`/`fetch_sub` — no sleep — so one `rcu_read_lock()`
   // spanning the loop is legal under the held `hierarchy_lock` + `threadgroup_rwsem`.
   let guard = rcu_read_lock();
   for (thread, source_i, _lca_i) in snapshot.iter() {
       let w = thread.sched_weight();
       if let Some(cbs) = source_i.cpu.read(&guard).and_then(|c| c.cbs.read(&guard)) {
           cbs.total_weight.fetch_sub(w as u64, Ordering::Relaxed);
       }
       if let Some(cbs) = target.cpu.read(&guard).and_then(|c| c.cbs.read(&guard)) {
           cbs.total_weight.fetch_add(w as u64, Ordering::Relaxed);
       }
   }
   - If a source cgroup has a CBS-guaranteed `CpuController`, its
     `CbsGroupConfig.total_weight` shrinks by that thread's weight. The
     cgroup's total `cpu.guarantee` does NOT change — it is a cgroup
     property, not divisible per-task. Only `total_weight` (used for
     proportional budget distribution at replenishment) is adjusted.
   - If the target cgroup has a CBS-guaranteed `CpuController`, its
     `total_weight` grows by the same per-thread weights.
   - If the destination cgroup's total CBS guarantee commitments would
     exceed the system's `cpu.guarantee` admission limit, return ENOSPC
     after rolling back in reverse step order (each undo names the step
     whose effect it reverses — the labels below match the pseudocode
     step numbers exactly):
     1. **Step 9 undo**: restore `CbsGroupConfig.total_weight` per
        snapshot entry — `fetch_add` the thread's `sched_weight()` back
        on `source_i`, `fetch_sub` on target — for exactly the prefix of
        entries already applied when the admission check failed.
     2. **Step 8 undo**: reverse the PID counter updates per snapshot
        entry — decrement `PidsController::current` on target and
        ancestors up to `lca_i`, increment on `source_i` and ancestors
        up to `lca_i`.
     3. **Step 7 undo**: reverse the memory charge transfer via the
        opposite `fetch_add`/`fetch_sub` walks (target→`lca_dom`
        subtract, `domain_source`→`lca_dom` add).
     4. **Step 6 — no undo**: the `MemCgroupStock` drain returned per-CPU
        unconsumed pre-charge FROM `domain_source.memory.usage` (a `fetch_sub`),
        which the step-7 undo above accounts for as part of the bulk reversal.
        Draining the stock is correct regardless of migration outcome —
        deliberately NOT reversed.
     5. **Step 5 undo**: restore the cgroup pointer per thread via
        `task.cgroup.swap(Arc::clone(&source_i))`.
     6. **Step 3 undo**: reset each thread's `cgroup_migration_state` to
        `CgroupMigrationState::None` (Release) and decrement
        `root.migrations_in_flight` (`fetch_sub(1, Release)`).
     Lock release then follows the step-3 order: `hierarchy_lock` first,
     `threadgroup_rwsem` second.
     Steps 10-16 have not executed at this failure point; nothing else to
     undo. After rollback every task is fully in its source cgroup —
     re-migration is immediately possible (no permanently stuck
     `Migrating` state).
   - If neither any source nor the target has a CpuController, this step
     is a no-op.

10. Complete the task list move (phase 2 of two-phase protocol), per
    snapshot entry:
    - Acquire `source_i.tasks` and `target.tasks` write locks
      (`CGROUP_TASKS_LOCK`, level 215 — see the master table row and the
      `Cgroup.tasks` field doc) **in cgroup-ID order** (lower CgroupId
      first) to prevent ABBA deadlock when two tasks migrate in opposite
      directions between the same pair of cgroups concurrently. If
      `source_i` IS the target (a thread already resident in the target
      cgroup), acquire the single lock once and skip the membership
      mutation — the state store below still runs.
    - Remove TaskId from `source_i.tasks`.
    - Insert TaskId into `target.tasks`.
    - Set task.cgroup_migration_state = CgroupMigrationState::Complete
      (runqueue re-enqueue pending; transitions to None in step 16).
    - Release both locks (higher CgroupId first).
    Threads with distinct `source_i` values are processed as independent
    per-thread lock-pair sections — partial visibility between threads is
    covered by the migration-state convention (readers include
    `Migrating`/`Complete` tasks in a consistent single cgroup).
    Because the task was marked Migrating in step 3, it was continuously
    visible in `source_i.tasks` throughout the migration. It now becomes
    visible in target.tasks. The step-1c exit filter guarantees every
    entry here is a live, non-exiting thread whose `detach_exiting_task()` —
    should it begin after step 14 — will observe `task.cgroup` = target
    and remove exactly the membership this step inserted (no dead-TaskId
    residue, no double-remove).

11. Update population counts, per snapshot entry, using the same LOCAL count
    + published-bit flip reconcile as fork/exit (the `population` fetch return
    value identifies the direct-membership edge task; only the edge task
    reconciles the `subtree_populated` bit under `flip_lock` and walks):
    - Target: `let prev = target.population.fetch_add(1)`; if `prev == 0`,
      `cgroup_reconcile_self_populated(&target)`.
    - Source: `let prev = source_i.population.fetch_sub(1)`; if `prev == 1`,
      `cgroup_reconcile_self_populated(&source_i)`.
    Propagation stops at the first ancestor whose populated state does not
    flip, so no explicit LCA truncation is needed — the shared ancestors above
    the LCA carry the task in both states and never flip. (The old to-LCA
    ±walk is subsumed: it existed only for the exact-running-total model that
    the contention fix replaced.)

12. Propagate generation counters:
    - Increment each distinct source's `generation` (Relaxed — any
      observer that sees the task's new cgroup will also observe the
      updated generation).
    - Increment `target.generation`.
    Tasks that cached effective limits from either cgroup will detect the
    mismatch on the next resource charge and re-walk to recompute limits.

13. Release `root.hierarchy_lock` (read mode). This completes the
    hierarchy phase. Released BEFORE the rwsem (step 14): the rwsem
    release's waiter wake-up chain acquires `waiters_lock` and then
    `RQ_LOCK(50)` via `scheduler::unblock()`
    ([Section 3.5](03-concurrency.md#locking-strategy) — `RwLock<T>` release semantics), and running that chain while
    still holding a level-210 spin-class lock would be a descending
    acquisition — exactly the violation the ascending-order invariant
    forbids. Nothing between steps 12 and 14 touches the cgroup tree, so
    releasing the tree lock first is semantically free. The step-15
    RQ_LOCK acquisitions below are likewise never nested under
    HIERARCHY_LOCK.

14. Release `process.threadgroup_rwsem` (write mode, acquired at step
    1a) — with NO spin-class lock held (step 13 released the last one),
    so the internal release-then-wake sequence (dequeue waiter under
    `waiters_lock` → drop → `scheduler::unblock` → `RQ_LOCK(50)`) starts
    from an empty lock set. All threads' cgroup membership (task lists,
    counters, pointers) is final at this point, so a `fork()` that
    proceeds now creates its child in the TARGET cgroup via
    `task.cgroup` (swapped at step 5), and a blocked `detach_exiting_task()`
    that proceeds now uncharges the TARGET (where steps 7/8/11 put the
    thread's charges) — both interleavings are consistent. Released
    HERE — before the per-thread RQ phase (steps 15-16) — to bound the
    fork/exit blackout window: holding the group-wide writer across
    O(threads) runqueue dequeue/enqueue work would block every fork()
    and every exit in the process for the whole per-thread phase, and
    the membership state the writer protects is already final.

15. For EACH snapshot entry: acquire RQ_LOCK on the thread's
    current CPU's runqueue via the lockfree `lock_task_rq()` protocol
    ([Section 7.1](07-scheduling.md#scheduler--task-to-runqueue-lookup-protocol-lockfree-cpuid-retry)).

    **Dispatch form and flags (applies to every dequeue/enqueue below)**:
    `dequeue_task` / `enqueue_task` are `SchedClassOps` methods
    ([Section 7.1](07-scheduling.md#scheduler--scheduler-classes)), dispatched by matching the
    entity's `sched_class` on the locked runqueue (enum dispatch — the
    trait is documentation-only). The migration pair uses:
    `dequeue_task(task, DequeueFlags::DEQUEUE_SAVE)` — the task remains
    runnable, removal is immediate (never deferred), and
    `vruntime`/`vlag` are preserved in the task struct for the paired
    re-enqueue; `enqueue_task(task, EnqueueFlags::ENQUEUE_RESTORE)` —
    re-enqueue preserving the saved virtual state (no wakeup placement,
    no fork placement), OR'd with `ENQUEUE_MIGRATED` when the enqueue
    lands on a different CPU (cpuset move). Semantics match Linux's
    Linux `DEQUEUE_SAVE|DEQUEUE_MOVE` / `ENQUEUE_RESTORE|ENQUEUE_MOVE` pair on
    the same Linux path (`kernel/sched/core.c sched_move_task()`).

    **Tree resolution rule (applies to every dequeue below)**: the
    scheduler locates the tree the task is PHYSICALLY linked into via the
    cached `EevdfTask.cgroup_id` (still = SOURCE — refreshed only inside
    `enqueue_task()`), NEVER via `task.cgroup` (already swapped to target
    in step 5). See the `EevdfTask.cgroup_id` tree-membership invariant in
    [Section 7.1](07-scheduling.md#scheduler--eevdf-algorithm-specification) and the note at step 5.
    The enqueue side reads `task.cgroup` (= target), resolves the target's
    GroupEntity / CBS server, and refreshes `cgroup_id` to the target ID.

    **Cpuset application** (if step 4e computed a `target_effective` mask):
    under the thread's RQ_LOCK, set `task.cpu_affinity =
    target_effective` and refresh `task.nr_cpus_allowed` to the mask's
    popcount (the two fields are updated together at every affinity
    write). Linux `cpuset_attach_task()` → `set_cpus_allowed_ptr()`
    semantics, verified against `kernel/cgroup/cpuset.c` master: the
    applied mask is derived from the cpuset's effective CPUs, NOT
    intersected with the thread's user-requested mask. The
    user-requested mask survives in `task.user_cpu_affinity`
    ([Section 8.1](08-process.md#process-and-task-management--task-model)) — written only by
    `sched_setaffinity(2)` — and is re-intersected on a later move to a
    permissive cpuset (matching Linux `user_cpus_ptr` restoration):
    `cpu_affinity = user_cpu_affinity ∩ target_effective` when
    `user_cpu_affinity` is `Some` and the intersection is non-empty,
    else `target_effective` alone.

    If the thread's current CPU ∉ `target_effective`, the thread must
    leave the excluded CPU — a CPU-bound task would otherwise run there
    indefinitely, violating the cpuset contract. HOW it leaves depends
    on whether it is currently executing:
      - **Queued (not curr)**: move it immediately. Select `new_cpu`
        from `target_effective` via `select_task_rq()` (which already
        honors `cpu_affinity`; prefers NUMA-near, idle). The
        dequeue/enqueue pair below spans two runqueues: dequeue under
        the current CPU's RQ_LOCK, enqueue under `new_cpu`'s RQ_LOCK.
        Acquire both locks in CPU-ID order (the work-stealing
        convention — release and reacquire if the first-locked CPU has
        the higher ID). **Revalidation**: after any release-and-
        reacquire reordering, re-read the thread's CPU; if it no longer
        matches the first-locked runqueue (the thread migrated while
        the locks were dropped), release both locks and retry the
        `lock_task_rq()` protocol from the top — the same
        re-read-after-acquire rule the single-lock protocol applies.
      - **Currently RUNNING on the excluded CPU (`task == rq.curr`)**:
        the move is DEFERRED to the context-switch boundary — the one
        point where the register state is provably saved. Publishing
        the task on `new_cpu`'s tree while its registers are live on
        the old CPU would let `new_cpu` pick and run a context that was
        never saved (same task executing on two CPUs). Instead: run
        the curr branch below (a-d) on the CURRENT runqueue — this
        transfers the cgroup accounting source→target correctly while
        the task remains curr for a bounded few microseconds more —
        then send `resched_curr(rq, ReschedUrgency::Eager)` + reschedule
        IPI. At the excluded CPU's next `schedule()`, `put_prev_task()`
        observes a still-RUNNING prev whose `cpu_affinity` excludes
        this CPU and does NOT re-insert it into this runqueue
        (deferred-placement rule, [Section 7.1](07-scheduling.md#scheduler--schedule-the-dispatch-loop));
        `finish_task_switch()` — running on the next task, after the
        register state is saved — completes the placement via
        `select_task_rq()` + `activate_task(…, ENQUEUE_RESTORE |
        ENQUEUE_MIGRATED)` on an allowed CPU. The bounded window in
        which the task still executes on the excluded CPU (mask write →
        IPI-driven switch) matches Linux's window between
        Linux `set_cpus_allowed_ptr()` and its `migration_cpu_stop` run.

    Then, per-thread, by `on_rq` state (exhaustive over `OnRqState`):

    **If task == rq.curr** (currently running):
    The migration MUST use the full `put_prev_task` / `set_next_task`
    protocol via `SchedClassOps` trait methods to ensure PELT, WaiterCount,
    and CBS accumulators are updated correctly. Direct tree manipulation
    would bypass accumulator flushing and corrupt scheduling state.

      a. `put_prev_task(rq, curr)` — flush vruntime delta, update PELT load
         averages, flush CBS charge accumulators (all against the SOURCE
         structures, resolved via the cached `cgroup_id`).
      b. `dequeue_task(task, DequeueFlags::DEQUEUE_SAVE)` (class-enum
         dispatch on `rq`) — remove from the SOURCE cgroup's tree
         (via cached `cgroup_id`), adjust source GroupEntity weight, update
         source WaiterCount.
      c. `enqueue_task(task, EnqueueFlags::ENQUEUE_RESTORE)` — reads
         `task.cgroup` (= target, swapped
         in step 5), finds/creates the target GroupEntity, inserts the
         task, updates target WaiterCount, and refreshes
         `task.cgroup_id = target.id` (the invariant maintenance point).
      d. `set_next_task(rq, task)` — re-set as curr with new cgroup context,
         reset accounting accumulators for the new cgroup. This runs in
         the excluded-CPU case too (the task genuinely remains curr
         until the IPI-driven `schedule()` — the deferred-placement rule
         above governs where it goes NEXT, not whether it is curr now).

    **If task != rq.curr and on_rq == OnRqState::Queued** (runnable,
    waiting):

      a. `dequeue_task(task, DequeueFlags::DEQUEUE_SAVE)` — removes from
         the SOURCE tree via the cached `cgroup_id`.
      b. `enqueue_task(task, EnqueueFlags::ENQUEUE_RESTORE)` — inserts
         into the TARGET tree via `task.cgroup`, refreshes `cgroup_id`
         (OR'd with `ENQUEUE_MIGRATED` on `new_cpu`'s runqueue per the
         cpuset rule above).

    **If on_rq == OnRqState::CbsThrottled** (dequeued, waiting for source
    server replenishment): see **CBS throttle state on migration** below.

    **If on_rq == OnRqState::Deferred** (sleeping but physically in the
    source tree with negative vlag): finalize the deferred removal —
    dequeue from the source tree preserving `vruntime`/`lag` in the task
    struct, transition to `OnRqState::Off`. Do NOT enqueue into the target
    (the task is sleeping); its next wakeup enqueues it into the target
    tree via `place_entity()`, which applies the preserved lag there.

    **If on_rq == OnRqState::Off** (sleeping, not in any tree): nothing to
    dequeue or enqueue. The stale `cgroup_id` is harmless — it is refreshed
    from `task.cgroup` (= target) by `enqueue_task()` at the next wakeup.

    **CBS server migration** (same-CPU, cross-cgroup) is handled within
    the dequeue/enqueue SchedClassOps paths:
      - If source cgroup has CBS: dequeue_task removes task from source
        `CbsCpuServer.tree`, updates
        `source_server.local_weight -= task.sched_weight()`.
        If this was the last task, the server becomes a steal donor.
      - If target cgroup has CBS: enqueue_task finds or creates `CbsCpuServer`
        for target on this CPU. Sets
        `target_server.local_weight += task.sched_weight()`.
        Enqueues task in `target_server.tree`.
      - Cross CBS/non-CBS: migration between CBS server tree and main EEVDF
        tree is handled by the dequeue (from source) + enqueue (to target)
        pair. See [Section 7.6](07-scheduling.md#cpu-bandwidth-guarantees--cpumax-ceiling-enforcement-bandwidth-throttling).

    **CBS throttle state on migration**: `OnRqState::CbsThrottled` is a
    property of the CBS SERVER the task is bound to, not of the task
    itself — it does NOT transfer across a cgroup change. The target
    cgroup's server has its own budget and its own throttle state; the
    task's runnability under the target is re-evaluated at enqueue:

      - Dequeue side (task was `CbsThrottled` under source): remove the
        task from `source_server.throttled_tasks`, under the SOURCE CPU's
        runqueue lock (it is NOT in any tree — `CbsThrottled` tasks are
        fully dequeued). The task's `vruntime`/`vlag` remain preserved in
        the task struct.
      - Enqueue side, per the TARGET's state:
        * target has no CBS server / server not throttled → enqueue into
          the target tree, `on_rq = OnRqState::Queued`. The task runs
          against the target's budget; the target's own `cbs_charge()`
          enforcement takes over from the first tick.
        * target's server currently throttled
          (`target_server.throttled == true`) → append to
          `target_server.throttled_tasks`, under the TARGET CPU's runqueue
          lock, `on_rq = OnRqState::CbsThrottled`. The task unblocks at the
          TARGET's next replenishment (`cbs_replenish()` →
          `cbs_unthrottle_tasks()`). For a cpuset-forced CPU move both
          runqueue locks are held, acquired in CPU-ID order per the
          existing same-level rule.
      - No transient `OnRqState` variant is needed: the dequeue and the
        enqueue re-evaluation occur within one RQ_LOCK critical section
        (or the two-lock section for a cpuset-forced CPU move), so no
        scheduler path can observe an intermediate state. Between the two
        operations the task is simply not linked anywhere — identical to
        the instant inside any dequeue/enqueue pair.
      - Symmetric rule for the reverse direction: a `Queued` task moving
        into a throttled target server is dequeued from the source tree
        and appended to `target_server.throttled_tasks` as `CbsThrottled` —
        the second enqueue bullet above already covers this; the task
        does not get a free pass around the target's exhausted budget.
      - Rationale for non-transfer: carrying source throttle state into
        an unthrottled target would strand the task (nothing at the
        target ever unthrottles it — the source's replenishment no longer
        knows the task, and the target's replenishment only walks its own
        `throttled_tasks`: the task would be skipped forever). Clearing it
        unconditionally would let a task burst past an exhausted target
        budget. Re-evaluation against the target server is the only
        semantics with no stranding and no budget bypass. Cross-reference:
        the same server-owns-throttle-state principle governs same-cgroup
        CPU→CPU migration in
        [Section 7.6](07-scheduling.md#cpu-bandwidth-guarantees--cpumax-ceiling-enforcement-bandwidth-throttling)
        (zero-budget edge case: task enqueues `CbsThrottled` under the
        NEW CPU's server).

    Release RQ_LOCK (both locks, reverse CPU-ID order, for a cpuset move).

    If the task is currently running on a tickless core
    ([Section 7.1](07-scheduling.md#scheduler--architecture)), send a reschedule IPI to that CPU.
    This forces the scheduler to re-evaluate the task with updated
    GroupEntity parameters from the target cgroup. Without this IPI,
    the task continues running with stale cpu.weight indefinitely
    (tickless cores have no periodic tick to trigger rescheduling).

16. Set `task.cgroup_migration_state = CgroupMigrationState::None` (Release)
    for each snapshot entry (a store on a thread that exited after step 14
    is harmless — the entry's `Arc<Task>` keeps the struct alive), then
    decrement
    `root.migrations_in_flight.fetch_sub(1, Release)` (once per migration
    operation — pairs with step 3's increment).
    This completes the migration and re-enables bandwidth enforcement.
    The `Complete -> None` transition is performed here, not deferred,
    because bandwidth enforcement should resume immediately after the
    runqueue reassignment in step 15.

The LCA (Lowest Common Ancestor) walks in steps 7–8 and 11 are bounded by the
maximum cgroup nesting depth (`CGROUP_MAX_DEPTH` in UmkaOS). In the
common case (migration within the same subtree, depth ≤ 4, all threads
sharing one source cgroup), the walk touches ≤ 8 nodes.

Population propagation (step 11) does not hold `hierarchy_lock`: the LCA walk
needs only per-cgroup serialization, provided by each visited cgroup's cold
`flip_lock` (taken only on a 0↔1 edge — the common same-subtree migration into
an already-populated target takes NO flip lock at all, claiming no edge). It
does not need `hierarchy_lock` because cgroup destruction requires the
population to be zero (enforced before rmdir proceeds). The steady-state
counter update is a single `AtomicU64` `fetch_add`/`fetch_sub`; the flip_lock is
entered only when the fetch return value marks an edge.

17.2.1.9 Single-Thread Migration (cgroup.threads write)¶

Writing a TID to cgroup.threads moves ONLY the specified thread — the per-thread counterpart of the cgroup.procs thread-group protocol above. It follows the SAME step sequence with the differences enumerated below; everything not listed is identical (including all rollback rules, the migrations_in_flight pairing, and the step-15 tree-resolution / cpuset / CBS-throttle rules).

Preconditions (checked under hierarchy_lock read, before step 3; Linux-compatible errnos per kernel/cgroup/cgroup.c Linux cgroup_migrate_vet_dst() and cgroup-v2 ABI):

Source and target cgroups must belong to the SAME threaded domain: both are cgroup.type = threaded members (or the threaded domain root) of one common domain root. Threads cannot escape their threaded root. Violation → EOPNOTSUPP.
A target that is a plain DOMAIN cgroup with subtree_control != 0 fails the no-internal-process rule → EBUSY (step 4b applies unchanged). Inside threaded subtrees the rule is relaxed: the threaded domain root MAY host threads directly even with controllers delegated to threaded children — only domain controllers are constrained by the no-internal-process rule; threaded controllers (cpu, pids, perf_event) distribute per-thread.
Domain controllers (memory, io) are NOT affected by a single-thread move: they charge at the process level, and the process remains in its domain cgroup. Steps 6-7 (MemCgroupStock drain, bulk RSS transfer) are therefore SKIPPED — memory charges do not move.

Protocol deltas vs cgroup.procs:

Step 1b: same three permission checks, with perm_source = the target THREAD's cgroup (not the leader's).
Step 1a: acquire process.threadgroup_rwsem in read mode, not write. Rationale: group-wide membership atomicity is not at stake — exactly one thread moves, and a concurrent fork() (also read mode) creates the child in the PARENT thread's cgroup via task.cgroup, which single-thread migration of a different thread does not disturb. Read mode still excludes a concurrent cgroup.procs whole-group migration (write mode), which IS a real conflict. Exit of THIS thread (also read mode) is serialized not by the rwsem but by the cgroup_migration_state CAS gate — see detach_exiting_task() step 0a and delta 3 below. Better than Linux: Linux takes the global Linux cgroup_threadgroup_rwsem even for single-thread moves; UmkaOS takes the per-Process rwsem in shared mode.
Steps 1c + 3 collapse: the "snapshot" is the single target thread. If the thread has PF_EXITING set, OR the step-3 CAS fails against the terminal Migrating claim installed by detach_exiting_task() step 0a, return SUCCESS as a no-op — matching this Linux behavior: Linux cgroup_migrate_add_task() silently skips exiting tasks.
Steps 6-7: skipped (see above — domain charges stay with the process).
Step 8: PID counters ±1 along the single thread's (source, lca) pair, not ± thread_group.count.
Step 9: CBS weight delta is the single thread's sched_weight(), not the sum across the group. The step-9 ENOSPC rollback shrinks accordingly (undo steps 9, 8, 5, 3 — steps 6-7 never ran).
Steps 10-12: move/account exactly one TaskId; population ±1.
Steps 15-16: run for the single thread only.

17.2.1.10 Interaction with Live Evolution¶

Cgroup task migration and the live-evolution engine (Section 13.18) interact in exactly two ways, both bounded:

1. Evolving cgroup-subsystem components (or layouts they touch). Any evolution batch that swaps cgroup-subsystem code (the migration protocol itself, migrate_task_charge, controller implementations) or changes a migration-tracked layout referenced mid-protocol (Cgroup, the cgroup fields of Task) MUST quiesce migrations in Phase A' — otherwise a migration frozen mid-protocol would resume old code on its call stack and then call into new-ABI functions (undefined behavior). The Phase A' quiescence recipe for the cgroup component:

Gate new entries: cgroup.procs/cgroup.threads/mkdir/rmdir writers block on the component's quiescence gate (they are sleeping process-context paths — the wait is invisible to userspace; this is the standard Phase A' "new operations are queued" behavior, not EAGAIN flakiness).
Acquire root.hierarchy_lock in write mode. This drains all Phase-1 migration sections (steps 2-13, which hold the read lock) and all mkdir/rmdir writers. Lock-ordering legality: the orchestrator holds the sleeping EVOLUTION_MUTEX and acquires a spinlock-class RwLock (level 210) inside it — Mutex→SpinLock nesting is permitted; only the reverse (acquiring a sleeping Mutex with a SpinLock held) is forbidden (Section 3.4). The "EVOLUTION_MUTEX must be outermost" rule constrains OTHER code paths from holding locks when acquiring EVOLUTION_MUTEX; it does not prevent the orchestrator from taking inner locks during quiescence.
Spin-wait root.migrations_in_flight.load(Acquire) == 0. This drains Phase-2 sections (steps 15-16, which run AFTER the read lock is released). Phase 2 never sleeps — it is RQ_LOCK critical sections plus a few atomics — so the wait is bounded to microseconds.
Hold the write lock through Phase B; release it in Phase C. Queued writers then proceed under the new code.

2. Evolving unrelated components. Phase B's stop-the-world NMI halts CPUs at arbitrary instruction boundaries — a migration may be frozen mid-protocol (even holding hierarchy_lock or an RQ_LOCK; the Phase B transfer protocol tolerates held runqueue locks via its try_lock bypass). This is SAFE without quiescence: the frozen migration resumes after release_all_cpus() and continues executing unchanged cgroup code against unchanged cgroup layouts — the swapped component is elsewhere. No cross-generation ABI mismatch is possible because the batch touched neither the code on the frozen call stack nor the types it dereferences.

Invariant: Phase B never observes a mid-flight migration of the evolving cgroup component — Phase A' step 2-3 above drained them. A migration can therefore never be "rolled back by evolution"; it either completed before the swap or starts fresh under the new code. The migrations_in_flight counter and its increment/decrement points are specified in the CgroupRoot struct and migration steps 3/16.

17.2.2 Cgroup Filesystem and Hierarchy¶

Cgroups are exposed via a pseudo-filesystem mounted at /sys/fs/cgroup:

/sys/fs/cgroup/
├── cgroup.controllers      # Available controllers (cpu cpuset io memory pids rdma hugetlb misc perf_event accel) — see ROOT_AVAILABLE_CONTROLLERS
├── cgroup.subtree_control  # Controllers enabled for children
├── cgroup.type             # "domain" (default) or "threaded" (thread-mode subtree)
├── cgroup.procs            # TGIDs in this cgroup (writes move all threads)
├── cgroup.threads          # TIDs in this cgroup (threaded mode only; writes move single thread)
├── system.slice/           # Systemd system services
├── user.slice/             # User sessions
└── docker/                 # Container cgroups
    └── <container-id>/
        ├── cpu.max
        ├── cpu.weight
        ├── cpu.guarantee   # UmkaOS extension ([Section 7.6](07-scheduling.md#cpu-bandwidth-guarantees))
        ├── memory.max
        ├── memory.current
        ├── io.max
        ├── pids.max
        └── cpuset.cpus

cgroup_mkdir() — cgroup creation algorithm:

cgroup_mkdir(parent: &Arc<Cgroup>, name: &str) -> Result<Arc<Cgroup>, CgroupError>:
  1. Validate name: build `CgroupName::new(name)?` — rejects '/' and '\0',
     and length > 255 bytes → EINVAL (the `CgroupName` fixed 255-byte inline
     buffer cannot overflow: `new` returns EINVAL rather than truncating; this
     is NOT `ArrayString::from`, which the struct never used). Also reject
     '..' traversal → EINVAL.
  2. Acquire root.hierarchy_lock (RwLock write) — the SLEEPING tree-wide
     serializer. `hierarchy_lock` is a `RwLock` (sleeping): the allocations in
     steps 4/5/8 run legally under it, and it excludes every concurrent
     mkdir/rmdir/`subtree_control` on this tree, so `parent.children` is frozen
     for the whole operation. It is acquired BEFORE the spin-class
     `children_lock` (per the master lock table's "acquired before per-cgroup
     subsystem locks"); a sleeping lock legally nests a SpinLock under it, and no
     allocation ever runs while `children_lock` is held (see step 6).
  3. Check: parent.lifecycle.load(Acquire) == Active, else release
     hierarchy_lock and return ENODEV (Draining/Dead).
  3a. **Depth enforcement**: `let depth = parent.depth + 1;` if
      `depth > CGROUP_MAX_DEPTH as u32`, release hierarchy_lock and return EAGAIN
      (Linux `cgroup_check_hierarchy_limits()` returns -EAGAIN when the new
      level would exceed `cgroup.max.depth`, `kernel/cgroup/cgroup.c`,
      `torvalds/linux` master; the same errno covers the max-descendants
      limit). This is the enforcement site the `for_each_descendant` and
      `pids_precharge_fork` `ArrayVec<_, CGROUP_MAX_DEPTH>` stacks depend on — an
      over-depth mkdir is refused here, so no walk can overflow.
  4. Allocate Cgroup struct:
     - id = root.next_id.fetch_add(1, Relaxed).  // CgroupId, never reused
     - parent = Some(Weak::clone(&Arc::downgrade(parent))).
     - depth = depth  (from step 3a — parent.depth + 1).
     - name = the CgroupName from step 1.
     - lifecycle = AtomicU8::new(Active); population = 0; nr_populated_children = 0;
       subtree_populated = AtomicU8::new(0) (unpopulated).
     - subtree_control = AtomicU32::new(0); config_lock = Mutex::new(());
     - inode = OnceCell::new(); dyn_subsys = [null; MAX_DYN_CGROUP_SUBSYS].
  5. Initialize subsystem controllers with DEFAULT values (NOT copies
     of parent state) — a controller field is a non-null `RcuPtr::new(state)` iff
     its bit is set in `ControllerMask::from_bits(parent.subtree_control.load(Acquire))`,
     else `RcuPtr::null()`:
     - cpu: cpu.weight = 100, cpu.max = "max 100000",
       cbs = RcuPtr::null(), allowed_core_types = 0 (all), usage/user/system_ns = 0
     - memory: memory.max/high/swap_max = u64::MAX; usage/swap_usage = 0
     - io: io.weight = 100; io.max unset; throttle map empty
     - pids: pids.max = u64::MAX (unlimited)
     - cpuset: allowed_cpus/allowed_mems = empty (inherit); partition = Member
     - rdma/hugetlb/misc: empty limit maps; all fields default to their
       "unlimited" sentinel (u32::MAX / u64::MAX) — see each controller struct
     - perf_event: no per-cgroup limit by default
  6. Insert into hierarchy (prepare/commit — allocation OUTSIDE the SpinLock):
     first build `new_vec` by cloning `parent.children` and pushing the new
     `Arc<Cgroup>` — a sleeping allocation done under the held `hierarchy_lock`
     write, race-free without `children_lock` because that write already excludes
     every other structural editor of `parent.children`. THEN acquire
     `parent.children_lock` (SpinLock) and publish the pre-built vector via
     `RcuCell::update(new_vec, &children_lock_guard)` (old Vec freed after an RCU
     grace period; the `SpinLockGuard` satisfies the sealed `WriterProof` trait),
     then release `children_lock`. The critical section is a single pointer swap
     — no allocation runs under the SpinLock.
  7. Register in the ONE canonical registry:
     CGROUP_REGISTRY.store(id, Arc::clone(&cg)).  // NOT a per-root id_map —
     this is the same XArray the scheduler's cgroup_from_id() reads.
  8. Create cgroupfs directory: allocate the inode, `cg.inode.set(inode)`,
     mkdir in the parent's cgroupfs inode, populate control files
     (cgroup.procs, cgroup.subtree_control, per-controller knobs).
  9. Release root.hierarchy_lock (children_lock was already released in step 6).
  10. Return Ok(cg).

cgroup_rmdir() — cgroup destruction algorithm:

rmdir(2) on a cgroupfs directory removes the cgroup. It is referenced by the registry-unpin discipline, the population == 0 gate, the Active → Draining lifecycle transition that migration step 4a's ENODEV check depends on, and the HIBERNATE_CANDIDATES removal — all of which need one concrete algorithm:

cgroup_rmdir(cgroup: &Arc<Cgroup>) -> Result<(), Errno>:
  1. Acquire root.hierarchy_lock (RwLock write) — the SLEEPING tree-wide
     serializer (same discipline as cgroup_mkdir: hierarchy_lock is the outer
     sleeping lock; the spin-class `children_lock` is taken only for the step-4a
     O(1) publish, with the vector clone done under `hierarchy_lock` beforehand
     so no allocation runs under the SpinLock).
  2. Emptiness gate — the membership half (Linux `cgroup_destroy_locked()`
     returns -EBUSY for BOTH conditions — cgroup rmdir does NOT use ENOTEMPTY;
     verified against `kernel/cgroup/cgroup.c`, `torvalds/linux` master). These
     two are serialized against task migration by the held `hierarchy_lock`, so
     they may be checked before the lifecycle store:
     a. if cgroup.is_populated()  → EBUSY   // tasks in self or any descendant
     b. if !cgroup.children.read(&guard).is_empty()  → EBUSY   // child cgroups
  3. **Lifecycle CAS Active → Draining (SeqCst), BEFORE loading `fork_pins`.**
     If the CAS fails (state was not Active) → EBUSY (a concurrent rmdir already
     claimed it). This store MUST precede step 3a's `fork_pins` load: it pairs
     with `precharge_fork`'s store-before-load (it `fetch_add`s `fork_pins`
     SeqCst, THEN loads `lifecycle` SeqCst) as a Dekker interlock. **All four
     gate ops — this CAS, the step-3a `fork_pins` load, and can_fork's
     `fork_pins` `fetch_add` + `lifecycle` load — are `SeqCst`** (NOT
     Acquire/Release): this is a store-buffering (StoreLoad) shape — each side
     stores one location then loads the other — and under the Rust/C++ memory
     model Acquire/Release order a store only against LATER loads on the SAME
     location, so they permit the both-loads-miss execution (both sides' store
     is reordered after its own subsequent load). x86's locked-RMW full fence
     hides this, but AArch64/POWER can observe it, re-opening the
     fork-into-Dead-cgroup hole. `SeqCst` on all four (equivalently a
     `fence(SeqCst)` between each side's store and load) forces one total order,
     so the two orders CANNOT both miss: either the fork observes Draining (→
     ENODEV, it backs its pin out) or rmdir observes the pin (→ EBUSY at 3a, it
     rolls Draining back). If instead this side loaded
     `fork_pins` before storing Draining (the earlier inverted order), a fork
     could `fetch_add` its pin and read `lifecycle == Active` in the window after
     rmdir's load and before its store — both gates miss and the fork lands a
     task in a cgroup rmdir then drains and frees. From this point migration
     step 4a's `lifecycle != Active` check rejects new tasks (ENODEV), and
     `precharge_fork` rejects CLONE_INTO_CGROUP into it (ENODEV) — the Draining
     state is published under hierarchy_lock write, serial with the migration
     readers' hierarchy_lock/lifecycle loads (the fork path is serialized by the
     Dekker interlock, not the lock).
     3a. **Live CLONE_INTO_CGROUP gate** (AFTER the Draining store): if
         cgroup.fork_pins.load(SeqCst) != 0 → **CAS `lifecycle` Draining →
         Active (Release) to undo step 3, then return EBUSY**. A clone3 with
         CLONE_INTO_CGROUP that resolved this cgroup as its target but has not
         yet completed `commit_fork()` holds a fork-pin (see
         `precharge_fork` below); destroying the target underneath it would
         land a task in a Dead cgroup. Rolling Draining back on this EBUSY leaves
         the cgroup fully Active and immediately re-rmdir-able (no stuck Draining
         state). This closes the fd-across-rmdir hole.
  4. Structural unlink (under hierarchy_lock; children_lock only for the publish):
     a. Remove from parent.children: build `new_vec` by cloning the Vec and
        removing this Arc — a sleeping allocation under the held `hierarchy_lock`
        write (race-free without `children_lock`, which that write already
        excludes). THEN acquire `parent.children_lock` (SpinLock), publish via
        `RcuCell::update(new_vec, &children_lock_guard)`, and release
        `children_lock` — a single allocation-free pointer swap under the SpinLock.
     b. Do NOT erase the registry entry here — leave it as a resolvable
        TOMBSTONE (the entry's `Arc<Cgroup>` is the keep-alive). A residual
        charge that outlives this rmdir (RDMA/perf/hugetlb/in-flight-bio) must
        still resolve its anchored counter via `CGROUP_REGISTRY.get(id)`, which
        the tombstone provides until the last residual uncharge. Final erase (and
        the last-Arc drop that frees the cgroup) happens in drain Phase 6 / the
        last residual uncharge — see
        [Section 17.2](#control-groups--registry-tombstone-and-residual-charge-drain). IDs
        are never reused, so the tombstone always resolves to THIS cgroup (now
        Dead), never a different one. The io-throttle/scheduler resolve paths
        tolerate a Dead resolution (a Dead cgroup has no tasks, so no NEW charge
        is stamped with its id; only already-in-flight residuals resolve it).
     c. Remove this cgroup's `Arc<MemCgroup>` from HIBERNATE_CANDIDATES
        ([Section 4.5](04-memory.md#oom-killer)) — ordered BEFORE controller teardown so OOM Step 0
        never selects a dying cgroup.
  5. **Trigger the residual drain**: flush every per-CPU `MemCgroupStock`
     cached for this cgroup via `smp_call_function_all` (the SAME IPI mechanism
     `memory.high` breach uses, drain trigger 3), RETURNING each CPU's unconsumed
     pre-charge via `memory.usage.fetch_sub(cached_charge)` and clearing the
     stock (refill pre-charges via `fetch_add`, so the flush subtracts — see
     drain trigger 5). Without this, a stock pre-charged on a remote CPU keeps
     `usage` inflated and the drain's Phase 5 (`usage == 0`) is unreachable,
     stalling the full CGROUP_MEM_DRAIN_TIMEOUT_MS. See
     [Section 17.2](#control-groups--per-cpu-memory-charge-batching-memcgroupstock).
  6. Release root.hierarchy_lock (children_lock was already released in step 4a).
  7. Schedule `cgroup_drain_residual(cgroup)` on a kworker (the 7-phase drain
     below), which reparents the enumerable (memory) charges and, at Phase 6,
     stores lifecycle = Dead and then either erases the registry tombstone (if
     `residual_refs == 0`) or leaves the tombstone for the last residual
     uncharge to erase. Return Ok(()).

Hierarchy delegation: A cgroup can delegate control to a subtree by enabling controllers in cgroup.subtree_control. Only controllers enabled in the parent's subtree_control are available in child cgroups. This matches Linux semantics for unprivileged container runtimes.

17.2.2.1 `cgroup.subtree_control` Write Handler¶

Writing cgroup.subtree_control (e.g. "+cpu -memory") enables/disables controllers FOR THIS CGROUP'S CHILDREN. This is the write path the field doc and every child mkdir read depend on:

/// cgroupfs write handler for `cgroup.subtree_control`.
/// Parses `+ctrl`/`-ctrl` tokens and applies them atomically (all-or-nothing).
pub fn write_subtree_control(
    cgroup: &Arc<Cgroup>,
    buf: &[u8],
) -> Result<usize, Errno> {
    // Parse into an enable set and a disable set of controller bits.
    let (enable, disable) = parse_subtree_control_tokens(buf)?; // EINVAL on bad token

    // Serialize config mutation + structural child-controller alloc/free.
    let cfg = cgroup.config_lock.lock();
    let _hier = cgroup_root().hierarchy_lock.write(); // child alloc/free is structural

    let cur = ControllerMask::from_bits(cgroup.subtree_control.load(Ordering::Acquire));

    // ── Validation (reject BEFORE any mutation) ──────────────────────────
    // (1) Availability: every +ctrl must be available to this cgroup, i.e. set
    //     in the PARENT's subtree_control (or, for the root, in cgroup.controllers).
    let available = match cgroup.parent {
        Some(ref p) => ControllerMask::from_bits(
            p.upgrade().expect("parent alive: child pins it")
             .subtree_control.load(Ordering::Acquire)),
        None => ROOT_AVAILABLE_CONTROLLERS, // every built-in + accel
    };
    for bit in iter_bits(enable) {
        if !available.has(bit) { return Err(Errno::ENOENT); }
    }
    // (2) No-internal-process rule (v2): a cgroup with member tasks of its own
    //     may not enable domain controllers for children (that would create an
    //     internal node holding both tasks and child-distributed resources).
    //     Linux `cgroup_subtree_control_write()` returns -EBUSY.
    if enable.bits() != 0 && cgroup.population.load(Ordering::Acquire) != 0 {
        return Err(Errno::EBUSY);
    }
    // (3) Disable safety: a -ctrl is refused if any child still has that
    //     controller enabled in ITS subtree_control (a live grandchild depends
    //     on it). Linux returns -EBUSY.
    {
        let guard = rcu_read_lock();
        for child in cgroup.children.read(&guard).iter() {
            let child_sc = ControllerMask::from_bits(
                child.subtree_control.load(Ordering::Acquire));
            for bit in iter_bits(disable) {
                if child_sc.has(bit) { return Err(Errno::EBUSY); }
            }
        }
    }

    // ── Apply ────────────────────────────────────────────────────────────
    // Enable side: instantiate the newly-enabled controller state (with mkdir
    // defaults) on EXISTING children that lack it; disable side: tear it down.
    //
    // **`pids` counter reconciliation (fork-charge coherence)**: toggling the
    // `pids` controller on an already-populated child interacts with the
    // per-fork pids reservation. Two rules keep `pids.current` balanced against
    // the un-re-walked `precharge_fork`/`detach_exiting_task` decrements:
    //   - ENABLE: `child_controller_alloc(child, PIDS)` must initialize the new
    //     `pids.current` to the child subtree's CURRENT live task count (walk
    //     `child`'s task set + descendants under the held config_lock +
    //     hierarchy_lock), NOT the mkdir default of 0. Otherwise every later
    //     `detach_exiting_task` of a task that predates the enable would `fetch_sub` a
    //     counter that never counted it (underflow). Because `precharge_fork`
    //     RECORDS its charged set in `CgroupForkCharge` and `rollback_fork`
    //     decrements exactly that set (never a fresh walk), an in-flight fork's
    //     can/cancel pair stays balanced regardless of an enable landing between
    //     them — this handler need not exclude in-flight forks.
    //   - DISABLE: `child_controller_free(child, PIDS)` drops the whole
    //     `PidsController` (counter included), so there is nothing left to
    //     under/over-count; the disable-safety check above already guaranteed no
    //     grandchild still has `pids` enabled.
    // See [Section 17.2](#control-groups--cgroup-fork-hooks) `CgroupForkCharge` for the
    // anchored-charge invariant this reconciliation completes.
    // ENABLE side is FALLIBLE (child_controller_alloc heap-allocates state, e.g.
    // `Arc<MemCgroup>`) and MUST be all-or-nothing to honour the handler's
    // "applies them atomically" contract. Pre-apply every (child, enable-bit),
    // recording each success; on the FIRST ENOMEM, free everything recorded and
    // return WITHOUT publishing the mask — no partial enable is ever visible.
    // Children are a stable snapshot here (hierarchy_lock write held), so the
    // recorded set exactly matches what must be rolled back. Cold path (a
    // cgroupfs config write): a `Vec` for the rollback log is acceptable.
    // Two-phase to keep the fallible allocation OUT of the RCU read section:
    // snapshot the child Arcs under a short `rcu_read_lock()`, DROP the guard,
    // THEN run the alloc/rollback loop on the owned snapshot.
    // `child_controller_alloc` heap-allocates (e.g. `Arc<MemCgroup>`) — a
    // sleeping allocation — and UmkaOS RCU readers are tick-reported via a
    // CpuLocal nesting counter (Ch 3), so a reader MUST NOT sleep. The held
    // `hierarchy_lock` write already freezes the child set, so the owned
    // snapshot is exactly what the alloc/rollback loop must act on and roll back.
    let children: Vec<Arc<Cgroup>> = {
        let guard = rcu_read_lock();
        cgroup.children.read(&guard).iter().map(Arc::clone).collect()
    }; // guard dropped here — no RCU read section held across the alloc below
    let mut allocated: Vec<(Arc<Cgroup>, u32)> = Vec::new();
    // `&cfg` (the parent's held `config_lock` guard) is the `WriterProof` the
    // child's controller-`RcuPtr` publishes require; `hierarchy_lock` write (held)
    // additionally serializes the structural alloc/free tree-wide.
    for child in &children {
        for bit in iter_bits(enable) {
            match child_controller_alloc(child, bit, &cfg) {
                Ok(()) => allocated.push((Arc::clone(child), bit)),
                Err(e) => {
                    // Roll back every controller allocated so far, in reverse.
                    for (c, b) in allocated.iter().rev() {
                        child_controller_free(c, *b, &cfg);
                    }
                    return Err(e); // ENOMEM — mask NOT published, nothing changed
                }
            }
        }
    }
    // DISABLE side is infallible (frees only) and runs only after every enable
    // succeeded — so a failed enable never tears down a live controller. Reuse
    // the same owned snapshot (no second RCU section needed).
    for child in &children {
        for bit in iter_bits(disable) { child_controller_free(child, bit, &cfg); }
    }
    // If this write toggled `pids` on any child, bump the pids-topology
    // generation so any in-flight fork (can_fork done, post_fork pending) detects
    // the change and reconciles its charge — see
    // [Section 17.2](#control-groups--pids-enable-time-in-flight-reconciliation).
    if (enable.bits() | disable.bits()) & ControllerMask::PIDS != 0 {
        PIDS_TOPOLOGY_GEN.fetch_add(1, Ordering::Release);
    }
    // Publish the new mask (single atomic store — a concurrent child mkdir
    // observes either the whole old or whole new mask). Reached only after the
    // enable side fully succeeded, so mask and controller state are consistent.
    let next = cur.union(enable).without(disable.bits());
    cgroup.subtree_control.store(next.bits(), Ordering::Release);
    let _ = cfg; // config_lock held until here (writer proof for child alloc RCU)
    Ok(buf.len())
}

The two child-controller helpers the handler and child_controller_free at cgroup_rmdir/drain rely on:

/// Instantiate the controller state named by `bit` on `child` with `mkdir`
/// defaults, if not already present, publishing it into the child's per-controller
/// `RcuPtr<T>` via `update(Some(state), proof)`. FALLIBLE: several controllers
/// heap-allocate (`memory` → `Arc<MemCgroup>`, `io` → `IoController` with its
/// XArrays) and `RcuPtr::update` boxes the state, so this returns
/// `Err(Errno::ENOMEM)` on allocation failure and the caller treats the whole
/// `subtree_control` write as all-or-nothing. Idempotent: a bit already present is
/// a no-op `Ok(())`. Runs under the parent's `hierarchy_lock` write + `config_lock`
/// (structural child mutation), so no concurrent mkdir/rmdir of `child` races it;
/// `proof` is the parent's `config_lock` guard, the `WriterProof` each controller
/// `RcuPtr` publish requires (`hierarchy_lock` write serializes these tree-wide).
/// The allocation runs OUTSIDE any RCU read section (the handler snapshots children
/// under a short guard, drops it, THEN calls this on the owned snapshot).
///
/// **`pids` special case** (fork-charge coherence): when `bit == PIDS`, the new
/// `PidsController.current` is initialized NOT to 0 but to the child subtree's
/// CURRENT live task count (walk `child`'s task set + descendants under the held
/// locks), so a later `detach_exiting_task` of a task that predates the enable does not
/// `fetch_sub` a counter that never counted it, and its `id` is stamped from
/// `PIDS_CONTROLLER_SERIAL.fetch_add(1, Relaxed)` so a fork charge taken on a
/// PRIOR box on this same cgroup (since freed by an interleaving `-pids`) is
/// never mistaken for a charge on this fresh box. See the fork-charge invariant
/// in [Section 17.2](#control-groups--cgroup-fork-hooks) and the in-flight-fork window
/// handled by [Section 17.2](#control-groups--pids-enable-time-in-flight-reconciliation).
fn child_controller_alloc(
    child: &Arc<Cgroup>,
    bit: u32,
    proof: &impl WriterProof,
) -> Result<(), Errno>;

/// Tear down the controller state named by `bit` on `child` by clearing the
/// controller's `RcuPtr` (`update(None, proof)`, counter included); the old state
/// box is reclaimed after an RCU grace period, so a concurrent lock-free reader
/// never observes freed state. INFALLIBLE (`update(None, ..)` allocates nothing).
/// Idempotent: a bit already absent (NULL `RcuPtr`) is a no-op. Same lock context
/// and `proof` as `child_controller_alloc`. The disable-safety EBUSY check in the
/// handler guarantees no grandchild still depends on the controller before this runs.
fn child_controller_free(child: &Arc<Cgroup>, bit: u32, proof: &impl WriterProof);

Cgroup namespace integration: CLONE_NEWCGROUP creates a new cgroup namespace where the process's current cgroup becomes the root of its view. Processes see /sys/fs/cgroup/ starting from their namespace's cgroup root, enabling rootless container runtimes to manage their own cgroup hierarchy.

/proc/[pid]/cgroup path relativity: When a process reads /proc/[pid]/cgroup, the kernel computes the cgroup path relative to the reader's cgroup namespace root, not the global cgroup root:

/// Compute the cgroup path as seen by the reading process.
///
/// The path is relative to the reader's cgroup namespace root.
/// If the target task's cgroup is outside the reader's cgroupns
/// subtree, the path is shown as "/../<path>" (indicating the task
/// is in an ancestor cgroup — visible but not navigable).
///
/// This function backs /proc/[pid]/cgroup and /proc/[pid]/cgroup.controllers.
///
/// **Buffer sizing and overflow** (ABI-visible /proc file): the assembled
/// path is bounded only by the tree depth × per-level `CgroupName` (≤255
/// bytes each), so a deep hierarchy can exceed any small fixed buffer. The
/// output buffer is therefore `PATH_MAX` (4096), matching the
/// Linux `proc_cgroup_show()` path, which `kmalloc(PATH_MAX)`s a buffer and calls
/// Linux `cgroup_path_ns(cgrp, buf, PATH_MAX, ns)`, mapping a too-long path
/// when Linux `kernfs_path_from_node` returns `>= PATH_MAX` (surfaced as `-E2BIG`)
/// to `-ENAMETOOLONG` (`kernel/cgroup/cgroup.c`, `torvalds/linux` master).
/// UmkaOS mirrors this exactly: on overflow the function returns
/// `Err(Errno::ENAMETOOLONG)` and the /proc read fails with that errno —
/// it never silently truncates, because container runtimes parse
/// `/proc/self/cgroup` and a truncated path would misidentify the cgroup.
fn cgroup_path_from_reader(
    target_cgroup: &Cgroup,
    reader_cgroupns_root: &Cgroup,
) -> Result<ArrayString<PATH_MAX>, Errno> {   // PATH_MAX = 4096
    // Walk the parent/name chain — UmkaOS cgroups have NO kernfs `kn` node
    // (there is no `Cgroup.kn` field; cgroupfs uses `Cgroup.inode` + the
    // `parent`/`name` tree). Collect `name` components from target toward the
    // root, stopping at reader_cgroupns_root. The component-reference stack is
    // bounded by the mkdir-enforced depth (≤ CGROUP_MAX_DEPTH names between a
    // node and the root), so it cannot overflow; only the JOINED byte string
    // can exceed the buffer, which `join_names_rev` reports.
    //
    //   // OWNED-Arc cursor + OWNED component Arcs — NOT `&Cgroup` into a
    //   // temporary. `p.upgrade()` yields a temporary Arc dropped at end of
    //   // statement, so `p.upgrade().expect().as_ref()` and `comps.push(&cur.name)`
    //   // would store references into a released borrow — the exact
    //   // dangling-temporary bug pids_precharge_fork documents avoiding. Keep each
    //   // component's Arc alive in `comps` and read its `name` when joining.
    //   let mut comps: ArrayVec<Arc<Cgroup>, CGROUP_MAX_DEPTH> = ArrayVec::new();
    //   let mut cur: Arc<Cgroup> = adopt_arc(target_cgroup);  // Arc::clone of the caller's ref
    //   loop {
    //       if core::ptr::eq(Arc::as_ptr(&cur), reader_cgroupns_root as *const _) {
    //           // Reached the ns root from below → relative path.
    //           return join_names_rev(&comps);            // "/" if comps empty
    //       }
    //       match cur.parent {                            // Weak<Cgroup>
    //           Some(ref p) => {
    //               let parent = p.upgrade()
    //                   .expect("ancestor pinned by populated/pinned descendant");
    //               comps.push(Arc::clone(&cur));         // keep this component alive
    //               cur = parent;                         // owned move to the parent
    //           }
    //           // Hit the global root without meeting reader's root → target is
    //           // in an ANCESTOR of the reader's ns → prefix "/.." (visible,
    //           // not navigable), matching Linux cgroupns behavior.
    //           None => return prepend_dotdot(join_names_rev(&comps)),
    //       }
    //   }
    // `join_names_rev` reads `c.name.as_str()` from each OWNED `Arc<Cgroup>` in
    // `comps` (reverse order) — the Arcs live for the whole join, so no borrow
    // dangles. If target IS reader's root: comps is empty → "/".
    //
    // `join_names_rev` (and `prepend_dotdot`) write into the PATH_MAX
    // `ArrayString`; each is `try_push_str`-based and returns
    // `Err(Errno::ENAMETOOLONG)` the instant the accumulated length would
    // exceed PATH_MAX — the loop's `return` arms propagate that `Result` so
    // every case is either the full path or a clean too-long error.
}

Effect on container processes: - A container process whose cgroup namespace root is /sys/fs/cgroup/system.slice/docker-abc123.scope sees /proc/self/cgroup as 0::/ (it thinks it is at the cgroup root). - A process in a child cgroup within the container (e.g., .../docker-abc123.scope/app) sees 0::/app. - A host process reading the same container process's cgroup sees the full path from the global root: 0::/system.slice/docker-abc123.scope.

This is essential for container compatibility: systemd inside a container expects to see 0::/ and then create child cgroups relative to that root. Without cgroupns path relativity, systemd would see the host's full cgroup hierarchy and fail to manage cgroups correctly.

17.2.3 CPU Controller Integration¶

The cgroup cpu controller maps to the UmkaOS scheduler:

cpu.weight: Hierarchical group proportional share (range 1-10000, default 100). A cgroup's effective share of CPU time is (my_weight / sum_sibling_weights) × parent_effective_share. This is a group-level multiplier, not a direct EEVDF per-entity weight — the scheduler distributes the group's CPU share among its member tasks using their individual nice-derived weights. When a cgroup with weight 200 competes with a sibling at weight 100, it receives approximately 2× as much CPU time at the group level, but individual task scheduling within the group follows standard EEVDF lag/vruntime mechanics.
cpu.max: Sets the bandwidth ceiling via the cgroup's CpuBandwidthThrottle (the CpuController.bandwidth state), enforced per-CPU. Format: "<quota> <period>" (both in microseconds). Example: "400000 1000000" limits the cgroup to 40% CPU (400ms per 1000ms period). This is a maximum limit, not a guarantee. When throttled, the cgroup's tasks are removed from the run queue until the next period begins. This matches standard Linux cgroup v2 semantics.
cpu.guarantee: (UmkaOS extension, see Section 7.6) Sets the bandwidth floor using Constant Bandwidth Server (CBS). Format: "<budget> <period>". Guarantees minimum CPU time regardless of other load. This is distinct from cpu.max: a cgroup can have both a guarantee (floor) and a limit (ceiling).

Relationship between cpu.max and cpu.guarantee: | Setting | Effect | Use Case | |---------|--------|----------| | cpu.max only | Limits maximum, no minimum | Prevent runaway containers | | cpu.guarantee only | Guarantees minimum, no maximum | RT workloads that need bounded latency | | Both | Guarantees minimum AND limits maximum | Mixed workloads with SLA |

When a cgroup is throttled (by either mechanism), the scheduler removes its tasks from the EEVDF tree until the next period or until budget is replenished.

17.2.3.1 `cpu.weight` Write Handler¶

When userspace writes a new value to cpu.weight, the cgroup subsystem validates the value, updates the CpuController.weight atomic, and then calls the scheduler to propagate the new weight to all per-CPU GroupEntity instances (Section 7.2).

/// cgroupfs write handler for `cpu.weight`.
///
/// **Serialization**: UmkaOS has no global cgroup configuration mutex (the design
/// deliberately eliminates Linux's global cgroup mutex — see
/// [Section 24.9](24-roadmap.md#what-umkaos-provides-that-linux-cannot)). Two concurrent `cpu.weight`
/// writers to the SAME cgroup must not interleave their per-CPU propagation
/// loops (each loop writes ITS value to every `GroupEntity`; interleaved
/// CPU-visit orders would leave a mix of old and new weights across CPUs with
/// no final-writer-wins guarantee). This handler therefore takes the per-cgroup
/// `config_lock` (the `Cgroup.config_lock` writer proof defined in
/// [Section 17.2](#control-groups--cgroup-node)) for the whole
/// store+propagate window — the same non-atomic-config writer proof the cpuset
/// and subtree_control handlers use. `config_lock` is a sleeping `Mutex`; the
/// cgroupfs write path runs in process context (no spin-class lock held), so
/// sleeping is legal.
pub fn cpu_weight_write(cgroup: &CgroupNode, buf: &[u8]) -> Result<usize, KernelError> {
    let value = parse_u32(buf)?;
    if !(1..=10000).contains(&value) {
        return Err(KernelError::InvalidArgument);
    }

    // Serialize concurrent writers to this cgroup: the store below and the
    // per-CPU propagation are one atomic operation w.r.t. another cpu.weight
    // write to THIS cgroup (both take this cgroup's OWN `config_lock`), so the
    // last writer's value wins on every CPU. This lock does NOT exclude a
    // concurrent `-cpu` disable: `child_controller_free` runs under the PARENT's
    // `config_lock` (the parent is the cgroup whose `subtree_control` file was
    // written, and it passes ITS guard down as the `proof`) — a DIFFERENT `Mutex`
    // instance. What keeps the `RcuPtr` read valid across the store+propagate
    // window is RCU deferred reclamation alone: `update(None, ..)` reclaims the
    // old `CpuController` box only after a grace period, so this read section's
    // borrow cannot observe freed state even if the controller is disabled
    // mid-handler (a mid-handler disable that lands before the read simply makes
    // `cpu.read(&guard)` return `None` → `NotSupported`/EOPNOTSUPP).
    let _cfg = cgroup.config_lock.lock();

    // `cpu` is `RcuPtr<CpuController>`; read it under an RCU guard. The store and
    // the (non-sleeping) per-CPU propagation below run inside this section — legal
    // because neither sleeps and `config_lock` was taken before the section opened.
    let guard = rcu_read_lock();
    let cpu_ctrl = cgroup.cpu.read(&guard)
        .ok_or(KernelError::NotSupported)?; // EOPNOTSUPP — cpu controller not enabled

    // Update the authoritative weight in the CpuController.
    // Relaxed ordering is sufficient: the scheduler reads this field under
    // per-CPU runqueue locks, and sched_group_set_weight() acquires those
    // locks with Acquire semantics after this store.
    cpu_ctrl.weight.store(value, Ordering::Relaxed);

    // Propagate to all per-CPU GroupEntity instances. This iterates all
    // online CPUs, acquires each runqueue lock, and updates the GroupEntity's
    // weight + vdeadline. On tickless cores with a running task from this
    // cgroup, a reschedule IPI is sent.
    sched_group_set_weight(cgroup.id, value);

    Ok(buf.len())   // config_lock (_cfg) released here
}

This two-phase design (atomic store + per-CPU propagation) ensures that: 1. A concurrent fork()/wake_up() that creates a new GroupEntity on a CPU not yet visited by the propagation loop reads the new weight from CpuController.weight (the authoritative source for new entity creation). 2. Existing GroupEntity instances are updated in-place with proper vdeadline recalculation, preserving accumulated lag for fairness continuity. 3. Tickless cores are explicitly poked so the running task is rescheduled under the new weight — without the IPI, a tickless core would run indefinitely at the stale weight.

17.2.4 Memory Controller Integration¶

The memory controller tracks physical page allocations per cgroup:

memory.current: The sum of all pages charged to this cgroup (in bytes).
memory.max: Hard limit (bytes). When exceeded, the per-cgroup OOM killer is invoked.
memory.high: Soft limit (bytes). When exceeded, the cgroup is throttled and its pages are prioritized for reclaim, but no OOM occurs.
memory.low: Memory protection (bytes). Pages below this threshold are protected from reclaim unless the system is under severe pressure.
memory.swap.max: Limits swap usage for this cgroup (Section 22.7).

Per-cgroup OOM killer: When memory.current exceeds memory.max, the OOM killer selects a victim within the cgroup subtree only — processes outside this cgroup are never killed for a memory.max breach (OomScope::CgroupLimit never escalates, Section 4.5). This is independent of global OOM: per-cgroup OOM can trigger even when global memory is not exhausted. Serialization by the global OOM_LOCK (same lock as global OOM — no per-cgroup OOM lock) happens INTERNALLY in invoke_oom_killer(); the charge path never touches OOM_LOCK itself. Victim selection uses the canonical oom_score_of() formula defined in Section 4.5: points = rss_pages + swap_pages + pgtable_pages; score = points + (oom_score_adj * totalpages) / 1000. Note: dirty_kb is NOT included (dirty pages are already counted in RSS — double-counting would inflate scores); child_rss/8 is NOT included (removed from Linux oom_badness() in the 2010 OOM rewrite, commit a63d83f427fb). The only difference from global OOM is scope: per-cgroup OOM considers only tasks within the cgroup subtree. Tasks with oom_score_adj == -1000 are OOM-immune (matching Linux semantics). When memory.oom.group is set to true, the OOM killer kills all tasks in the cgroup atomically rather than selecting a single victim (Linux memory.oom.group semantics). Without this flag (default), only the highest-scoring victim is killed.

try_charge(cgroup, nr_pages) flow (all steps run under one rcu_read_lock(), resolving memcg = cgroup.memcg(&guard) once since memory is RcuPtr<Arc<MemCgroup>>): (1) Atomic add to memcg.usage. (2) If usage > memory.max: (a) try direct reclaim within cgroup. (b) If reclaim insufficient: build the OOM context and invoke the canonical entry point — invoke_oom_killer(&OomContext { scope: OomScope::CgroupLimit, memcg: Some(Arc::clone(memcg)), nodemask: None, gfp_mask, order, task: Arc::clone(current_task()) }) (Section 4.5). OOM_LOCK acquisition, the two-phase drop-before-kill discipline, Step 0 hibernation, oom_group_root() group-kill resolution, memory.events increments, and the /dev/oom notification are ALL internal to invoke_oom_killer() — the charge path MUST NOT acquire OOM_LOCK around the call: invoke_oom_killer() takes it itself (a caller-held OOM_LOCK is an instant self-deadlock on the non-reentrant Mutex), and holding it across SIGKILL delivery is exactly the OOM_LOCK/SIGLOCK ordering hazard the two-phase design exists to prevent (Section 4.5, "SIGLOCK + OOM_LOCK deadlock prevention"). (c) If invoke_oom_killer() returns true (victim killed or a subtree cgroup hibernated): retry the charge. (d) If it returns false, or the retried charge still exceeds memory.max after a bounded number of retries: return -ENOMEM (the allocation fails; the task may then be killed by its own error handling, not by this path). CgroupLimit scope guarantees no global victim is ever killed for this breach. See Section 4.5 for the canonical kill sequence.

Ripple flagged (not fixed here — out of the R3f subtree-iteration task's scope; needs its own follow-up): the try_charge() flow above and MemCgroup::charge()'s definition (see Section 4.2) both charge ONLY the exact cgroup's usage counter — neither walks cgroup.parent to propagate the charge to ancestors. Linux cgroup v2 memory.current is hierarchical (a parent's counter includes all descendants' charges), enforced by Linux page_counter_charge() walking every ancestor up to the root at charge time (mm/page_counter.c, torvalds/linux master). As specified, a non-leaf cgroup's memory.current reflects only tasks charged directly to it, not its subtree, and a parent's memory.max never sees pressure from descendant charges. This is an ancestor-ward (parent-chain) propagation gap in the CHARGE counters specifically — the OOM killer's memcg_event_hierarchy() (Section 4.5) already does the equivalent parent-chain walk for the (cold) memory.events counters, so events ARE hierarchical; only usage propagation (hot path, per page charge) remains open. A fix needs either (a) MemCgroup::charge()/MemCgroup::uncharge() walking cgroup.parent to the root doing fetch_add/fetch_sub per ancestor (the same O(depth) pattern migrate_task_charge() already uses above), or (b) memory.current reads computed on-demand via self.usage + Σ for_each_descendant(|cg| cg.memcg(&guard).map_or(0, |m| m.usage)). Option (a) matches Linux's actual mechanism and keeps memory.current reads O(1); option (b) avoids touching the charge hot path but makes every read O(subtree size). This needs its own task — the hot-path cost analysis (per-ancestor atomic on every page charge/uncharge) is a real trade-off warranting dedicated design review, not a decision to make as a side effect of subtree iteration.

memory.min / memory.low enforcement: memory.min — reclaim scanner unconditionally skips pages belonging to cgroups where usage <= memory.min (hard protection). memory.low — reclaim scanner deprioritizes pages from cgroups where usage <= memory.low (soft protection — reclaimed only when no other reclaimable memory is available).

Cgroup-scoped vs global reclaim: Cgroup-scoped reclaim scans only the cgroup's LRU lists (not global). reclaimd is NOT invoked for cgroup-scoped reclaim — only direct reclaim in the charging task's context. Global reclaimd runs only for global memory pressure (zone watermarks).

Memory accounting has low overhead (~1 atomic increment per page charge); the charge operation piggybacks on existing page table allocation routines in umka-nucleus.

17.2.5 I/O Controller Integration¶

The io controller limits block I/O bandwidth and IOPS per cgroup:

io.max: Per-device limits. Format: "<major>:<minor> rbps=<bytes> wbps=<bytes> riops=<ops> wiops=<ops>". Example: "8:0 rbps=10485760 wbps=5242880" limits reads to 10 MB/s and writes to 5 MB/s on device 8:0.
io.weight: Proportional weight for best-effort I/O scheduling (1-10000, default 100).

Device identity for stacking filesystems: overlayfs, device-mapper, and other stacking layers do not have their own block device identity for io.max purposes. The <major>:<minor> always refers to the underlying physical block device (e.g., /dev/sda = 8:0). The block I/O layer resolves I/O from stacking filesystems to the backing device before applying cgroup throttling, matching Linux behavior.

The block I/O subsystem (Section 15.2) integrates with cgroup accounting: each bio (block I/O request) is tagged with its originating cgroup, and the I/O scheduler enforces per-cgroup limits.

The block layer's bio submission path calls cgroup_io_throttle() — the ONE canonical entry point from block I/O into the io controller:

/// Block-layer entry point for cgroup I/O throttling. Called from the bio
/// submission path ([Section 15.2](15-storage.md#block-io-and-volume-management)) with `bio.cgroup_id`
/// already stamped. Resolves the submitting cgroup's `IoController` and, if a
/// STATIC `io.max` limit applies to the bio's device, returns a `Throttle`
/// ticket pinning the per-(cgroup, device) `IoThrottleState`. `None` means
/// unthrottled — no cgroup, the root cgroup, or no `io.max` for this device —
/// and the hot path dispatches with zero further work.
///
/// Context: process context only. Takes/releases `rcu_read_lock()` internally
/// for the registry + controller lookup and drops it before returning; the
/// returned `Throttle` keeps the `IoThrottleState` alive via its own counted
/// reference, so `wait_for_token()` may sleep after the RCU guard is gone.
pub fn cgroup_io_throttle(bio: &Bio) -> Option<Throttle> {
    let guard = rcu_read_lock();
    // CGROUP_REGISTRY is the ONE canonical id→cgroup registry; `cgroup.io` is
    // the RcuPtr<IoController> read-side.
    let cgroup = CGROUP_REGISTRY.get(bio.cgroup_id)?;   // None → unthrottled
    let io_ctl = cgroup.io.read(&guard)?;               // None → unthrottled
    let dev = bio.bdev.dev;
    // No STATIC io.max limit for this device → unthrottled.
    io_ctl.devices_snapshot.load().get(dev)?;
    // Pin the MUTABLE token-bucket state (created lazily on first io.max write).
    // `get(dev)` borrows the stored `Arc` under the RCU guard; `Arc::clone`
    // hands the ticket a counted reference that survives after the guard drops.
    let state = io_ctl.throttle.get(dev).map(Arc::clone)?;
    Some(Throttle { state })
}

/// Throttle ticket returned by `cgroup_io_throttle`. Holds a counted reference
/// to the per-(cgroup, device) `IoThrottleState`
/// ([Section 15.2](15-storage.md#block-io-and-volume-management--cgroup-io-throttling)); the reference
/// keeps the token-bucket state alive after the lookup's RCU guard is dropped.
pub struct Throttle {
    state: Arc<IoThrottleState>,
}

impl Throttle {
    /// Charge `bio`'s I/O cost against the token bucket, blocking on the
    /// bucket's wait queue until tokens are available (rbps/wbps/riops/wiops
    /// rate limits; 1 ms refill tick). Process context only — may sleep.
    pub fn wait_for_token(&self, bio: &Bio) {
        self.state.charge_and_wait(bio);
    }
}

17.2.6 PIDs Controller (Fork Bomb Prevention)¶

pids.max: Maximum number of tasks (threads + processes) in the cgroup subtree. Prevents fork bombs from exhausting system-wide PID space. fork()/clone() returns EAGAIN when the limit is reached.
pids.current: Current number of tasks in the cgroup.

This is critical for container isolation: a misbehaving container cannot exhaust the host's PID space.

17.2.6.1 Cgroup Fork Hooks¶

The cgroup subsystem participates in fork()/clone() via two hook functions called by the process creation path (Section 8.1):

precharge_fork(parent, target_cgroup) — Called before the child task struct is fully initialized. Performs pre-allocation checks and counter reservations on the RESOLVED cgroup — the CLONE_INTO_CGROUP target when the fork passed one (resolved at create_task step 3, BEFORE this call), the parent's cgroup otherwise.

CLONE_INTO_CGROUP target resolution — before precharge_fork(), create_task step 3 resolves the explicit target from the clone3() cgroup fd and checks that the caller may place a task there:

/// Resolve a `CLONE_INTO_CGROUP` target from the `clone3()` `cgroup` field
/// (a `u64` holding an O_RDONLY fd on a cgroupfs directory). Returns the
/// target cgroup, `Arc`-pinned. Called at create_task step 3.
///
/// Errors: `KernelError::InvalidCapability` (EBADF) — the value is not an
/// open fd, or not a cgroupfs directory fd.
pub fn from_fd(cgroup_fd: u64) -> Result<Arc<Cgroup>, KernelError> {
    // SAFETY: current_task() returns the running task, valid for this call.
    let task = unsafe { &*current_task() };
    let file = task.files.load_full().get(cgroup_fd as i32)
        .ok_or(KernelError::InvalidCapability)?;
    // Recover the Cgroup the directory inode backs; a non-cgroupfs directory
    // (or a non-directory fd) is EBADF.
    file.cgroupfs_dir_target().ok_or(KernelError::InvalidCapability)
}

/// Permission gate for forking INTO an explicit `CLONE_INTO_CGROUP` target:
/// the caller must hold write access (`CGRP_WRITE`) to `cgrp` — the same
/// delegation authority required to write `cgroup.procs`
/// ([Section 17.2](#control-groups--task-migration-cgroupprocs-write)). Distinct from
/// `precharge_fork()`, which reserves controller counters; this is the
/// delegation-boundary check on the target directory. Called at create_task
/// step 3, immediately after `Cgroup::from_fd()`.
///
/// Errors: `KernelError::PermissionDenied` (EACCES) — the caller lacks
/// `CGRP_WRITE` on `cgrp`.
pub fn precharge_fork_into(cgrp: &Arc<Cgroup>) -> Result<(), KernelError> {
    if cgrp.writable_by_current() {
        Ok(())
    } else {
        Err(KernelError::PermissionDenied)
    }
}

/// Bumped once whenever the `pids` controller is ENABLED or DISABLED on any
/// cgroup (`child_controller_alloc`/`child_controller_free` for `PIDS`, and
/// `mkdir`/drain when they create/drop a `PidsController`). A fork captures this
/// at `precharge_fork` and `commit_fork` compares: an unchanged value
/// proves no pids-topology change occurred in the fork window, so the recorded
/// `pids_charged` set is still exact and no reconcile is needed (the fast path).
/// `AtomicU64`, monotonic, cold writer (a cgroupfs config write) — never wraps.
pub static PIDS_TOPOLOGY_GEN: AtomicU64 = AtomicU64::new(0);

/// One ancestor a fork charged `pids.current` on, recorded as the pair
/// (cgroup, controller-box serial). The `Arc<Cgroup>` keeps the cgroup alive
/// so the paired uncharge re-resolves `cgroup.pids` without `Weak::upgrade`;
/// `controller_id` pins the EXACT `PidsController` box that was incremented so
/// the uncharge decrements THAT box, not whatever box currently sits on the
/// cgroup. The distinction matters only when a `-pids`+`+pids` in the fork
/// window swaps the box — see `CgroupForkCharge`.
pub struct PidsCharge {
    /// The charged cgroup (identity for the reconcile; keeps it alive for the
    /// re-resolve).
    pub cgroup: Arc<Cgroup>,
    /// `PidsController::id` of the box incremented at `precharge_fork` time.
    pub controller_id: u64,
}

/// Fork-time cgroup charge record, carried from `precharge_fork` to its
/// paired `commit_fork` (commit) or `rollback_fork` (rollback).
///
/// **Why the pids set is RECORDED, not re-walked in cancel**:
/// `write_subtree_control` can enable or disable the `pids` controller
/// on an EXISTING ancestor (`child_controller_alloc`/`child_controller_free`
/// on live children) at any time — including between this fork's
/// `precharge_fork` and its `rollback_fork`. So the set of cgroups with
/// `Some(pids)` on the resolved-target→root chain is NOT stable across the fork
/// window. A fresh root-walk in cancel would touch a DIFFERENT set than
/// can_fork incremented: an ancestor with `pids` newly ENABLED in between gets
/// a `fetch_sub` with no matching `fetch_add` (u64 underflow to ~2^64 — every
/// later fork in that subtree then reads `current ≈ u64::MAX > max` and fails
/// EAGAIN permanently), and a newly-DISABLED ancestor leaks its increment.
/// INVARIANT: the multiset `rollback_fork` decrements is exactly the
/// multiset `precharge_fork` incremented — guaranteed by decrementing
/// `pids_charged` rather than re-deriving the chain.
///
/// **Why the controller-BOX id is recorded, not just the cgroup**: recording
/// the cgroup identity is not enough. `-pids` (disable) FREES the
/// `PidsController` box (`RcuPtr::update(None)`) and a later `+pids` (re-enable)
/// on the SAME cgroup allocates a FRESH box whose `current` is re-initialized to
/// the subtree's live-task count. If both land in this fork's window, the
/// charged cgroup is still in `pids_charged`, but its live box is no longer the
/// one `precharge_fork` incremented: a blind `cgroup.pids.read()` in cancel
/// would `fetch_sub` the FRESH box (which this fork never `fetch_add`ed),
/// underflowing it, while the original box's increment vanished with the box.
/// So each entry pins `PidsController::id` (`PidsCharge.controller_id`) and both
/// the uncharge and the post_fork reconcile act only when the CURRENT box's id
/// matches — a swapped-in box is left untouched (its charge, on the freed box,
/// is moot). This is the same anchored-charge shape the perf_event controller
/// uses (`event.charged_cgroup` + whole-chain pinning): the charge names the
/// exact box it must reverse, never a re-resolved one.
///
/// This is the binding constraint the [Section 17.2](#control-groups--cgroupsubtreecontrol-write-handler)
/// must respect: it may freely toggle `pids` on a cgroup, because the
/// per-fork charge is anchored here and never re-derived. (The COMPLEMENTARY
/// requirement — that enabling `pids` on an already-populated cgroup
/// initializes `pids.current` to that subtree's live task count so the
/// unmatched `detach_exiting_task` decrements stay balanced — is stated on that
/// handler.)
pub struct CgroupForkCharge {
    /// Ancestors this fork incremented `pids.current` on (resolved target →
    /// root, only those with `Some(pids)` at reservation time), each paired with
    /// the incremented box's `PidsController::id`. Owned `Arc`s keep every
    /// charged cgroup alive until its matching decrement, so the cancel walk
    /// never needs `Weak::upgrade`; the recorded id makes that decrement hit the
    /// exact box that was charged. Bounded by `CGROUP_MAX_PATH_NODES`.
    pub pids_charged: ArrayVec<PidsCharge, CGROUP_MAX_PATH_NODES>,
    /// CLONE_INTO_CGROUP fork-pin target (`Some` only for an explicit target
    /// fd). Released by whichever of post_fork / cancel_fork consumes the charge.
    pub pin_target: Option<Arc<Cgroup>>,
    /// Snapshot of `PIDS_TOPOLOGY_GEN` at `precharge_fork` time. `commit_fork`
    /// compares it against the live value: if unchanged (the overwhelmingly common
    /// case — no `pids` controller was enabled/disabled anywhere during this
    /// fork's window), the recorded `pids_charged` set still exactly matches the
    /// resolved chain and post_fork skips the reconcile walk entirely (fast
    /// path). If it changed, post_fork reconciles the enable-time in-flight-fork
    /// hole (see [Section 17.2](#control-groups--pids-enable-time-in-flight-reconciliation)).
    pub pids_topology_gen: u64,
    /// Pre-allocated task-set XArray interior nodes (the Linux `radix_tree_preload`
    /// analogue), consumed by `commit_fork`'s allocation-free insert or
    /// freed back to slab by `rollback_fork`.
    pub task_slot: XaInsertReservation,
}

/// Pre-allocated XArray interior nodes reserved for exactly one future
/// insertion: `reserve_insertion()` runs the fallible slab allocation up
/// front and hands back this owning token, so the paired `insert_reserved()`
/// stores the value without allocating and cannot fail.
///
/// `RwLock<XArray<()>>::reserve_insertion()` performs the slab-fallible
/// allocation (`Err(EAGAIN)` on OOM) of every interior node the deepest path to
/// a not-yet-present key can require and hands back this OWNING token; the paired
/// `insert_reserved(key, token)` then places the value WITHOUT allocating, so
/// that insert is infallible (it runs at `create_task` step 18, past the last
/// rollback point). Unlike Linux's per-CPU preload cache, the reservation is
/// owned by this token, so it survives preemption and CPU migration between the
/// reserving call (`precharge_fork`) and the consuming insert
/// (`commit_fork`, possibly on another CPU). Dropping the token unconsumed
/// (`rollback_fork`) returns the reserved nodes to the slab — no leak.
///
/// The interior-node layout is private to the collections crate that owns
/// `XArray`; this token is an opaque owning handle over a bounded set of those
/// nodes. A `u64` key over the 64-way (6-bit) radix fanout descends at most 11
/// levels, so at most 11 nodes are ever held.
pub struct XaInsertReservation {
    /// Owned pre-allocated interior nodes (slab-backed; collections-crate
    /// internal node layout), returned to the slab on `insert_reserved`
    /// consumption or on `Drop`. Bounded by the maximum radix-tree height.
    nodes: ArrayVec<SlabBox<[u8]>, 11>,
}

/// The paired reserve/insert API on the cgroup task-set `XArray<()>`
/// (`Cgroup.tasks`, keyed by `TaskId`). `reserve_insertion` performs the
/// slab-fallible pre-allocation; `insert_reserved` consumes the token in an
/// allocation-free, infallible store. Both take `&mut self` — the caller holds
/// the write side of `RwLock<XArray<()>>`.
impl XArray<()> {
    /// Pre-allocate every interior radix node the deepest path to a
    /// not-yet-present key can require and hand back an OWNING
    /// `XaInsertReservation`. UmkaOS's analogue of Linux `radix_tree_preload()`.
    ///
    /// This is the ONLY fallible step of the reserve/insert pair: on slab
    /// exhaustion it returns `Err(TryAgain)` (EAGAIN), still a legal `fork()`
    /// failure at `precharge_fork()` — before `create_task`'s last rollback point.
    /// The reservation is owned by the returned token (NOT a per-CPU cache), so
    /// it survives preemption and CPU migration between the reserving call
    /// (`precharge_fork`) and the consuming `insert_reserved`
    /// (`commit_fork`, possibly on another CPU). Dropping the token
    /// unconsumed returns the nodes to the slab.
    pub fn reserve_insertion(&mut self) -> Result<XaInsertReservation, KernelError> {
        // Worst-case interior path for a u64 key over the 64-way (6-bit) radix:
        // ceil(64 / 6) = 11 levels ⇒ at most 11 slab nodes, one per level.
        const XA_MAX_DEPTH: usize = 11;
        // Interior-node byte size (64 × u64 slots + node header). The exact
        // layout is private to the collections crate; the reservation allocates
        // node-sized slab buffers so the paired insert never touches the slab.
        const XA_NODE_BYTES: usize = 576;
        let mut nodes: ArrayVec<SlabBox<[u8]>, 11> = ArrayVec::new();
        for _ in 0..XA_MAX_DEPTH {
            // Fallible slab reservation. On exhaustion the partially-filled
            // `nodes` vec drops here (freeing its nodes) and the whole call
            // yields EAGAIN.
            match SlabBox::try_new([0u8; XA_NODE_BYTES]) {
                Some(node) => nodes.push(node), // SlabBox<[u8; N]> unsizes to SlabBox<[u8]>
                None => return Err(KernelError::TryAgain),
            }
        }
        Ok(XaInsertReservation { nodes })
    }

    /// Store `()` at `key`, consuming `reservation`'s pre-allocated interior
    /// nodes so the store performs NO allocation and therefore cannot fail —
    /// safe at `create_task` step 18, past the last rollback point. The collections
    /// crate splices one reserved node in for each interior radix level the path
    /// to `key` is missing (private node layout), then publishes the `()` leaf;
    /// reserved nodes the path did not need are returned to the slab as
    /// `reservation` drops at end of scope. `key` is a fresh `tid` on the fork
    /// path, so it is never already present.
    pub fn insert_reserved(&mut self, key: TaskId, reservation: XaInsertReservation) {
        // The reservation backs the (otherwise allocating) interior-node
        // creation, making this store allocation-free and infallible. The `()`
        // leaf is published at `key`; unconsumed nodes free on `reservation`'s
        // drop below.
        self.store(key, ());
        drop(reservation);
    }
}

/// Pre-fork cgroup check. Walks the resolved cgroup's hierarchy and
/// invokes each controller's can_fork hook. On success returns the exact set
/// of cgroups whose `pids.current` it incremented; returns Err(EAGAIN) if any
/// controller rejects.
///
/// `target_cgroup`: Some(&target) for CLONE_INTO_CGROUP (create_task step 3
/// early resolution), None to use the parent's cgroup. The SAME resolved
/// target must be passed to rollback_fork() on every fork rollback —
/// reserving on one hierarchy and cancelling on another permanently
/// drifts pids.current on both.
///
/// On failure, already-approved controllers are rolled back internally.
///
/// Returns a `CgroupForkCharge` recording the EXACT reservation so the paired
/// `rollback_fork` / `commit_fork` reverse or commit precisely what
/// was reserved — see the struct doc for why a fresh walk in cancel is unsafe.
pub fn precharge_fork(
    parent: &Task,
    target_cgroup: Option<&Arc<Cgroup>>,
) -> Result<CgroupForkCharge, KernelError> {
    // CLONE_INTO_CGROUP lifecycle gate + fork-pin (Some(target) only). A plain
    // fork into the parent's own cgroup cannot race rmdir (the parent is a live
    // member, so the cgroup is populated and un-rmdir-able); an explicit target
    // fd CAN name a cgroup that rmdir is tearing down.
    if let Some(target) = target_cgroup {
        // Publish the pin, THEN check lifecycle: rmdir CASes Draining (SeqCst)
        // under hierarchy_lock, then step 3a loads fork_pins (SeqCst). The
        // store-before-check here pairs with rmdir's set-before-load so the two
        // orders cannot both miss — either rmdir sees our pin (→ EBUSY) or we
        // see Draining (→ ENODEV). Coherent with migration step 4a's ENODEV.
        // BOTH ops here are SeqCst (not Acquire): this is a store-buffering
        // shape (store fork_pins, then load lifecycle), and only a total
        // SeqCst order rules out the both-miss execution on AArch64/POWER —
        // see rmdir step 3. The rollback fetch_sub stays Release (not part of
        // the interlock).
        target.fork_pins.fetch_add(1, Ordering::SeqCst);
        if target.lifecycle.load(Ordering::SeqCst) != CgroupLifecycle::Active as u8 {
            target.fork_pins.fetch_sub(1, Ordering::Release);
            return Err(KernelError::NoDevice); // ENODEV — target is Draining/Dead
        }
    }
    // For each CSS in the resolved cgroup set, call controller.can_fork().
    // Currently only the pids controller has a can_fork hook. On success it
    // returns the EXACT set of cgroups it incremented, recorded into the charge
    // so cancel_fork decrements that same set (see CgroupForkCharge).
    let pids_charged = match pids_precharge_fork(parent, target_cgroup) {
        Ok(set) => set,
        Err(e) => {
            if let Some(target) = target_cgroup {
                target.fork_pins.fetch_sub(1, Ordering::Release);
            }
            return Err(e);
        }
    };
    // Pre-reserve the resolved target's task-set XArray interior nodes NOW,
    // where EAGAIN is still a legal fork failure. This is the slab-fallible
    // allocation that commit_fork's "insert" would otherwise perform at
    // create_task step 18 — past the last rollback point, where it MUST NOT fail.
    // Same model as Linux `radix_tree_preload()` → `radix_tree_insert()`: the
    // reservation is carried in the charge (NOT a per-CPU cache, so it survives
    // preemption/migration between here and post_fork) and consumed by an
    // allocation-free `insert_reserved`.
    let reserve_in: Arc<Cgroup> = target_cgroup
        .map_or_else(|| parent.cgroup.load_full(), Arc::clone);
    let task_slot = match reserve_in.tasks.write().reserve_insertion() {
        Ok(res) => res,
        Err(_) => {
            // Undo the pids reservation and the pin before failing. Decrement
            // only the box each ancestor was charged on (id-guarded), same as
            // rollback_fork.
            let g = rcu_read_lock(); // `pids` is RcuPtr; recorded set stable under hierarchy_lock
            for charge in &pids_charged {
                if let Some(pids) = charge.cgroup.pids.read(&g) {
                    if pids.id == charge.controller_id {
                        pids.current.fetch_sub(1, Ordering::Relaxed);
                    }
                }
            }
            if let Some(target) = target_cgroup {
                target.fork_pins.fetch_sub(1, Ordering::Release);
            }
            return Err(KernelError::TryAgain); // EAGAIN — node reservation failed, retry
        }
    };
    // The fork-pin, pids reservation, and task-slot are consumed by
    // commit_fork() (commit) or rollback_fork() (rollback). Snapshot
    // the pids-topology generation so post_fork can detect (and only then
    // reconcile) a `pids` enable/disable that landed in this fork's window.
    Ok(CgroupForkCharge {
        pids_charged,
        pin_target: target_cgroup.map(Arc::clone),
        task_slot,
        pids_topology_gen: PIDS_TOPOLOGY_GEN.load(Ordering::Acquire),
    })
}

/// Pids controller fork check. Atomically increments pids.current for
/// the RESOLVED cgroup and every ancestor up to the root.
///
/// If any ancestor's pids.current exceeds pids.max after increment,
/// the increment is rolled back for all already-incremented cgroups,
/// pids.events_max is bumped on the failing cgroup, and EAGAIN is returned.
fn pids_precharge_fork(
    parent: &Task,
    target_cgroup: Option<&Arc<Cgroup>>,
) -> Result<ArrayVec<PidsCharge, CGROUP_MAX_PATH_NODES>, KernelError> {
    // OWNED rollback/record storage and an OWNED Arc cursor. Storing `&Cgroup`
    // would dangle: each `Weak::upgrade()` yields a temporary `Arc`
    // dropped at the end of its loop iteration, leaving the stored
    // reference pointing at a released borrow — the identical
    // dangling-temporary bug this file documents fixing in
    // `migrate_task_charge()`. Cap = `CGROUP_MAX_PATH_NODES`
    // (`CGROUP_MAX_DEPTH + 1` = 257) — the root-INCLUSIVE bound. This walk
    // visits the resolved cgroup (depth ≤ CGROUP_MAX_DEPTH) up to AND INCLUDING
    // the root (depth 0), so it can push up to `depth + 1` = 257 entries; a
    // 256-slot vec would `push`-panic on the deepest legal tree. `push` is
    // therefore infallible here — mkdir step 3a rejects `depth >
    // CGROUP_MAX_DEPTH` with EAGAIN, so no reachable tree exceeds this bound,
    // and the vec is sized to the exact maximum. (No `try_push` fallback is
    // needed: the depth invariant, not a runtime check, guarantees capacity.)
    // Warm path, stack-allocated (257 × 8 bytes ≈ 2 KiB).
    // Ancestor liveness for the upgrade(): the resolved cgroup is kept
    // un-rmdir-able for the whole walk — either it is the parent's own
    // (populated) cgroup, or it is a CLONE_INTO_CGROUP target held by
    // fork_pins (bumped in precharge_fork), which blocks rmdir step 3a.
    // An un-rmdir-able cgroup and all its ancestors stay strongly reachable
    // from the root, so every upgrade() succeeds.
    let mut incremented: ArrayVec<PidsCharge, CGROUP_MAX_PATH_NODES> = ArrayVec::new();

    // Walk from the RESOLVED cgroup to the root, incrementing each.
    let mut cg: Arc<Cgroup> = match target_cgroup {
        Some(t) => Arc::clone(t),
        None => parent.cgroup.load_full(),
    };
    // `pids` is `RcuPtr<PidsController>`; one guard spans the walk + any rollback.
    // The fork holds `hierarchy_lock` READ, which excludes the `-pids` disabler
    // (`hierarchy_lock` WRITE), so the chain's controllers are stable; the guard
    // only satisfies `RcuPtr::read`. All ops are atomic — no sleep in the section.
    let g = rcu_read_lock();
    loop {
        if let Some(pids) = cg.pids.read(&g) {
            let prev = pids.current.fetch_add(1, Ordering::Relaxed);
            let max = pids.max.load(Ordering::Relaxed);
            if prev + 1 > max && max != u64::MAX {
                // Over limit — rollback this increment and all prior.
                pids.current.fetch_sub(1, Ordering::Relaxed);
                pids.events_max.fetch_add(1, Ordering::Relaxed);
                // Rollback already-incremented ancestors — decrement only the
                // SAME box each was charged on (id-guarded). Under this single
                // RCU section a box swap is not expected, but the guard keeps
                // the rollback exact regardless, mirroring rollback_fork.
                for ancestor in &incremented {
                    if let Some(apids) = ancestor.cgroup.pids.read(&g) {
                        if apids.id == ancestor.controller_id {
                            apids.current.fetch_sub(1, Ordering::Relaxed);
                        }
                    }
                }
                return Err(KernelError::TryAgain); // EAGAIN — retry
            }
            incremented.push(PidsCharge { cgroup: Arc::clone(&cg), controller_id: pids.id });
        }
        let next = match cg.parent {
            Some(ref p) => p.upgrade()
                .expect("ancestor population > 0 guarantees liveness -- see hierarchy_lock"),
            None => break,
        };
        cg = next;
    }
    Ok(incremented)   // the EXACT charged set, recorded into CgroupForkCharge
}

/// Rollback precharge_fork() reservations when fork fails after the
/// cgroup checks succeeded but a later step (RLIMIT_NPROC, PID alloc,
/// memory) failed.
///
/// Consumes the `CgroupForkCharge` returned by `precharge_fork`. It
/// decrements `pids.current` on EXACTLY the cgroups recorded in
/// `charge.pids_charged` — NOT a fresh resolved-target→root walk. Re-walking
/// is unsafe because `write_subtree_control` may have toggled the `pids`
/// controller on an ancestor since can_fork ran, so the live chain no longer
/// matches the charged set (see `CgroupForkCharge` for the underflow/leak this
/// prevents). The owned `Arc`s in `pids_charged` also keep every charged cgroup
/// alive here, so no `Weak::upgrade` is needed.
pub fn rollback_fork(charge: CgroupForkCharge) {
    // `pids_topology_gen` is unused on the cancel path: cancel reverses EXACTLY
    // the recorded `pids_charged` set, which is stable regardless of any `pids`
    // enable/disable in this fork's window — only `commit_fork` (commit)
    // reconciles a topology change. Bind it to `_`.
    let CgroupForkCharge { pids_charged, pin_target, task_slot, pids_topology_gen: _ } = charge;

    // Reverse the pids reservation on precisely the recorded set. Decrement
    // only the box each ancestor was charged on: `charge.controller_id` pins
    // the exact `PidsController` that can_fork incremented. If a `-pids`+`+pids`
    // in the fork window swapped the box, the current box's id will NOT match
    // (or the controller is now absent) and we leave it untouched — the fresh
    // box was never charged by this fork, and the original box's increment
    // vanished with the box. Without this guard the `fetch_sub` would underflow
    // the fresh box. `population` is NOT touched: commit_fork() never ran on
    // this failure path (it runs at create_task step 18, strictly after every
    // fallible step), so there is no population increment to reverse.
    let g = rcu_read_lock(); // `pids` is RcuPtr; the guard only satisfies `read`
    for charge in &pids_charged {
        if let Some(pids) = charge.cgroup.pids.read(&g) {
            if pids.id == charge.controller_id {
                pids.current.fetch_sub(1, Ordering::Relaxed);
            }
        }
    }

    // Release the CLONE_INTO_CGROUP fork-pin (Some only for an explicit target
    // fd). Pairs with rmdir step 3a so a Draining target can complete its drain
    // once no fork holds it.
    if let Some(ref target) = pin_target {
        target.fork_pins.fetch_sub(1, Ordering::Release);
    }

    // The unused task-set node reservation returns to slab when `task_slot`
    // drops at end of scope.
    drop(task_slot);
}

commit_fork(child, target_cgroup, charge) — Called after the child task struct is initialized and linked (create_task step 18) but before enqueue_new_task(). Attaches the child to the resolved cgroup's task set (either the parent's cgroup, or the target specified by CLONE_INTO_CGROUP — the same Arc<Cgroup> resolved in create_task step 15 and installed into child.cgroup by the step-15b Task literal).

Failure and rollback semantics:

The fork path has a three-phase cgroup protocol:

precharge_fork() — Atomically increments pids.current for the RESOLVED cgroup (the CLONE_INTO_CGROUP target when the fork passed one, the parent's cgroup otherwise) and all its ancestors. If any ancestor exceeds pids.max, the increment is rolled back and the fork fails with EAGAIN before the child task is allocated.
commit_fork(child, target, charge) — Attaches the child to the cgroup's task set. This step is infallible by design: the pids reservation was secured in step 1, AND the task-set XArray insertion consumes the interior nodes that precharge_fork pre-reserved into charge.task_slot (step 1, where slab-allocation failure was still a legal EAGAIN), so the insert here allocates NOTHING and cannot fail — closing the "fallible insert past the last rollback point" gap. No re-check of pids.max is performed here because the counter was already incremented atomically in step 1.
If any fork step between precharge_fork() and commit_fork() fails (e.g., PID allocation exhaustion, RLIMIT_NPROC, memory allocation for the child task struct), the fork path calls rollback_fork(charge) with the CgroupForkCharge returned by precharge_fork(), decrementing pids.current on exactly the cgroups the charge recorded (never a fresh walk — see CgroupForkCharge). The child task is never added to the cgroup's task set, the pre-reserved node returns to slab, and no cgroup state is leaked.

TOCTOU note: An administrator may lower pids.max between precharge_fork() and commit_fork(), causing pids.current > pids.max temporarily. This is benign and matches Linux behavior: the lowered limit prevents new forks but does not kill existing tasks. The overshoot is bounded (at most one task per concurrent fork) and resolves when tasks exit.

/// Post-fork cgroup attachment. Adds the child task to the resolved
/// cgroup's task set. Must complete before the child is made runnable.
///
/// This function is infallible: the pids.current counter was already
/// incremented by precharge_fork(), AND the task-set insert consumes the
/// interior nodes pre-reserved into `charge.task_slot` (so it allocates
/// nothing). No counter update or limit check is needed here. If the child
/// must not be added (fork failure after precharge_fork()), the caller
/// invokes rollback_fork(charge) instead — commit_fork() is never
/// called for failed forks.
///
/// `target_cgroup` is the cgroup create_task step 15 resolved (the
/// CLONE_INTO_CGROUP target, or the parent's cgroup) — the SAME
/// Arc<Cgroup> already installed into `child.cgroup` by the step-15b
/// Task literal. This function does NOT write `child.cgroup` — the
/// membership pointer is the fork path's responsibility; this function
/// performs the task-set insertion and consumes the charge (committing the
/// pids reservation, releasing the fork-pin).
///
/// The cgroup's task set is updated atomically (under the cgroup's
/// tasks RwLock).
pub fn commit_fork(child: &Task, target_cgroup: &Arc<Cgroup>, charge: CgroupForkCharge) {
    let CgroupForkCharge { pids_charged, pin_target, task_slot, pids_topology_gen } = charge;

    // Serialize the task-set insert AND the pids reconcile against topology
    // writes. `write_subtree_control` holds `hierarchy_lock` WRITE across
    // its ENTIRE subtree live-task count + `PIDS_TOPOLOGY_GEN` bump, so taking
    // `hierarchy_lock` READ here (level 210, below CGROUP_TASKS_LOCK(215)) forces
    // a concurrent `pids` enable to be EITHER fully-before this insert (its count
    // excluded the not-yet-inserted child → the gen-compare below sees the bump →
    // we charge the child) OR fully-after (its count WILL include the child → gen
    // unchanged → we do NOT charge). The two cannot interleave, so the child is
    // counted on the newly-enabled ancestor EXACTLY once. A bare
    // `PIDS_TOPOLOGY_GEN` compare WITHOUT this lock is unsound in BOTH
    // directions: (under-count) an enable slipping between an un-serialized
    // gen-read and the insert, and (mirror double-count) the enable's count
    // AND this reconcile both charging the child when the insert lands between
    // the enable's count and its gen bump — the global gen cannot attribute a
    // bump to a particular ancestor's count ordering, only the topology lock can.
    // Reads are concurrent (parallel forks do not block each other); only a rare
    // topology write (mkdir/rmdir/subtree_control) excludes them — the same
    // css_set_lock-on-fork discipline Linux uses. Legal nesting: sleeping
    // threadgroup_rwsem → hierarchy_lock(210) → tasks(215).
    let _hier = cgroup_root().hierarchy_lock.read();

    // Add child to the cgroup's task set (CGROUP_TASKS_LOCK, level 215).
    // This set is used by:
    // - cgroup.procs reads (enumerate tasks)
    // - Freezer (iterate tasks to set TaskState::FROZEN)
    // - OOM killer (select victim within cgroup)
    // - Migration (remove from old cgroup's set, add to new)
    // `insert_reserved` consumes the pre-allocated `task_slot` nodes and cannot
    // allocate — the insert is infallible here (create_task step 18, past the last
    // rollback point). See precharge_fork's reservation step.
    target_cgroup.tasks.write().insert_reserved(child.tid, task_slot);

    // The pids reservation (pids_charged) is now COMMITTED: the child is a live
    // member, so its charge stays until detach_exiting_task() decrements it. Normally we
    // keep the counts and drop the owned Arcs without touching pids.current —
    // the increments already happened in can_fork.
    //
    // **In-flight-fork reconcile** (only when a pids enable/disable landed in
    // this fork's window — the `PIDS_TOPOLOGY_GEN` fast-path check makes this a
    // no-op for the common fork). AUTHORITATIVE under the `hierarchy_lock` read
    // held above: gen-changed ⟺ some ancestor's `pids` controller was
    // enabled/disabled before this insert. Membership is tested by CONTROLLER-BOX
    // id (`PidsCharge.controller_id`), NOT cgroup id: any ancestor whose CURRENT
    // `pids` box id is not among the charged box ids has a live box that
    // `precharge_fork` did not charge — either a freshly-ENABLED ancestor, or
    // one whose box was SWAPPED by a `-pids`+`+pids` since the charge (same
    // cgroup, different box). Such a box was initialized (at enable) BEFORE this
    // insert committed (the insert is under the same read guard, after any
    // committed enable's write), so its count did NOT include this child, yet the
    // child's later `detach_exiting_task` fresh-walk WILL decrement it. Charge it here so
    // the exit is balanced; the moot charge on the now-freed old box is left
    // behind with it. A cgroup-id test would wrongly treat a swapped ancestor as
    // already-charged and under-count the fresh box. (A DISABLE with no re-enable
    // needs no action: the controller is gone, exit's Some(pids) filter skips it.)
    // See [Section 17.2](#control-groups--pids-enable-time-in-flight-reconciliation).
    if pids_topology_gen != PIDS_TOPOLOGY_GEN.load(Ordering::Acquire) {
        // Alloc-free membership: `pids_charged` (an `ArrayVec`, ≤
        // `CGROUP_MAX_PATH_NODES`) IS the recorded charged-box set. Test it with a
        // bounded linear scan rather than building a `BTreeSet` — a heap
        // allocation here would break this function's documented-infallible
        // contract (it runs at create_task step 18, past the last rollback point).
        // This branch runs only when a `pids` enable/disable landed in the fork
        // window (rare admin op, gated by the `PIDS_TOPOLOGY_GEN` compare), so the
        // O(depth²) scan over a ≤257-node chain is off every hot path; the
        // reconcile stays allocation-free and infallible. Membership is an
        // inline bounded `.any()` scan over `pids_charged` (no closure/alloc).
        let mut cg: Arc<Cgroup> = Arc::clone(target_cgroup);
        loop {
            {
                // `pids` is `RcuPtr<PidsController>`; one guarded read serves both
                // the presence test and the increment. The guard's deferred
                // reclamation keeps a concurrently-disabled controller alive for
                // the `fetch_add` (a `-pids` in the window is harmless — see the
                // DISABLE note above).
                let g = rcu_read_lock();
                if let Some(pids) = cg.pids.read(&g) {
                    if !pids_charged.iter().any(|c| c.controller_id == pids.id) {
                        // Live box can_fork did not charge (newly-enabled ancestor
                        // or a box swapped in since the charge).
                        pids.current.fetch_add(1, Ordering::Relaxed);
                    }
                }
            }
            cg = match cg.parent {
                Some(ref p) => p.upgrade()
                    .expect("child is a live member → every ancestor strongly reachable"),
                None => break,
            };
        }
    }
    // Topology-stable window ends. The population flip below is flip_lock-
    // serialized and needs no hierarchy_lock.
    drop(_hier);
    drop(pids_charged);

    // Bump the target's LOCAL population by one (this cgroup only — NOT a
    // to-root walk). The 0→1 subtree flip is claimed by the fetch_add RETURN
    // VALUE, not a separate racy is_populated() read: `prev == 0` means THIS
    // op drove `population` 0→1, so this is the (at most one) up-edge task.
    // Any `prev > 0` is a steady-state fork into an already-populated cgroup —
    // it returns here touching exactly one cacheline, no flip_lock, no ancestor
    // walk (eliminating the root-cacheline bounce). Only the edge task enters
    // the flip path and reconciles the leaf's `subtree_populated` bit under
    // flip_lock (is_populated()-vs-published, so a concurrent child edge cannot
    // fool it), notifying ancestors only on a genuine subtree flip. Zero counter
    // drift over uptime: detach_exiting_task() step 4 is the exact inverse (down-edge
    // claimed by fetch_sub `prev == 1`). Atomics + one cold per-cgroup lock —
    // the function stays infallible.
    let prev = target_cgroup.population.fetch_add(1, Ordering::Release);
    if prev == 0 {
        // population 0→1 at this cgroup — MAY be a subtree flip. Decide under
        // flip_lock against the PUBLISHED `subtree_populated` bit, NOT a bare
        // `nr_populated_children == 0` read: a concurrent child-subtree edge may
        // have changed `nr_populated_children` between this fetch_add and the
        // lock, so only is_populated()-vs-published is race-free.
        cgroup_reconcile_self_populated(target_cgroup);
    }

    // Release the CLONE_INTO_CGROUP fork-pin (Some only for an explicit target
    // fd). By now the child is a counted member, so the cgroup is un-rmdir-able
    // on its own (population > 0) and the pin is no longer needed — symmetric
    // with rollback_fork releasing it on the failure path.
    if let Some(ref target) = pin_target {
        target.fork_pins.fetch_sub(1, Ordering::Release);
    }
}

17.2.6.1.1 Pids Enable-Time In-Flight Reconciliation¶

Enabling the pids controller on a cgroup whose DESCENDANTS hold tasks (the cgroup itself must be process-free — the no-internal-process EBUSY check) creates PidsController.current and initializes it to the subtree's CURRENT live task count (child_controller_alloc(child, PIDS) walks the task set + descendants under the held hierarchy_lock + config_lock). That live-task walk is exact for tasks already inserted into a task set, but it MISSES an in-flight fork — one whose precharge_fork completed (charging the ancestors that had pids THEN, i.e. NOT this newly-enabled one) but whose commit_fork task-set insert has not yet run (create_task steps ~5–18). Such a child is invisible to the walk, was never charged on the new controller, yet its later detach_exiting_task does a fresh Some(pids) walk and decrements it — a 1-count underflow per race, accumulating toward the u64 wrap the pids.current doc warns of.

The reconcile closes this on the fork side, symmetric with the exit-side fresh walk, WITHOUT the enable handler having to exclude in-flight forks (which it cannot see — a fork that has not yet reached commit_fork is in no task set):

PIDS_TOPOLOGY_GEN is bumped on every pids enable/disable, under the hierarchy_lock WRITE the enable handler holds across its whole subtree count.
precharge_fork snapshots it into CgroupForkCharge.pids_topology_gen.
commit_fork takes hierarchy_lock READ across BOTH its task-set insert AND this gen-compare (the interlock — see the code), then compares. Unchanged (the common case) → the recorded pids_charged set still exactly matches the resolved chain; nothing to do beyond the RwLock read-acquire. Changed → walk the target→root chain and, for each ancestor whose CURRENT pids box id is NOT among the charged box ids (PidsCharge.controller_id), fetch_add(1) — its live box was enabled (or swapped in by a -pids++pids) during this fork's window, so can_fork never charged it; charge it now so its later exit decrement is balanced. Membership is by box id, not cgroup id, so a box swapped on a previously-charged ancestor (same cgroup, fresh box whose count excludes this child) is correctly recharged rather than skipped.

Why the insert and the gen-compare share ONE hierarchy_lock read hold: the enable's subtree count and its PIDS_TOPOLOGY_GEN bump both run under hierarchy_lock WRITE, so a fork's insert+compare under hierarchy_lock READ is serialized fully-before or fully-after any enable — never interleaved. This closes BOTH failure directions with one interlock: the original 1-count UNDER-count (an enable landing between the count and the insert), AND the mirror 1-count DOUBLE-count that a bare gen-compare admits (the insert landing between the enable's count and its gen bump, so the enable counts the child AND the reconcile recharges it — the global gen cannot attribute a bump to a particular ancestor's count ordering; only the topology lock resolves it). The cost is a concurrent RwLock read-acquire on the fork attach path (reads never block each other, only the rare topology write excludes them — the same discipline Linux's css_set_lock-on-fork uses).

This makes the invariant "a task increments and later decrements exactly the pids counters charged on its behalf, once each" hold across a mid-fork enable, completing the can/cancel symmetry (which CgroupForkCharge already guarantees) with a can/exit symmetry. A mid-fork DISABLE needs no action: the controller is gone and both the charge (can_fork) and the uncharge (exit) skip it via their Some(pids) filters.

Limit change propagation: When a cgroup's cpu.max or cpu.guarantee is changed, the new limit takes effect lazily via generation counters (incremented on write). A child task forked before the limit change may temporarily exceed the new limit for up to one scheduler slice (~4ms at HZ=250). This one-slice transient overshoot is acceptable: the child is re-evaluated at its first scheduler tick. The generation counter is checked on the warm path (wakeup / replenishment), not the hot path (context switch), to avoid per-switch overhead.

Exit path: When a task exits, detach_exiting_task(task) performs the inverse of the precharge_fork() + commit_fork() pair, ensuring zero counter drift over the kernel's lifetime:

detach_exiting_task(task):
  0. Acquire `task.process.threadgroup_rwsem` in READ mode (sleeping
     rwsem, ordering entry 3a — exit_task Step 8a runs in process context
     with no SpinLock held, so sleeping is legal). This serializes the
     charge release against a whole-group `cgroup.procs` migration
     (which write-holds the rwsem, steps 1a-14): an exit then runs
     entirely-before the migration (charges leave the thread's
     pre-migration cgroup, and the migration's step-1c PF_EXITING
     filter excludes the thread) or entirely-after it (task.cgroup =
     target, and the charges leave the target — exactly where migration
     steps 7/8/10/11 put them). Without this serialization, an exit
     interleaved with steps 5-11 double-frees or leaks pids/population
     on one of the two chains.
  0a. Claim the per-task accounting gate — the same CAS trylock the
     migration protocol uses:
       loop {
         if task.cgroup_migration_state
              .compare_exchange(0 /*None*/, 1 /*Migrating*/,
                                Ordering::Acquire, Ordering::Relaxed)
              .is_ok() { break; }
         cond_resched();   // bounded wait, see below
       }
     A CAS failure means a single-thread `cgroup.threads` move of THIS
     task is in flight (it read-holds the rwsem, so step 0 does not
     exclude it). Its Phase 1 + Phase 2 never sleep while the state is
     held (atomics, level-215/RQ_LOCK sections), so the retry loop is
     bounded to microseconds. The claim is TERMINAL: exit never stores
     `None` back — any later migration CAS on this task permanently
     fails, and both migration entry points already treat an exiting
     task as not-migratable (`cgroup.procs`: step-1c PF_EXITING filter;
     `cgroup.threads`: no-op success, threaded-delta list).
  1. let cg = task.cgroup.load_full();  // ArcSwap load of current cgroup
  2. For each subsystem controller attached to the cgroup hierarchy:
       a. controller.exit(task)  // controller-specific cleanup:
          - pids: pids.current.fetch_sub(1, Release) for task's cgroup
            and every ancestor up to the root.
          - memory: uncharge residual kernel memory (see zero-residual
            drain in [Section 17.2](#control-groups--memory-controller-state)).
          - cpu: remove task from CFS bandwidth tracking.
          - io: remove task from blkio weight accounting.
  3. Remove task from the cgroup's task list (write-acquires the
     cgroup's `CGROUP_TASKS_LOCK`, level 215 — legal: no other
     spin-class lock is held here):
     cg.tasks.write().remove(task.task_id());
  4. Update population count (the exact inverse of commit_fork's local
     increment + 0↔1 propagation — NOT a to-root walk). The 1→0 subtree flip
     is claimed by the fetch_sub RETURN VALUE (not a racy is_populated() read):
       let prev = cg.population.fetch_sub(1, Release);
       if prev == 1 {                    // THIS op drove population 1→0
           // Decide the subtree flip under flip_lock via the published
           // `subtree_populated` bit (NOT a bare nr_populated_children read) —
           // a concurrent child edge may have changed nr since the fetch_sub.
           cgroup_reconcile_self_populated(&cg);
       }
     // Only the down-edge task (prev == 1) enters the flip path; any prev > 1
     // is a steady-state exit leaving siblings and returns touching one
     // cacheline. cgroup_reconcile_self_populated republishes the bit and, on a
     // genuine 1→0 flip, notifies ancestors, stopping at the first still-
     // populated ancestor. Zero drift (mirrors post_fork's up-edge claim).
  5. Set task.cgroup.store(root_cgroup);
     // Point to root cgroup, preventing use-after-free if any subsystem
     // accesses the task's cgroup after exit but before Task struct free.
  6. Release `threadgroup_rwsem` (read mode).

17.2.6.2 Taskstats Exit Notification (`record_task_exit`)¶

The per-task-exit statistics message consumed by delay-accounting daemons (iotop, container runtimes). Called from exit_task() Step 10c (Section 8.2), last thread only in the aggregate case; Linux parity: taskstats_exit(tsk, group_dead) in kernel/taskstats.c.

/// A netlink socket's port id (`nlmsg_pid` / `snd_portid`) — the destination
/// the taskstats exit path unicasts a record to. `u32` to match the netlink wire
/// ABI (Linux `__u32 nl_pid`), not the kernel-internal-id `u64` rule: this is an
/// address assigned by the netlink layer (usually the listener's pid, or a
/// kernel-allocated unique port), reused across sockets — never a monotonic
/// counter incremented toward overflow, so no widening is required.
pub struct NetlinkPortId(pub u32);

/// One CPU's taskstats listener list. A daemon that registers for a CPU mask
/// gets one `NetlinkPortId` entry appended to EACH masked CPU's list.
///
/// **Not `PerCpu<T>`**: every access path touches a SPECIFIC, often-remote
/// CPU's list — the register/deregister fan-out writes an arbitrary set of
/// CPUs' lists, and the exit path's dead-listener prune writes the exiting
/// CPU's list from that CPU. `PerCpu<T>` forbids cross-CPU access (its `get()`
/// requires the OWNING CPU's `PreemptGuard`, Ch 3), so this is a plain
/// CPU-indexed array whose entries carry their own lock — the same shape as
/// Linux's `DEFINE_PER_CPU(struct listener_list)` reached via `per_cpu(.., cpu)`.
pub struct ListenerList {
    /// Sleeping RW-semaphore: register/deregister/prune take it in WRITE, the
    /// exit-path snapshot reads under RCU without it. (Sleeping is legal on all
    /// three write paths — genl command context and exit_task process context.)
    pub sem: RwLock<()>,
    /// Netlink port ids listening on this CPU. RCU-readable; writers hold `sem`.
    pub list: RcuList<NetlinkPortId>,
}

/// Registry for the taskstats genetlink family. Boot-sized to
/// `num_possible_cpus()` (NOT online — a listener registered for a CPU that
/// later offlines keeps its entry). Exits on unlistened CPUs cost one atomic
/// load of `nr_listeners`.
pub struct TaskstatsListeners {
    /// One `ListenerList` per possible CPU, indexed by CPU id. Accessed
    /// cross-CPU by index (see `ListenerList` — this is why it is NOT
    /// `PerCpu<ListenerList>`). Boot-allocated `Box<[ListenerList]>`.
    pub by_cpu: Box<[ListenerList]>,
    /// Fast global gate: total registered listeners across all CPUs. 0 →
    /// record_task_exit returns immediately (the common case — one Relaxed load).
    pub nr_listeners: AtomicU32,
}

Registration protocol (the genl "other side") — without this, by_cpu is never populated and the exit path is dead. UmkaOS registers the Linux-ABI genl family and handles its commands (Linux parity: kernel/taskstats.c):

/// Generic-netlink command invocation context (Linux `struct genl_info`), handed
/// to every genl command handler. Carries the sender's netlink port id (the reply
/// destination and the identity a listener registration is keyed on), the parsed
/// command attribute set, and the `NETLINK_GENERIC` socket the reply is unicast
/// on. Borrows the receive buffer for the duration of command dispatch (`'a`).
pub struct GenlInfo<'a> {
    /// Sending socket's netlink port id (`nlmsg_pid` / `snd_portid`).
    pub snd_portid: NetlinkPortId,
    /// Parsed command attributes (`TASKSTATS_CMD_ATTR_*`), read via
    /// `NlAttrSet::get` / `get_u32`. Zero-copy view over the request buffer.
    pub attrs: NlAttrSet<'a>,
    /// The `NETLINK_GENERIC` socket that received the command; the one-shot
    /// `_PID`/`_TGID` GET reply is unicast on it.
    pub sock: &'a NetlinkSocket,
}

/// genl family: name `TASKSTATS_GENL_NAME`, version `TASKSTATS_GENL_VERSION`,
/// registered once at boot ([Section 16.17](16-networking.md#netlink-socket-interface) NETLINK_GENERIC).
/// Command `TASKSTATS_CMD_GET` (CAP_SYS_ADMIN) carries exactly one of:
///   - TASKSTATS_CMD_ATTR_PID / _TGID       → one-shot stats for a task/group
///   - TASKSTATS_CMD_ATTR_REGISTER_CPUMASK  → add   this port to a CPU set
///   - TASKSTATS_CMD_ATTR_DEREGISTER_CPUMASK→ remove this port from a CPU set
pub fn taskstats_cmd_get(info: &GenlInfo) -> Result<(), Errno> {
    // REGISTER/DEREGISTER path — Linux add_del_listener():
    // 1. Parse the cpumask attribute; reject bits outside cpu_possible_mask.
    // 2. Require the init user AND init pid namespace (Linux restriction —
    //    listeners are host-global; a container cannot register). EPERM else.
    // 3. For each CPU in the mask, take by_cpu[cpu].sem.write():
    //      register:   dedup on info.snd_portid, then push the port (RcuList
    //                  write under sem); nr_listeners.fetch_add(1, AcqRel) per
    //                  newly-added (cpu, port).
    //      deregister: remove the matching port; nr_listeners.fetch_sub(1,...).
    // The port id is info.snd_portid (the sender's netlink port), stored so
    // the exit path can unicast to it.
    // (The one-shot _PID/_TGID GET arm assembles the record like the exit path
    //  and replies to the caller directly — no listener list involved.)
}

Mutation sources (contradiction resolved): by_cpu[*].list is mutated on TWO paths, both under by_cpu[cpu].sem.write() — the earlier "mutation only via genl commands" wording was wrong: 1. genl register/deregister (above) — cross-CPU fan-out over the mask. 2. exit-path prune — in record_task_exit, a listener whose unicast fails with ECONNREFUSED (daemon exited) is removed from the EXITING CPU's list and nr_listeners decremented, matching Linux send_cpu_listeners() marking valid = 0 then deleting under the write lock. This is a exit_task-context mutation of the local CPU's list — legal because it takes that list's sem (a real writer lock), NOT a PerCpu::get owning-CPU fast path.

/// Emit the exit statistics record. `group_dead` is supplied by exit_task
/// (the just-decremented `signal.live` reached 0 — matching Linux
/// Linux `taskstats_exit(tsk, group_dead)`), so this function does NOT re-test
/// `thread_group.count` and is immune to the count-decrement ordering.
/// 1. Fast path: `nr_listeners == 0` → return (no allocation, no locks).
/// 2. Assemble the Linux-ABI `struct taskstats` record (genl family
///    TASKSTATS_GENL_NAME, versioned TASKSTATS_GENL_VERSION; native-endian —
///    netlink is a local, same-host ABI, not a cross-node wire format)
///    from task.rusage (CPU times, minor/major faults, I/O bytes) and
///    the per-task delay accumulators in `task.delays`
///    ([Section 8.8](08-process.md#resource-limits-and-accounting)): `blkio_delay_ns` (exists),
///    plus `cpu_delay_ns` / `swapin_delay_ns` / `freepages_delay_ns`
///    (reclaim) — the three delay fields this record needs that
///    create_task/exit_task must maintain (cross-file field additions on Task,
///    see the aggregate note below).
/// 3. Fold this thread's stats into the process aggregate, then — if
///    `group_dead` — emit the TASKSTATS_TYPE_AGGR_TGID record from that
///    aggregate (see `TaskstatsAggregate`).
/// 4. RCU-snapshot the exiting CPU's listener list (`by_cpu[this_cpu].list` is
///    an `RcuList` — a lock-free RCU read, NO sem, matching the `ListenerList.sem`
///    doc's "exit-path snapshot reads under RCU without it") and unicast to each
///    listener. A send failure (ECONNREFUSED = daemon gone) then re-takes
///    `by_cpu[this_cpu].sem.write()` to PRUNE that listener and decrement
///    `nr_listeners` — the write path of the "mutation sources" note. So the
///    common all-alive exit holds no sem at all; only an actual prune takes it.
///    (A prune racing a concurrent RCU snapshot is safe: the snapshot either
///    sees the listener and gets ECONNREFUSED itself, or misses it — both benign.)
/// Cold path; may allocate (netlink skb) — exit_task task context only.
pub fn record_task_exit(task: &Task, group_dead: bool);

Process-level aggregate (TASKSTATS_TYPE_AGGR_TGID) — the per-group sum "accumulated in Process as threads exited" needs a real accumulator; none existed. Define it and its fold point:

/// Per-thread-group taskstats accumulator, folded at EACH thread's exit and
/// emitted once when the last thread exits. Home: the `Process` struct
/// (thread-group leader's `signal`-equivalent), lazily boxed on first use —
/// matching Linux `signal_struct.stats` / `taskstats_tgid_alloc()`.
///
/// **Field addition (cross-file handoff)**: `Process` gains
/// `taskstats_agg: OnceCell<Box<TaskstatsAggregate>>`
/// ([Section 8.1](08-process.md#process-and-task-management)); `task.delays` gains
/// `cpu_delay_ns`, `swapin_delay_ns`, `freepages_delay_ns`
/// ([Section 8.8](08-process.md#resource-limits-and-accounting) — `blkio_delay_ns` already present).
pub struct TaskstatsAggregate {
    /// Serializes the fold against concurrent thread exits in the same group.
    /// Held only for the fold (a few field adds) — the exiting thread's
    /// `sighand`/`siglock`-class lock, matching Linux's `sighand->siglock`
    /// protection of `sig->stats`.
    pub lock: SpinLock<TaskstatsAggFields>,
}

/// Summed fields: CPU/user/system time, min/maj faults, I/O bytes, and the
/// four delay accumulators — the same members `struct taskstats` exposes.
pub struct TaskstatsAggFields { /* u64 sums, mirrors struct taskstats numeric fields */ }

/// Fold one exiting thread into its group aggregate. Called from record_task_exit
/// step 3 (and, so the aggregate is complete even with no listeners, whenever
/// delay accounting is enabled). Add-only; no reset — the aggregate is dropped
/// with the Process when the group is reaped.
fn taskstats_fold_thread(agg: &TaskstatsAggregate, task: &Task);

Zero-residual drain (memory controller): When the last task exits a cgroup, residual kernel memory charges (slab objects, page tables, socket buffers) may still reference the cgroup's mem_cgroup. These are drained asynchronously: the cgroup transitions to a "dying" state where new charges are rejected but existing charges are allowed to drain naturally as their owning objects are freed. The mem_cgroup is freed only when charge_count reaches zero. A watchdog timer logs a warning if a dying cgroup's charges do not drain within 60 seconds (potential leak).

17.2.7 Cpuset Controller (CPU and NUMA Pinning)¶

cpuset.cpus: CPUs allowed for tasks in this cgroup. Format: "0-3,8-11" (CPU list).
cpuset.mems: NUMA nodes allowed for memory allocation. Format: "0,2" (node list).
cpuset.cpus.partition: Partition mode (root, member, isolated). Isolated partitions have exclusive CPU access.

The scheduler respects cpuset constraints when selecting a CPU for a task. NUMA-aware allocation (Section 4.1) respects the cpuset.mems mask.

17.2.7.1 Cpuset Write Path and Hotplug¶

Migration-time application of a cpuset (moving a task INTO a cgroup) is handled by migration step 15. This section specifies the two paths migration does NOT cover: changing a populated cgroup's cpuset.cpus in place, and CPU hotplug.

cpuset.cpus / cpuset.mems write handler (cpuset_cpus_write):

/// cgroupfs write handler for `cpuset.cpus`. Recomputes the effective CPU set
/// and re-homes tasks already in the cgroup that no longer fit — the case
/// migration step 15 does not cover (it applies a cpuset only at attach time).
pub fn cpuset_cpus_write(cgroup: &Arc<Cgroup>, buf: &[u8]) -> Result<usize, Errno> {
    let new_mask = parse_cpu_list(buf)?;                 // EINVAL on bad list
    // Acquire this cgroup's OWN `config_lock` BEFORE opening the RCU read
    // section. It is the `RcuCell`/`RcuPtr` writer proof for the `cs.allowed_cpus`
    // publish below and serializes concurrent `cpuset.cpus` writers to THIS
    // cgroup. It does NOT exclude a concurrent `-cpuset` disable:
    // `child_controller_free` runs under the PARENT's `config_lock` (a DIFFERENT
    // `Mutex` instance) and passes that guard as its `proof`. What keeps the
    // borrowed `cs` controller live for the whole handler — across the
    // `allowed_cpus` publish AND the task re-homing — is RCU deferred
    // reclamation: a concurrent disable's `update(None, ..)` reclaims the old
    // `CpusetController` box only after a grace period, i.e. after this read
    // section ends, so `cs` cannot dangle. (A racing disable makes the
    // `allowed_cpus` publish a benign lost update into a controller being torn
    // down anyway.)
    let cfg = cgroup.config_lock.lock();
    // Compute the effective set BEFORE the RCU read section: `cpu_online_mask()`
    // and `parent_effective_cpus()` allocate `CpuMaskBuf`s
    // ([Section 9.1](09-security.md#capability-based-foundation)), which is ILLEGAL under `rcu_read_lock`
    // (the preempt-disable read side). Both BOUNDS may exceed 128 CPUs, so they
    // are `CpuMaskBuf`s; `effective` itself is a Copy of `new_mask` (already sized
    // for the system by `parse_cpu_list`'s `empty_system()` — pool-backed above 128
    // CPUs), so it stays a plain `CpuMask` and the narrowing below allocates
    // nothing (Copy + in-place bit clears).
    // effective = requested ∩ parent.effective ∩ online — narrow `new_mask` by
    // testing each set bit against both bounds.
    let online = cpu_online_mask().ok_or(Errno::ENOMEM)?;
    let parent_eff = parent_effective_cpus(cgroup).ok_or(Errno::ENOMEM)?;
    let mut effective = new_mask;                        // CpuMask (Copy)
    let mut c = effective.first_set();
    while let Some(cpu) = c {
        if !online.test(cpu) || !parent_eff.test(cpu) {
            effective.clear(cpu);
        }
        c = effective.next_set(cpu + 1);
    }
    // Empty is rejected for a populated cgroup (Linux -ENOSPC), permitted for an
    // empty one.
    if effective.is_empty() && cgroup.is_populated() {
        return Err(Errno::ENOSPC);
    }
    // Publish the new mask under a SHORT RCU section (the `cpuset` `RcuPtr` read +
    // the `RcuCell::update` publish are the only steps that need it), then DROP the
    // guard BEFORE any task re-homing: `tasks.read()` is a sleeping `RwLock`
    // (`CGROUP_TASKS_LOCK(215)`) and `cpu_online_mask()`/`parent_effective_cpus()`
    // allocate `CpuMaskBuf`s — both are illegal under `rcu_read_lock()` (the
    // preempt-disable read side, tick-reported via a CpuLocal nesting counter).
    {
        let guard = rcu_read_lock();
        let cs = cgroup.cpuset.read(&guard).ok_or(Errno::ENODEV)?;
        cs.allowed_cpus.update(new_mask, &cfg)?;         // publish (RcuCell)
    } // guard dropped — no RCU section held across the sleeping re-home below
    // Re-home tasks that are now on an excluded CPU — the SAME machinery
    // migration step 15 uses (per-thread RQ_LOCK; queued task moved via
    // select_task_rq; running task deferred to its context-switch boundary via
    // resched IPI). Narrowing THIS cgroup's mask also narrows every DESCENDANT's
    // effective set (a descendant's effective = its requested ∩ parent.effective
    // ∩ online), so descendants' tasks can now be on an excluded CPU too — they
    // must be re-homed as well, exactly as the hotplug path walks "every cgroup".
    // A recompute-effective-then-rehome subtree walk (Linux `update_cpumasks_hier()`).
    // First THIS cgroup's tasks against `effective` (computed above), then every
    // descendant via the file's own `for_each_descendant` method (pre-order,
    // depth-bounded, acquires its own RCU guard) — each descendant's effective
    // set is `its allowed_cpus ∩ parent.effective ∩ online`.
    for tid in cgroup.tasks.read().iter() {
        apply_cpuset_affinity(tid, &effective);          // [Section 17.2](#control-groups--task-migration-cgroupprocs-write) step 15 machinery
    }
    // `for_each_descendant` holds ONE RCU read section for the whole DFS, so its
    // callback must not sleep, allocate, or take a sleeping lock (the `tasks`
    // `RwLock` at `CGROUP_TASKS_LOCK(215)`) — see its contract at
    // [Section 17.2](#control-groups--core-data-structures). GROWING the `rehome` worklist is
    // an allocation, so it MUST NOT happen inside the callback (a nested
    // `rcu_read_lock()` does not lift the walk's outer guard). The Vec is therefore
    // capacity-reserved OUTSIDE the walk, and the callback does only RCU-safe work:
    // `Arc::clone` each descendant, snapshot its `allowed` mask (a `Copy` `CpuMask`),
    // and push into the reserved spare capacity — a `push` with `len < capacity`
    // never reallocates. If a walk fills the reservation (the subtree grew, or the
    // initial hint was low), the callback signals overflow and breaks; the Vec is
    // then regrown OUTSIDE the RCU section and the walk is retried from scratch.
    // Termination: the subtree is finite (depth-bounded by `CGROUP_MAX_DEPTH`) and
    // the capacity doubles each retry, so a walk completes within capacity after a
    // bounded number of retries. The sleeping per-descendant re-home — the
    // `CpuMaskBuf` bound allocations AND `d.tasks.read()` — runs AFTER the final
    // walk returns, on the owned snapshot, with no RCU held. Cold path (a cgroupfs
    // `cpuset.cpus` write): a `Vec` worklist is acceptable per
    // [Section 3.13](03-concurrency.md#collection-usage-policy).
    let mut rehome: Vec<(Arc<Cgroup>, CpuMask)> = Vec::new();
    // Initial capacity hint (a small subtree is the common case); the
    // regrow-and-rewalk loop below makes correctness independent of it — an
    // undersized guess only costs an extra walk.
    const REHOME_HINT: usize = 16;
    rehome.reserve(REHOME_HINT);
    loop {
        rehome.clear();                 // reuse the reserved allocation across retries
        let mut overflowed = false;
        cgroup.for_each_descendant(|d| {
            // Nested short read guard (RCU nesting under the walk's guard is legal):
            // read the descendant's `cpuset` `RcuPtr` and copy its `allowed` mask
            // (`Copy`, pool-backed above 128 CPUs — no allocation). Skip descendants
            // with no cpuset controller.
            let g = rcu_read_lock();
            if let Some(dcs) = d.cpuset.read(&g) {
                let entry = (Arc::clone(d), *dcs.allowed_cpus.read(&g));
                // Non-allocating push: only into reserved spare capacity. At
                // capacity, signal overflow and break — the regrow happens OUTSIDE
                // this RCU section, never inside the walk.
                if rehome.len() < rehome.capacity() {
                    rehome.push(entry);
                } else {
                    overflowed = true;
                    return core::ops::ControlFlow::Break(());
                }
            }
            core::ops::ControlFlow::Continue(())
        });
        if !overflowed {
            break;
        }
        // OUT of RCU (the walk has returned): double the reservation, then re-walk.
        rehome.reserve(rehome.capacity());
    }
    // Now OUT of RCU: narrow each descendant's snapshot against the live
    // online/parent-effective bounds and re-home its tasks. `d.tasks.read()`
    // (sleeping) and the `CpuMaskBuf` bound allocations are legal here.
    for (d, allowed) in &rehome {
        let mut eff = *allowed;                          // CpuMask (Copy)
        // ENOMEM on either bound → skip re-homing this descendant THIS pass
        // (best-effort; the next hotplug/cpuset event retries) — never abort the
        // whole subtree.
        if let (Some(online), Some(parent_eff)) =
            (cpu_online_mask(), parent_effective_cpus(d))
        {
            let mut c = eff.first_set();
            while let Some(cpu) = c {
                if !online.test(cpu) || !parent_eff.test(cpu) {
                    eff.clear(cpu);
                }
                c = eff.next_set(cpu + 1);
            }
            for tid in d.tasks.read().iter() {
                apply_cpuset_affinity(tid, &eff);
            }
        }
    }
    // Without the descendant walk, a CPU-bound task in a child cgroup would run
    // forever on a CPU the parent just excluded — violating the section's own
    // "never CONTINUES running on a CPU outside its cgroup's allowed_cpus beyond
    // the resched-IPI window" contract for that descendant.
    Ok(buf.len())
}

/// Parse a cpuset CPU-list string (`cpuset.cpus` syntax) into a `CpuMask`.
/// Comma-separated single ids or `lo-hi` ranges, e.g. `"0-3,8,10-11"`, with
/// surrounding whitespace / a trailing newline tolerated. An empty write clears
/// the mask (inherit the parent). The returned mask is sized for the running
/// system via `CpuMask::empty_system()` ([Section 9.1](09-security.md#capability-based-foundation)), so it
/// addresses every possible CPU — there is no 128-CPU cap. Any malformed field, an
/// inverted range, or a CPU id `>= num_possible_cpus()` → `Errno::EINVAL` (never
/// silent truncation).
fn parse_cpu_list(buf: &[u8]) -> Result<CpuMask, Errno> {
    let s = core::str::from_utf8(buf).map_err(|_| Errno::EINVAL)?.trim();
    // Runtime construction site: `empty_system()` (NOT the `const empty()`) so a
    // system with more than `CPU_MASK_INLINE_WORDS * 64` CPUs gets a pool-backed
    // mask that can address CPUs 128+ ([Section 9.1](09-security.md#capability-based-foundation)). This is
    // a `cpuset.cpus` write — a cold/admin config path — so the pool-backed branch
    // obeys `pool_alloc`'s kernel-lifetime contract.
    let mut mask = CpuMask::empty_system();
    if s.is_empty() {
        return Ok(mask); // empty → inherit parent (all bits clear)
    }
    for field in s.split(',') {
        let (lo, hi) = match field.split_once('-') {
            Some((a, b)) => (
                a.trim().parse::<u32>().map_err(|_| Errno::EINVAL)?,
                b.trim().parse::<u32>().map_err(|_| Errno::EINVAL)?,
            ),
            None => {
                let v = field.trim().parse::<u32>().map_err(|_| Errno::EINVAL)?;
                (v, v)
            }
        };
        if lo > hi {
            return Err(Errno::EINVAL); // inverted range
        }
        for cpu in lo..=hi {
            // Reject any CPU id the running system cannot have. The ceiling is the
            // boot-discovered CPU count (`num_possible_cpus()`,
            // [Section 2.3](02-boot-hardware.md#boot-init-cross-arch)) — NOT a hardcoded 128: `empty_system()`
            // sized `mask` for this system (pool-backed above 128), so `set(cpu)`
            // no longer truncates at the inline bound. `CpuMask::set` still returns
            // false past the mask's word capacity (defence in depth); either check
            // rejects the id rather than silently dropping the bit.
            if cpu >= num_possible_cpus() as u32 || !mask.set(cpu) {
                return Err(Errno::EINVAL);
            }
        }
    }
    Ok(mask)
}

/// The PARENT cgroup's effective CPU set as an owned `CpuMaskBuf`
/// ([Section 9.1](09-security.md#capability-based-foundation)) — the upper bound on `cgroup`'s own
/// effective set (Linux `effective_cpus` propagation). `None` on allocation
/// failure (callers propagate `ENOMEM`). Root (no parent) or a parent with no
/// configured cpuset → the online set (which may exceed 128 CPUs, hence the
/// `CpuMaskBuf`). A parent whose cpuset is empty inherits transitively, so
/// recurse to the grandparent.
fn parent_effective_cpus(cgroup: &Arc<Cgroup>) -> Option<CpuMaskBuf> {
    // Resolve the parent UNDER RCU, then DROP the guard before allocating the
    // online `CpuMaskBuf` (alloc is illegal under the preempt-disable RCU read
    // side); the `Arc<Cgroup>` clone keeps the parent live past the guard.
    let parent = {
        let guard = rcu_read_lock();
        match cgroup.parent.as_ref().and_then(|w| w.upgrade()) {
            Some(p) => p,
            None => { drop(guard); return cpu_online_mask(); } // root: online set
        }
    };
    // Snapshot the parent's cpuset decision (Copy) UNDER RCU, drop the guard
    // before the online-mask allocation. Encoding: `Some(Some(req))` = non-empty
    // cpuset (bound by `req`); `Some(None)` = empty cpuset (inherit from the
    // grandparent); `None` = no cpuset controller (bound only by online).
    let decision: Option<Option<CpuMask>> = {
        let guard = rcu_read_lock();
        match parent.cpuset.read(&guard) {
            Some(pcs) => {
                let requested = pcs.allowed_cpus.read(&guard);
                if requested.is_empty() { Some(None) } else { Some(Some(*requested)) } // CpuMask: Copy
            }
            None => None, // parent has no cpuset controller enabled
        }
    };
    match decision {
        None => cpu_online_mask(),                         // bound only by online
        Some(None) => parent_effective_cpus(&parent),      // empty cpuset inherits
        Some(Some(req)) => {
            let mut eff = cpu_online_mask()?; // online CpuMaskBuf
            eff.intersect_with(&req);          // ∩ parent.allowed
            Some(eff)
        }
    }
}

/// Apply an effective cpuset mask to ONE already-attached thread — the
/// per-thread machinery of cgroup-migration step 15
/// ([Section 17.2](#control-groups--task-migration-cgroupprocs-write)), reused by in-place
/// `cpuset.cpus` writes and by CPU hotplug. A thread that exited between the
/// task-set snapshot and here is simply skipped (its slot is already gone).
fn apply_cpuset_affinity(tid: TaskId, effective: &CpuMask) {
    let task = match find_task_by_tid(tid) {
        Some(t) => t,
        None => return,
    };
    // Honor a `sched_setaffinity(2)` request when it still intersects the new
    // effective set (Linux `user_cpus_ptr` restoration); otherwise the cpuset's
    // effective mask stands alone.
    let applied = match &task.user_cpu_affinity {
        Some(u) if !u.intersection(effective).is_empty() => u.intersection(effective),
        _ => effective.clone(),
    };
    // Publish `cpu_affinity` + `nr_cpus_allowed` together under the thread's
    // RQ_LOCK. `lock_task_rq()` resolves the owning CPU from `task.cpu_id` and
    // revalidates it after acquiring; the held lock provides the exclusive
    // access these two co-updated fields require. The guard derefs to
    // `&mut RunQueueData`.
    let mut rq = lock_task_rq(&task);
    task.cpu_affinity = applied.clone();
    task.nr_cpus_allowed = applied.count();
    // If the thread's current CPU is now excluded it must leave it. A queued
    // thread is moved immediately: `deactivate_task` off the source runqueue,
    // `select_task_rq` (which honors the just-published mask) picks the
    // destination, `activate_task` on it — the destination runqueue lock is
    // acquired in CPU-ID order (the work-stealing convention) to avoid deadlock.
    // A thread currently RUNNING on the excluded CPU is deferred to its next
    // context-switch boundary by an eager reschedule IPI (registers are only
    // provably saved there); `resched_curr` on the source runqueue drives it.
    if !applied.test(task.cpu_id.load(Acquire)) {
        deactivate_task(&mut rq, &task, DequeueFlags::DEQUEUE_SAVE);
        let dst = select_task_rq(&task);
        let mut drq = dst.lock.lock();
        activate_task(&mut drq, &task, EnqueueFlags::ENQUEUE_MIGRATED);
        resched_curr(&mut rq, ReschedUrgency::Eager);
    }
}

cpuset.mems is analogous but has no running-task migration (v2 has no memory_migrate; only future allocations follow the new mask — see CpusetController.mem_migrate). cpuset.cpus.partition writes update CpusetController.partition under config_lock and, for root/isolated, carve an exclusive CPU slice from the parent partition (rejecting overlap with sibling partitions → EINVAL/EBUSY).

Hotplug: when a CPU goes offline, the cpuset hotplug handler recomputes every cgroup's effective set (allowed_cpus ∩ online_cpus) and re-homes tasks on the departed CPU via the same apply_cpuset_affinity path; if a cgroup's effective set becomes empty, its tasks fall back to the nearest ancestor with a non-empty effective set (mem_migrate gates whether their memory follows).

17.2.8 Freezer (Cgroup Pause/Resume)¶

cgroup.freeze: Write 1 to freeze all tasks in the cgroup subtree; write 0 to thaw.
cgroup.events: Contains frozen 0/1 indicating current frozen state.

Frozen tasks are removed from the run queue and cannot be scheduled. Used by docker pause and checkpoint/restore.

Freeze/thaw algorithm:

Freeze initiation: Write 1 to cgroup.freeze.
The handler takes the cgroup's config_lock to serialize concurrent freeze/thaw writers (UmkaOS has no global cgroup configuration mutex — the design eliminates Linux's global cgroup mutex; the cpu.weight/subtree_control/ cpuset handlers make the same statement), then walks the cgroup's task list under CGROUP_TASKS_LOCK (read) — the same lock cgroup.procs enumeration uses.
For each task: set the TaskState::FROZEN (0x0000_8000) bit in task.state while preserving its other state bits. If the task is currently on a CPU, send a reschedule IPI; the task observes the frozen bit at the next preemption point or syscall return and calls schedule(), where the scheduler sees TaskState::FROZEN and does not re-enqueue.
For tasks already in INTERRUPTIBLE/UNINTERRUPTIBLE sleep: set TaskState::FROZEN directly. The task remains sleeping; the scheduler will not select it.
Report an RCU quiescent state for each frozen task's CPU (implicit quiescent state — frozen tasks cannot hold RCU read locks).
Set cgroup.events.frozen = 1 after all tasks have entered TaskState::FROZEN.
Wakeup-during-freeze: If a wakeup event arrives for a TaskState::FROZEN task (e.g., I/O completion, timer expiry), the task remains frozen. The wakeup is recorded as a pending wakeup flag so the task is immediately schedulable when thawed.
Thaw: Write 0 to cgroup.freeze.
Kernel walks the cgroup's task list.
For each task: clear the TaskState::FROZEN bit. If the task has a pending wakeup, call try_to_wake_up() to enqueue it on the appropriate run queue with its preserved vruntime and lag.
A task in OnRqState::Deferred (EEVDF lag compensation) is removed from the EEVDF tree during freeze, transitioning to OnRqState::Off + TaskState::FROZEN. On thaw, it re-enters as Deferred (not Queued) to preserve EEVDF fairness -- the task's lag value is restored so it does not gain unfair scheduling advantage from being frozen.
Set cgroup.events.frozen = 0.
Nested freeze: A cgroup can be frozen by both its own freeze field and an ancestor cgroup's freeze field. The effective frozen state (e_freeze) is propagated downward: child.e_freeze = child.freeze || parent.e_freeze. A task remains frozen until ALL ancestor cgroups are thawed AND the task's own cgroup has freeze = false.
Fatal signals while frozen: Processes in TaskState::FROZEN can be killed by SIGKILL. The signal delivery path checks for TaskState::FROZEN and transitions the task to TaskState::RUNNING to allow exit processing. This ensures kill -9 always works regardless of cgroup freeze state.

RCU Interaction. Frozen tasks cannot execute code and therefore cannot report RCU quiescent states. To prevent RCU grace periods from blocking indefinitely, UmkaOS's RCU subsystem treats entry into TaskState::FROZEN as an implicit quiescent state: the cgroup freezer uses the task-exit reporting path to report a quiescent state on behalf of the frozen task's CPU at the moment the task is frozen. This is safe because a frozen task holds no RCU read-side critical sections — it is not executing, so it cannot be inside rcu_read_lock(). When the task is thawed it re-enters the normal quiescent-state reporting cycle with no special handling required. This design ensures that container pause/resume, whole-cgroup SIGSTOP, and checkpoint-restore operations never stall RCU grace periods regardless of freeze duration.

CBS timer interaction: When a cgroup enters the Frozen state, all CBS replenishment timers for tasks in that cgroup are cancelled (hrtimer_cancel). When the cgroup thaws, CBS timers are re-armed with fresh budgets — the throttled flag is cleared and budget is set to the full CBS period to avoid spurious throttling on resume.

17.2.8.1 Freeze and Network Interaction¶

In-flight TX packets: When a cgroup is frozen, VETH TX packets already enqueued in the peer's RX ring are delivered normally. The receiving side processes them. If the receiver is also frozen, the delivery wakeup is recorded as pending.
TCP mid-send: If a task is frozen during sendmsg():
Partially-queued data remains in the socket send buffer (not discarded).
TCP retransmit timers continue. Timer expiry wakeups are recorded as pending.
On thaw, the task resumes. Retransmit timer may have expired; TCP handles this via standard timeout/retransmit.
UDP receives: Queued in socket receive buffer. Wakeup recorded as pending. On thaw, data is available immediately.
VETH cross-namespace freeze: Packets flow in both directions regardless of freeze state. The freeze is per-cgroup-per-task, not per-interface. VETH pair connectivity is unaffected; only task scheduling is frozen.
Cleanup on cgroup destroy after freeze: If a frozen cgroup is destroyed, all frozen tasks receive SIGKILL. Socket close triggers TCP RST/FIN. No packet leaks.

17.2.8.2 Cgroup Zero-Residual Destruction¶

17.2.8.2.1 Problem¶

When rmdir removes a cgroup directory, the cgroup's population is zero (no tasks), but residual resource charges may remain:

Memory controller: Pages on LRU lists still charged to the cgroup's MemCgroup::usage. These pages were allocated by tasks that have since migrated out or exited, but the page charge persists until the page is freed (reclaimed, munmap'd, or process exits).
I/O controller: In-flight block I/O requests tagged with the cgroup's ID. These drain naturally when I/O completes (~10 ms typical).
Hugetlb controller: Huge pages still charged. These may persist indefinitely if a process in another cgroup holds a mapping to a page originally charged here.

In Linux, these zombie cgroups persist until all charged pages are freed, which may be never (a process in cgroup B holding a page allocated while it was in cgroup A keeps cgroup A's kernel metadata alive). Over 50-year uptime, this leaks ~2-8 KB of kernel memory per zombie cgroup. At 100 cgroup creates/destroys per hour (typical Kubernetes pod churn), this accumulates ~875 MB over 50 years — significant on memory-constrained systems.

17.2.8.2.2 Design: Bounded Residual Drain¶

UmkaOS enforces a zero-residual invariant: after rmdir, all residual resource charges are drained within a bounded deadline. The cgroup's kernel memory (Arc<Cgroup> and all controller state) is freed within this deadline, guaranteed.

/// Drain timeout for residual memory charges after cgroup rmdir, in ms.
/// Runtime-writable backing storage — a `const` cannot back the promised
/// `/proc/sys/kernel/cgroup_drain_timeout_ms` tunable. Registered with the
/// typed kernel-parameter store ([Section 20.9](20-observability.md#kernel-parameter-store)) so writes go
/// through the range-validated setter; the drain protocol reads it with a
/// single `.load(Relaxed)`.
static CGROUP_MEM_DRAIN_TIMEOUT_MS: AtomicU64 = AtomicU64::new(30_000);

kernel_param! {
    name:          "kernel.cgroup_drain_timeout_ms",
    schema:        ParamSchema::U64 { min: 1_000, max: 300_000, default: 30_000 },
    description:   "Deadline (ms) for draining residual charges after cgroup rmdir before force-reparent.",
    privileged:    true,
    per_namespace: false,   // host-global (kernel.* scope)
    getter:        || ParamValue::U64(CGROUP_MEM_DRAIN_TIMEOUT_MS.load(Ordering::Relaxed)),
    setter:        |v| {
        // The store's schema already clamped/range-checked to [1000, 300000].
        if let ParamValue::U64(ms) = v {
            CGROUP_MEM_DRAIN_TIMEOUT_MS.store(ms, Ordering::Release);
            Ok(())
        } else {
            Err(ParamError::TypeMismatch)
        }
    },
}

/// Lifecycle state machine for cgroup destruction.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
#[repr(u8)]
pub enum CgroupLifecycle {
    /// Normal operation. Tasks may be present.
    Active = 0,
    /// `rmdir` called, population == 0, draining residual charges.
    /// No new tasks can be migrated into this cgroup (`cgroup.procs`
    /// write returns ENODEV; mkdir in parent does not find it).
    Draining = 1,
    /// All charges drained (or force-drained). Kernel memory freed.
    Dead = 2,
}

17.2.8.2.3 Drain Protocol¶

The drain runs as a kworker task (cgroup_drain_residual) triggered after rmdir confirms population == 0:

cgroup_drain_residual(cgroup):
  Precondition: population == 0, lifecycle == Draining.

  Phase 0 — the per-CPU MemCgroupStock flush already ran at rmdir step 5
    (Draining transition), so no remote stock still hides bytes from Phase 5.

  Phase 1 — REPARENT memory charges to the parent cgroup:
    Resolve both memcgs by Arc-cloning them out under a SHORT RCU guard, then
    DROP the guard BEFORE the walk (memory is RcuPtr<Arc<MemCgroup>>; a reader
    loads `&Arc<MemCgroup>` under the guard, `Arc::clone`s it, and drops the
    guard — the `memory` field-doc pattern). The page-lock-taking walk below runs
    for up to ~12 ms and MUST NOT hold an RCU read section (this file's own
    no-sleep-in-RCU rule); the owned `Arc` clones keep both memcgs alive without
    one:
      let (memcg, parent_memcg) = {
          let g = rcu_read_lock();
          let memcg = Arc::clone(
              self.memory.read(&g).expect("memory controller present on a charged cgroup"));
          let parent_memcg = Arc::clone(
              parent.memory.read(&g).expect("parent has memory enabled"));
          (memcg, parent_memcg)
      }; // guard dropped here — the walk below holds no RCU read section
    For each NUMA node's `memcg.lru_gen[node]` (the per-node MGLRU state):
      walk its generation folio lists; for each page:
        parent_memcg.usage.fetch_add(PAGE_SIZE, Relaxed)
        memcg.usage.fetch_sub(PAGE_SIZE, Relaxed)
        page.mem_cgroup = parent   // Under page lock; relink onto parent's
                                   // lru_gen[node]
    Bounded: at most `memcg.usage / PAGE_SIZE` pages.
    At 1 GB residual (extreme): ~256K pages, ~12 ms wall time
    (50 ns per page: one fetch_add + one fetch_sub + one pointer store).

  Phase 2 — hugetlb is NOT reparented (RESIDUAL-tracked, tombstone model).
    Huge folios are not on `lru_gen` and UmkaOS keeps no per-cgroup huge-folio
    list, so there is no enumeration by which to reparent them — and reparenting
    the COUNTER while leaving the folios' stored `CgroupId` anchor pointing here
    would double-subtract at their eventual `hugetlb_uncharge`. Instead, each
    huge-folio charge bumped `residual_refs` (see the residual-charge mechanism
    below); the folios keep their anchor, and a later `hugetlb_uncharge` resolves
    the registry TOMBSTONE and `fetch_sub`s this (Dead) cgroup's `hugetlb.usage`,
    dropping `residual_refs`. This is the "lazy uncharge-time redirect" mechanism
    the two prior candidates (page-walk reparent / eager counter transfer) could
    not provide. The cgroup is freed when the last such folio is uncharged (see
    Phase 6). No work in this phase beyond confirming hugetlb residuals are
    accounted in `residual_refs`.

  Phase 3 — DETACH all BPF programs attached to the cgroup.
    Detach BPF_CGROUP_INET_INGRESS, BPF_CGROUP_INET_EGRESS,
    BPF_CGROUP_INET_SOCK_CREATE, BPF_CGROUP_DEVICE, BPF_CGROUP_SYSCTL,
    BPF_CGROUP_GETSOCKOPT, BPF_CGROUP_SETSOCKOPT.
    Precondition: task list is empty (population == 0).
    Programs are detached BEFORE releasing memory charges (Phase 5)
    to prevent BPF callbacks from accessing freed cgroup state.
    Detach is O(n) in number of attached programs (typically 0-3).

  Phase 3.5 — FREE registered-subsystem state (dyn_subsys):
    For each non-null slot in `self.dyn_subsys`, call the subsystem's
    `free_state` (from `CGROUP_SUBSYS_OPS[id]`), which drains the subsystem's
    per-cgroup state (e.g. MlPolicyCss overrides, accel limits) and frees it
    after an RCU grace period, then null the slot. Runs after the task list is
    empty (population == 0) so no walker can still resolve a slot to live state.

  Phase 4 — WAIT (bounded) for in-flight I/O to complete — an OPTIMIZATION so
    the common case frees the cgroup promptly, NOT a correctness gate.
    Predicate (O(1)): { let g = rcu_read_lock();
      self.io.read(&g).map_or(true, |io| io.inflight_bios.load(Acquire) == 0) }.
    `IoController.inflight_bios` is the per-cgroup in-flight bio COUNTER
    maintained by the block layer — incremented when a bio is enrolled in its
    device's `BlkInflightTable` and decremented when it is removed at
    completion ([Section 15.2](15-storage.md#block-io-and-volume-management--per-device-in-flight-bio-table);
    the same two hooks that carry `bio.cgroup_id`). Each enrolled bio also bumped
    `residual_refs`; its completion decrement resolves the registry TOMBSTONE
    (which rmdir step 4b deliberately did NOT erase) — so the decrement reaches
    the right counter even after rmdir, and the predicate CAN reach 0. The drain
    kworker polls this single atomic — it does NOT scan every device's inflight
    table, which is racy (`capture_all()` is crash-path-only) and O(devices ×
    shards). Loop with a bounded backoff until the counter is 0 or the Phase-7
    deadline elapses; on timeout the in-flight bios stay `residual_refs`-tracked
    and complete lazily (Phase 6's residual path frees the cgroup then).

  Phase 5 — VERIFY the ENUMERABLE (memory) charge is drained:
    memory.usage.load(Acquire) == 0
    (io/hugetlb/rdma/perf are NOT verified here — they are `residual_refs`-tracked
    and may legitimately outlive the drain kworker; the tombstone keeps their
    counters resolvable until they drain. Verifying `usage == 0` for them would
    stall every drain that has a resource held by another cgroup, exactly the
    zombie problem the tombstone solves.)

  Phase 6 — Store Dead, then FINALIZE via the residual-drain interlock:
    lifecycle.store(Dead, Release);              // publish BEFORE the load
    if residual_refs.load(Acquire) == 0 {
        // No outliving charge → erase the tombstone now, dropping the last
        // Arc<Cgroup> (the registry entry held it) and freeing the struct +
        // all controller state.
        CGROUP_REGISTRY.erase(cgroup.id);        // idempotent; returns the Arc
    }
    // else: leave the tombstone. The LAST residual uncharge (which does
    // fetch_sub then, seeing 0 AND Dead, erases) frees it. store-Dead-then-load
    // here pairs with fetch_sub-then-load-Dead there (Dekker) so at least one
    // side observes the drained state; `CGROUP_REGISTRY.erase` returning the Arc
    // exactly once makes the free happen exactly once.

  Phase 7 — If Phase 5 fails after CGROUP_MEM_DRAIN_TIMEOUT_MS.load(Relaxed):
    Force-reparent the remaining ENUMERABLE memory charges to the parent (the
    only force-reparentable class — `lru_gen` is walkable; residual
    rdma/perf/hugetlb/io held by external resources cannot be force-completed and
    stay tombstoned until their real uncharge).
    Log FMA warning: "cgroup {id} force-drained after timeout;
      {n} bytes reparented to parent {parent_id}."
    Then run the Phase-6 store-Dead + residual-interlock finalize (memory now 0,
    so the cgroup frees as soon as residual_refs reaches 0).

17.2.8.2.4 Registry Tombstone and Residual Charge Drain¶

Four controllers charge resources whose lifetime is NOT bounded by the cgroup's task set, so their charges can outlive rmdir (which requires only population == 0): RDMA verbs objects (fds passed to processes in other cgroups), perf_event fds (held by tasks elsewhere), hugetlb folios (mapped by other cgroups), and in-flight bios (async completion after the charging task exits). Each anchors its charge on a CgroupId and uncharges by resolving that id via CGROUP_REGISTRY. If rmdir erased the registry entry immediately, every such late uncharge would resolve None, its counter would never decrement, and the drain would either stall the full timeout or free a cgroup whose counters a later uncharge then corrupts. All four share ONE mechanism.

Mechanism: registry tombstone + residual_refs.

Why a tombstone, not an Arc captured at charge time. The alternative — have each charge hold a strong Arc<Cgroup> so uncharge needs no registry — is impossible here: the charge anchors are #[repr(C)], size-const_asserted structs storing a CgroupId integer (RdmaResourceAnchor = 16 B; PerfEvent.charged_cgroup: u64; bio.cgroup_id; the huge folio's stored id). A #[repr(C)] struct cannot hold an Arc (no stable layout, rule 9), and the whole CgroupId-not-Arc anchor design exists to avoid strong back-edges that recreate the MemCgroup::owner cycle. The tombstone keeps the existing 16-byte integer anchors unchanged and is exactly Linux's model (an offline css stays alive under its Linux percpu_ref until the last charge drops).
residual_refs (a per-cgroup AtomicU64). Every one of the four charges does cgroup.residual_refs.fetch_add(1, Relaxed) at charge and fetch_sub(1, ...) at uncharge (alongside the per-kind counter). It counts outstanding outlives-rmdir charges only — the bounded, task-set-bounded charges (memory RSS, pids, population) are NOT counted here (they drain when tasks exit/migrate).
Tombstone window. cgroup_rmdir step 4b leaves the CGROUP_REGISTRY entry in place (the entry's Arc<Cgroup> is the sole keep-alive once the tree edge is dropped). get(id) therefore still resolves the (now Dead) cgroup, so a late rdma_uncharge/perf_event_release/hugetlb_uncharge/bio-completion finds its counter. IDs are never reused, so this always resolves to the same cgroup, never a different one.
Exactly-once free (Dekker interlock). Two parties may observe the fully drained state: the drain kworker (Phase 6) and the last residual uncharge. Both do store-then-load:
drain Phase 6: lifecycle.store(Dead, Release); if residual_refs.load(Acquire) == 0 { erase }
residual uncharge: let n = residual_refs.fetch_sub(1, Release); if n == 1 && lifecycle.load(Acquire) == Dead { erase }

With store-Dead-before-load-refs on one side and sub-refs-before-load-Dead on the other, at least one party observes the drained state and calls CGROUP_REGISTRY.erase(id). erase removes the entry atomically and returns the previous Arc exactly once (a racing second erase returns None), so the last-Arc drop — which frees the Cgroup and all inline controller state — happens exactly once. Before Dead (during Draining), residual uncharges only decrement residual_refs; they never erase.

What the drain kworker still frees eagerly (Phases 1/3/3.5): the enumerable memory charges (reparented via lru_gen), BPF programs, and registered-subsystem state — none of which a residual uncharge touches. The Cgroup SHELL and its RcuPtr-held RdmaController/HugetlbController/PerfEventController/ IoController counters stay allocated — the controller boxes live as long as the Cgroup does (their RcuPtrs are dropped, reclaiming the boxes after a grace period, only when the last Arc<Cgroup> drops at tombstone erase) — so residual uncharges always have a valid counter.
Bounded in practice. The tombstone persists only while a real external resource still references the cgroup; it is bounded by resource lifetime, not monotonic, and collapses to immediate free (residual_refs == 0 at Phase 6) for the overwhelmingly common cgroup with no outliving charges. This is strictly better than Linux zombies (which also persist for held pages) because the enumerable memory charge is force-reparented at the timeout rather than pinning the shell on page-cache pages.

17.2.8.2.5 Performance Impact¶

Steady state: Zero. The lifecycle field is never checked on the hot path (task scheduling, memory allocation, I/O submission). It is only read during cgroup.procs writes (to reject migration into a Draining cgroup) and during rmdir processing — both cold paths.

During drain: The per-page reparent is ~50 ns/page (one fetch_add + one fetch_sub + one pointer store under page lock). At 256K pages (1 GB residual), total drain time is ~12 ms. The page lock is the same lock used by reclaim — no new lock is introduced. Other cgroups are unaffected.

Invariant: After cgroup_drain_residual returns, the Cgroup struct has exactly zero residual resource charges. No zombie cgroups persist beyond the drain timeout. Over 50-year uptime, zero memory is leaked to cgroup metadata.

17.2.9 Additional Controllers¶

Controller	Key Interface	Description
`hugetlb`	`hugetlb.<size>.max`	Limits huge page allocations per cgroup
`rdma`	`rdma.max`	Limits RDMA/InfiniBand resources
`misc`	`misc.max`	Limits miscellaneous resources (e.g., SGX EPC)
`accel`	`accel.devices`, `accel.memory.max`, `accel.compute.max`, `accel.priority`	Accelerator compute and memory limits per cgroup (Section 22.5).

17.2.9.1 Device Access Control (BPF-based)¶

Linux cgroup v2 replaces the v1 devices controller with a BPF-based enforcement mechanism. UmkaOS implements the same model: device access decisions are made by a BPF_PROG_TYPE_CGROUP_DEVICE program attached to the cgroup.

/// BPF context passed to BPF_PROG_TYPE_CGROUP_DEVICE programs.
/// The BPF program returns 0 (deny) or 1 (allow).
///
/// **External ABI — MUST match Linux `struct bpf_cgroup_dev_ctx` byte-for-byte**
/// (`include/uapi/linux/bpf.h`, verified against `torvalds/linux` master): three
/// `u32` fields = 12 bytes, with the DEVICE TYPE PACKED INTO `access_type`, not a
/// separate field. This is a BPF-program-visible layout: an unmodified Linux device
/// program (systemd's, Docker/runc's default device policy) reads `major` at offset
/// 4 and `minor` at offset 8 and decodes the packed `access_type`. A 4-field /
/// 16-byte split (a separate `dev_type` at offset 4) would shift `major`→offset 8
/// and `minor`→offset 12 and mis-decode the access bits, silently breaking every
/// device-cgroup policy. "Linux outside" applies here — no internal-layout divergence
/// is available.
#[repr(C)]
pub struct BpfCgroupDevCtx {
    /// Packed access + device type, encoded EXACTLY as Linux:
    /// `(BPF_DEVCG_ACC_* << 16) | BPF_DEVCG_DEV_*`.
    /// Access bits (high half): BPF_DEVCG_ACC_MKNOD=1, ACC_READ=2, ACC_WRITE=4.
    /// Device bits (low half):  BPF_DEVCG_DEV_BLOCK=1, DEV_CHAR=2.
    /// Construct with `(acc << 16) | dev`; recover access as `access_type >> 16`
    /// and device type as `access_type & 0xffff`.
    pub access_type: u32,
    /// Major device number.
    pub major: u32,
    /// Minor device number.
    pub minor: u32,
}
// BPF UAPI context struct — MUST equal Linux's 3 × u32 = 12 bytes.
const_assert!(size_of::<BpfCgroupDevCtx>() == 12);

Enforcement hook point: - The VFS device-open path (device_node_open, Section 14.5) calls cgroup_device_permitted() before granting access; that entry point runs cgroup_bpf_run(BPF_CGROUP_DEVICE, &ctx) as specified below:

/// Run the `BPF_CGROUP_DEVICE` policy for a device-node access: builds the
/// `BpfCgroupDevCtx` (`access_type = (acc << 16) | dev_type`, with
/// `dev_type = BPF_DEVCG_DEV_BLOCK` for block / `BPF_DEVCG_DEV_CHAR` for char,
/// and `acc = (read ? BPF_DEVCG_ACC_READ : 0) | (write ? BPF_DEVCG_ACC_WRITE : 0)`),
/// evaluates every ancestor cgroup's device program bottom-up, and returns
/// `true` iff access is permitted (or no program is attached — allow-by-default).
///
/// `read`/`write` are the caller's requested access, derived from `O_ACCMODE`
/// exactly as Linux's `ACC_MODE` maps `O_RDONLY → MAY_READ`, `O_WRONLY →
/// MAY_WRITE` (write only), `O_RDWR → both` (`include/linux/device_cgroup.h`:
/// `mask & MAY_WRITE ⇒ ACC_WRITE`, `mask & MAY_READ ⇒ ACC_READ`). `ACC_READ` is
/// NOT set unconditionally: an `O_WRONLY` open under a write-only `DeviceAllow`
/// policy must be permitted, so the BPF program must see `ACC_WRITE` alone. (The
/// `mknod(2)` path invokes the hook with the `BPF_DEVCG_ACC_MKNOD` bit instead;
/// see the dispatch prose above.)
pub fn cgroup_device_permitted(is_block: bool, major: u32, minor: u32, read: bool, write: bool) -> bool;

mknod() syscall calls the same hook before creating the device node in the filesystem.
Default (no BPF program attached): allow all device access. This matches the principle of least surprise — cgroups without explicit device policy impose no restrictions.

Legacy v1 shim translation (see the v1→v2 translation table below): The v1 devices.allow / devices.deny interface is translated into BPF programs attached to the cgroup: - devices.allow a *:* rwm → generates and attaches a permissive BPF program (unconditional return 1). - devices.deny a → attaches a deny-all BPF program (unconditional return 0). - Specific rules (e.g., devices.allow c 1:3 rwm) → generates a BPF program that matches the (dev_type, major, minor, access_type) tuple and returns 1 for matching entries, 0 otherwise. Multiple rules accumulate into a single match-list BPF program that is atomically replaced on each devices.allow / devices.deny write.

Hierarchy enforcement: BPF device programs are evaluated bottom-up from the task's cgroup to the root. Access is allowed only if every ancestor's BPF program (if any) returns 1. This ensures a parent cgroup can restrict device access for all descendants regardless of their individual policies.

17.2.9.2 Cgroup BPF Program Attachment¶

The enforcement side above (cgroup_bpf_run call sites) needs an ATTACH side: how programs land in Cgroup.bpf_progs and how multi-program / replace / hierarchy semantics work. BPF_PROG_ATTACH/BPF_PROG_DETACH themselves are the bpf(2) commands BpfCmd::ProgAttach = 8 / ProgDetach = 9, and a cgroup attach target is BpfAttachTarget::CgroupSkb(cgroup_id, direction) and its siblings, all in Section 19.2 (the enforcement points here are the consumer side of those). This section specifies the cgroup-local storage, handler, and semantics that Section 19.2 defers here.

/// Number of cgroup-BPF attach types (`bpf_effective` slot count) — one per
/// `enum bpf_attach_type` value that targets a cgroup (INET_INGRESS/EGRESS,
/// INET_SOCK_CREATE, DEVICE, SYSCTL, GETSOCKOPT, SETSOCKOPT, SOCK_OPS, …). A
/// compile-time slot-array bound (like `MAX_DYN_CGROUP_SUBSYS`), NOT a system
/// limit; the authoritative numbering is Linux `enum bpf_attach_type`
/// ([Section 19.2](19-sysapi.md#ebpf-subsystem)). Validated at registration.
pub const NR_CGROUP_BPF_ATTACH_TYPES: usize = 24;

/// One attachment of a BPF program to a cgroup. Stored inline in
/// `Cgroup.bpf_progs: SpinLock<ArrayVec<BpfCgroupLink, 64>>` — a FLAT list
/// across ALL cgroup attach types; per-type views are obtained by filtering on
/// `attach_type` (there is no per-type sub-array). The 64-slot cap bounds the
/// sum of all cgroup-BPF attachments on one cgroup.
///
/// Linux parity: `struct bpf_cgroup_link { struct bpf_link link; struct cgroup
/// *cgroup; }` plus the per-type `struct bpf_prog_list { prog; link; flags; }`.
pub struct BpfCgroupLink {
    /// Which cgroup attach type this entry serves (INET_INGRESS, INET_EGRESS,
    /// DEVICE, SYSCTL, GETSOCKOPT, SETSOCKOPT, ...). The per-attach-type lookup
    /// filters `bpf_progs` on this field. Values match Linux `enum
    /// bpf_attach_type` ([Section 19.2](19-sysapi.md#ebpf-subsystem)); a bare `u32` (not a new enum)
    /// keeps the ABI numbering authoritative in one place.
    pub attach_type: u32,
    /// Attach flags: BPF_F_ALLOW_MULTI | BPF_F_ALLOW_OVERRIDE (mutually
    /// exclusive per Linux), recorded per entry for effective-set computation.
    pub flags: u32,
    /// The attached program. `Arc` because a program may be attached to several
    /// cgroups and also held by a userspace fd; freed when the last ref drops.
    pub prog: Arc<BpfProg>,
    /// `true` if this attachment is owned by a `bpf_link` fd (detach = link fd
    /// close), `false` if it is a bare `BPF_PROG_ATTACH` (detach =
    /// `BPF_PROG_DETACH` or the last prog fd close). Distinguishes the two
    /// detach lifecycles Linux draws between `bpf_link` and legacy attach.
    /// (Kernel-internal struct — never crosses a KABI/wire boundary, so a
    /// `bool` here is not a validity-invariant hazard.)
    pub link_owned: bool,
}

Attach handler (BPF_PROG_ATTACH with a cgroup fd → cgroup_bpf_attach):

Resolve the cgroup fd → Arc<Cgroup> (EBADF on a non-cgroup fd). Require CAP_BPF + write permission on the cgroup.
Take cgroup.bpf_progs.lock(). Compute the current per-type view = entries with matching attach_type.
Dispatch on flags (mirrors Linux __cgroup_bpf_attach):
BPF_F_ALLOW_MULTI: append a new BpfCgroupLink (multiple programs of this type run in attach order). BPF_F_REPLACE (valid ONLY with MULTI) swaps the entry whose prog equals the caller-supplied replace_prog, preserving order; ENOENT if not found.
BPF_F_ALLOW_OVERRIDE: at most one program of this type at this level, but a descendant cgroup MAY attach its own overriding program. Replaces the existing single entry of this type.
neither flag (exclusive): at most one program of this type, and it forbids any descendant from attaching this type (EPERM on the child).
EBUSY if the requested flag conflicts with the existing entries' flags (Linux: you cannot mix MULTI and non-MULTI for the same type).
ENOSPC if the 64-slot ArrayVec is full (try_push, never a panic).

Detach: BPF_PROG_DETACH (bare) or link-fd close removes the matching BpfCgroupLink under bpf_progs.lock() and drops its prog Arc. rmdir's drain Phase 3 auto-detaches ALL remaining entries (any link_owned links are marked defunct — the link fd stays valid but its program no longer runs, matching Linux cgroup_bpf_release → bpf_cgroup_link_auto_detach), so programs never outlive the cgroup's task set.

Effective program set & cgroup_bpf_run: the effective list for a (cgroup, attach_type) is, conceptually, this cgroup's MULTI entries of that type followed by each ancestor's, EXCEPT that an OVERRIDE/exclusive program in a descendant suppresses the ancestor's (descendant wins), matching Linux Linux compute_effective_progs. This bottom-up composition is computed ONCE per attach/detach/rmdir — NOT per packet — and cached in the target's bpf_effective[attach_type] slot. cgroup_bpf_run on the hot path does a single RCU read of cgroup.bpf_effective[attach_type], iterates the flat Arc<[…]> slice, and combines results per attach type (DEVICE: logical-AND of the 0/1 verdicts — deny if ANY returns 0, as the hierarchy-enforcement paragraph above requires; SKB: first non-OK verdict stops the pipeline) — no ancestor walk on the packet/open path.

Recompute (subtree walk): on any attach/detach at cgroup C, and on C's rmdir-drain auto-detach, the handler holds C.bpf_progs.lock(), recomputes the effective list for C AND every descendant of C (an attach on C changes the inherited-from-ancestor portion of every descendant's list), and publishes each node's bpf_effective[attach_type] via RcuCell::update under that node's bpf_progs writer proof (the SpinLockGuard<'_, ArrayVec<BpfCgroupLink, 64>> from bpf_progs.lock() — a payload guard satisfies WriterProof directly, Section 3.4). The subtree walk is for_each_descendant-bounded (depth ≤ CGROUP_MAX_DEPTH); it is a COLD path (program attach is rare), so the O(subtree) republish is acceptable and buys the per-packet lock-free read. Old published slices are freed after an RCU grace period (readers mid-iteration keep their Arc<[…]> alive).

v1 devices shim: the "atomically replaced on each devices.allow / devices.deny write" behavior above is the BPF_F_ALLOW_MULTI + BPF_F_REPLACE path — the shim keeps ONE synthesized BPF_PROG_TYPE_CGROUP_DEVICE program per cgroup and REPLACEs it on each write.

Cross-file handoff (resolved): Section 19.2 BpfAttachTarget enumerates the cgroup targets this handler dispatches — CgroupSkb(cgroup_id, direction), CgroupDevice(cgroup_id), CgroupSysctl(cgroup_id), and CgroupSockopt(cgroup_id, direction) — so the detach-by-id path there resolves to Cgroup.bpf_progs.

17.2.9.3 perf_event Cgroup Controller¶

The perf_event controller limits per-cgroup PMU (Performance Monitoring Unit) resource usage. Without limits, a container running perf record can exhaust all available hardware performance counters, denying monitoring to other containers and the host.

The Cgroup struct includes:

    /// perf_event cgroup controller. Limits per-cgroup PMU resource usage
    /// to prevent container perf_event exhaustion. `RcuPtr` (not `Option`) for the
    /// same runtime enable/disable reason as every built-in controller — see the
    /// `Cgroup` "Resource controller state" field group.
    pub perf_event: RcuPtr<PerfEventController>,

pub struct PerfEventController {
    /// Maximum number of concurrent perf events for this cgroup subtree.
    /// `0` = unlimited — the internal sentinel. The cgroupfs `perf_event.max`
    /// file renders `0` as the string `"max"` and parses `"max"` back to `0`
    /// (so the struct default `0` and the file default `"max"` are the SAME
    /// state, not a mismatch). Write `perf_event.max` to set.
    pub max_events: AtomicU32,
    /// Current active perf event count charged to this cgroup subtree.
    pub nr_events: AtomicU32,
}

Charge anchoring (the fix for migration/exit drift): the charge is ANCHORED on the event, not re-derived from the task's current cgroup at close. perf_event_open records the id of the cgroup it charged onto the event; close decrements THAT chain. Without this, an event opened in cgroup A then whose task migrates to B would decrement B's chain at close — inflating A forever (eventually ENOSPC-ing all opens under A) and wrapping B's AtomicU32 fetch_sub below zero to ~4×10⁹ (making max_events vacuous).

Field addition (cross-file handoff): PerfEvent (Section 20.8) gains charged_cgroup: u64 — the CgroupId whose chain perf_event_open charged (0 if the controller was disabled, i.e. no charge taken). This is the same charge-anchor pattern RdmaResourceAnchor/bio.cgroup_id use.
No migration transfer: nr_events is NOT moved when the task migrates cgroups (migration steps 8/9 transfer pids and CBS weight only). The perf event's charge stays on its open-time cgroup for the event's lifetime — the anchor makes this correct, and it matches Linux, where a perf event's cgroup association is fixed at open.

Enforcement: The perf_event_open() syscall, after validating the event attributes, checks the calling task's cgroup: 1. Resolve the charge chain = the calling task's cgroup → root. If the task's cgroup has no perf_event controller (max_events conceptually unlimited at every level), set event.charged_cgroup = 0 and allow (no charge). 2. Walk the chain root-ward; at EACH level fetch_add(1, Relaxed) on nr_events AND fetch_add(1, Relaxed) on that level's residual_refs (pinning the WHOLE charged chain, not just the anchor — see below); if any level with max_events != 0 now exceeds its max_events, roll back BOTH increments on every level done so far on THIS walk and return -ENOSPC. On success, record event.charged_cgroup = <task's cgroup id> as the resolution anchor (the event can outlive the cgroup's task set — its fd may be held by a task in another cgroup). 3. On perf_event_release() (close of the fd): if event.charged_cgroup != 0, resolve that CgroupId via CGROUP_REGISTRY and walk ITS chain to the root, fetch_sub(1, Relaxed) on each level's nr_events AND residual_refs — the SAME chain charged at open, so no underflow and no cross-cgroup drift. If the anchored cgroup (or any charged ancestor) was already rmdir'd, the registry entry is a resolvable TOMBSTONE (rmdir does NOT erase it while residuals remain), so get(charged_cgroup) still returns the Dead cgroup AND every ancestor's residual_refs (bumped in step 2) keeps ITS Arc alive, so the root-ward walk's parent.upgrade() reaches every level even when an INTERMEDIATE ancestor has emptied and been rmdir'd — closing the drift where a freed intermediate broke the walk and stranded +1 on the live root. Each level's last residual_refs drop frees that level. There is NO drain-phase nr_events reparent — no drain phase performs one — the tombstone replaces it. Whole-chain pinning (why not anchor-only): nr_events is decremented at every level, so every charged level must stay resolvable until its own decrement; pinning only the anchor left an intermediate ancestor free to be freed at ITS Phase 6 (perf is excluded from Phase-5 usage==0 verification) with nr_events still 1, permanently inflating every still-live higher ancestor toward ENOSPC. Bumping residual_refs on the whole chain (bounded by the events charged in that subtree, dropped as they close) makes the model correct; the same shape applies to the rdma and misc hierarchical uncharges (their charge walks bump residual_refs per level too). See Section 17.2.

Hierarchy: Child cgroups inherit the parent's limit as an upper bound. A child can set a lower perf_event.max but cannot exceed the parent's value. The effective limit for any cgroup is min(own max_events, parent effective limit). The root-ward charge walk in step 2 enforces this: an open fails if ANY ancestor is at its limit.

cgroupfs interface: - perf_event.max: read/write, integer or "max" (unlimited). Default: "max". - perf_event.current: read-only, current number of active perf events in this subtree.

17.2.10 Cgroup v1 Compatibility Translation¶

UmkaOS exposes cgroup v2 exclusively inside the kernel. For userspace processes that set cgroup v1 knobs (Docker Engine ≤20.10, systemd pre-247, legacy orchestrators), UmkaOS provides a v1-to-v2 translation shim implemented as a virtual filesystem (cgroupv1fs) that mounts the legacy hierarchy paths at /sys/fs/cgroup/cpu, /sys/fs/cgroup/memory, etc. The full shim specification is in Section 19.1. This section documents the authoritative translation table and the cpu.shares → cpu.weight formula that the shim applies.

Translation table (v1 write → v2 equivalent):

Subsystem	v1 knob	v2 equivalent	Conversion formula
memory	`memory.limit_in_bytes`	`memory.max`	Direct (bytes); -1 → "max"
memory	`memory.soft_limit_in_bytes`	`memory.high`	Direct (bytes)
memory	`memory.memsw.limit_in_bytes`	`memory.swap.max`	`swap_max = memsw - mem`
memory	`memory.kmem.limit_in_bytes`	(no v2 equivalent)	Silently ignored (kmem tracking removed in v2)
memory	`memory.oom_control` (disable OOM)	(no direct v2 equivalent)	Incompatible: v1 `oom_kill_disable=1` means "do not kill tasks in this cgroup"; v2 has no equivalent knob. `memory.oom.group` is a different feature (kills all tasks atomically on OOM rather than selecting one victim). The shim silently drops `oom_kill_disable` writes and logs a compat warning.
cpu	`cpu.shares`	`cpu.weight`	`weight = clamp(1 + (shares − 2) × 9999 / 262142, 1, 10000)`
cpu	`cpu.cfs_quota_us` + `cpu.cfs_period_us`	`cpu.max`	`"$quota $period"` (µs); quota=-1 → `"max $period"`
cpuacct	`cpuacct.usage`	`cpu.stat` (usage_usec)	Read-only; ns→µs unit conversion
blkio	`blkio.throttle.read_bps_device`	`io.max` (rbps=N)	`MAJ:MIN rbps=N`
blkio	`blkio.throttle.write_bps_device`	`io.max` (wbps=N)	`MAJ:MIN wbps=N`
blkio	`blkio.throttle.read_iops_device`	`io.max` (riops=N)	`MAJ:MIN riops=N`
blkio	`blkio.throttle.write_iops_device`	`io.max` (wiops=N)	`MAJ:MIN wiops=N`
blkio	`blkio.weight`	`io.weight`	v1 range 10–1000 → v2 range 100–10000 via `weight × 10`
freezer	`freezer.state`	`cgroup.freeze`	`FROZEN` → `"1"`, `THAWED` → `"0"`
net_cls	`net_cls.classid`	(no v2 equivalent)	Logged and ignored; use eBPF for network classification
net_prio	`net_prio.ifpriomap`	(no v2 equivalent)	Logged and ignored
pids	`pids.max`	`pids.max`	Direct
devices	`devices.allow` / `devices.deny`	`BPF_PROG_TYPE_CGROUP_DEVICE`	Translated to eBPF program attached to the cgroup
hugetlb	`hugetlb.Xm.limit_in_bytes`	`hugetlb.Xm.max`	Direct
rdma	`rdma.max`	`rdma.max`	Direct
(any v1)	`cgroup.event_control`	(no v2 equivalent)	Silently ignored — v1-only eventfd notification mechanism, replaced by inotify on cgroupfs in v2
(any v1)	`notify_on_release`	(no v2 equivalent)	Silently ignored — v1-only automatic agent notification, no v2 equivalent (cgroup v2 uses systemd scope/slice lifecycle)

cpu.shares formula derivation: Linux cpu.shares range is [2, 262144]; cpu.weight range is [1, 10000]. The formula is a linear interpolation that maps the full v1 range onto the full v2 range:

weight = clamp(1 + (shares - 2) × 9999 / 262142, 1, 10000)

This is the formula used by runc (the OCI reference runtime), containerd, and crun as of 2025. Key values:

v1 `cpu.shares`	v2 `cpu.weight`
2 (minimum)	1
1024 (Docker default)	~39
262144 (maximum)	10000

Systemd divergence: systemd (≥247) writes cpu.weight directly when operating in cgroup v2 mode, using its own unit mapping (default weight = 100) rather than the runc formula. When systemd writes v2 files natively, those writes bypass the shim entirely and go straight to the cgroupfs. The shim translates only raw v1 cgroupfs writes from programs that open the legacy v1 paths directly (older Docker daemons, legacy orchestrators).

Implementation: The shim is implemented in umka-sysapi as cgroupv1fs, a VFS pseudo-filesystem that mounts legacy controller directories. Writes to v1 paths invoke the translation function below and apply the result to the v2 cgroupfs. Reads return translated v2 values in v1 format.

/// Result of translating a cgroup v1 write to its v2 equivalent.
pub struct CgroupV2Write {
    /// Relative path of the v2 control file (e.g., "memory.max", "cpu.weight").
    pub path: &'static str,
    /// Value to write (already formatted for v2 semantics).
    pub value: ArrayString<64>,
}

/// The cgroup v1 subsystem (controller) a legacy write targets, parsed from the
/// v1 controller directory the write arrived through. The `cgroupv1fs` shim tags
/// every raw v1 write with its originating subsystem so `cgroupv1_translate` can
/// route it. Variants cover the full Linux v1 controller set; controllers with
/// no v2 equivalent (`NetCls`, `NetPrio`) route to warn-and-ignore, and
/// `CpuAcct` is served on the READ path (its counters derive from v2 `cpu.stat`).
/// Kernel-internal (never crosses a KABI/wire boundary), so a plain Rust enum.
/// `Copy` because `cgroupv1_translate` matches on `(subsystem, knob)` (moving
/// `subsystem` into the tuple) and still passes `subsystem` to
/// `cgroupv1_translate_full` in the catch-all arm — the enum is fieldless, so
/// `Copy` is trivial.
#[derive(Clone, Copy)]
pub enum CgroupV1Subsystem {
    Cpu,
    CpuAcct,
    Cpuset,
    Memory,
    Devices,
    Freezer,
    NetCls,
    Blkio,
    PerfEvent,
    NetPrio,
    Hugetlb,
    Pids,
    Rdma,
    Misc,
}

/// Outcome of a v1→v2 translation. Distinguishes the THREE dispositions the
/// translation table draws (an `Option` conflated the two ignore cases):
///   - `Write`         — apply this v2 write.
///   - `IgnoreSilent`  — v1 knob with no v2 meaning, dropped without a trace
///                       (`cgroup.event_control`, `notify_on_release`).
///   - `IgnoreWarn`    — v1 knob deliberately unsupported; log a one-time compat
///                       warning (`memory.oom_control` disable, `net_cls.classid`,
///                       `net_prio.ifpriomap`, `memory.kmem.limit_in_bytes`).
///   - `Err(Errno)`    — malformed value (e.g. unparseable number → EINVAL).
pub enum CgroupV1Translation {
    Write(CgroupV2Write),
    IgnoreSilent,
    IgnoreWarn(&'static str),   // the message to log once
}

/// Translate a cgroup v1 write to its v2 equivalent.
///
/// `cgroup` is REQUIRED: two table rows are cross-knob and need the cgroup's
/// CURRENT v2 state (a stateless per-(knob,value) function cannot produce them):
///   - `cpu.cfs_quota_us` / `cpu.cfs_period_us` → `cpu.max "$quota $period"`:
///     writing one half must be combined with the cgroup's current OTHER half
///     (read back from `cpu.max`).
///   - `memory.memsw.limit_in_bytes` → `memory.swap.max = memsw - mem`: needs
///     the cgroup's current `memory.max`.
/// The caller applies the returned `CgroupV2Write` to the SAME cgroup.
pub fn cgroupv1_translate(
    cgroup: &Cgroup,
    subsystem: CgroupV1Subsystem,
    knob: &str,
    value: &[u8],
) -> Result<CgroupV1Translation, Errno> {
    use CgroupV1Translation::*;
    match (subsystem, knob) {
        (CgroupV1Subsystem::Memory, "memory.limit_in_bytes") => {
            let bytes = parse_bytes_or_max(value).ok_or(Errno::EINVAL)?;
            Ok(Write(CgroupV2Write { path: "memory.max", value: format_bytes_or_max(bytes) }))
        }
        (CgroupV1Subsystem::Cpu, "cpu.shares") => {
            let shares: u64 = parse_u64(value).map_err(|_| Errno::EINVAL)?;
            let weight = 1u64.saturating_add(
                shares.saturating_sub(2).saturating_mul(9999) / 262142
            ).clamp(1, 10000);
            // `CgroupV2Write.value` is `ArrayString<64>`, NOT heap `String`:
            // `format_u64` renders into an ArrayString (same family as the
            // `format_bytes_or_max`/`format_cpu_max` producers the other arms use);
            // `to_string()` would be a type error here.
            Ok(Write(CgroupV2Write { path: "cpu.weight", value: format_u64(weight) }))
        }
        // Cross-knob rows: read back the cgroup's current v2 state to combine.
        (CgroupV1Subsystem::Cpu, "cpu.cfs_quota_us") => {
            let quota = parse_quota(value)?;                 // -1 → "max"
            let period = cgroup_cpu_max_period(cgroup);      // current period half
            Ok(Write(CgroupV2Write { path: "cpu.max", value: format_cpu_max(quota, period) }))
        }
        (CgroupV1Subsystem::Cpu, "cpu.cfs_period_us") => {
            let period = parse_u64(value).map_err(|_| Errno::EINVAL)?;
            let quota = cgroup_cpu_max_quota(cgroup);        // current quota half
            Ok(Write(CgroupV2Write { path: "cpu.max", value: format_cpu_max(quota, period) }))
        }
        (CgroupV1Subsystem::Memory, "memory.memsw.limit_in_bytes") => {
            let memsw = parse_bytes_or_max(value).ok_or(Errno::EINVAL)?;
            let mem = cgroup_memory_max(cgroup);             // current memory.max
            Ok(Write(CgroupV2Write { path: "memory.swap.max",
                value: format_bytes_or_max(memsw.saturating_sub(mem)) }))
        }
        // Deliberately unsupported → warn-and-ignore (distinct from silent).
        (CgroupV1Subsystem::NetCls, _) =>
            Ok(IgnoreWarn("net_cls.classid ignored; use eBPF for network classification")),
        (CgroupV1Subsystem::NetPrio, _) =>
            Ok(IgnoreWarn("net_prio.ifpriomap ignored; no v2 equivalent")),
        (CgroupV1Subsystem::Memory, "memory.kmem.limit_in_bytes") =>
            Ok(IgnoreWarn("memory.kmem tracking removed in cgroup v2")),
        // v1-only mechanisms with no v2 meaning → drop silently.
        (_, "cgroup.event_control") | (_, "notify_on_release") => Ok(IgnoreSilent),
        // Every remaining table row (all DIRECT or simple-format, no cross-knob
        // read-back needed) → the continuation below. NOT a cross-file handoff:
        // `cgroupv1_translate_full` is defined in THIS file, immediately after.
        _ => cgroupv1_translate_full(subsystem, knob, value),
    }
}

/// Continuation of `cgroupv1_translate` for the table rows that need NO
/// cross-knob read-back (so they take no `&Cgroup`): the direct-passthrough and
/// simple-format rows of the translation table above. Defined here (not a
/// handoff) so the catch-all resolves to real code. Any knob not in the table is
/// a v1 knob UmkaOS does not model → `EINVAL` (the shim rejects it rather than
/// silently dropping an unknown write).
fn cgroupv1_translate_full(
    subsystem: CgroupV1Subsystem,
    knob: &str,
    value: &[u8],
) -> Result<CgroupV1Translation, Errno> {
    use CgroupV1Translation::*;
    use CgroupV1Subsystem as S;
    match (subsystem, knob) {
        // DIRECT byte/count passthrough.
        (S::Memory, "memory.soft_limit_in_bytes") => {
            let b = parse_bytes_or_max(value).ok_or(Errno::EINVAL)?;
            Ok(Write(CgroupV2Write { path: "memory.high", value: format_bytes_or_max(b) }))
        }
        (S::Pids, "pids.max") => {
            let v = parse_bytes_or_max(value).ok_or(Errno::EINVAL)?; // "max" or N
            Ok(Write(CgroupV2Write { path: "pids.max", value: format_bytes_or_max(v) }))
        }
        (S::Rdma, "rdma.max") =>
            Ok(Write(CgroupV2Write { path: "rdma.max", value: array_str_from(value)? })),
        (S::Hugetlb, k) if k.ends_with(".limit_in_bytes") => {
            // hugetlb.<size>.limit_in_bytes → hugetlb.<size>.max (direct bytes).
            let size = k.strip_suffix(".limit_in_bytes").unwrap();
            let b = parse_bytes_or_max(value).ok_or(Errno::EINVAL)?;
            Ok(Write(CgroupV2Write { path: hugetlb_v2_path(size), value: format_bytes_or_max(b) }))
        }
        // blkio throttle rows → io.max token (device-qualified, formatted from value).
        (S::Blkio, "blkio.throttle.read_bps_device")   => Ok(Write(io_max_token("rbps",  value)?)),
        (S::Blkio, "blkio.throttle.write_bps_device")  => Ok(Write(io_max_token("wbps",  value)?)),
        (S::Blkio, "blkio.throttle.read_iops_device")  => Ok(Write(io_max_token("riops", value)?)),
        (S::Blkio, "blkio.throttle.write_iops_device") => Ok(Write(io_max_token("wiops", value)?)),
        (S::Blkio, "blkio.weight") => {
            // v1 [10,1000] → v2 [100,10000] via ×10.
            let w = parse_u64(value).map_err(|_| Errno::EINVAL)?.clamp(10, 1000) * 10;
            Ok(Write(CgroupV2Write { path: "io.weight", value: format_u64(w) }))
        }
        // freezer.state FROZEN/THAWED → cgroup.freeze 1/0.
        (S::Freezer, "freezer.state") => {
            let s = core::str::from_utf8(value).map_err(|_| Errno::EINVAL)?.trim();
            let v = match s { "FROZEN" => "1", "THAWED" => "0", _ => return Err(Errno::EINVAL) };
            Ok(Write(CgroupV2Write { path: "cgroup.freeze", value: array_str_from(v.as_bytes())? }))
        }
        // Read-only cpuacct.usage is handled on the READ path, not here.
        // Anything else: a v1 knob UmkaOS does not model.
        _ => Err(Errno::EINVAL),
    }
}

// --- Shared value parsers/formatters (v2 write handlers + v1→v2 shim) ---
// All run on the warm cgroupfs write path (process context, bounded buffer);
// formatters render into the fixed-capacity `CgroupV2Write.value`
// (`ArrayString<64>`) with no heap allocation. `write!` targets the
// `core::fmt::Write` impl of `ArrayString`.
use core::fmt::Write as _;

/// Parse a base-10 `u32` from an ASCII cgroupfs write buffer, tolerating
/// surrounding whitespace / a trailing newline. Empty, non-numeric, or
/// out-of-range input → `InvalidArgument`. Used by the v2 write handlers (e.g.
/// `cpu.weight`) and the v1 shim.
fn parse_u32(buf: &[u8]) -> Result<u32, KernelError> {
    core::str::from_utf8(buf)
        .map_err(|_| KernelError::InvalidArgument)?
        .trim()
        .parse::<u32>()
        .map_err(|_| KernelError::InvalidArgument)
}

/// Parse a base-10 `u64` (same trimming rules as `parse_u32`). Returns `EINVAL`
/// on malformed input; callers `.map_err` it into their local error type.
fn parse_u64(value: &[u8]) -> Result<u64, Errno> {
    core::str::from_utf8(value)
        .map_err(|_| Errno::EINVAL)?
        .trim()
        .parse::<u64>()
        .map_err(|_| Errno::EINVAL)
}

/// Parse a byte count that MAY be the cgroup-v2 literal `"max"` (the "no limit"
/// sentinel). `"max"` → `u64::MAX`; any malformed input → `None` (callers map to
/// `EINVAL`). Distinct return shape (`Option`) from `parse_u64` because the
/// translation arms treat `"max"` as a valid value, not an error.
fn parse_bytes_or_max(value: &[u8]) -> Option<u64> {
    let s = core::str::from_utf8(value).ok()?.trim();
    if s == "max" { Some(u64::MAX) } else { s.parse::<u64>().ok() }
}

/// Render a byte count into a fresh `CgroupV2Write.value` string, emitting the
/// v2 `"max"` sentinel for `u64::MAX` and the decimal value otherwise. A `u64`
/// is at most 20 digits, so the `ArrayString<64>` never overflows (the `write!`
/// / `try_push_str` results are therefore infallible here).
fn format_bytes_or_max(bytes: u64) -> ArrayString<64> {
    let mut out = ArrayString::<64>::new();
    if bytes == u64::MAX {
        let _ = out.try_push_str("max");
    } else {
        let _ = write!(&mut out, "{bytes}");
    }
    out
}

/// Render a `u64` decimal into a fresh `CgroupV2Write.value` (`ArrayString<64>`).
/// A `u64` is at most 20 digits, so the buffer never overflows (the `write!`
/// result is therefore infallible here). Used for `cpu.weight` / `io.weight`.
fn format_u64(v: u64) -> ArrayString<64> {
    let mut out = ArrayString::<64>::new();
    let _ = write!(&mut out, "{v}");
    out
}

/// Parse a v1 `cpu.cfs_quota_us` value. `-1` is the "no limit" sentinel (→ v2
/// `"max"`, see `format_cpu_max`); any other value is a positive-microsecond
/// quota. Malformed input → `Errno::EINVAL`.
fn parse_quota(value: &[u8]) -> Result<i64, Errno> {
    core::str::from_utf8(value)
        .map_err(|_| Errno::EINVAL)?
        .trim()
        .parse::<i64>()
        .map_err(|_| Errno::EINVAL)
}

/// Render a v2 `cpu.max` value `"$quota $period"` from a signed v1 quota
/// (`quota < 0` → the `"max"` sentinel) and a period in microseconds. At most
/// "max " + two 20-digit numbers, so `ArrayString<64>` never overflows.
fn format_cpu_max(quota: i64, period: u64) -> ArrayString<64> {
    let mut out = ArrayString::<64>::new();
    if quota < 0 {
        let _ = write!(&mut out, "max {period}");
    } else {
        let _ = write!(&mut out, "{quota} {period}");
    }
    out
}

/// The cgroup's current `cpu.max` period half (microseconds). Defaults to
/// 100,000 (100 ms) when no cpu controller is enabled — the value the cross-knob
/// `cpu.cfs_quota_us` translation combines with the newly written quota.
fn cgroup_cpu_max_period(cgroup: &Cgroup) -> u64 {
    let guard = rcu_read_lock();
    cgroup.cpu.read(&guard)
        .map_or(100_000, |c| c.period_us.load(Ordering::Relaxed))
}

/// The cgroup's current `cpu.max` quota half in the v1 signed convention: `-1`
/// ("max", unlimited) when the stored quota is `u64::MAX` or no cpu controller is
/// enabled, else the microsecond quota. Combined with a newly written
/// `cpu.cfs_period_us` by the cross-knob translation.
fn cgroup_cpu_max_quota(cgroup: &Cgroup) -> i64 {
    let guard = rcu_read_lock();
    match cgroup.cpu.read(&guard).map(|c| c.max_us.load(Ordering::Relaxed)) {
        Some(q) if q != u64::MAX => q as i64,
        _ => -1,
    }
}

/// The cgroup's current `memory.max` (bytes); `u64::MAX` ("max", unlimited) when
/// no memory controller is enabled. Used by the `memory.memsw.limit_in_bytes`
/// translation (`memory.swap.max = memsw - memory.max`).
fn cgroup_memory_max(cgroup: &Cgroup) -> u64 {
    let guard = rcu_read_lock();
    cgroup.memcg(&guard).map_or(u64::MAX, |m| m.max.load(Ordering::Relaxed))
}

/// Copy a trimmed cgroupfs write buffer verbatim into a `CgroupV2Write.value`
/// (`ArrayString<64>`) for direct-passthrough rows whose v1 and v2 encodings are
/// identical (e.g. `rdma.max`, the freezer `1`/`0` token). Input longer than 64
/// bytes → `Errno::EINVAL` (no silent truncation).
fn array_str_from(value: &[u8]) -> Result<ArrayString<64>, Errno> {
    let s = core::str::from_utf8(value).map_err(|_| Errno::EINVAL)?.trim();
    ArrayString::<64>::from(s).map_err(|_| Errno::EINVAL)
}

/// Map a v1 `hugetlb.<size>` knob prefix to its v2 `hugetlb.<size>.max` control
/// file. The `<size>` token is one of the architecturally fixed hugepage-size
/// spellings across the eight supported arches (64 KiB … 16 GiB); the mapping is
/// a fixed vocabulary of `&'static str` file names, NOT a runtime-discovered
/// capacity. The v1 knob exists only for a live hstate, so an unrecognized
/// spelling is unreachable for a well-formed shim input; the fallthrough returns
/// a non-existent path that the cgroupfs write layer rejects (safe failure, no
/// silent misdirection).
fn hugetlb_v2_path(size: &str) -> &'static str {
    match size {
        "hugetlb.64KB"  => "hugetlb.64KB.max",
        "hugetlb.512KB" => "hugetlb.512KB.max",
        "hugetlb.1MB"   => "hugetlb.1MB.max",
        "hugetlb.2MB"   => "hugetlb.2MB.max",
        "hugetlb.8MB"   => "hugetlb.8MB.max",
        "hugetlb.16MB"  => "hugetlb.16MB.max",
        "hugetlb.32MB"  => "hugetlb.32MB.max",
        "hugetlb.256MB" => "hugetlb.256MB.max",
        "hugetlb.512MB" => "hugetlb.512MB.max",
        "hugetlb.1GB"   => "hugetlb.1GB.max",
        "hugetlb.2GB"   => "hugetlb.2GB.max",
        "hugetlb.16GB"  => "hugetlb.16GB.max",
        _               => "hugetlb.__unknown_size.max",
    }
}

/// Build a v2 `io.max` write from a v1 blkio throttle row. The v1 value is
/// `"MAJ:MIN RATE"`; the v2 encoding is `"MAJ:MIN <kind>=RATE"` where `kind` is
/// one of `rbps`/`wbps`/`riops`/`wiops`. Malformed input, or a formatted result
/// exceeding the 64-byte `CgroupV2Write.value`, → `Errno::EINVAL`.
fn io_max_token(kind: &str, value: &[u8]) -> Result<CgroupV2Write, Errno> {
    let s = core::str::from_utf8(value).map_err(|_| Errno::EINVAL)?.trim();
    let (dev, rate) = s.split_once(' ').ok_or(Errno::EINVAL)?;
    let mut out = ArrayString::<64>::new();
    write!(&mut out, "{dev} {kind}={rate}").map_err(|_| Errno::EINVAL)?;
    Ok(CgroupV2Write { path: "io.max", value: out })
}

format_u64, format_bytes_or_max, format_cpu_max, array_str_from, io_max_token, and hugetlb_v2_path all produce ArrayString<64> (or a CgroupV2Write wrapping one; io_max_token returns Result on a <64-byte overflow) — none allocate a heap String, so every CgroupV2Write.value is the fixed-capacity type the struct declares. They are the same trivial pseudocode-formatter convention as the pre-existing format_bytes_or_max / format_cpu_max used by the arms above.

17.3 POSIX Inter-Process Communication (IPC)¶

UmkaOS supports standard POSIX IPC mechanisms, optimized using UmkaOS's native zero-copy primitives where possible.

17.3.1 AF_UNIX Sockets¶

Local domain sockets (AF_UNIX) are heavily used in containerized environments (e.g., Docker, Kubernetes).

Zero-Copy Process-to-Process Rings: For SOCK_STREAM sockets, UmkaOS maps the connection to a pair of single-producer/single-consumer (SPSC) ring buffers shared directly between the two processes. These are distinct from the kernel-domain KABI ring buffers (Section 11.7), which are fixed-size command/completion rings for Tier 0/Tier 1 communication. The AF_UNIX ring buffer is:

/// Cache-line-aligned wrapper to prevent false sharing between fields
/// accessed by different CPUs or different producer/consumer threads.
///
/// The alignment is platform-dependent, defined by `CACHE_LINE_SIZE`:
/// - x86-64: 64 bytes (but spatial prefetcher pairs → 128-byte effective)
/// - AArch64: 64 or 128 bytes (Neoverse V2 / Apple M-series)
/// - ARMv7: 32 or 64 bytes
/// - RISC-V: 64 bytes (typical)
/// - PPC32: 32 bytes
/// - PPC64LE: 128 bytes (POWER9/10)
/// - s390x: 256 bytes (z13+)
/// - LoongArch64: 64 bytes (3A5000/6000)
///
/// `CACHE_LINE_SIZE` is a compile-time constant per target:
///   32  — PPC32
///   64  — x86-64, AArch64 (default), ARMv7, RISC-V, LoongArch64
///   128 — PPC64LE, AArch64 with the `cache-line-128` build feature
///   256 — s390x
///
/// The `CacheAligned` wrapper uses the target's `CACHE_LINE_SIZE` to ensure
/// no false sharing on any supported platform.
///
/// **AArch64 128-byte build config.** AArch64 microarchitectures differ in
/// effective cache-line/prefetch granularity: Cortex-A cores use 64 bytes,
/// but Apple M-series and Neoverse V2 pair adjacent 64-byte lines into a
/// 128-byte prefetch unit. Because `#[repr(align(N))]` demands a literal,
/// UmkaOS selects the AArch64 alignment with a Cargo **build feature**
/// (`cache-line-128`, declared in the crate `[features]` table) — the
/// UmkaOS analogue of Linux's `CONFIG_ARM64_L1_CACHE_SHIFT` Kconfig option.
/// A distro building for Apple/Neoverse-V2 targets enables the feature; the
/// default AArch64 build keeps 64. The feature is meaningful only on
/// `target_arch = "aarch64"` (ignored elsewhere).
///
/// Implementation: Rust's `#[repr(align(N))]` requires a literal, so we use
/// `cfg_attr` to select the correct alignment per target. The `not(any(...))`
/// fall-through arm below EXCLUDES the aarch64+`cache-line-128` case so it
/// does not collide with the 128 arm.
#[cfg_attr(target_arch = "s390x", repr(C, align(256)))]
#[cfg_attr(target_arch = "powerpc64", repr(C, align(128)))]
#[cfg_attr(all(target_arch = "aarch64", feature = "cache-line-128"), repr(C, align(128)))]
#[cfg_attr(target_arch = "powerpc", repr(C, align(32)))]
#[cfg_attr(
    not(any(
        target_arch = "s390x",
        target_arch = "powerpc64",
        target_arch = "powerpc",
        all(target_arch = "aarch64", feature = "cache-line-128"),
    )),
    repr(C, align(64))
)]
pub struct CacheAligned<T>(pub T);

/// Platform-dependent cache line size constant.
/// Must agree with the `CacheAligned` alignment above — the `const_assert!`
/// below turns "must agree" from prose into a compile-time check.
#[cfg(target_arch = "s390x")]
pub const CACHE_LINE_SIZE: usize = 256;
#[cfg(target_arch = "powerpc64")]
pub const CACHE_LINE_SIZE: usize = 128;
#[cfg(all(target_arch = "aarch64", feature = "cache-line-128"))]
pub const CACHE_LINE_SIZE: usize = 128;
#[cfg(target_arch = "powerpc")]
pub const CACHE_LINE_SIZE: usize = 32;
#[cfg(not(any(
    target_arch = "s390x",
    target_arch = "powerpc64",
    target_arch = "powerpc",
    all(target_arch = "aarch64", feature = "cache-line-128"),
)))]
pub const CACHE_LINE_SIZE: usize = 64;

// The constant and the wrapper alignment are selected by two independent
// `cfg`/`cfg_attr` cascades; this assertion fails the build if they ever
// drift apart (e.g. a new arm added to one but not the other).
const_assert!(CACHE_LINE_SIZE == align_of::<CacheAligned<u8>>());

/// Process-to-process SPSC ring for AF_UNIX **connected** sockets
/// (`SOCK_STREAM` and `SOCK_SEQPACKET` — exactly two endpoints). It is a
/// data-plane *fast path* under the canonical AF_UNIX object defined in
/// [Section 16.20](16-networking.md#afunix-socket-specification): that section remains authoritative for
/// addressing, credentials (`SO_PEERCRED`, `SCM_CREDENTIALS`), fd passing
/// (`SCM_RIGHTS`), and LSM checks. `SOCK_DGRAM` does NOT use this ring (it is
/// connectionless and multi-sender — see "SOCK_DGRAM" below).
///
/// # Two mapped regions with DISTINCT permissions
///
/// A connection owns TWO kernel-allocated, pinned (non-swappable) regions,
/// both created atomically at `connect()`/`socketpair()` time:
///
/// 1. **Control block** (`SpscControl`, below): mapped **read-write into BOTH
///    endpoints**. It holds the two byte cursors and the rendezvous futex.
///    Both parties must WRITE it (the receiver advances `read_pos` and clears
///    the futex; the sender advances `write_pos` and clears the futex), so a
///    read-only receiver mapping — the flaw in the previous design — cannot
///    work. The control block is the userspace-visible ABI operated by
///    `umka-sysapi`.
/// 2. **Data buffer** (`buffer`, `capacity` bytes, power of two): mapped
///    **read-write in the sender, read-only in the receiver**. The receiver
///    only ever *reads* stream bytes, so the RO mapping is a defense-in-depth
///    barrier (a buggy/hostile receiver cannot corrupt bytes the sender
///    committed) while still letting it advance its cursor in the RW control
///    block. One data buffer per direction; a bidirectional stream is a PAIR
///    of `UserSpscRing`s (each with its own control block + data buffer).
///
/// # Safety / lifetime
///
/// - **Allocation**: kernel allocates the control page + `capacity` data bytes
///   from the page allocator at connection time; pages are pinned.
/// - **Lifetime**: refcounted via `Arc<UserSpscRing>`; the last `Arc` drop (on
///   socket close) unmaps both sides and returns the pages. If one process
///   exits while the other holds a reference, the survivor's mappings remain
///   valid until its socket closes.
pub struct UserSpscRing {
    /// Data buffer (stream bytes). RW in the sender, RO in the receiver.
    /// Kernel virtual address of the pinned region; the same physical pages
    /// are mapped into both processes with the per-endpoint permissions above.
    pub buffer: *mut u8,

    /// Data buffer size in bytes (power of two for efficient masking).
    pub capacity: usize,

    /// Kernel virtual address of the shared `SpscControl` page (the same page
    /// mapped RW into both endpoints). Cursors and the futex live HERE, not in
    /// this kernel-side struct, so that userspace can operate them directly.
    pub control: *mut SpscControl,
}

/// Shared control block for a `UserSpscRing`. Mapped **read-write into both
/// endpoints**. Its layout is process-visible ABI: `umka-sysapi` reads/writes
/// the cursors and futex in userspace and enters the kernel only via `futex`
/// when a party must block. `#[repr(C)]` with a const-asserted, cache-line
/// layout so field offsets are identical in both address spaces.
#[repr(C)]
pub struct SpscControl {
    /// Sender's free-running byte cursor (bytes ever written; wraps mod 2^64).
    /// Written by the sender with `Release` after copying data; read by the
    /// receiver with `Acquire`. On the wake-side handshake (the sender has just
    /// produced into an empty ring) the store is `SeqCst` so it cannot reorder
    /// past the following `futex_word` read — see `futex_word`. On its own
    /// cache line (written by the sender, read by the receiver → false-sharing
    /// hazard without the separation).
    pub write_pos: CacheAligned<AtomicU64>,

    /// Receiver's free-running byte cursor (bytes ever consumed; wraps mod
    /// 2^64). Written by the receiver with `Release`; read by the sender with
    /// `Acquire`. On the wake-side handshake (the receiver has just freed space
    /// in a full ring) the store is `SeqCst` so it cannot reorder past the
    /// following `futex_word` read — see `futex_word`. Own cache line.
    pub read_pos: CacheAligned<AtomicU64>,

    /// Rendezvous futex. Both parties write it, so it lives in the RW control
    /// block (never a read-only mapping) and sits on its own cache line so it
    /// does not falsely share with either cursor.
    ///
    /// Protocol (SPSC ⇒ at most one party blocks at a time; a ring cannot be
    /// simultaneously full and empty, so the `1` and `2` states are mutually
    /// exclusive and the two writers never clobber a live value):
    ///   0 = IDLE (no waiter).
    ///   1 = SENDER_WAITING (sender blocked; ring full).
    ///   2 = RECEIVER_WAITING (receiver blocked; ring empty).
    ///
    /// The block/wake handshake is the canonical futex **arm → RE-CHECK →
    /// wait** sequence. Arming the flag and re-reading the ring condition MUST
    /// NOT reorder, and the wake side's cursor advance and flag read carry the
    /// same StoreLoad ordering — so BOTH the flag access that pairs with a
    /// cursor and the paired cursor access are `SeqCst` (equivalently an
    /// explicit `fence(SeqCst)` between them). The `FUTEX_WAIT` value compare
    /// ALONE is insufficient: it only rejects a clear that lands *after* the
    /// arm and *before* the syscall; it does NOT cover a consume that lands
    /// *before* the arm (T0 sender sees full; T1 receiver drains and reads the
    /// flag while it is still 0, so it wakes nobody; T2 sender arms and waits
    /// on a ring that already has space — deadlock). The RE-CHECK after arming
    /// closes exactly that window.
    ///
    /// Sender, ring full (`write_pos.wrapping_sub(read_pos) == capacity`):
    ///   1. `futex_word.store(1, SeqCst)`             // arm SENDER_WAITING
    ///   2. RE-CHECK: reload `write_pos.wrapping_sub(read_pos)`. If `< capacity`
    ///      the receiver drained in the window → `futex_word.store(0, Release)`
    ///      and resume sending (do NOT wait).
    ///   3. else `FUTEX_WAIT(&futex_word, 1)`         // kernel re-compares == 1;
    ///      a clear landing here returns the wait at once. On wake, reload and
    ///      re-evaluate fullness before retrying step 1.
    /// Receiver, after advancing `read_pos` (freeing space): publish the cursor
    ///   with `read_pos.store(.., SeqCst)`, then if `futex_word.load(SeqCst) == 1`
    ///   → `futex_word.store(0, Release)` + `FUTEX_WAKE(&futex_word)`.
    ///
    /// Receiver empty (`write_pos == read_pos`) is the mirror with value 2:
    ///   arm `futex_word.store(2, SeqCst)`, RE-CHECK `write_pos != read_pos`
    ///   (sender produced in the window → `store(0, Release)` and resume
    ///   consuming), else `FUTEX_WAIT(&futex_word, 2)`. The sender, after
    ///   advancing `write_pos` with `SeqCst`, wakes a value-2 waiter
    ///   (`futex_word.load(SeqCst) == 2` → `store(0, Release)` + `FUTEX_WAKE`).
    pub futex_word: CacheAligned<AtomicU32>,

    /// Ring capacity in bytes (power of two). Immutable after creation;
    /// duplicated here so userspace can mask cursors without a kernel call.
    pub capacity: u64,
}
// Three cache-line-isolated fields plus the immutable `capacity` tail, which
// the struct's cache-line alignment rounds up to a fourth line. Ties the ABI
// size to CACHE_LINE_SIZE on every target (256/512/1024 B for 64/128/256).
const_assert!(size_of::<SpscControl>() == 4 * CACHE_LINE_SIZE);

Cursor wrap safety (50-year budget). write_pos/read_pos are free-running u64 byte counters, never reset. At a sustained 100 GB/s AF_UNIX stream (well above any real workload) 2^64 bytes elapse in ~5.8 years — inside the 50-year envelope — so the wrap MUST be correct, not merely improbable. It is correct because every comparison uses wrapping (modular) arithmetic on the unsigned difference, never an ordered </> between the raw positions:

Bytes in flight: used = write_pos.wrapping_sub(read_pos). The invariant 0 ≤ used ≤ capacity holds across the wrap because the true difference is bounded by capacity and wrapping_sub yields it exactly mod 2^64.
Space available: capacity - used. Ring full ⇔ used == capacity. Ring empty ⇔ used == 0 (equivalently write_pos == read_pos).

A naïve implementation comparing the raw positions (write_pos < read_pos) would misfire the instant write_pos wraps past read_pos — deadlocking the sender or corrupting the ring. umka-sysapi and any external operator of the control block MUST use wrapping_sub; this is the documented wrap-safety analysis the counter-longevity rule requires.

The umka-sysapi layer intercepts plain send()/recv()/read()/write() (no ancillary data) and translates them into ring enqueues/dequeues.
Data is copied twice: once from the sender's buffer into the shared ring, once from the ring into the receiver's buffer. This eliminates the traditional kernel-buffer intermediate copy, reducing the path from 3 copies to 2.
For the plain-data path the kernel is invoked only via futex when a ring is full/empty and the process must block. Connection setup/teardown, credentials, and ancillary data are always kernel-mediated (next bullet).

Ancillary data and credentials are kernel-mediated (reconciliation with Section 16.20). The ring carries only the ordinary byte/message stream; it never carries SCM_RIGHTS/SCM_CREDENTIALS/ SCM_SECURITY, which require fd translation, credential capture, and LSM checks that only the kernel can perform. A sendmsg() that carries a control message therefore enters the kernel, which:

Processes the cmsg exactly as Section 16.20 specifies (fd File-reference duplication, SO_PEERCRED/ucred capture, two-pass LSM validation).
Anchors the cmsg to the sender's current write_pos — the free-running stream byte offset — in a kernel-side control record for the connection, and advances write_pos past any accompanying in-band bytes normally. Because write_pos/read_pos are a byte offset both sides agree on, the kernel needs no view of the byte stream to place the boundary: when the receiver's recvmsg() reaches a read_pos that equals a record's anchor offset, the kernel returns that record's ancillary data and stops the read at the cmsg boundary (POSIX recvmsg() boundary semantics). This is the UmkaOS design choice that lets a zero-copy userspace ring coexist with Linux-faithful SCM_* semantics; the "attached to the in-flight sk_buff" wording in Section 16.20 is the same mechanism expressed against the kernel-mediated queue — the ancillary record is keyed by ring stream offset rather than by an in-flight kernel queue position. (Deferred handoff by symbol: Section 16.20 SCM_RIGHTS / SCM_CREDENTIALS — state that for the connected-ring fast path the ancillary record anchors to the UserSpscRing stream offset.)
SO_PEERCRED is snapshotted at connect()/socketpair() (kernel-side), independent of the ring, per Section 16.20.

SOCK_SEQPACKET message boundaries: SOCK_SEQPACKET (also connected, exactly two endpoints) uses the ring with a 4-byte length header before each message so recv() returns exactly one message per call:

/// Message format in SOCK_SEQPACKET ring:
/// | msg_len: u32 | data: [u8; msg_len] | msg_len: u32 | data: ... |
///
/// The receiver reads msg_len, then reads exactly that many bytes.
/// Short reads (buffer smaller than msg_len) discard the remainder of the message.

SOCK_DGRAM does NOT use the SPSC ring. A SOCK_DGRAM AF_UNIX socket is connectionless: arbitrarily many senders may sendto() the same receiver, which violates the single-producer invariant the ring depends on. SOCK_DGRAM uses the kernel-mediated MPSC receive queue of Section 16.20 — a per-socket queue of datagrams protected by the socket lock, preserving message boundaries and per-datagram credentials/ancillary data. The zero-copy ring is strictly an optimization for the two-endpoint connected types (SOCK_STREAM, SOCK_SEQPACKET).

17.3.2 Pipes and FIFOs¶

Standard pipes are implemented as bounded in-memory buffers managed by the VFS. - For high-throughput scenarios, applications can use vmsplice() to zero-copy data from a pipe into a memory-mapped region. - Internally, a pipe is a specialized VfsNode that maintains a wait queue for readers and writers.

17.3.2.1 Pipe Buffer¶

/// Default pipe ring size: **16 pages** (`PIPE_DEF_BUFFERS` in Linux
/// `include/linux/pipe_fs_i.h`), matching Linux since 2.6.11. This is a page
/// COUNT, not a byte count — the **page count is authoritative** and the byte
/// capacity is derived at runtime as `PIPE_DEFAULT_PAGES * PAGE_SIZE`.
pub const PIPE_DEFAULT_PAGES: usize = 16;

/// Pipe buffer: inline storage for the common case (≤ 16 pages), with heap
/// fallback for pipes expanded via `fcntl(F_SETPIPE_SZ)`.
///
/// Allocated when `pipe(2)`/`pipe2(2)` (or FIFO open) is called. At creation
/// `capacity` is set to `PIPE_DEFAULT_PAGES * PAGE_SIZE` and the page ring
/// starts EMPTY — no `PhysPage` is allocated yet (see "Page population" in the
/// write algorithm); the inline slots hold `MaybeUninit<PipePage>`.
///
/// **Default capacity is `16 * PAGE_SIZE`, NOT a hardcoded 65536.** On the
/// common 4 KiB-page targets this is 65536 bytes; on 16 KiB / 64 KiB-page
/// targets (some AArch64 and PPC64LE configurations) it is 256 KiB / 1 MiB —
/// exactly Linux, whose default is likewise `PIPE_DEF_BUFFERS << PAGE_SHIFT`
/// (Linux `fs/pipe.c alloc_pipe_info()`, verified against torvalds/linux master).
/// `fcntl(F_GETPIPE_SZ)` therefore returns `16 * PAGE_SIZE` for a fresh pipe.
/// Baking in 4096 would break `F_GETPIPE_SZ` binary compatibility on
/// large-page hosts, so no code in this section may assume `PAGE_SIZE == 4096`
/// (CLAUDE.md, no 4 KiB assumptions). The size is configurable via
/// `fcntl(F_SETPIPE_SZ)` (rounded up to a whole number of `PAGE_SIZE` pages,
/// minimum one page) up to `/proc/sys/fs/pipe-max-size` (default 1 MiB; root
/// with `CAP_SYS_RESOURCE` may raise further, hard limit `2^31` bytes per
/// Linux `round_pipe_size()`).
///
/// (Deferred handoff by symbol: [Section 14.17](14-vfs.md#pipes-and-fifos) "Capacity" says
/// "Default pipe capacity: 65536 bytes" unconditionally — it should read
/// `16 * PAGE_SIZE` (65536 on 4 KiB pages) to stay correct on large-page
/// targets. Its "Key `PipeBuffer` fields" summary also lists field names
/// (`pages: ArrayVec`, `r_idx`/`w_idx`, `capacity: u32`,
/// `active_writer: AtomicU64`) that DIVERGE from the canonical struct below;
/// reconcile it to the field names here — this file is canonical.)
///
/// **Zero-copy optimization**: When a pipe page is "gifted" via vmsplice()
/// with SPLICE_F_GIFT, the page is transferred to the pipe without copying.
/// The gifted page is unmapped from the sender's address space and becomes
/// owned by the pipe until read. This enables zero-copy data pipelines.
///
/// **Allocation model**: The inline `pages_small` array covers the standard
/// default pipe (16 pages = `16 * PAGE_SIZE`). When `fcntl(F_SETPIPE_SZ)` sets
/// capacity beyond 16 pages, the buffer transitions to `pages_large` (a heap-allocated
/// `Vec<PipePage>`). This hybrid approach keeps the struct compact (384 bytes of
/// inline page storage vs. the previous 6144 bytes) while supporting the full
/// Linux pipe size range.
pub struct PipeBuffer {
    // === First cache line(s): hot-path lock-free atomic fields ===
    // These fields are accessed on every read/write syscall without holding
    // any lock. Placing them first ensures they occupy the initial cache
    // lines of the heap-allocated struct, minimising cache misses on the
    // common single-reader/single-writer path.

    /// Index of the first page with data (read cursor).
    pub read_idx: AtomicU32,

    /// Index of the first empty page (write cursor).
    pub write_idx: AtomicU32,

    /// Byte offset within pages[read_idx] for partial reads.
    pub read_offset: AtomicU32,

    /// Byte offset within pages[write_idx] for partial writes.
    pub write_offset: AtomicU32,

    /// Total bytes currently in the pipe (atomic for lock-free size check).
    pub len: AtomicU32,

    /// Total pipe capacity in bytes. Set at creation to `16 * PAGE_SIZE`
    /// (65536 on 4 KiB-page targets), retuned by `fcntl(F_SETPIPE_SZ)`. The
    /// ring SLOT count `page_count()` is `capacity / PAGE_SIZE` — always ≥ 1.
    pub capacity: AtomicU32,

    /// Seqlock for detecting concurrent fcntl(F_SETPIPE_SZ) during lock-free writes.
    /// Uses the `SeqLock` protocol defined in
    /// [Section 3.6](03-concurrency.md#lock-free-data-structures--seqlockt-sequence-lock): odd values indicate
    /// resize in progress; even values indicate stable. Writers read before and after;
    /// if changed, retry. Incremented twice per F_SETPIPE_SZ resize. At 1 resize/sec
    /// (unrealistic), wraps in ~292 billion years with u64.
    pub resize_seq: AtomicU64,

    /// Count of active single-writer fast-path operations.
    /// fcntl(F_SETPIPE_SZ) waits for this to reach 0 before resizing.
    pub active_writer: AtomicU32,

    // === Warm fields: reader/writer reference counts and page count ===

    /// Number of readers (for detecting write-side SIGPIPE).
    /// When this drops to 0, write() returns EPIPE.
    pub reader_count: AtomicU32,

    /// Number of writers (for detecting read-side EOF).
    /// When this drops to 0 and the pipe is empty, read() returns 0.
    pub writer_count: AtomicU32,

    /// Count of **populated** inline slots in `pages_small` — slots `[0,
    /// small_len)` hold an initialized `PipePage` with an allocated `PhysPage`;
    /// slots `[small_len, page_count())` are still `MaybeUninit`. Starts at 0
    /// on a fresh pipe and grows by one each time a write reaches the
    /// population frontier (`write_idx == small_len`), so it is the lazy-
    /// allocation high-water mark, capped at `page_count()` (≤
    /// `PIPE_DEFAULT_PAGES`). Because `write_idx` advances by 1 (mod
    /// `page_count()`) it never runs ahead of `small_len`, keeping the
    /// populated region a contiguous prefix. `0` also once `pages_large` is in
    /// use (the heap path pre-populates its slots at resize — see below).
    pub small_len: AtomicU8,

    // === Cold fields: locks, wait queues, and page storage ===
    // Only accessed on blocked paths (empty/full) and on resize.

    /// Wait queue for blocked readers (pipe empty).
    pub read_wait: WaitQueueHead,

    /// Wait queue for blocked writers (pipe full).
    pub write_wait: WaitQueueHead,

    /// Lock for modifying the page ring (growing/shrinking, page population)
    /// and the multi-writer path. The lock-free single-writer path does not
    /// hold this lock.
    ///
    /// **Lock class**: `ring_lock` is a **sleeping `Mutex`** (Linux
    /// `pipe_inode_info.mutex` parity), NOT a `Lock<T, LEVEL>` spin lock, so it
    /// carries no numeric spin level. Per the sleeping-mutex discipline
    /// ([Section 3.4](03-concurrency.md#cumulative-performance-budget--lock-ordering)), it MAY nest the
    /// page-allocator spin locks (`BUDDY_LOCK`/`SLAB_LOCK` — `SLAB_LOCK` is
    /// level **130**, not 13) UNDER it when a write must allocate a `PhysPage`
    /// to populate a fresh ring slot, and it is never acquired while already
    /// holding any `Lock<T, LEVEL>` spin lock. Sleeping under it (blocking
    /// `PhysPage` allocation) is therefore legal, which is precisely why it is
    /// a sleeping mutex rather than a spin lock. (The prior comment cited a
    /// nonexistent `PIPE_LOCK(14)` above a mis-numbered `SLAB_LOCK(13)`;
    /// neither is in the master lock table.)
    pub ring_lock: Mutex<()>,

    /// Reader serialization lock for multi-reader FIFOs.
    ///
    /// The read cursor is a PAIR — `(read_idx, read_offset)` — which no single
    /// atomic can advance indivisibly, so concurrent readers CANNOT lock-free
    /// "claim" a byte range (the previous "`read_idx.fetch_add()`" description
    /// was unsound: `read_idx` is a page index and the read algorithm advances
    /// the pair with `load`+`store`, never `fetch_add`). Multiple readers
    /// therefore SERIALIZE: when `reader_count > 1`, each reader acquires this
    /// lock in **exclusive** mode for the duration of its cursor advance, so
    /// exactly one reader mutates `(read_idx, read_offset)` at a time (Linux
    /// serializes all pipe reads under `pipe->mutex`). The lock-free read fast
    /// path (`load`+`store` on the cursor, no lock) is used ONLY when a single
    /// reader is present (`reader_count == 1`). Writers do NOT acquire this
    /// lock; the write path uses `ring_lock` (multi-writer) or the lock-free
    /// single-writer path. (`RwLock` rather than `Mutex` only so a future
    /// read-side snapshot operation — e.g. `FIONREAD` — can take it shared;
    /// cursor-advancing readers always take it exclusive.)
    pub read_lock: RwLock<()>,

    /// Heap-allocated pages for pipes expanded beyond 16 pages.
    /// `None` until the first `fcntl(F_SETPIPE_SZ)` exceeding 16 pages.
    /// Allocated from the general kernel heap (not slab) because expanded
    /// pipe buffers are rare and size varies. Accessed only while holding
    /// `ring_lock`.
    pub pages_large: Option<Vec<PipePage>>,

    /// Inline storage for ≤ 16 pages (covers the `16 * PAGE_SIZE` default pipe).
    /// Zero-allocation fast path for the common case.
    /// `MaybeUninit` avoids initialization cost for unused slots while
    /// keeping stack safety — only `small_len` entries are valid.
    /// Placed last so the hot atomic fields above occupy the initial cache lines.
    pub pages_small: [MaybeUninit<PipePage>; PIPE_DEFAULT_PAGES],
}

> **Design rationale**: `PipeBuffer` is heap-allocated (not stack-allocated). The hot-path
> atomic counters (`read_idx`, `write_idx`, `len`, `resize_seq`, `active_writer`) are placed
> **first** so they occupy the initial cache lines of the allocation; the 384-byte
> `pages_small` array is placed **last** so it does not evict the hot counters on
> lock-free read/write paths.
>
> **Inline vs. heap page storage**: `pages_small` covers the standard default
> pipe (16 pages = `16 * PAGE_SIZE`). The previous design used a 256-slot inline array
> (`[PipePage; 256]` = ~6144 bytes) sized for the maximum possible pipe (1 MB via
> `F_SETPIPE_SZ`), which is rarely reached in practice — default Linux pipes are 16
> pages (64 KiB on 4 KiB-page hosts), and most pipes never exceed this. The 16-slot inline array reduces the
> struct's page-storage footprint from 6144 bytes to 384 bytes (16 x 24B), a 16x
> reduction that dramatically improves slab allocator cache density.
>
> **Transition to heap**: When `fcntl(F_SETPIPE_SZ)` sets capacity beyond 16 pages
> (> `16 * PAGE_SIZE`), the buffer transitions to `pages_large` (`Vec<PipePage>`) and `small_len`
> is set to 0. This transition is uncommon in production workloads. The `Vec` is
> allocated from the general kernel heap (not slab) because expanded pipe sizes vary.
>
> **`MaybeUninit` wrapper**: The `MaybeUninit<PipePage>` wrapper avoids initialization
> cost for unused inline slots while maintaining stack safety. Only the first
> `small_len` entries contain valid data; the remainder are uninitialised memory.

/// A single page in the pipe buffer — the UmkaOS analogue of Linux
/// `struct pipe_buffer` (`page` + `offset` + `len`). Kernel-internal (not a
/// KABI/wire struct), so `AtomicBool`/`AtomicU32` interior mutability is fine.
pub struct PipePage {
    /// Physical page containing the data.
    /// Allocated from the page allocator or gifted via vmsplice.
    pub page: PhysPage,

    /// Byte offset within `page` where this buffer's valid data STARTS
    /// (Linux `pipe_buffer.offset`). `0` for a normal `write()`-populated
    /// page. Non-zero for splice: `splice(file → pipe)` from mid-file and
    /// pipe-to-pipe splice of a partially-consumed page produce buffers whose
    /// data does not begin at byte 0. The read algorithm reads from
    /// `page[offset + read_offset ..]`, so a spliced page with a starting
    /// offset is expressible (it was not, before this field existed).
    pub offset: AtomicU32,

    /// Number of valid bytes in this buffer, starting at `offset`
    /// (`offset + len <= PAGE_SIZE`). `0` = empty. For a full standard write
    /// this is `PAGE_SIZE` (with `offset == 0`); partial pages are possible.
    /// `u32` (not `usize`) to match `read_offset`/`write_offset` and the other
    /// pipe cursors — a page length never exceeds `PAGE_SIZE ≤ 64 KiB`.
    pub len: AtomicU32,

    /// True if this page was gifted via vmsplice(SPLICE_F_GIFT).
    /// Gifted pages are unmapped from the sender and transferred to
    /// the reader; standard pages are copied.
    pub is_gifted: AtomicBool,
}

Page ring model and on-demand population. The page ring has a fixed number of SLOTS — page_count() — and each slot's backing PhysPage is allocated the first time a write reaches it (lazy allocation, matching Linux fs/pipe.c, which Linux alloc_page()s in pipe_write). Two definitions the algorithms below depend on:

// Ring SLOT count = the modulus for index wraparound. Derived from CAPACITY
// (the byte size set at creation / F_SETPIPE_SZ), NOT from small_len (the
// populated-slot high-water mark). Always ≥ 1 — a pipe's capacity is never
// less than one page — so `% page_count()` can NEVER divide by zero, even on
// a brand-new pipe whose small_len is still 0.
fn page_count(pipe) -> u32:
    return (pipe.capacity.load(Acquire) / PAGE_SIZE) as u32   // ≥ 1

// Return &pages[idx], allocating its PhysPage on first use. Called before any
// copy into a slot. `idx < page_count()` always (the caller masks with
// page_count()). Returns ENOMEM if a fresh page cannot be allocated.
fn ensure_slot(pipe, idx) -> Result<&PipePage, ENOMEM>:
    if pipe.small_len == 0 && pipe.pages_large.is_some():
        // Heap ring: slots are pre-populated at resize time (cold path), so
        // every idx in range already has a PhysPage.
        return Ok(&pipe.pages_large[idx])
    // Inline ring: slots [0, small_len) are populated; grow the prefix by one
    // when the write frontier reaches an unpopulated slot. Because write_idx
    // advances by 1 (mod page_count), the frontier is always exactly small_len.
    let sl = pipe.small_len.load(Acquire) as u32
    if idx == sl:                         // fresh slot at the frontier
        let page = alloc_phys_page()?     // ENOMEM path (see write step)
        pipe.pages_small[idx].write(PipePage {
            page, offset: AtomicU32::new(0),
            len: AtomicU32::new(0), is_gifted: AtomicBool::new(false),
        })
        pipe.small_len.store((idx + 1) as u8, Release)   // publish population
    // else idx < sl: already populated (reused as the ring cycles) — reuse it.
    return Ok(pipe.pages_small[idx].assume_init_ref())

The inline ring never over-allocates: for a default pipe at most PIPE_DEFAULT_PAGES PhysPages are ever allocated, and only as data actually fills the pipe. pages_large (the F_SETPIPE_SZ > 16 heap ring) pre-allocates all page_count() PhysPages at resize — a cold path the caller opted into by asking for a large pipe — so its slots need no per-write population branch.

Pipe write algorithm (lock-free fast path):

Note: This algorithm assumes single-writer semantics for the lock-free fast path. POSIX pipes technically allow multiple concurrent writers, but such usage requires atomic writes smaller than PIPE_BUF (4096 bytes) to guarantee data integrity. For UmkaOS's high-performance path, the lock-free algorithm below requires exactly one concurrent writer — multi-writer scenarios fall back to the mutex-protected slow path described below.

Multi-writer slow path (POSIX atomicity guarantee for writes ≤ PIPE_BUF):

When multiple writers are detected (via writer_count.load(Acquire) > 1), all writers acquire ring_lock (a mutex) before writing. Under ring_lock: 1. The writer checks available space (same as step 3 of the fast path). 2. If remaining ≤ PIPE_BUF (4096), the write is performed atomically: all bytes are written to contiguous pages before write_idx is advanced. If insufficient contiguous space exists, the writer sleeps on the pipe's wait queue until space is available (matching Linux POSIX behaviour). 3. If remaining > PIPE_BUF, POSIX does not guarantee atomicity. The write proceeds page-by-page under the mutex (may interleave with other large writes). 4. write_idx and len are updated under the mutex, then ring_lock is released.

The interaction with the lock-free reader is safe because the reader only reads committed pages (visible via len.load(Acquire)), and the reader's read_idx advancement is atomic. The resize_seq seqlock interaction is the same as the fast path — fcntl(F_SETPIPE_SZ) acquires ring_lock and waits for active writers.

Multi-reader coordination: When multiple readers exist on a FIFO (reader_count > 1), they SERIALIZE — each reader acquires read_lock in exclusive mode for the duration of its cursor advance, so exactly one reader mutates the (read_idx, read_offset) pair at a time. Concurrent lock-free "byte claiming" is NOT possible: the two-word cursor cannot be advanced by a single atomic, so read_idx.fetch_add() (a page-index increment) would never give a reader an exclusive, contiguous byte range. This matches Linux, which serializes all pipe reads under pipe->mutex. The lock-free read fast path is reserved for the single-reader case (reader_count == 1).

Splice / tee structural moves: splice() and tee() MOVE PipePage entries into or out of the ring (page-reference transfer, Section 14.17). Because that mutates ring structure (which slots are populated, their offset/len), a splice/tee acquires ring_lock (the structural-mutation lock, same as F_SETPIPE_SZ) and bumps resize_seq around the move, so any concurrent lock-free reader/writer detects the change via the seqlock and retries — the reader never observes a half-moved page. A gifted or spliced page carries a nonzero offset; the read algorithm above reads from page.offset + read_off, so partially-consumed spliced pages are handled without a separate code path.

Resize safety: The lock-free write path uses a resize_seq: AtomicU64 seqlock to detect concurrent fcntl(F_SETPIPE_SZ) operations. Before starting the write loop, the writer reads the seqlock; after completing each page, it re-checks. If the seqlock changed, the writer retries from the beginning with the new page count. fcntl(F_SETPIPE_SZ) acquires ring_lock, waits for in-flight single-writers via an active_writer count, increments resize_seq, performs the resize (potentially transitioning from pages_small to pages_large when expanding beyond 16 pages), and increments resize_seq again. This ensures the lock-free path never observes an inconsistent buffer size.

write(pipe, data, len):
  0. remaining = len; written = 0
  1. seq_start = resize_seq.load(Acquire)  // Capture resize generation
  2. If reader_count.load(Acquire) == 0: return EPIPE (SIGPIPE to caller)
     // TOCTOU note: reader may close between this check and write. This is
     // acceptable per POSIX — data written to a pipe with no readers is simply
     // discarded, and the next write() will observe reader_count == 0 and
     // return EPIPE. The pipe remains consistent; no data corruption occurs.
  3. // Try to claim fast path via compare-and-swap
     if !active_writer.compare_exchange(0, 1, Acquire, Relaxed).is_ok():
         // Another writer active — take slow path with ring_lock
         return write_slow_path(pipe, data, len)
  4. current_num_pages = page_count(pipe)  // ring SLOT count (capacity/PAGE_SIZE), ≥ 1
  5. while remaining > 0:
       a. If len.load() >= capacity.load():
          // Pipe full — block on write_wait
          active_writer.store(0, Release)  // Release during wait
          if write_wait.wait_event(|| len.load() < capacity).is_err():
              return written  // bytes successfully written before interrupt
          // Re-acquire fast path and re-check
          if !active_writer.compare_exchange(0, 1, Acquire, Relaxed).is_ok():
              // Lost to another writer during wait — take slow path
              return written + write_slow_path(pipe, &data[written..], remaining)
          if reader_count.load(Acquire) == 0:
              active_writer.store(0, Release)
              return EPIPE
          // Check for resize during wait
          if resize_seq.load(Acquire) != seq_start:
              active_writer.store(0, Release)
              goto 1  // Retry with new seq_start; written/remaining preserved

       b. write_idx_val = write_idx.load(Relaxed)
       c. write_off = write_offset.load(Relaxed)
       d. // Populate the target slot on demand (allocates a PhysPage the first
          //    time this slot is written). This is the step whose absence made
          //    a fresh pipe copy into uninitialised MaybeUninit slots.
          page = match ensure_slot(pipe, write_idx_val):
              Ok(p) => p,
              Err(ENOMEM) =>
                  active_writer.store(0, Release);
                  // POSIX: report ENOMEM only if NOTHING was written; otherwise
                  // return the short count (bytes already committed are visible).
                  return if written > 0 { written } else { -ENOMEM };
       e. available = min(PAGE_SIZE - write_off, remaining)
       f. // A write-populated page always starts at offset 0. When write_off ==
          //    0 we are BEGINNING a fresh page in this slot — reset offset,
          //    because the slot may still carry a nonzero offset left by a
          //    splice-populated page that occupied it before the ring wrapped
          //    and reused the slot. The reader reads from page.offset + read_off
          //    (step e of read), so a stale nonzero offset would misalign every
          //    byte the reader copies (garbage after splice → wrap → reuse).
          //    Published Release, so a reader that Acquire-loads the len bumped
          //    in step i also observes offset == 0.
          if write_off == 0: page.offset.store(0, Release)
          copy data[written:written+available] to page.data[write_off : write_off + available]
       g. page.len.store(write_off + available, Release)  // per-page valid length (offset reset in step f)
       h. write_offset.store(write_off + available, Release)
       i. len.fetch_add(available, Release)  // Publishing barrier for data in step f
       j. If write_offset == PAGE_SIZE:
          // Page full, advance to next — but first check for concurrent resize
          if resize_seq.load(Acquire) != seq_start:
              active_writer.store(0, Release)
              goto 1  // Retry with new seq_start; written/remaining preserved
          write_idx.store((write_idx_val + 1) % current_num_pages, Release)  // ≥1 divisor
          write_offset.store(0, Release)
       k. written += available; remaining -= available
  6. active_writer.store(0, Release)  // Release fast-path lock
  7. wake_up(read_wait)  // Notify any blocked readers
  8. return written

Memory ordering rationale for write path: The Release on len.fetch_add() (step i) is the publishing barrier that synchronizes with the reader's Acquire load of global len. This ensures all prior stores (the data memcpy in step f, the per-page len update in step g) are visible to the reader before it observes the new len value. The reader must use the global len Acquire→per-page len Acquire chain.

fcntl(F_SETPIPE_SZ) implementation:

fcntl_setpipe_sz(pipe, new_size):
  1. ring_lock.lock()
  2. // Wait for active single-writers to complete using futex
     while active_writer.load(Acquire) > 0:
         // Use futex wait instead of busy-spin to avoid priority inversion
         futex_wait(&active_writer, expected=1, timeout=1ms)
  3. resize_seq.fetch_add(1, Release)  // Start resize
  4. old_pages = pages  // Save pointer to old pages array
  5. // Perform resize: if new_pages > 16, transition to pages_large (Vec);
     //   copy data from old pages, update small_len or pages_large
  6. resize_seq.fetch_add(1, Release)  // End resize
  7. rcu_call(old_pages, free_pages_callback)  // Defer freeing old pages array
  8. ring_lock.unlock()

Design note — lock ordering during resize: The resize path holds ring_lock while waiting for active_writer to drain (with a 1ms timeout). This prevents permanent deadlock but creates a retry loop if the writer is blocked on an unrelated resource. The implementation SHOULD drop ring_lock before the futex wait, re-acquire it after wake-up, and re-validate the resize preconditions. This two-phase approach (validate → release → wait → re-acquire → re-validate) eliminates the lock-while-wait pattern at the cost of one extra validation pass.

Memory safety during resize: When fcntl(F_SETPIPE_SZ) replaces the pages array, the OLD pages array is freed via rcu_call() (deferred until the next RCU grace period). This ensures that any concurrent reader in step 4a-4e, which executes under an implicit RCU read-side critical section (preemption disabled during the pipe read fast path), will not access freed memory. The seqlock (resize_seq) detects that a resize occurred and triggers a retry, but the deferred freeing guarantees that the stale pointer is still valid for the duration of the read attempt.

Multi-writer support: When multiple threads write to the same pipe concurrently, the lock-free path cannot be used. The kernel detects multi-writer scenarios using a compare-and-swap pattern: a writer performs active_writer.compare_exchange(0, 1, Acquire, Relaxed). If successful (previous value was 0), it proceeds on the fast path. If it fails (another writer is active), it acquires ring_lock and takes the slow path. This ensures exactly one writer can be on the fast path at a time, preserving POSIX atomic write guarantees for writes ≤ PIPE_BUF.

Pipe read algorithm (lock-free, requires single reader or mutex for multi-reader):

read(pipe, buffer, len):
  0. seq_start = resize_seq.load(Acquire)  // Capture resize generation
  1. If len.load(Acquire) == 0:
       // Pipe empty — check for EOF or block
       if writer_count.load(Acquire) == 0:
           return 0  // EOF — all writers closed
       if read_wait.wait_event(|| len.load(Acquire) > 0 || writer_count.load(Acquire) == 0).is_err():
           return 0
       if len.load(Acquire) == 0 && writer_count.load(Acquire) == 0:
           return 0  // EOF after wakeup
       // Check for resize during wait
       if resize_seq.load(Acquire) != seq_start:
           goto 0  // Retry with new parameters

  2. bytes_read = 0
  3. current_num_pages = page_count(pipe)  // ring SLOT count (capacity/PAGE_SIZE), ≥ 1
  4. while bytes_read < len && len.load(Acquire) > 0:
       a. // Check for concurrent resize BEFORE accessing pages[] array.
          // If resize occurred, the old pages[] pointer may be deallocated.
          if resize_seq.load(Acquire) != seq_start:
              seq_start = resize_seq.load(Acquire)
              current_num_pages = page_count(pipe)
       b. read_idx_val = read_idx.load(Acquire)  // Acquire to see writer's stores
       c. read_off = read_offset.load(Acquire)
       d. // Determine bytes available in the current page. Data begins at the
          //    page's `offset` (nonzero only for splice-populated pages); the
          //    read cursor read_off counts bytes already consumed FROM offset.
          page = &pages[read_idx_val]      // populated: len>0 implies a PhysPage
          page_off = page.offset.load(Acquire)
          page_len = page.len.load(Acquire)
          available = min(page_len - read_off, len - bytes_read)
       e. // Copy data from page to user buffer (Acquire ensures data is visible)
          copy page.data[page_off + read_off : page_off + read_off + available] to buffer[bytes_read:]
       f. // Post-copy validation: if a resize raced with the copy, the data
          // may be stale. Discard this iteration and retry.
          if resize_seq.load(Acquire) != seq_start:
              seq_start = resize_seq.load(Acquire)
              current_num_pages = pipe.page_count()
              // Re-read page index and offset — resize may have moved data
              // to different page indices or changed the page array size.
              // Without this, stale read_idx_val may index a different page
              // (data corruption) or exceed current_num_pages (OOB access).
              read_idx_val = read_idx.load(Acquire)
              read_off = read_offset.load(Acquire)
              continue  // Retry — do not commit read_offset or len changes
       g. read_offset.store(read_off + available, Release)
       h. If (read_off + available) >= page_len:
          // Page consumed — advance index BEFORE decrementing len. This ensures
          // a concurrent writer observing free space (via len) sees the updated
          // read_idx and does not overwrite the page the reader just finished.
          read_idx.store((read_idx_val + 1) % current_num_pages, Release)
          read_offset.store(0, Release)
       i. len.fetch_sub(available, Release)  // Must be AFTER read_idx advance
       j. bytes_read += available

  5. wake_up(write_wait)  // Notify any blocked writers
  6. return bytes_read

Memory ordering rationale: The reader uses Acquire loads on len, read_idx, read_offset, and pages[].len to synchronize with the writer's Release stores. This ensures the reader observes all data written before the writer updated these indices. On weakly-ordered architectures (AArch64, RISC-V, ARMv7, PPC), this ordering is critical to prevent the reader from seeing stale data.

pipe_poll implementation: Polls a pipe for readiness without blocking. Two ABI requirements Linux (fs/pipe.c pipe_poll()) enforces and UmkaOS must match:

The reported events depend on WHICH END is being polled. A pipe fd is opened read-only (pipe[0]) or write-only (pipe[1]); epoll on a read fd must NEVER see EPOLLOUT, and a write fd must never see EPOLLIN. pipe_poll therefore takes the end's access mode and branches on it — the read end reports EPOLLIN/EPOLLHUP only, the write end EPOLLOUT/ EPOLLERR only.
EPOLLERR/EPOLLHUP are never masked out. Per Section 19.1 they are "always reported, cannot be masked". pipe_poll therefore returns the FULL readiness mask (it does NOT & events — masking by the requested set is the poll/select/epoll CORE's job, which applies & (requested | EPOLLERR | EPOLLHUP), always re-admitting the two unmaskable bits). This matches the FileOps::poll contract (Section 14.1), whose regular- file case likewise returns the full mask unmasked.

/// Which end of the pipe an open file descriptor refers to. Recovered from the
/// open file's access mode (FMODE_READ vs FMODE_WRITE): pipe(2)/pipe2(2) set
/// pages[0] = Read, pages[1] = Write; a FIFO open() derives it from O_RDONLY /
/// O_WRONLY. Carried in the pipe's per-open file-private value so FileOps::poll
/// can recover it without an f_mode parameter (see note below).
pub enum PipeEnd { Read, Write }

fn pipe_poll(pipe: &PipeBuffer, access: PipeEnd,
             events: PollEvents, pt: Option<&mut PollTable>) -> PollEvents {
    // Register on BOTH wait queues on the first pass; re-poll passes skip.
    if let Some(pt) = pt {
        poll_wait(&pipe.read_wait, pt);
        poll_wait(&pipe.write_wait, pt);
    }
    let mut ready = PollEvents::empty();
    let avail = pipe.len.load(Acquire);
    match access {
        PipeEnd::Read => {
            if avail > 0 { ready |= EPOLLIN | EPOLLRDNORM; }
            // Writers all gone → hang-up on the read end.
            if pipe.writer_count.load(Acquire) == 0 { ready |= EPOLLHUP; }
        }
        PipeEnd::Write => {
            if avail < pipe.capacity.load(Acquire) { ready |= EPOLLOUT | EPOLLWRNORM; }
            // Readers all gone → error on the write end (writes would EPIPE).
            if pipe.reader_count.load(Acquire) == 0 { ready |= EPOLLERR; }
        }
    }
    ready   // full mask; the poll core applies `& (events | EPOLLERR | EPOLLHUP)`
}

access sourcing (self-contained, no FileOps::poll signature change). FileOps::poll (Section 14.1) receives the per-open private: u64 returned by open(). The pipe/FIFO layer stores the end discriminant (PipeEnd) in that value — pipe(2)/pipe2(2) set it directly for the two new descriptors, a FIFO open() derives it from the open access mode — so the pipe's FileOps::poll recovers access from private and calls pipe_poll with it. This keeps the fix inside the pipe subsystem. (The broader gap the finding notes — that FileOps::poll carries no f_mode at all — is deferred: if the VFS later threads the open-file access mode through the poll chain, the pipe impl can use that instead. Deferred handoff by symbol: Section 19.1 poll/select/epoll core — ensure the returned mask is combined as & (requested_events | EPOLLERR | EPOLLHUP) so the two unmaskable bits reach a poller that did not request them.)

FIFOs (named pipes): A FIFO is a VFS node (VfsNode) that, when opened, creates a reference to an existing PipeBuffer or creates a new one. Multiple readers and writers can open a FIFO; the reader_count and writer_count fields track opens/closes. Writers use the multi-writer slow path when concurrent writes are detected. When the last reader and last writer close, the buffer is freed.

17.3.3 Shared Memory (POSIX and SysV)¶

POSIX shm_open(): Implemented as a memory-mapped file (mmap) backed by a hidden tmpfs instance.
SysV shmget(): Maps to the same underlying physical memory allocation mechanism, but managed via the CLONE_NEWIPC namespace tables.

Both mechanisms result in direct page table entries (PTEs) mapping the same physical frames into multiple Capability Domains.

17.3.3.1 POSIX Message Queues (mqueuefs)¶

POSIX message queues are backed by the in-memory PosixMqueue object family (PosixMqueue / PosixMqueueInner / MqueueAttr / PosixMessage / MqueueNotify), defined with its IPC-namespace siblings (ShmSegment, SemSet, MsgQueue) in Section 17.1. The per-IPC-namespace mqueuefs filesystem (conventionally mounted at /dev/mqueue) is the VFS surface over those objects — it provides naming, permission checks, and fd lifecycle, exactly as Linux's mqueue filesystem wraps struct mqueue_inode_info (ipc/mqueue.c, verified against torvalds/linux master):

mq_open() creates (or looks up) a mqueuefs inode whose private data is the Arc<PosixMqueue> — the same Arc registered in IpcNamespace.posix_mqueues below. The namespace is the CALLER's IPC namespace (current_ipc_ns(), name-based — never a mounted /dev/mqueue path). The returned descriptor is an ordinary VFS file handle; file.mqueue_inner() resolves the inode's private data to &SpinLock<PosixMqueueInner>.
mq_open() limit validation (the per-namespace /proc/sys/fs/mqueue caps). On the O_CREAT create path, against ipc_ns.limits:
attr == NULL → the new queue uses mq_msg_default / mq_msgsize_default.
attr.mq_maxmsg <= 0 or attr.mq_maxmsg > mq_msg_max, or attr.mq_msgsize <= 0 or attr.mq_msgsize > mq_msgsize_max → EINVAL (an unprivileged caller; CAP_SYS_RESOURCE raises the ceiling to the hard limits HARD_MSGMAX / HARD_MSGSIZEMAX, Linux parity).
ipc_ns.posix_mqueues.len() >= mq_queues_max → ENOSPC (the queue-COUNT cap; the RLIMIT_MSGQUEUE per-UID byte charge, Section 8.8, is a SEPARATE, additional bound and does not cap the count for root). These are the userspace-visible /proc/sys/fs/mqueue ABI checks Docker and systemd rely on; without the IpcLimits mqueue fields added above there was no per-namespace bound on queue count, message count, or message size.
mq_send() / mq_receive() / mq_timedsend() / mq_timedreceive() are dedicated syscalls (they carry a message priority, which VFS read/write cannot express) dispatching directly to the queue via file.mqueue_inner() — priority-ordered insertion/removal on PosixMqueueInner.queue, blocking on PosixMqueue.recv_waiters/send_waiters.
read(2) on an mqueue fd does NOT dequeue a message. It returns the Linux-compatible status string ("QSIZE:%-10lu NOTIFY:%-5d SIGNO:%-5d NOTIFY_PID:%-6d\n", Linux Linux ipc/mqueue.c mqueue_read_file(), verified against torvalds/linux master) — required for cat /dev/mqueue/<name> compatibility. write(2) on an mqueue fd returns EINVAL (Linux mqueue_file_operations defines no write method).
Queue attributes (mq_maxmsg, mq_msgsize) live in the typed MqueueAttr field of the struct, NOT in inode extended attributes; mq_getattr() reads them plus the current message count from PosixMqueueInner under the queue lock.
mq_unlink() removes the name from IpcNamespace.posix_mqueues and the mqueuefs directory entry; the queue object is destroyed when the last open descriptor closes (Arc refcount), matching POSIX unlink semantics.

mqueuefs is namespace-scoped: each IPC namespace has its own superblock and name space (see the per-namespace filesystem table in Section 17.1). The mount at /dev/mqueue is convention (performed by init/systemd), not a kernel-fixed path.

17.3.3.2 mq_notify Ownership and Exit Cleanup¶

mq_notify(2) registration is per-process and single-owner: at most one process may hold a queue's notification at a time (EBUSY otherwise), and the registration records the owner's kernel ProcessId (MqueueNotify.pid, Section 17.1). Deregistration has exactly three triggers — none of them an exit_task() step:

Explicit: mq_notify(mqd, NULL) by the owner.
Fired: the one-shot notification auto-deregisters after delivery (POSIX.1-2017).
Owner close — the exit path: the mqueue file's flush operation (invoked on EVERY close(), including the implicit closes when the exiting process's FdTable drops in exit_task Step 5) removes the registration iff the closing process is the registered owner:

/// mqueuefs `FileOps::flush` override — runs on EVERY close of an mqueue fd
/// (unlike `release()`, which runs only when the LAST descriptor drops). This
/// distinction is load-bearing and is dictated by the observable mq_notify
/// contract: a dup'd or fork-inherited fd keeps the file alive, so the OWNER's
/// close must deregister the notification IMMEDIATELY even when the file
/// survives. A last-reference-only hook could not honor that, so the `FileOps`
/// trait ([Section 14.1](14-vfs.md#virtual-filesystem-layer)) carries a distinct every-close
/// `flush` method. (Linux surfaces the same observable behavior through its
/// per-close file flush hook.)
impl FileOps for MqueueFileOps {
    fn flush(&self, file: &OpenFile, closer: &Process) -> Result<()> {
        let mut inner = file.mqueue_inner().lock();
        if inner.notify.as_ref().is_some_and(|n| n.pid == closer.pid) {
            inner.notify = None; // drops the SIGEV registration state
        }
        Ok(())
    }
    // open/read/write/release/poll/... elsewhere in the mqueuefs FileOps impl.
}

The flush hook must exist and be invoked on every close. The FileOps trait (Section 14.1) gains a flush method with a no-op default, so no existing filesystem impl changes:
/// Called on EVERY close(2) of a descriptor referencing this open file
/// (before the file's refcount is decremented), NOT only on last-reference
/// release(). `closer` is the process performing the close. Default: no-op,
/// so no existing filesystem impl changes. mqueuefs overrides it for
/// mq_notify owner cleanup; most filesystems do not.
fn flush(&self, file: &OpenFile, closer: &Process) -> Result<()> { Ok(()) }
(Deferred handoff by symbol: the fd-close path — explicit close(2) and exit_task Step 5 FdTable::drop (Section 8.2) — must call file_ops.flush(file, current().process) on each descriptor close, before the file-refcount decrement, so the every-close guarantee holds. The default no-op makes this cheap for all other file types.)

Because mq_notify() requires an open descriptor and the owner's exit always closes every descriptor through this hook, a dead process can never leave a dangling registration (Section 8.2). A shared fd table (CLONE_FILES across processes) can defer the final close past the owner's exit; the registration then targets a dead ProcessId and the delivery-time find_process_by_pid() fails harmlessly (ProcessIds are never reused) — the same behavior as Linux.

17.3.4 IPC Namespace Dispatch (SysV IPC)¶

SysV IPC objects (shared memory segments, semaphore arrays, and message queues) are isolated per IPC namespace. Each IPC namespace maintains independent key-to-ID mappings so that the same key_t value in two different containers refers to two entirely separate IPC objects.

Dispatch path: All SysV IPC syscalls resolve the IPC namespace from the calling task's NamespaceSet before performing any lookup or creation:

/// Per-namespace SysV + POSIX IPC resource limits. Matches Linux defaults from
/// `include/uapi/linux/ipc.h`, `/proc/sys/kernel/`, and `/proc/sys/fs/mqueue/`.
///
/// **Interior mutability (why every field is atomic).** `IpcLimits` lives
/// inline in `IpcNamespace`, which is shared as `Arc<IpcNamespace>`
/// (`NamespaceSet.ipc_ns`, [Section 17.1](#namespace-architecture)). `Arc` has no
/// `DerefMut`, so a plain field could never be written through the shared
/// namespace — yet Docker/systemd write `/proc/sys/kernel/shmmax` etc. at
/// container start. Each limit is therefore an atomic, written by the sysctl
/// setter (see "IPC sysctl registration" below) and read on the allocation
/// path. Readers use `load(Relaxed)` — a limit check racing a concurrent
/// sysctl write may use either the old or new bound (both are valid points in
/// time; the sysctl is monotone-agnostic and Linux gives the same
/// no-serialization guarantee). Values are cross-checked against the min/max in
/// each `kernel_param!` schema, so an out-of-range store cannot occur.
pub struct IpcLimits {
    // --- SysV shared memory limits (`/proc/sys/kernel/`) ---
    /// `shmmax`: max size of a single shared memory segment (bytes).
    /// Linux default: SHMMAX = ULONG_MAX - (1UL << 24) ≈ 16 EiB on 64-bit.
    pub shmmax: AtomicU64,
    /// `shmall`: max total shared memory PAGES (namespace-wide).
    /// Linux default: SHMALL = ULONG_MAX - (1UL << 24). Enforced against the
    /// `IpcNamespace.shm_tot` running page counter.
    pub shmall: AtomicU64,
    /// `shmmni`: max number of shared memory segments (namespace-wide).
    /// Linux default: SHMMNI = 4096. Must be ≤ `IPC_MNI` (id-space bound).
    pub shmmni: AtomicU32,

    // --- SysV semaphore limits (`/proc/sys/kernel/sem`, 4-tuple) ---
    /// `semmsl`: max semaphores per set. Linux default 32000.
    pub semmsl: AtomicU32,
    /// `semmns`: max semaphores namespace-wide. Linux default 1024000000.
    /// Enforced against the `IpcNamespace.used_sems` running counter.
    pub semmns: AtomicU32,
    /// `semopm`: max operations per `semop()` call. Linux default 500.
    pub semopm: AtomicU32,
    /// `semvmx`: max semaphore value. Linux default 32767.
    pub semvmx: AtomicU32,
    /// `semmni`: max number of semaphore sets (namespace-wide). Linux default
    /// 32000. Must be ≤ `IPC_MNI`.
    pub semmni: AtomicU32,

    // --- SysV message queue limits (`/proc/sys/kernel/`) ---
    /// `msgmax`: max size of a single message (bytes). Linux default 8192.
    pub msgmax: AtomicU32,
    /// `msgmnb`: max total bytes in a single message queue. Linux default
    /// 16384. Seeds a new queue's `MsgQueueInner.max_bytes` at `msgget()`.
    pub msgmnb: AtomicU32,
    /// `msgmni`: max number of message queues (namespace-wide). Linux default
    /// 32000. Must be ≤ `IPC_MNI`.
    pub msgmni: AtomicU32,

    // --- POSIX message queue limits (`/proc/sys/fs/mqueue/`) ---
    // These are the userspace-visible /proc/sys/fs/mqueue ABI (absent before;
    // mq_open() had no per-namespace validation). Linux ipc_namespace fields
    // mq_queues_max / mq_msg_max / mq_msgsize_max / mq_msg_default /
    // mq_msgsize_default (ipc/mqueue.c, verified against torvalds/linux master).
    /// `queues_max`: max POSIX message queues (namespace-wide). Linux default
    /// 256. The `fs.mqueue.queues_max` sysctl has NO upper
    /// bound in Linux (there is no HARD_QUEUESMAX constant — the old 1024
    /// ceiling was removed in 2014, torvalds/linux commit f3713fd9cff7).
    /// Enforced against `IpcNamespace.posix_mqueues.len()`.
    pub mq_queues_max: AtomicU32,
    /// `msg_max`: max `mq_maxmsg` accepted by `mq_open()`. Linux DFLT_MSGMAX =
    /// 10 (minimum 1, hard maximum 65536 — a FLAT constant on
    /// every arch, `include/linux/ipc_namespace.h`, verified against
    /// torvalds/linux master; NOT the arch-dependent `sizeof`-scaled formula).
    /// `mq_open(attr.mq_maxmsg > msg_max)` → EINVAL (unprivileged).
    pub mq_msg_max: AtomicU32,
    /// `msgsize_max`: max `mq_msgsize` accepted by `mq_open()`. Linux
    /// Default 8192 (minimum 128, hard maximum 16 MiB).
    pub mq_msgsize_max: AtomicU32,
    /// `msg_default`: `mq_maxmsg` used when `mq_open()` is passed `attr = NULL`.
    /// Linux DFLT_MSG = 10.
    pub mq_msg_default: AtomicU32,
    /// `msgsize_default`: `mq_msgsize` used when `attr = NULL`. Linux
    /// Default 8192.
    pub mq_msgsize_default: AtomicU32,
}

impl Default for IpcLimits {
    /// Returns Linux-compatible defaults. Called by `create_ipc_namespace()` for
    /// EVERY new IPC namespace — CLONE_NEWIPC resets to defaults (it does NOT
    /// inherit the parent's tuned limits), matching Linux `create_ipc_ns()`.
    fn default() -> Self {
        Self {
            shmmax: AtomicU64::new(u64::MAX - (1 << 24)),
            shmall: AtomicU64::new(u64::MAX - (1 << 24)),
            shmmni: AtomicU32::new(4096),
            semmsl: AtomicU32::new(32000),
            semmns: AtomicU32::new(1_024_000_000),
            semopm: AtomicU32::new(500),
            semvmx: AtomicU32::new(32767),
            semmni: AtomicU32::new(32000),
            msgmax: AtomicU32::new(8192),
            msgmnb: AtomicU32::new(16384),
            msgmni: AtomicU32::new(32000),
            mq_queues_max: AtomicU32::new(256),
            mq_msg_max: AtomicU32::new(10),
            mq_msgsize_max: AtomicU32::new(8192),
            mq_msg_default: AtomicU32::new(10),
            mq_msgsize_default: AtomicU32::new(8192),
        }
    }
}

IPC sysctl registration (/proc/sys/kernel/*, /proc/sys/fs/mqueue/*). Every limit is registered as a typed kernel parameter (Section 20.9) whose getter/setter resolve the caller's IPC namespace via current_ipc_ns() and load/store the matching IpcLimits atom. Because /proc/sys is read and written by a task, current_ipc_ns() binds the file to that task's IPC namespace — so a write inside a container tunes only that container's limits (per-namespace tunability, Linux setup_ipc_sysctls() parity, achieved without Linux's per-namespace table allocation because the storage is the namespace's own atomics). Static registration (one descriptor per name, emitted into .umka_params) fits the compile-time kernel_param! model; the per-namespace behaviour comes from the getter/setter, not from a runtime-allocated table.

kernel_param! {
    name:          "kernel.shmmax",
    schema:        ParamSchema::U64 { min: 0, max: u64::MAX, default: u64::MAX - (1 << 24) },
    description:   "Max size of a single SysV shared memory segment (bytes).",
    privileged:    true,
    per_namespace: true,   // IPC namespace (getter/setter resolve current_ipc_ns)
    getter:        || ParamValue::U64(current_ipc_ns().limits.shmmax.load(Ordering::Relaxed)),
    setter:        |v| match v {
        ParamValue::U64(n) => { current_ipc_ns().limits.shmmax.store(n, Ordering::Release); Ok(()) }
        _ => Err(ParamError::TypeMismatch),
    },
}
// kernel.sem is the SINGLE 4-value ABI file "SEMMSL SEMMNS SEMOPM SEMMNI"
// (Linux `/proc/sys/kernel/sem` is ONE entry, not four — verified against
// torvalds/linux master). It uses the ParamSchema::U32Array variant
// ([Section 20.9](20-observability.md#kernel-parameter-store)): the setter writes the four
// semmsl/semmns/semopm/semmni atoms in order, the getter reads them back as a
// 4-vector (space-separated in the text view). The ABI bounds each element to
// the int range [0, INT_MAX] and additionally rejects semmni (element 3) >
// the `IPC_MNI` bound — both enforced directly by the setter below. All OTHER names below
// follow the single-atom pattern shown for `kernel.shmmax`.
kernel_param! {
    name:          "kernel.sem",
    schema:        ParamSchema::U32Array {
                       len: 4, min: 0, max: i32::MAX as u32,
                       default: [32000, 1_024_000_000, 500, 32000],   // SEMMSL SEMMNS SEMOPM SEMMNI
                   },
    description:   "SysV semaphore limits: SEMMSL SEMMNS SEMOPM SEMMNI.",
    privileged:    true,
    per_namespace: true,   // IPC namespace (getter/setter resolve current_ipc_ns)
    getter:        || {
        let l = &current_ipc_ns().limits;
        ParamValue::U32Array { len: 4, vals: [
            l.semmsl.load(Ordering::Relaxed), l.semmns.load(Ordering::Relaxed),
            l.semopm.load(Ordering::Relaxed), l.semmni.load(Ordering::Relaxed),
        ] }
    },
    setter:        |v| match v {
        ParamValue::U32Array { len: 4, vals } => {
            if vals[3] > IPC_MNI { return Err(ParamError::SetterRejected); }  // semmni ≤ IPC_MNI (ABI bound)
            let l = &current_ipc_ns().limits;
            l.semmsl.store(vals[0], Ordering::Release);
            l.semmns.store(vals[1], Ordering::Release);
            l.semopm.store(vals[2], Ordering::Release);
            l.semmni.store(vals[3], Ordering::Release);
            Ok(())
        }
        _ => Err(ParamError::TypeMismatch),
    },
}

Registered parameters (name → schema → backing IpcLimits atom):

`/proc` path	Schema (min, max, default)	Backing atom
`kernel.shmmax`	U64 (0, u64::MAX, u64::MAX−2^24)	`shmmax`
`kernel.shmall`	U64 (0, u64::MAX, u64::MAX−2^24)	`shmall`
`kernel.shmmni`	U32 (0, IPC_MNI, 4096)	`shmmni`
`kernel.sem`	U32Array len 4 (each 0..i32::MAX; semmni≤IPC_MNI; defaults 32000/1024000000/500/32000)	`semmsl,semmns,semopm,semmni`
`kernel.msgmax`	U32 (0, i32::MAX, 8192)	`msgmax`
`kernel.msgmnb`	U32 (0, i32::MAX, 16384)	`msgmnb`
`kernel.msgmni`	U32 (0, IPC_MNI, 32000)	`msgmni`
`fs.mqueue.queues_max`	U32 (0, i32::MAX, 256)	`mq_queues_max`
`fs.mqueue.msg_max`	U32 (1, 65536, 10)	`mq_msg_max`
`fs.mqueue.msgsize_max`	U32 (128, 16·2^20, 8192)	`mq_msgsize_max`
`fs.mqueue.msg_default`	U32 (1, 65536, 10)	`mq_msg_default`
`fs.mqueue.msgsize_default`	U32 (128, 16·2^20, 8192)	`mq_msgsize_default`

Bounds are the exact Linux sysctl ranges (verified against torvalds/linux master, ipc/ipc_sysctl.c + ipc/mq_sysctl.c), because these are ABI: a container that writes kernel.msgmax = 1048576 or raises fs.mqueue.queues_max above 1024 must succeed exactly as on Linux. Key points: SysV kernel.msgmax/kernel.msgmnb accept the full [0, INT_MAX] range, NOT capped at 65536; kernel.shmmni/kernel.msgmni cap at IPC_MNI; fs.mqueue.queues_max is UNBOUNDED (no HARD_QUEUESMAX); POSIX fs.mqueue.msg_max/msg_default cap at the flat maximum 65536, and msgsize_max/msgsize_default at the hard maximum 16 MiB (the defaults are bounded by these hard maxima, not by the live max — so msg_default > msg_max is settable, matching Linux).

Note: semvmx has no /proc file in Linux either (it is a compile-time ceiling, enforced by semctl(SETVAL)), so it is not registered — it remains a per-namespace IpcLimits atom read on the semctl path. (Deferred handoff by symbol: Section 20.9 "Per-Namespace Parameter Scoping" table — add an IPC-namespace row so kernel.{shm*,sem,msg*} and fs.mqueue.* are scoped per IPC namespace rather than host-global; the getter/setter already do the per-ns resolution, this only aligns the prose scoping table.)

/// Per-IPC-namespace state. One instance per IPC namespace, created by
/// clone(CLONE_NEWIPC) or unshare(CLONE_NEWIPC).
///
/// **Locking**: Each IPC type (shm, sem, msg) has a single `RwLock<IpcIdTable>`
/// that protects BOTH the ID allocator (Idr) and the key-to-ID map (XArray)
/// atomically. One lock over both maps is required for correctness: separate
/// locks would open a TOCTOU race where two threads both see "key not present"
/// then both allocate, producing duplicate IPC objects for one key — which
/// would violate the observable contract that
/// `shmget(key, size, IPC_CREAT|IPC_EXCL)` returns EEXIST when the key exists.
///
/// `RwLock` (not SpinLock) allows concurrent `shmat()`/`shmdt()` read-side
/// access without writer contention.
pub struct IpcNamespace {
    /// Unique namespace ID. Allocated from the global `NEXT_NS_ID` counter
    /// ([Section 17.1](#namespace-architecture)), which its doc states is "shared by all
    /// namespace types" — so IPC namespaces draw from the same u64 space.
    pub ns_id: u64,

    /// SysV shared memory: ID allocator + key-to-ID map under single lock.
    /// key_t is i32 on Linux; XArray key uses `key as u32 as u64` to avoid
    /// sign-extension overlap.
    pub shm: RwLock<IpcIdTable<ShmSegment>>,

    /// SysV semaphore sets: ID allocator + key-to-ID map under single lock.
    pub sem: RwLock<IpcIdTable<SemSet>>,

    /// SysV message queues: ID allocator + key-to-ID map under single lock.
    pub msg: RwLock<IpcIdTable<MsgQueue>>,

    /// Running total of shared-memory PAGES charged in this namespace — the
    /// accounting counter `shmall` enforcement requires (Linux
    /// `ipc_namespace.shm_tot`). Incremented at `shmget()` by the segment's
    /// page count, decremented when a segment's pages are freed. `shmget()`
    /// rejects a request whose `shm_tot + ceil(size/PAGE_SIZE) > shmall` with
    /// ENOSPC.
    pub shm_tot: AtomicU64,

    /// Running total of semaphores across all sets in this namespace — the
    /// accounting counter `semmns` enforcement requires (Linux
    /// `ipc_namespace.used_sems`). Incremented at `semget()` by `nsems`,
    /// decremented when a set is destroyed. `semget()` rejects
    /// `used_sems + nsems > semmns` with ENOSPC.
    pub used_sems: AtomicU64,

    /// System-wide limits (per-namespace). All atomic; written by the
    /// registered IPC sysctls (see "IPC sysctl registration" above), read on
    /// the allocation paths.
    pub limits: IpcLimits,

    /// Owning user namespace (for permission checks — `has_ns_cap()` on the
    /// SysV/POSIX dispatch paths). STRONG `Arc`: an IPC namespace pins its
    /// owning user namespace alive (Linux `get_user_ns()` parity; acyclic —
    /// a user namespace never references an IPC namespace).
    pub user_ns: Arc<UserNamespace>,

    /// Handle to the SysV message NODE cache (`SysVMessage`, fixed object
    /// size), referenced by the MsgQueue spec ([Section 17.1](#namespace-architecture)
    /// "Allocation strategy") — a message node is slab-allocated from THIS
    /// handle BEFORE the queue SpinLock is taken, so no heap allocation occurs
    /// under the lock. Because the node is fixed-size with no ctor/dtor, the
    /// handle MERGES into the shared kmalloc size class for that size
    /// ([Section 4.3](04-memory.md#slab-allocator) merging conditions) — the SAME PERMANENT cache
    /// backs every IPC namespace (Linux allocates `struct msg_msg` from kmalloc
    /// too; no dedicated per-namespace cache, hence no cache proliferation
    /// across thousands of containers). It is stored per-namespace only as a
    /// ready-to-use `kmem_cache_alloc` handle. The variable message BODY
    /// (`SysVMessage.data`) is a separate allocation from the sized kmalloc
    /// classes (the 64/256/1024/4096/`MSGMAX` "fixed-size buckets" the MsgQueue
    /// doc names — the standard kmalloc size classes). Created by
    /// `create_ipc_namespace()` via `kmem_cache_create()`; dropped at namespace
    /// teardown — dropping a merged handle is harmless (it aliases the
    /// PERMANENT kmalloc cache; nothing to GC), and the Drop below still drains
    /// every queued message's node back to the cache first (leak-free).
    /// (Deferred handoff by symbol: [Section 17.1](#namespace-architecture) MsgQueue
    /// "Allocation strategy" — clarify that `msg_slab` is the NODE handle and
    /// message bodies come from the sized kmalloc classes.)
    pub msg_slab: SlabCacheHandle,

    /// Per-namespace internal `mqueuefs` mount, constructed by
    /// `internal_mount()`
    /// ([Section 14.6](14-vfs.md#mount-tree-data-structures-and-operations--kernel-internal-mounts-mntinternal)).
    /// Linux equivalent: `ipc_namespace.mq_mnt`, built by Linux `kern_mount`.
    /// Holds the namespace's own `mqueuefs` superblock — the VFS
    /// surface over `posix_mqueues`; the user-visible `/dev/mqueue` is a bind
    /// of this internal mount (convention, performed by init/systemd).
    /// **Reference direction (cycle-free):** `IpcNamespace` →(strong)→ `Mount`
    /// →(strong)→ `SuperBlock`; the superblock does NOT hold a strong
    /// `Arc<IpcNamespace>` back-reference (that would cycle), so mq_open's
    /// namespace is resolved from the CALLER (`current_ipc_ns()`), never from a
    /// mounted path. Created by `create_ipc_namespace()`; torn down at ns teardown.
    pub mqueue_mnt: Arc<Mount>,

    /// POSIX message queues (name -> queue). String-keyed; BTreeMap is
    /// appropriate since mq_open() is a setup-time operation, not per-message.
    /// Growth bounded by `limits.mq_queues_max` (checked at mq_open).
    pub posix_mqueues: RwLock<BTreeMap<String, Arc<PosixMqueue>>>,
}

/// IPC id-space parameters (Linux `IPCMNI` / sequence multiplier, `ipc/util.c`,
/// `ipc/util.h`, verified against torvalds/linux master). A userspace IPC id
/// is `(seq << IPC_SEQ_SHIFT) | idx` — the slot index in the low bits, a
/// per-table sequence number in the high bits. `IPC_PRIVATE` is the SysV
/// "create-anonymous" key.
pub const IPC_PRIVATE: i32 = 0;                 // uapi `IPC_PRIVATE`
pub const IPC_MNI: u32 = 32768;                 // max slot index (2^15), Linux IPCMNI
pub const IPC_SEQ_SHIFT: u32 = 15;              // log2(IPC_MNI)
/// Sequence ceiling: keeps `(seq << IPC_SEQ_SHIFT) | idx` ≤ `i32::MAX`, so
/// every id is a POSITIVE i32 (an id ≥ 2^31 would return negative and alias
/// `-errno`). = 65535.
pub const IPC_SEQ_MAX: u32 = (i32::MAX as u32) >> IPC_SEQ_SHIFT;

/// One IPC id-table entry: the shared object plus the sequence number and key
/// captured at allocation. Storing `Arc<T>` (NOT `T`) is load-bearing —
/// `idr_remove()` returns the WHOLE entry, so it moves the `Arc` (a refcount
/// handle), never the live object; concurrent RCU-guard borrows,
/// sleeping `semop()`/`msgsnd()`/`msgrcv()` waiters, and `nattach > 0`
/// attachments all keep their own `Arc<T>` and the object outlives its table
/// slot exactly as `IPC_RMID`-while-attached requires. Corpus precedent:
/// `Process.posix_timers: Idr<Arc<PosixTimer>>` ([Section 7.8](07-scheduling.md#timekeeping-and-clock-management)).
pub struct IpcIdEntry<T> {
    /// Sequence number embedded in this object's id (ABA/reuse defense).
    pub seq: u32,
    /// Creation key (for `key_map` removal at `IPC_RMID`). `IPC_PRIVATE` for
    /// anonymous objects (never entered into `key_map`).
    pub key: i32,
    /// The shared IPC object.
    pub obj: Arc<T>,
}

/// Bundled IPC ID allocator and key-to-ID map, held in one struct so a single
/// `RwLock` in `IpcNamespace` covers key lookup, ID allocation, the sequence
/// counter, and the live-object count atomically (closing the lookup→allocate
/// TOCTOU). `Idr` is the right allocator here: SysV ids are small dense
/// integers that must recycle the lowest free slot, which is exactly `Idr`'s
/// lowest-free allocation discipline.
pub struct IpcIdTable<T> {
    /// ID allocator, indexed by SLOT INDEX (`idx`, not the composite id).
    /// Stores `IpcIdEntry<T>` (an `Arc<T>` inside) — see `IpcIdEntry` for why
    /// by-value `Idr<T>` was unsound.
    pub ids: Idr<IpcIdEntry<T>>,
    /// Key-to-ID reverse map. XArray keyed by `key_t as u32 as u64`, storing
    /// the composite i32 id. `IPC_PRIVATE` is NEVER inserted (all anonymous
    /// segments would collide at key 0).
    pub key_map: XArray<i32>,
    /// Next sequence number. Bumped on every allocation
    /// so a freed id's `seq` differs from the next occupant of the same slot —
    /// a stale id fails `lookup()`'s seq check instead of silently addressing
    /// the wrong object (the opaque-handle ABA rule). Wraps at `IPC_SEQ_MAX`.
    pub seq: u32,
    /// Live object count. Enforces the per-type
    /// object-count limit (`shmmni`/`semmni`/`msgmni`), which — being ≤
    /// `IPC_MNI` — also bounds `idx` and thus keeps ids positive.
    pub in_use: u32,
}

/// Stamp-the-id hook implemented by the three SysV object types
/// (`ShmSegment`/`SemSet`/`MsgQueue`). The userspace-visible id is
/// `(seq << IPC_SEQ_SHIFT) | idx`, and `idx` is only known AFTER the slot is
/// allocated — so the object cannot be constructed already carrying its id.
/// `insert` therefore builds the object with a zero id, allocates the slot,
/// composes the id, and calls `set_id()` to write it in — exactly once, still
/// under the id-table write lock, BEFORE the object becomes findable
/// (`key_map` publish / return to the creator). The id field is `AtomicI32`
/// ONLY to permit this one
/// set-once store through the shared `Arc`; it never changes again, so it is
/// semantically immutable (the `IPC_STAT` reader observes the final value).
///
/// Race-freedom of the stamp: the only lock-free reader of the object is the
/// RCU `lookup(idx)` path, which is reachable solely with a valid id — and no
/// thread can hold this object's NEW id until `insert` returns it. A stale id
/// (a prior occupant of `idx`) carries a different `seq`, so it fails the
/// `IpcIdEntry.seq` check and never dereferences the object. Thus no reader
/// observes the pre-stamp zero id.
pub trait IpcObject {
    /// Write the userspace id. Called once by `IpcIdTable::insert`, under the
    /// id-table write lock, immediately after slot allocation.
    fn set_id(&self, id: i32);
}

// The id fields are `AtomicI32` ([Section 17.1](#namespace-architecture)); `Release` so a
// later `IPC_STAT` reader that Acquire-loads it sees the stamped value. A local
// trait for same-crate types — the orphan rule is satisfied.
impl IpcObject for ShmSegment {
    fn set_id(&self, id: i32) { self.shmid.store(id, Ordering::Release); }
}
impl IpcObject for SemSet {
    fn set_id(&self, id: i32) { self.semid.store(id, Ordering::Release); }
}
impl IpcObject for MsgQueue {
    fn set_id(&self, id: i32) { self.msqid.store(id, Ordering::Release); }
}

impl<T> IpcIdTable<T> {
    /// Empty table. `const` so `INIT_IPC_NS` can build its tables at boot.
    pub const fn new() -> Self {
        Self { ids: Idr::new(), key_map: XArray::new(), seq: 0, in_use: 0 }
    }

    /// Compose the userspace-visible i32 id from a slot index and a sequence.
    pub const fn build_id(idx: u32, seq: u32) -> i32 { ((seq << IPC_SEQ_SHIFT) | idx) as i32 }
    /// Extract the slot index from an id.
    pub const fn id_to_idx(id: i32) -> u32 { (id as u32) & (IPC_MNI - 1) }
    /// Extract the sequence number from an id.
    pub const fn id_to_seq(id: i32) -> u32 { (id as u32) >> IPC_SEQ_SHIFT }

    /// Allocate a slot, stamp the object's id, and (unless `IPC_PRIVATE`)
    /// publish the key. `max_count` is the per-type limit
    /// (`shmmni`/`semmni`/`msgmni`). Caller holds the `IpcNamespace` RwLock in
    /// WRITE mode (this method takes `&mut self`). `T: IpcObject` so the id can
    /// be set once, post-allocation, under this write lock (see `IpcObject`).
    pub fn insert(&mut self, key: i32, obj: Arc<T>, max_count: u32) -> Result<i32, Errno>
    where
        T: IpcObject,
    {
        if self.in_use >= max_count { return Err(Errno::ENOSPC); }  // shmmni/semmni/msgmni
        let this_seq = self.seq;
        // idx in [0, IPC_MNI): keeps the composite id a positive i32. Clone the
        // Arc into the table entry and keep the original to stamp the id (both
        // point at the same object).
        let idx = self.ids
            .idr_alloc_range(0, IPC_MNI,
                             IpcIdEntry { seq: this_seq, key, obj: Arc::clone(&obj) })
            .map_err(|_| Errno::ENOSPC)?;
        self.in_use += 1;
        self.seq = (self.seq + 1) % IPC_SEQ_MAX;   // ABA generation
        let id = Self::build_id(idx, this_seq);
        obj.set_id(id);   // set-once id stamp, before key_map
                          // publish → no lookup can observe the pre-stamp id.
        if key != IPC_PRIVATE {                     // anonymous objects never enter key_map
            self.key_map.store(key as u32 as u64, id);
        }
        Ok(id)
    }

    /// Resolve an id to an OWNED `Arc<T>` (safe to hold across a later sleep,
    /// after the RwLock is dropped), validating the embedded sequence. Caller
    /// holds the RwLock in READ (or WRITE) mode.
    pub fn lookup(&self, id: i32) -> Option<Arc<T>> {
        if id < 0 { return None; }                 // i32 ABI: no valid id is negative
        let idx = Self::id_to_idx(id);
        let guard = rcu_read_lock();
        let entry = self.ids.lookup(idx, &guard)?;
        if entry.seq != Self::id_to_seq(id) { return None; }   // stale/reused id → EINVAL/EIDRM
        Some(Arc::clone(&entry.obj))
    }

    /// Find an existing object's id by key (`shmget`/`semget`/`msgget`).
    pub fn lookup_by_key(&self, key: i32) -> Option<i32> {
        self.key_map.load(key as u32 as u64)
    }

    /// `IPC_RMID` removal: unlink from both maps and return the `Arc<T>` (the
    /// object survives while other `Arc`s — attachments, sleeping waiters —
    /// exist). Caller holds the RwLock in WRITE mode.
    pub fn remove(&mut self, id: i32) -> Option<Arc<T>> {
        if id < 0 { return None; }
        let idx = Self::id_to_idx(id);
        {   // validate seq before removing, so a stale id cannot evict a live slot
            let guard = rcu_read_lock();
            let entry = self.ids.lookup(idx, &guard)?;
            if entry.seq != Self::id_to_seq(id) { return None; }
        }
        let entry = self.ids.idr_remove(idx)?;     // moves the Arc handle, not the object
        self.in_use -= 1;
        if entry.key != IPC_PRIVATE { self.key_map.remove(entry.key as u32 as u64); }
        Some(entry.obj)
    }
}

/// Resolve the IPC namespace for a SysV OR POSIX IPC syscall.
///
/// Called at the entry of the SysV syscalls — shmget/semget/msgget/shmctl/
/// semctl/msgctl/shmat/shmdt/semop/msgsnd/msgrcv — AND the POSIX message-queue
/// syscalls — mq_open/mq_unlink/mq_notify/mq_getsetattr/mq_timedsend/
/// mq_timedreceive. POSIX mqueues resolve to `IpcNamespace.posix_mqueues` /
/// the per-ns `mqueue_mnt` superblock the SAME way: via the CALLER's IPC
/// namespace. mq_open() is **name-based** (`mq_open("/name", ...)`), never
/// resolved through a mounted `/dev/mqueue` path — a task with another
/// namespace's mqueuefs bind-mounted still creates in ITS OWN namespace
/// (Linux `current->nsproxy->ipc_ns`, not the mount's namespace). This is the
/// load-bearing resolution rule an implementing agent needs; without it
/// mq_open's namespace was unspecified.
///
/// Returns an OWNED `Arc` (one clone per syscall — cold/warm path, and
/// IPC namespace switches are rare). `Task.namespace_set` is an
/// `ArcSwap<NamespaceSet>`: returning `&Arc<IpcNamespace>` is
/// impossible — the reference would borrow the `load()` guard, a
/// temporary that dies at the end of this function (the exact
/// guard-lifetime footgun documented at `pid_nr_in()`,
/// [Section 17.1](#namespace-architecture--namespace-implementation)).
fn current_ipc_ns() -> Arc<IpcNamespace> {
    Arc::clone(&current_task().namespace_set.load().ipc_ns)
}

/// Create a fresh IPC namespace — the CLONE_NEWIPC / unshare(CLONE_NEWIPC)
/// constructor AND the boot-time `INIT_IPC_NS` builder. This is the
/// fully-specified sequence the CLONE_NEWIPC switch arm
/// ([Section 17.1](#namespace-architecture), "Create empty IPC namespace") must call,
/// mirroring CLONE_NEWNET's three-phase sequence.
///
/// `owner_user_ns` is the creating task's user namespace (its
/// `pending_cred.user_ns`, exactly as the NET path uses the creating cred's
/// `user_ns`). It becomes `user_ns` and backs every `has_ns_cap()` check on
/// the SysV/POSIX dispatch paths.
///
/// IPC namespaces are NOT migration-tracked (UTS/IPC/Cgroup/Time/User/IMA keep
/// bare `Arc::new` allocation, [Section 17.1](#namespace-architecture--tracked-allocation-namespaceset-pidnamespace-netnamespace)).
///
/// Errors: `ENOMEM` from slab-cache creation or mounting the internal `mqueuefs`.
/// Both fallible steps run BEFORE `Arc::new`, so a failure drops the already-
/// created handle with no partially-published namespace (a dropped `msg_slab`
/// handle is harmless — it aliases the PERMANENT kmalloc cache, nothing to
/// reclaim; a built `mqueue_mnt` unmounts).
///
/// Linux also charges `max_ipc_namespaces` (`UCOUNT_IPC_NAMESPACES`). UmkaOS's
/// `NsUcountKind` currently enumerates only `UserNs`/`PidNs`; charging IPC is a
/// follow-on when that enum gains an `IpcNs` variant (deferred handoff by
/// symbol: [Section 17.1](#namespace-architecture) `NsUcountKind`).
fn create_ipc_namespace(owner_user_ns: &Arc<UserNamespace>) -> Result<Arc<IpcNamespace>, Errno> {
    let ns_id = NEXT_NS_ID.fetch_add(1, Ordering::Relaxed);
    // SysVMessage NODE slab handle (msg_slab), fixed object size. With no
    // ctor/dtor and natural alignment a node-sized cache MERGES into the shared
    // kmalloc size class for `size_of::<SysVMessage>()` ([Section 4.3](04-memory.md#slab-allocator)
    // merging conditions), so `kmem_cache_create` returns a handle ALIASING the
    // PERMANENT kmalloc-N cache — one shared node cache, not a per-namespace
    // dedicated cache (no cache proliferation across thousands of containers).
    // The name is therefore a fixed `&'static str` label (dynamic per-ns names
    // are neither possible with `&'static str` nor meaningful for a merged
    // alias). The variable message BODY (SysVMessage.data) is a separate
    // allocation from the sized kmalloc classes (the 64/256/1024/4096/MSGMAX
    // buckets) — see the msg_slab field doc for the node/body split.
    let msg_slab = kmem_cache_create(
        "umka_sysv_msg",
        size_of::<SysVMessage>(),
        align_of::<SysVMessage>(),
        SlabCacheFlags::empty(),
        None,   // no ctor
        None,   // no dtor
    )?;
    // Internal mqueuefs mount ([Section 14.6](14-vfs.md#mount-tree-data-structures-and-operations--kernel-internal-mounts-mntinternal)).
    // Holds this namespace's mqueuefs superblock; user /dev/mqueue binds it.
    // Linux equivalent: `ns->mq_mnt = kern_mount(&mqueue_fs_type)`.
    let mqueue_mnt = internal_mount(&MQUEUEFS_FS_TYPE)?;
    Ok(Arc::new(IpcNamespace {
        ns_id,
        shm: RwLock::new(IpcIdTable::new()),
        sem: RwLock::new(IpcIdTable::new()),
        msg: RwLock::new(IpcIdTable::new()),
        shm_tot: AtomicU64::new(0),
        used_sems: AtomicU64::new(0),
        // CLONE_NEWIPC RESETS limits to defaults — it does NOT inherit the
        // parent's tuned values (Linux `create_ipc_ns()` parity; load-bearing
        // for container isolation now that limits are tunable).
        limits: IpcLimits::default(),
        user_ns: Arc::clone(owner_user_ns),   // creating cred's user_ns
        msg_slab,
        mqueue_mnt,
        posix_mqueues: RwLock::new(BTreeMap::new()),
    }))
}

/// Namespace teardown lives in `impl Drop` (the framework mandate — teardown
/// is never duplicated at call sites, [Section 17.1](#namespace-architecture)). Reached
/// when the last `Arc<IpcNamespace>` drops (rollback, the last task leaving the
/// namespace, or `namespace_set` replacement). No task lives here, so no NEW IPC op
/// can arrive; force-`IPC_RMID` every surviving object and wake any residual
/// cross-namespace waiter with `EIDRM` BEFORE the tables drop — the same
/// obligation Linux discharges via `shm_exit_ns`/`sem_exit_ns`/`msg_exit_ns`
/// (Linux `free_ipcs`). Drop rule 2 ([Section 17.1](#namespace-architecture)) applies to the
/// cross-namespace `memcg`/`mq_bytes` uncharges below (`Weak::upgrade()` +
/// `None`-is-noop, never a strong ref that would pin the target).
impl Drop for IpcNamespace {
    fn drop(&mut self) {
        let g = rcu_read_lock();

        // SysV semaphores: RMID-mark each set and wake its waiters (they return
        // EIDRM). Dropping the table's Arc<SemSet> then frees each set that has
        // no external ref; a set still referenced (a live undo list from a task
        // that setns'd away) survives until that ref drops. `detach_sysv_sem()`
        // tolerates a destroyed set ("None => continue",
        // [Section 8.2](08-process.md#process-lifecycle-teardown--step-4c-sysv-semaphore-undo-operations)).
        for (_i, e) in self.sem.get_mut().ids.iter(&g) {
            sem_rmid_mark_and_wake(&e.obj);   // the SemSet IPC_RMID path (below)
        }

        // SysV message queues: RMID-mark + wake (EIDRM), then drain every queued
        // SysVMessage node back to `msg_slab` (returning it to the shared kmalloc
        // node cache), so no message memory leaks before the handle drops.
        for (_i, e) in self.msg.get_mut().ids.iter(&g) {
            msg_rmid_mark_and_wake(&e.obj);   // the MsgQueue IPC_RMID path (below)
            msg_drain_to_slab(&e.obj);
        }

        // SysV shared memory: set the RMID mark. An UNATTACHED segment frees its
        // pages when the table's Arc<ShmSegment> drops (last ref → PhysPages
        // Drop → frame free + memcg uncharge). A segment still attached by a task
        // that setns'd away keeps its own Arc<ShmSegment> (held by the shm VMA)
        // and frees at the last shm_detach() after this mark. `shm_tot` (the
        // within-namespace `shmall` reservation counter) is discarded with the
        // namespace — surviving-segment memory stays memcg-accounted, which is
        // independent of `shm_tot`, so nothing dereferences the freed counter.
        for (_i, e) in self.shm.get_mut().ids.iter(&g) {
            // SeqCst, NOT Release: the shm free protocol's missed-free proof
            // ([Section 17.1](#namespace-architecture), `ShmSegment.shm_dest` /
            // `shm_free_pages`) argues over the single SeqCst total order of
            // the `shm_dest` store and the `nattach` fetch_sub — a weaker
            // store here would reopen the missed-free race against a
            // concurrent last `shm_detach()` from a setns'd-away attacher.
            e.obj.shm_dest.store(true, SeqCst);
        }

        // POSIX message queues: wake blocked senders/receivers (they re-check and
        // exit) and uncharge each queue's full creation-time (worst-case
        // footprint) charge against its creator UID before the
        // Arc<PosixMqueue> drops.
        for (_name, mq) in self.posix_mqueues.get_mut().iter() {
            mq.recv_waiters.wake_up_all();
            mq.send_waiters.wake_up_all();
            mq_drain_and_uncharge(mq);        // [Section 8.8](08-process.md#resource-limits-and-accounting) mq_uncharge(mq.uid, ..)
        }
        drop(g);

        // Remaining field drops run implicitly, in declaration order: the three
        // IpcIdTables drop their Arc<T> handles (objects with external Arcs
        // survive until those drop); `posix_mqueues` drops its handles;
        // `msg_slab`'s handle drops (harmless — a merged handle aliasing the
        // PERMANENT kmalloc node cache; the drain above already returned nodes);
        // `mqueue_mnt` drops, tearing down the internal mqueuefs mount + its
        // superblock (no strong Arc<IpcNamespace> back-ref, so there is no cycle
        // to break); `user_ns` (a forward strong ref) drops last.
    }
}

The three *_rmid_mark_and_wake / drain helpers are the SAME per-object IPC_RMID teardown that semctl/msgctl(IPC_RMID) already perform (set the object's removed flag; waiters.wake_up_all() so blocked callers return EIDRM; for message queues, unlink and free each SysVMessage). They are factored so both the syscall path and this namespace-teardown path share one implementation. (Deferred handoff by symbol: Section 17.1 SemSet/MsgQueue — expose sem_rmid_mark_and_wake/msg_rmid_mark_and_wake/ msg_drain_to_slab as the shared IPC_RMID primitive; Section 8.8 mq_drain_and_uncharge — drains any still-queued message memory and uncharges the queue's full creation-time footprint amount against mq.uid via mq_uncharge — the charge is footprint-based, applied once at mq_open(O_CREAT), never per message.)

detach_sysv_sem() field-name correctness (deferred handoff by symbol). The external caller in Section 8.2 reads ipc_ns.sem_ids.lookup(undo_ref.sem_id) — there is NO sem_ids field. The canonical accessor against the struct above is ipc_ns.sem.read().lookup(undo_ref.sem_id) (the single RwLock<IpcIdTable<SemSet>>, whose lookup() validates the id sequence and returns an owned Arc<SemSet> safe to hold across the sem_set.lock acquisition and the wake). SemSet.lock and SemSet.waiters DO exist (Section 17.1); SemUndo.sem_id is already i32, matching SemSet.semid. The msg_slab the MsgQueue allocation doc requires is the IpcNamespace.msg_slab field added above.

Syscall dispatch. The key/flags resolution, the error paths, and the IPC_PRIVATE special case are IDENTICAL across shmget/semget/msgget, so they are factored into one get_or_create() helper. The three syscalls differ only in the per-type LIMIT checks and object construction, passed as a closure.

// Shared get-or-create. `table_lock` is the IpcNamespace RwLock for this type.
// `create` does the type-specific limit checks (object COUNT first, against the
// borrowed table's `in_use`; then size/aggregate), builds the object, applies
// any per-namespace accounting increment, and returns (Arc<T>, max_count) — or
// an errno. Ordering matters: `create` performs the count check BEFORE any
// accounting mutation and returns `max_count = in_use ceiling ≤ IPC_MNI`, so a
// successful `create` GUARANTEES `insert` succeeds (free slot exists, count in
// range) — no rollback path is needed. The single write() makes key-lookup +
// allocate atomic, closing the TOCTOU that IPC_CREAT|IPC_EXCL's EEXIST
// guarantee depends on. `ipc_ns.user_ns` backs has_ns_cap() overrides.
fn get_or_create(ipc_ns, table_lock, key, flags,
          create: impl FnOnce(&IpcIdTable<T>) -> Result<(Arc<T>, u32), Errno>,
          perm_check: impl Fn(&T) -> Result<(), Errno>)   // EACCES + type-specific EINVAL
    -> Result<i32, Errno>
{
  1. let mut table = table_lock.write();   // covers ids + key_map + seq + in_use
  2. if key == IPC_PRIVATE {
       // ALWAYS a new anonymous object. Never consults or enters key_map —
       // every IPC_PRIVATE segment would otherwise collide at XArray key 0.
       let (obj, max_count) = create(&table)?;
       return table.insert(IPC_PRIVATE, obj, max_count);   // key not inserted
     }
  3. match table.lookup_by_key(key) {
       Some(id) => {                         // key exists
         if (flags & IPC_CREAT) != 0 && (flags & IPC_EXCL) != 0 {
             return Err(Errno::EEXIST);       // the race the single RwLock closes
         }
         let obj = table.lookup(id).expect("key_map id must resolve under the lock");
         perm_check(&obj)?;                   // mode bits → EACCES; size/nsems → EINVAL
         return Ok(id);
       }
       None => {                              // key absent
         if (flags & IPC_CREAT) == 0 { return Err(Errno::ENOENT); }
         let (obj, max_count) = create(&table)?;
         return table.insert(key, obj, max_count);   // key ≠ IPC_PRIVATE → entered
       }
     }
}

shmget(key, size, flags):
  ipc_ns = current_ipc_ns()
  return get_or_create(ipc_ns, &ipc_ns.shm, key, flags,
    create = |table| {
      let shmmni = ipc_ns.limits.shmmni.load(Relaxed);
      if table.in_use >= shmmni { return Err(ENOSPC); }               // shmmni: COUNT check first
      let shmmax = ipc_ns.limits.shmmax.load(Relaxed);
      if (size as u64) > shmmax || size == 0 { return Err(EINVAL); }   // size bound
      let pages = size.div_ceil(PAGE_SIZE) as u64;
      if ipc_ns.shm_tot.load(Relaxed) + pages > ipc_ns.limits.shmall.load(Relaxed) {
          return Err(ENOSPC);                                          // shmall
      }
      // All checks passed under the write lock ⇒ insert cannot fail, so this
      // accounting increment never needs a rollback.
      // Built with shmid = 0; insert() stamps the real id via IpcObject::set_id
      // (under the write lock, before the segment is findable). See IpcObject.
      let seg = Arc::new(ShmSegment::new(key, size, flags, ipc_ns));
      ipc_ns.shm_tot.fetch_add(pages, AcqRel);
      Ok((seg, shmmni))                                                // shmmni bounds count
    },
    perm_check = |seg| {
      check_perm(seg, flags, ipc_ns)?;                                   // EACCES
      if (size as u64) > seg.size as u64 { return Err(EINVAL); }       // request > segment
      Ok(())
    })
  // insert() has stamped seg.shmid (= build_id(idx, seq)) via IpcObject::set_id;
  // IPC_STAT reports it. The returned i32 id is the same value.

semget(key, nsems, flags):
  ipc_ns = current_ipc_ns()
  return get_or_create(ipc_ns, &ipc_ns.sem, key, flags,
    create = |table| {
      let semmni = ipc_ns.limits.semmni.load(Relaxed);
      if table.in_use >= semmni { return Err(ENOSPC); }               // semmni: COUNT check first
      let semmsl = ipc_ns.limits.semmsl.load(Relaxed);
      if nsems == 0 || nsems > semmsl { return Err(EINVAL); }          // nsems bound
      if ipc_ns.used_sems.load(Relaxed) + nsems as u64
           > ipc_ns.limits.semmns.load(Relaxed) as u64 { return Err(ENOSPC); }  // semmns
      let set = Arc::new(SemSet::new(key, nsems, flags, ipc_ns));      // nsems AtomicU16
      ipc_ns.used_sems.fetch_add(nsems as u64, AcqRel);                // insert cannot fail now
      Ok((set, semmni))                                                // semmni bounds count
    },
    perm_check = |set| {
      check_perm(set, flags, ipc_ns)?;                                   // EACCES
      if nsems != 0 && nsems > set.nsems as u32 { return Err(EINVAL); } // request > set
      Ok(())
    })

msgget(key, flags):
  ipc_ns = current_ipc_ns()
  return get_or_create(ipc_ns, &ipc_ns.msg, key, flags,
    create = |table| {
      let msgmni = ipc_ns.limits.msgmni.load(Relaxed);
      if table.in_use >= msgmni { return Err(ENOSPC); }               // msgmni: COUNT check first
      // Seed the new queue's byte cap from the LIVE limit (not a hardcoded
      // MSGMNB), so tuning kernel.msgmnb affects subsequently-created queues.
      let q = Arc::new(MsgQueue::new(key, flags, ipc_ns,
                                     /*max_bytes=*/ ipc_ns.limits.msgmnb.load(Relaxed) as usize));
      Ok((q, msgmni))                                                  // msgmni bounds count
    },
    perm_check = |_q| check_perm(_q, flags, ipc_ns))                     // EACCES (no extra EINVAL)

check_perm(obj, flags, ipc_ns) performs the SysV IPC permission check: it compares the requested access (derived from flags's low 9 mode bits) against the object's owner/group/mode, granting on a match or when the caller has CAP_IPC_OWNER via has_ns_cap(ipc_ns.user_ns, CAP_IPC_OWNER); otherwise EACCES. The per-type perm_check closures add the EINVAL cases Linux raises on an existing object (shmget size larger than the segment; semget nsems larger than the set). Every limit above reads the LIVE atomic from ipc_ns.limits, so a sysctl write takes effect on subsequent calls.

Isolation guarantee: A process in IPC namespace A cannot access or even detect the existence of IPC objects in namespace B. The key-to-ID mappings are entirely disjoint. ipcs inside a container shows only that container's IPC objects. This matches Linux IPC namespace semantics required by Docker and Kubernetes pod isolation (shareProcessNamespace: false implies separate IPC namespaces).