Chapter 3: Concurrency Model¶
Locking strategy, lock-free structures, PerCpu, RCU, atomic operations, memory ordering, interrupt handling
Concurrency is built on Rust ownership and lock-free primitives. There is no Big Kernel
Lock. Hot paths use per-CPU state (CpuLocal via dedicated register), RCU for read-mostly
data, and lock-free rings. Spinlocks protect short critical sections; mutexes are for
sleepable paths only. Memory ordering is explicit and per-architecture validated.
3.1 Rust Ownership for Lock-Free Paths¶
Rust's ownership system provides compile-time guarantees that replace
Linux runtime-only checking tools (lockdep, KASAN, KCSAN).
/// Guard returned by `preempt_disable()`. Held reference prevents the
/// current task from being preempted and migrated to another CPU.
/// Dropped by `preempt_enable()` (implicit on scope exit).
/// Cannot be sent across threads (`!Send`).
pub struct PreemptGuard {
_not_send: PhantomData<*const ()>,
}
/// Disable preemption on the current CPU. Returns a guard; preemption
/// is re-enabled when the guard is dropped. Nesting is tracked via
/// `CpuLocal.preempt_count` — the counter increments on acquire and
/// decrements on drop. Interrupts are **not** disabled by this call;
/// use `local_irq_save()` for interrupt masking.
///
/// Safe to call from any context: process, softirq, hardirq, NMI.
/// `preempt_count` is a nesting depth counter — nested calls from
/// interrupt context are the correct and expected behavior. (UmkaOS uses
/// separate `irq_count` / `softirq_count` / `preempt_count` fields in
/// CpuLocalBlock, not a packed bitfield with a "high bit" discriminator.)
pub fn preempt_disable() -> PreemptGuard {
// Single-instruction register-relative increment of preempt_count
// (safe accessor — the increment itself is what pins the task,
// [Section 3.2](#cpulocal-register-based-per-cpu-fast-path--generic-cpulocal-field-accessors)).
cpu_local::preempt_count_inc();
PreemptGuard { _not_send: PhantomData }
}
impl Drop for PreemptGuard {
fn drop(&mut self) {
// Matching decrement — the standard preempt_enable() epilogue:
// if the count reaches 0 and need_resched is set, schedule().
// Debug builds assert no underflow (guard lifetime guarantees it).
cpu_local::preempt_count_dec_and_test_resched();
}
}
impl PreemptGuard {
/// The CPU this guard pins the caller to, as a per-CPU-table index.
///
/// Reads `CpuLocalBlock.cpu_id` through the CpuLocal register base
/// (`cpu_local::cpu_id()`,
/// [Section 3.2](#cpulocal-register-based-per-cpu-fast-path--generic-cpulocal-field-accessors),
/// ~1-3 cycles). SAFE — no `unsafe` at call sites — precisely because
/// possession of the guard witnesses the CpuLocal access precondition:
/// preemption is disabled, so the caller cannot migrate, and the
/// returned index names the CPU it executes on for the guard's whole
/// lifetime. This is the PINNED (proof-token) form of the
/// current-CPU-id seam; the unpinned form is `cpu_local::cpu_id()`
/// itself, whose value may be stale the instant preemption re-enables.
#[inline(always)]
pub fn cpu_id(&self) -> usize {
cpu_local::cpu_id() as usize
}
}
/// Per-CPU data: compile-time prevention of cross-CPU access.
/// Access requires a proof token that can only be obtained with
/// preemption disabled on the current CPU.
///
/// The backing storage is dynamically sized at boot based on the actual
/// CPU count discovered from ACPI MADT / device tree / firmware tables.
/// No compile-time CPU count limit — MAX_CPUS (4096) is a static array
/// capacity for link-time allocation, not a runtime constraint. The actual
/// CPU count is discovered dynamically from ACPI MADT / device tree.
/// Allocation uses the boot allocator (Section 4.1) during early init,
/// before the general-purpose allocator is available.
pub struct PerCpu<T> {
/// Pointer to dynamically allocated array of per-CPU slots.
/// Length = num_possible_cpus(), discovered at boot.
data: *mut UnsafeCell<T>,
/// Number of CPU slots (set once at boot, never changes).
count: usize,
/// Per-slot borrow state tracking for runtime aliasing detection.
/// Length = count, parallel to data array. Each slot tracks:
/// - 0 = free (no active borrows)
/// - 1..=MAX-1 = that many active read borrows
/// - u32::MAX = mutably borrowed
/// This array enables detection of aliased borrows across multiple
/// PreemptGuard instances, which the type system cannot prevent alone.
borrow_state: *mut AtomicU32,
}
impl<T> PerCpu<T> {
/// Construct a new `PerCpu<T>`, allocating one slot per CPU.
///
/// **Bootstrap ordering**:
/// All allocations go through `alloc::alloc::alloc()`, which routes to
/// the currently-registered `#[global_allocator]`. During boot phases 0-7,
/// the global allocator IS the boot allocator ([Section 4.1](04-memory.md#boot-allocator)).
/// After phase 8, the slab allocator replaces it. The code uses a single
/// allocation path — the global allocator abstraction handles the routing
/// transparently, so no boot-vs-slab branching is needed here.
///
/// The `borrow_state` array (debug-only tracking) is always allocated
/// alongside the data array from the same allocator, initialized to zero
/// (`AtomicU32::new(0)` per slot — all slots start as "free").
///
/// # Safety
///
/// Must be called exactly once per `PerCpu<T>` instance, during boot on the
/// BSP with no concurrent access. The `count` parameter must equal
/// `num_possible_cpus()` discovered from firmware tables.
pub unsafe fn new(count: usize, init: impl Fn() -> T) -> Self {
let layout_data = core::alloc::Layout::array::<UnsafeCell<T>>(count).unwrap();
let data = alloc::alloc::alloc(layout_data) as *mut UnsafeCell<T>;
assert!(!data.is_null(), "PerCpu: data allocation failed");
for i in 0..count {
core::ptr::write(data.add(i), UnsafeCell::new(init()));
}
let layout_borrow = core::alloc::Layout::array::<AtomicU32>(count).unwrap();
let borrow_state = alloc::alloc::alloc_zeroed(layout_borrow) as *mut AtomicU32;
assert!(!borrow_state.is_null(), "PerCpu: borrow_state allocation failed");
// alloc_zeroed guarantees all bytes are zero, which is the correct
// representation for AtomicU32::new(0) on all architectures.
Self { data, count, borrow_state }
}
/// Returns a reference to the borrow_state atomic for the given CPU slot.
/// # Panics
/// Panics if cpu >= count (bounds check).
fn borrow_state(&self, cpu: usize) -> &'static AtomicU32 {
assert!(cpu < self.count, "PerCpu: cpu {} out of range (count {})", cpu, self.count);
// SAFETY: borrow_state was allocated with `count` elements at boot
// (in new()), and cpu < count is verified above. The pointer remains
// valid for the kernel's lifetime (static allocation). The returned
// reference is 'static because the borrow_state array outlives all
// PerCpu usages.
unsafe { &*self.borrow_state.add(cpu) }
}
/// # Safety contract
///
/// `PerCpu<T>` is a global singleton per data type, accessed via a static
/// reference. Soundness depends on preventing aliased mutable references
/// to the same CPU's slot. Two distinct hazards must be addressed:
///
/// 1. **Cross-thread aliasing**: Two threads on different CPUs access
/// *different* slots, so no aliasing occurs. A thread cannot migrate
/// mid-access because preemption is disabled by the guard.
///
/// 2. **Interrupt aliasing**: Interrupt handlers (including NMI) may fire
/// while preemption is disabled. If an interrupt handler accesses the
/// same per-CPU variable mutably, this creates aliased `&mut T`
/// references — undefined behavior in Rust. Therefore:
/// - **Read-only access via `get()`** is safe with only preemption
/// disabled, because multiple `&T` references are permitted.
/// `get()` returns a `PerCpuRefGuard` that only disables preemption.
/// - **Mutable access via `get_mut()`** requires BOTH preemption
/// disabled AND local interrupts disabled. `get_mut()` returns a
/// `PerCpuMutGuard` that disables local interrupts (via
/// `local_irq_save`/`local_irq_restore`, defined in `arch::current::interrupts`)
/// for the duration of the borrow, in addition to disabling
/// preemption. This prevents any maskable interrupt handler from
/// observing or mutating the slot while the caller holds `&mut T`.
///
/// 3. **NMI constraint**: NMIs (non-maskable interrupts) cannot be
/// disabled by `local_irq_save`. An NMI that fires during a
/// `get_mut()` borrow will see `borrow_state == u32::MAX`. If the
/// NMI handler calls `get()` on the **same** PerCpu variable, the
/// borrow_state check detects the conflict and panics. This is the
/// intended safety mechanism — panicking is preferable to silent
/// data corruption.
///
/// **NMI handler rules**:
/// - NMI handlers MUST NOT call `get_mut()` on any PerCpu variable.
/// - NMI handlers SHOULD avoid calling `get()` on PerCpu variables
/// that are commonly accessed via `get_mut()` elsewhere, because
/// the conflict causes a panic (which in NMI context halts the
/// CPU).
/// - NMI handlers that need per-CPU state MUST use dedicated
/// NMI-specific PerCpu variables that are only read (never
/// mutated) from non-NMI contexts, or use raw atomic fields
/// outside the PerCpu abstraction (e.g., the pre-allocated
/// per-CPU crash buffer used by the panic NMI handler).
/// - UmkaOS's NMI handlers (panic coordinator, watchdog, perf sampling)
/// follow this rule: they write only to dedicated pre-allocated
/// per-CPU buffers and never access the general PerCpu<T> API.
///
/// The `PreemptGuard` is `!Send`, `!Clone`, and pin-bound to the issuing
/// CPU, proving that the caller is on the correct CPU. The borrow checker
/// prevents calling `get()` and `get_mut()` on the same guard
/// simultaneously, preventing aliased `&T` + `&mut T` within the same
/// non-interrupt context.
///
/// **Aliasing safety**: The `&mut PreemptGuard` borrow prevents aliasing
/// through the *same* guard, but nothing in the type system prevents a
/// caller from creating two separate `PreemptGuard`s and using them to
/// obtain aliased references (e.g., `&T` from one guard and `&mut T` from
/// another, or two `&mut T`). To close this hole, `PerCpu<T>` maintains a
/// per-slot `borrow_state: AtomicU32` with the following encoding:
/// - `0` = slot is free (no active borrows)
/// - `1..=MAX-1` = slot has that many active read borrows (`get()` in use)
/// - `u32::MAX` = slot is mutably borrowed (`get_mut()` in use)
///
/// `get()` atomically increments the reader count (fails if currently
/// `u32::MAX`, i.e., a writer is active). `get_mut()` atomically
/// transitions from `0` to `u32::MAX` (fails if any readers OR another
/// writer are active). Both `PerCpuRefGuard` and `PerCpuMutGuard` restore
/// the borrow_state on drop. In debug builds, violations panic
/// unconditionally, catching aliasing bugs during development and testing.
/// In release builds, the borrow-state CAS is elided for performance
/// (Section 3.1.3) — the structural invariants (preemption disabled, IRQs
/// disabled for `get_mut()`) are sufficient to prevent aliased access.
/// For the hottest per-CPU fields, `CpuLocal` (Section 3.1.2) bypasses
/// `PerCpu<T>` entirely, using architecture-specific registers for
/// ~1-10 cycle access.
///
/// **Comparison with Linux**: Linux's `this_cpu_read/write` on x86-64 compiles
/// to a single `gs:`-prefixed instruction (no atomic, no preempt_disable needed
/// for the single-instruction case). UmkaOS addresses this gap with a two-tier
/// per-CPU model (Section 3.1.2-c):
///
/// - **CpuLocal** (Section 3.1.2): Register-based access (~1-10 cycles, arch-dependent)
/// for the hottest ~10 fields (current_task, runqueue, slab magazines, etc.).
/// Matches Linux's per-CPU register pattern on all eight architectures.
/// - **PerCpu\<T\>** (this struct): Generic abstraction for all other per-CPU data.
/// Borrow-state CAS is debug-only in release builds (Section 3.1.3), reducing cost
/// from ~20-30 cycles to ~3-8 cycles. The CAS catches aliasing bugs during
/// development; release builds trust the structural invariants (preemption
/// disabled + IRQs disabled = exclusive access).
///
/// See `get()` and `get_mut()` documentation below.
/// Read-only access to the current CPU's slot. Takes `&PreemptGuard`,
/// which only requires preemption to be disabled. Multiple `&T` references
/// are sound because `&T` is `Sync`-like — interrupt handlers may also
/// hold `&T` to the same slot without causing UB.
/// Returns a `PerCpuRefGuard<T>` that derefs to `&T`.
///
/// The lifetime `'g` ties the returned guard to the `PreemptGuard`, not
/// to `&self`. This prevents dropping the `PreemptGuard` (re-enabling
/// preemption and allowing migration) while still holding a reference
/// to this CPU's slot — which would be use-after-migrate unsoundness.
///
/// The `T: Sync` bound is required because interrupt handlers may
/// concurrently hold `&T` to the same slot. Without `Sync`, types like
/// `Cell<u32>` would allow data races between the caller and interrupt
/// handlers sharing the same per-CPU slot.
///
/// **Borrow-state protocol**: `get()` atomically increments the per-slot
/// `borrow_state` counter, provided the current value is not `u32::MAX`
/// (which indicates an active mutable borrow). If the slot is currently
/// mutably borrowed, `get()` panics. This prevents the `&T` + `&mut T`
/// aliasing UB that would arise if a caller obtained an immutable borrow
/// via one `PreemptGuard` while another context held a mutable borrow via
/// a second `PreemptGuard`. The `PerCpuRefGuard` decrements the counter
/// on drop, returning the slot to a lower reader count or back to 0.
pub fn get<'g>(&self, guard: &'g PreemptGuard) -> PerCpuRefGuard<'g, T>
where
T: Sync,
{
let cpu = guard.cpu_id();
assert!(cpu < self.count, "PerCpu: cpu_id {} out of range (count {})", cpu, self.count);
// Atomically increment the reader count. Fail if a writer holds the
// slot (borrow_state == u32::MAX). Use a compare-exchange loop so we
// can distinguish "writer active" from a successful increment.
// Active in debug builds only; elided in release (Section 3.1.3).
//
// Memory ordering: AcqRel on success ensures the borrow_state write
// (Release) is visible to NMI handlers or concurrent debug checks
// before we access the slot data (Acquire). Failure uses Acquire to
// see the writer's Release store.
#[cfg(debug_assertions)]
{
let state = self.borrow_state(cpu);
let prev = state.fetch_update(Ordering::AcqRel, Ordering::Acquire, |v| {
if v == u32::MAX { None } else { Some(v + 1) }
});
if prev.is_err() {
panic!("PerCpu: slot {} already mutably borrowed — cannot take shared borrow", cpu);
}
}
// SAFETY: PreemptGuard guarantees we are on cpu_id() and cannot
// migrate. In debug builds, the borrow_state increment above ensures
// no mutable borrow is active. In release builds, the structural
// invariant (preemption disabled = CPU pinned) is sufficient for
// read-only access. T: Sync guarantees shared references are sound
// across concurrent contexts (caller + interrupt handlers).
unsafe {
PerCpuRefGuard {
value: &*self.data.add(cpu).as_ref().unwrap().get(),
#[cfg(debug_assertions)]
borrow_state: self.borrow_state(cpu),
_guard: PhantomData,
}
}
}
/// Mutable access to the current CPU's slot. Takes `&mut PreemptGuard`
/// to prevent aliasing with any concurrent `get()` or `get_mut()` on
/// the same guard in the caller's context. Additionally, the returned
/// `PerCpuMutGuard` disables local interrupts (via `local_irq_save`)
/// to prevent interrupt handlers from accessing this slot while `&mut T`
/// is live. Interrupts are restored when the guard is dropped.
///
/// The lifetime `'g` ties the returned guard to the `PreemptGuard`,
/// preventing use-after-migrate (same rationale as `get()`).
///
/// This two-layer protection is necessary because preemption-disable
/// alone does NOT prevent interrupt handlers from firing and potentially
/// accessing the same per-CPU variable.
///
/// **Aliasing safety**: The `&mut PreemptGuard` borrow prevents aliasing
/// through the *same* guard, but callers could theoretically create a
/// second `PreemptGuard` and call `get()` or `get_mut()` again on the
/// same slot, producing `&T` + `&mut T` or two `&mut T` — both are UB.
/// To close this hole, `get_mut()` atomically transitions the per-slot
/// `borrow_state` from exactly `0` (free) to `u32::MAX` (writer) using
/// a compare-exchange. If the slot has any active readers (count > 0) or
/// another writer (count == u32::MAX), the CAS fails and `get_mut()`
/// panics.
///
/// **Debug vs release**: In debug builds (`cfg(debug_assertions)`), the
/// CAS is always active — catching aliasing bugs during development. In
/// release builds, the CAS is elided (Section 3.1.3): the structural invariants
/// (preemption disabled + IRQs disabled) are sufficient to guarantee
/// exclusive access. This reduces `get_mut()` cost from ~20-30 cycles to
/// ~3-8 cycles. The `borrow_state` array is still allocated for binary
/// compatibility with debug-built modules.
///
/// `PerCpu<T>` is designed to be used with a
/// single `PreemptGuard` per critical section — creating multiple guards
/// and using them to access the same `PerCpu<T>` is a logic error detected
/// at runtime.
pub fn get_mut<'g>(&self, guard: &'g mut PreemptGuard) -> PerCpuMutGuard<'g, T> {
let cpu = guard.cpu_id();
assert!(cpu < self.count, "PerCpu: cpu_id {} out of range (count {})", cpu, self.count);
// SAFETY: local_irq_save() MUST be called BEFORE updating borrow_state.
// If we update borrow_state first and an interrupt fires before IRQs
// are disabled, an interrupt handler calling get() on the same PerCpu
// variable would see borrow_state == u32::MAX and panic. Disabling IRQs
// first ensures no interrupt handler can observe or race with the
// borrow_state transition.
let saved_flags = local_irq_save();
// Runtime borrow-state check: active in debug builds
// (`cfg(debug_assertions)`); elided in release builds for performance
// (see Section 3.1.3). Atomically transitions borrow_state from 0
// (free) to u32::MAX (writer). Fails if any reader (count > 0) or
// another writer (u32::MAX) is active. Fires both for &mut T + &mut T
// (two writers) and for &T + &mut T (reader + writer) aliasing.
// This check is safe from interrupt-handler races because IRQs are
// already disabled above.
#[cfg(debug_assertions)]
{
let state = self.borrow_state(cpu);
if state.compare_exchange(0, u32::MAX, Ordering::AcqRel, Ordering::Acquire).is_err() {
local_irq_restore(saved_flags);
panic!("PerCpu: slot {} already borrowed (reader or writer active)", cpu);
// Note: The kernel is compiled with `panic = "abort"` (no unwinding),
// so borrow_state cannot leak: a panic immediately halts the core.
// For Tier 1 drivers sharing the address space, driver panics are caught
// by the driver fault handler (Section 11.7) which resets the driver's state
// including any held per-CPU borrows and any Core kernel locks acquired
// through KABI calls (see Section 11.7 ISOLATE step: KABI lock registry).
}
}
// SAFETY: PreemptGuard prevents migration. local_irq_save() prevents
// interrupt handlers from accessing this slot. In debug builds, the CAS
// above additionally verifies no aliased borrows exist. In release builds,
// we trust the structural invariants. Together, these guarantee exclusive
// access to this CPU's slot.
unsafe {
PerCpuMutGuard {
value: &mut *self.data.add(cpu).as_ref().unwrap().get(),
saved_flags,
#[cfg(debug_assertions)]
borrow_state: self.borrow_state(cpu),
_guard: PhantomData,
}
}
}
}
/// Guard for read-only per-CPU access. Only requires preemption disabled.
/// Implements `Deref<Target = T>` for ergonomic read access.
///
/// The lifetime `'a` is tied to the `PreemptGuard`, not to the `PerCpu<T>`
/// container. This ensures the guard cannot outlive the preemption-disabled
/// critical section, preventing use-after-migrate.
///
/// On creation, the per-slot `borrow_state` reader count was incremented by
/// `get()`. On drop, `PerCpuRefGuard` decrements the reader count, returning
/// the slot to its prior borrow state. This is necessary so that `get_mut()`
/// can detect when readers are no longer active and safely transition to
/// writer mode.
pub struct PerCpuRefGuard<'a, T> {
value: &'a T,
/// Reference to the per-slot borrow_state counter so Drop can decrement
/// the reader count. Only present in debug builds (`#[cfg(debug_assertions)]`);
/// in release builds, the borrow-state CAS is elided (Section 3.1.3).
#[cfg(debug_assertions)]
borrow_state: &'a AtomicU32,
/// Ties this guard's lifetime to the `PreemptGuard`, not to `PerCpu<T>`.
_guard: PhantomData<&'a PreemptGuard>,
}
/// `PerCpuRefGuard` must NOT be sent to another CPU/thread. It holds a
/// reference to a per-CPU slot that is only valid on the CPU where the
/// `PreemptGuard` was obtained. Sending it to another thread (which runs
/// on a different CPU) would allow reading another CPU's slot without any
/// synchronization, violating the per-CPU data invariant.
impl<T> !Send for PerCpuRefGuard<'_, T> {}
impl<'a, T> core::ops::Deref for PerCpuRefGuard<'a, T> {
type Target = T;
fn deref(&self) -> &T { self.value }
}
impl<'a, T> Drop for PerCpuRefGuard<'a, T> {
fn drop(&mut self) {
// Decrement the reader count with underflow detection.
// Only present in debug builds; release builds elide borrow tracking
// (Section 3.1.3).
#[cfg(debug_assertions)]
{
// CORRECTNESS: fetch_sub on 0 wraps to u32::MAX (the writer sentinel),
// which would corrupt the borrow state. We detect this case after the
// fact and panic rather than silently corrupting state.
//
// Normal case: borrow_state is 1..=u32::MAX-1 (one or more readers).
// Underflow case: borrow_state is 0 (no readers — this is a bug).
// Writer-corruption case: borrow_state is u32::MAX (impossible if get()
// correctly checked for writer before incrementing).
//
// We panic on error because it indicates a logic error in the PerCpu
// borrow tracking, and continuing would corrupt state. The recovery
// store attempts to restore a consistent state for debugging, but
// the kernel will halt anyway (panic = "abort" for kernel code).
//
// **Why no IRQ protection?** An interrupt handler that fires between
// fetch_sub and the panic check, and calls get() on the same PerCpu,
// will see borrow_state == u32::MAX (writer sentinel) and correctly
// fail its get() call. The handler does NOT proceed with corrupted
// state — it safely errors out. Adding IRQ save/restore to every guard
// drop (hot path) to protect a code path that already panics is
// unnecessary overhead.
let old = self.borrow_state.fetch_sub(1, Ordering::Release);
if old == 0 {
// Underflow: no reader was registered. This is a kernel bug.
// The fetch_sub wrapped to u32::MAX. Restore to 0 and panic.
self.borrow_state.store(0, Ordering::Release);
panic!("PerCpuRefGuard::drop: borrow_state underflow — double-drop or missing get()");
}
if old == u32::MAX {
// Was in writer mode — this should be impossible since get() fails
// when a writer is active. If we reach here, state is corrupted.
// The fetch_sub wrapped to u32::MAX-1. Restore sentinel and panic.
self.borrow_state.store(u32::MAX, Ordering::Release);
panic!("PerCpuRefGuard::drop: borrow_state was u32::MAX (writer sentinel) — corrupted state");
}
// Normal case: old was 1..=u32::MAX-1, now decremented successfully.
// No further action needed.
}
}
}
/// Guard for mutable per-CPU access. Disables local interrupts on creation,
/// restores them on drop. This prevents interrupt handlers from creating
/// aliased references to the same per-CPU slot.
///
/// The lifetime `'a` is tied to the `PreemptGuard` (via `PhantomData`),
/// not to the `PerCpu<T>` container. Same rationale as `PerCpuRefGuard`.
///
/// On creation, the per-slot `borrow_state` was set to `u32::MAX` (writer
/// sentinel) by `get_mut()`. On drop, `PerCpuMutGuard` resets `borrow_state`
/// to `0` (free) before restoring local IRQs, allowing subsequent `get()` or
/// `get_mut()` calls on this slot.
pub struct PerCpuMutGuard<'a, T> {
value: &'a mut T,
saved_flags: usize,
/// Reference to the per-slot borrow_state counter so Drop can reset it
/// to 0 (free), enabling detection of concurrent borrows in subsequent
/// callers. Only present in debug builds (`#[cfg(debug_assertions)]`);
/// in release builds, the borrow-state CAS is elided entirely
/// (Section 3.1.3) and the structural invariants (preemption disabled +
/// IRQs disabled) are sufficient to guarantee exclusive access.
#[cfg(debug_assertions)]
borrow_state: &'a AtomicU32,
/// Ties this guard's lifetime to the `PreemptGuard`, not to `PerCpu<T>`.
_guard: PhantomData<&'a mut PreemptGuard>,
}
/// `PerCpuMutGuard` must NOT be sent to another CPU/thread. The `saved_flags`
/// field contains the interrupt state of the originating CPU (saved via
/// `local_irq_save`). Restoring these flags on a different CPU would corrupt
/// that CPU's interrupt state — potentially enabling interrupts that should
/// be disabled or vice versa, leading to missed interrupts or unsafe re-entry.
impl<T> !Send for PerCpuMutGuard<'_, T> {}
impl<'a, T> core::ops::Deref for PerCpuMutGuard<'a, T> {
type Target = T;
fn deref(&self) -> &T { self.value }
}
impl<'a, T> core::ops::DerefMut for PerCpuMutGuard<'a, T> {
fn deref_mut(&mut self) -> &mut T { self.value }
}
impl<'a, T> Drop for PerCpuMutGuard<'a, T> {
fn drop(&mut self) {
// Reset borrow_state from u32::MAX (writer) back to 0 (free) BEFORE
// restoring IRQs. This ensures that any interrupt handler firing
// immediately after IRQ restoration sees the slot as free and can
// call get() or get_mut() without a false-positive panic.
// Only present in debug builds; release builds elide borrow tracking.
//
// NMI window note: Between borrow_state.store(0) and drop() completion,
// an NMI handler could see borrow_state == 0 and obtain &T while this
// guard still technically holds &mut T. This is safe in practice because:
// (1) no writes occur after the borrow_state reset — the mutation is
// complete, and (2) the remaining operation (local_irq_restore) does
// not access the PerCpu slot data. In release builds, borrow_state
// tracking is entirely compiled out, so no observable state transition
// exists. The reverse order (IRQ restore before borrow_state reset)
// would cause IRQ handlers to see borrow_state == MAX and panic.
#[cfg(debug_assertions)]
{
self.borrow_state.store(0, Ordering::Release);
}
local_irq_restore(self.saved_flags);
}
}
3.1.1 Guarded Position Claim¶
The guarded_claim primitive underpins lock-free ring/queue slot claiming.
Such algorithms claim a slot by advancing a single monotonic position counter
(a ring head or tail) and validating the target slot's sequence number relative
to that position. A value-compare CAS (compare_exchange(pos,
pos + 1)) is subject to ABA: a claimant that stalls between reading pos and
committing can, in principle, observe the counter wrap a full 2^BITS back to the
SAME numeric value and commit a claim for a DIFFERENT logical position. At 64-bit
width a full wrap is physically unreachable; but PPC32 has no native 64-bit atomic
and its 32-bit counter can wrap. UmkaOS closes this window WITHOUT a lock and
WITHOUT probabilistic acceptance by defining an arch-abstracted guarded claim
whose commit is conditioned on an UNBROKEN reservation, not a matching value — the
load-reserved/store-conditional (ll/sc) discipline. Scope is single-word
sequence/position claims ONLY: coupled multi-field state spans two granules, which
one reservation cannot cover, and MUST use a lock instead (e.g. the cfg-split
SpinLock TokenBucket,
Section 3.6).
Pseudocode convention: Code in this section uses Rust syntax and follows Rust ownership, borrowing, and type rules. Atomic fields use
.store()/.load()/.compare_exchange_weak(). See CLAUDE.md §Spec Pseudocode Quality Gates.
/// Width of a lock-free claim counter (a ring head/tail, or a per-slot sequence
/// number compared against one).
///
/// The width is selected by `target_has_atomic = "64"`, NOT by pointer width
/// (`usize`) — the two DIVERGE on ARMv7-A, where `usize` is 32-bit yet the target
/// has native 64-bit atomics (LDREXD/STREXD; Rust `armv7a-none-eabi` carries
/// `max_atomic_width: Some(64)`). A claim counter must be as wide as the widest
/// atomic the leg can carry in ONE instruction, so the split is on atomic width:
/// - `target_has_atomic = "64"` — x86-64, AArch64, ARMv7-A, RISC-V64, PPC64LE,
/// s390x, LoongArch64: 64-bit. A full `2^64` wrap is physically unreachable
/// (see `guarded_claim`), so `guarded_claim` lowers to a plain CAS loop with no
/// reservation guard.
/// - `not(target_has_atomic = "64")` — PPC32 alone among supported targets:
/// 32-bit native. The wrap window is closed STRUCTURALLY by the ll/sc lowering
/// of `guarded_claim`.
#[cfg(target_has_atomic = "64")]
pub type ClaimPos = u64;
#[cfg(target_has_atomic = "64")]
pub type AtomicClaimPos = AtomicU64;
/// Signed companion of `ClaimPos`, SAME width — used for the wrapping-RELATIVE
/// slot classification (`seq.wrapping_sub(pos) as SignedClaimPos`). Cast to this
/// type, never to `isize`: `isize` tracks pointer width and would truncate the
/// 64-bit difference to 32 bits on ARMv7-A.
#[cfg(target_has_atomic = "64")]
pub type SignedClaimPos = i64;
#[cfg(not(target_has_atomic = "64"))]
pub type ClaimPos = u32;
#[cfg(not(target_has_atomic = "64"))]
pub type AtomicClaimPos = AtomicU32;
#[cfg(not(target_has_atomic = "64"))]
pub type SignedClaimPos = i32;
/// Terminal result of `guarded_claim`: the structure cannot satisfy the claim
/// right now — a full producer ring, or an empty consumer ring. Distinguished
/// from a transient retry (a stale anchor), which the primitive handles
/// internally by re-anchoring.
pub struct ClaimUnavailable;
/// Guarded single-word position claim — a fetch-increment-if-valid over one
/// atomic counter, ABA-free at ANY counter width.
///
/// `word` is the claim counter (a ring head/tail). The paired per-slot sequence
/// number is loaded BY THE PRIMITIVE, INSIDE the claim window: the caller supplies
/// the slot ADDRESS (`seq_of`) and target arithmetic (`expected`), and the
/// primitive — not the caller — performs the deciding load. `seq_of(pos)` returns the sequence atomic to
/// load for the freshly-anchored position `pos`; `expected(pos)` returns the
/// sequence value that marks the slot claimable at `pos`. The primitive anchors
/// `pos`, loads `seq_of(pos)` with `Acquire`, and classifies the
/// wrapping-RELATIVE distance `dif = seq.wrapping_sub(expected(pos)) as
/// SignedClaimPos`:
/// - `dif == 0` → claimable: atomically advance `pos -> pos + 1`; on success
/// return `Ok(pos)`, on a lost race re-anchor.
/// - `dif > 0` → a concurrent claimant already moved past this anchor:
/// re-anchor and re-classify.
/// - `dif < 0` → the structure cannot satisfy the claim (a full producer ring
/// or an empty consumer ring): return `Err(ClaimUnavailable)` without
/// advancing.
///
/// **The deciding load is issued by the primitive, inside the window
/// (load-bearing)**: `seq_of(pos)` returns the sequence ATOMIC to read and
/// `expected(pos)` the value it is compared against, but the LOAD whose result
/// drives the commit decision — `seq_of(pos).load(Acquire)` — is performed by
/// `guarded_claim` itself, at the freshly-anchored `pos`, AFTER the anchor. A
/// caller cannot substitute its own earlier load for the DECIDING load, which the
/// primitive issues inside the window. It CAN, however, still taint the decision
/// through an impure `expected` — a closure that RETURNS a captured pre-anchor
/// value — which is exactly why closure purity is a documented obligation (next
/// paragraph), NOT a type-system guarantee. This enforced-load step is strictly
/// stronger than the `validate`-closure shape it replaces, which handed the whole
/// decision — the load included — to caller code.
///
/// This is NOT total immunity by construction, and must not be stated as such:
/// `seq_of` and `expected` are ordinary closures. A closure that CAPTURES a value
/// read before the anchor — e.g. `expected` returning a sequence number the caller
/// loaded earlier and closed over — would feed pre-anchor state into the
/// `wrapping_sub` classification and reintroduce the stale-window ABA. Excluding
/// that is a DOCUMENTED CALLER OBLIGATION, not a shape the type system forecloses:
/// `seq_of` MUST compute only a slot address from `pos`, and `expected` MUST
/// compute only arithmetic on `pos`; neither may perform a side load or return a
/// captured value that originates from a load. The ring call sites below honour
/// this (each passes a pure index into `self.slots` and pure `pos` arithmetic); a
/// future closure that violates it reintroduces the defect.
///
/// **Contract (load-bearing)**: a commit succeeds ONLY IF no other agent advanced
/// `word` between the anchor and the commit. Validation observes only post-anchor
/// state, and a stale-value window across preemption, interrupt, or another CPU's
/// store CANNOT commit. This is the ll/sc-vs-CAS distinction: success requires an
/// UNBROKEN reservation (or, on wide-atomic legs, an unwrapped counter), not a
/// matching value — so numeric aliasing after a full counter wrap cannot forge a
/// claim. Because a lost reservation only forces a re-anchor, preemption inside
/// the window is safe and NO preempt-disable is required (a context switch clears
/// any live reservation,
/// [Section 7.3](07-scheduling.md#context-switch-and-register-state--llsc-reservation-clearing-arm-risc-v-powerpc)).
/// On the ll/sc lowering the primitive additionally REQUIRES (i) that the arch
/// interrupt/exception-return path clear any outstanding reservation before
/// resuming the interrupted context
/// ([Section 3.8](#interrupt-handling--interrupt-return-reservation-clearing)), and (ii)
/// that the lowering itself consume the reservation on EVERY non-committing exit
/// from the window. Together these close the interrupt-nested case: a handler
/// that anchors and exits without committing must not leave a live reservation
/// the outer claimant could later commit against on stale validation.
///
/// Scope: single-word sequence/position counters ONLY (see the section intro).
///
/// The `word` is advanced with `Relaxed` ordering on both lowerings — the counter
/// is claim ARBITRATION only; all data-visibility ordering is carried by the
/// per-slot `seq` handshake (the `Acquire` load here pairs with the caller's
/// `Release` at publish).
///
/// **Lowering — legs with 64-bit atomics** (all but PPC32): a plain
/// `compare_exchange` loop on the 64-bit word — ONE generic definition, no
/// per-arch code. The ABA guarantee is VACUOUS by width: a 64-bit claim counter
/// cannot complete a `2^64` wrap within the operational lifetime (the wrap-safety
/// analysis on `BoundedMpmcRing`,
/// [Section 3.11](#workqueue-deferred-work--boundedmpmcring-memory-ordering-specification),
/// puts it at ~5.8x10^4 years even at 10^7 claims/s), so a matching value already
/// implies an unbroken count and a plain CAS is a sound commit. Spurious CAS
/// failure is absorbed by the retry loop
/// ([Section 3.6](#lock-free-data-structures--compare-and-swap-semantics-differ-by-architecture)).
/// This lowering holds NO reservation across the window (the `compare_exchange`
/// is self-contained), so requirements (i)/(ii) above are vacuous on it.
///
/// **RISC-V scoping note**: the guarded (ll/sc) lowering below is specified ONLY
/// for legs lacking 64-bit atomics. It is NOT portable to a RISC-V `LR`/`SC`
/// lowering — RISC-V's LR/SC forward-progress ("eventuality") guarantee is void
/// if the reserved window contains ANY load or store (RISC-V ISA, "A" extension),
/// and validation must load the slot sequence inside the window. This is moot for
/// UmkaOS because RISC-V64 has 64-bit atomics and takes the plain-CAS branch; the
/// note stands so no future port lowers `guarded_claim` onto LR/SC.
#[cfg(target_has_atomic = "64")]
pub fn guarded_claim<'a>(
word: &'a AtomicClaimPos,
seq_of: impl Fn(ClaimPos) -> &'a AtomicClaimPos,
expected: impl Fn(ClaimPos) -> ClaimPos,
) -> Result<ClaimPos, ClaimUnavailable> {
loop {
let pos = word.load(Ordering::Relaxed); // anchor
// The primitive issues the deciding load HERE, inside the window (Acquire
// pairs with the caller's Release publish); the caller supplied only the
// address (`seq_of`) and arithmetic (`expected`). Keeping `expected` from
// returning a captured pre-anchor value is the closure-purity obligation —
// see the deciding-load note above.
let dif = seq_of(pos).load(Ordering::Acquire)
.wrapping_sub(expected(pos)) as SignedClaimPos;
if dif == 0 {
// Width makes ABA vacuous: a matching count implies an unwrapped
// counter, so a plain CAS is a sound commit — no reservation.
match word.compare_exchange_weak(
pos, pos.wrapping_add(1),
Ordering::Relaxed, Ordering::Relaxed,
) {
Ok(_) => return Ok(pos),
Err(_) => continue, // lost race → re-anchor
}
} else if dif > 0 {
continue; // stale anchor → re-anchor
} else {
return Err(ClaimUnavailable); // full / empty
}
}
}
// ---- arch::current::atomic reservation seam (PPC32 leg only) ---------------
// The PPC32 arch module (`arch/powerpc/`, the only supported leg without 64-bit
// atomics) provides the three reservation primitives `arch::current::atomic`
// exposes to the guarded lowering below. That lowering is ONE of two callers:
// the other is `reserve_head_claim()`
// ([Section 14.3](14-vfs.md#vfs-per-cpu-ring-extension)), the head/tail-ring variant of this same
// ll/sc discipline — it anchors on a ring `head` word and validates against the
// `tail` (rather than a per-slot sequence), but on the PPC32 leg it calls these
// SAME three primitives directly. The 64-bit lowering above holds no reservation
// and needs none. These items are
// absent on every 64-bit-atomic leg. Naming `lwarx`/`stwcx.` in these cfg-gated
// per-arch items is the ESC-0120 exemption.
/// `lwarx`: anchor a reservation on `word` and return its current value. An
/// ordinary load by the reserving processor between this and `store_conditional`
/// leaves the reservation intact (Power ISA Book II, Storage Control).
pub fn load_reserved(word: &AtomicClaimPos) -> ClaimPos { /* lwarx */ }
/// `stwcx.`: conditionally store `val` into `word`. Returns `true` iff the
/// reservation anchored by `load_reserved` still held (the store committed); a
/// `false` return means the reservation was lost. EITHER outcome consumes it.
pub fn store_conditional(word: &AtomicClaimPos, val: ClaimPos) -> bool { /* stwcx. */ }
/// Consume any outstanding reservation held by this CPU WITHOUT committing a
/// claim: issue a `stwcx.` to this CPU's private `CpuLocalBlock.llsc_dummy`
/// scratch ([Section 3.2](#cpulocal-register-based-per-cpu-fast-path) — the `LlscDummy`
/// field). Whether that conditional store succeeds or fails, a `stwcx.` clears
/// the processor's reservation, so on return NO live reservation remains. The
/// scratch is WRITE-ONLY: its value is never read; the store exists only for its
/// reservation-clearing side effect. Same reservation-kill idiom the context
/// switch path uses
/// ([Section 7.3](07-scheduling.md#context-switch-and-register-state--llsc-reservation-clearing-arm-risc-v-powerpc)).
/// Called by the `guarded_claim` lowering below on every NON-committing window
/// exit (a committing claim consumes the reservation through its own `stwcx.`).
pub fn kill_reservation() { /* stwcx. to CpuLocalBlock.llsc_dummy.word */ }
/// PPC32 ll/sc lowering. `arch::current::atomic::load_reserved` == `lwarx`,
/// `store_conditional` == `stwcx.`, and `kill_reservation` == a `stwcx.` to this
/// CPU's per-CPU `LlscDummy` scratch word (all defined in `arch/powerpc/`; naming
/// the instructions in this cfg-gated per-arch item is the ESC-0120 exemption).
/// The scratch
/// `stwcx.` is the same per-CPU `LlscDummy` reservation-kill idiom the context
/// switch path uses
/// ([Section 7.3](07-scheduling.md#context-switch-and-register-state--llsc-reservation-clearing-arm-risc-v-powerpc)).
/// `lwarx` anchors a reservation on `word`; the paired-sequence load performed
/// inside the window does NOT clear it (Power ISA Book II, Storage Control —
/// ordinary loads by the reserving processor between larx and stcx. leave the
/// reservation intact); `stwcx.` commits `pos -> pos + 1` and SUCCEEDS ONLY IF
/// the reservation still holds. A single store to the reservation granule by
/// ANOTHER processor architecturally clears the reservation (Power ISA Book II;
/// Book E / e500 agree — a store or `dcbz` by another processor to the granule
/// clears it), so a claimant stalled across ANY other CPU's progress fails its
/// `stwcx.` by construction: the `2^32` stores to `word` a full wrap would
/// require include the one that already broke this reservation. Structural, not
/// probabilistic.
///
/// **Reservation consumed on EVERY window exit.** A committing claim consumes the
/// reservation through its own `stwcx.`. The two NON-committing exits — re-anchor
/// (`dif > 0`) and unavailable (`dif < 0`) — leave the reserved window WITHOUT a
/// commit, so each first calls `kill_reservation()`. Whether that scratch
/// `stwcx.` succeeds or fails is IRRELEVANT: a `stwcx.` clears the processor's
/// reservation either way, so both outcomes leave NO live reservation. This is
/// required because PPC32 interrupt return (`rfi`) does NOT itself clear a
/// reservation
/// ([Section 3.8](#interrupt-handling--interrupt-return-reservation-clearing)): a nested
/// interrupt handler that anchored on `word` and then exited via a non-committing
/// arm WITHOUT consuming would hand its live reservation to the resumed OUTER
/// claimant, whose `stwcx.` — validated against DIFFERENT, pre-interrupt state —
/// could then wrongly succeed, producing two owners of one slot. Consuming on
/// exit (belt) and the interrupt-return clearing invariant (suspenders) both
/// close that window.
///
/// (A benign spurious `stwcx.` failure — from an unrelated store to the same
/// granule, or from a context switch that clears the reservation — merely forces
/// a re-anchor; correctness is unaffected,
/// [Section 3.6](#lock-free-data-structures--compare-and-swap-semantics-differ-by-architecture).
/// An INTERRUPT is NOT made safe by any implicit reservation drop on `rfi` — `rfi`
/// drops nothing — but by the handler's own granule traffic plus the mandated
/// exit-path consumption above, backed by the interrupt-return clearing invariant.)
#[cfg(not(target_has_atomic = "64"))]
pub fn guarded_claim<'a>(
word: &'a AtomicClaimPos,
seq_of: impl Fn(ClaimPos) -> &'a AtomicClaimPos,
expected: impl Fn(ClaimPos) -> ClaimPos,
) -> Result<ClaimPos, ClaimUnavailable> {
loop {
let pos = arch::current::atomic::load_reserved(word); // lwarx: anchor
// Paired-sequence load INSIDE the reservation window (Acquire pairs with
// the caller's Release publish). Ordinary loads do not clear a reservation.
let dif = seq_of(pos).load(Ordering::Acquire)
.wrapping_sub(expected(pos)) as SignedClaimPos;
if dif == 0 {
// stwcx.: commits only on an unbroken reservation. `false` = the
// reservation was lost (another claimant, or a spurious clear). Either
// way the reservation is consumed by this stwcx.
if arch::current::atomic::store_conditional(word, pos.wrapping_add(1)) {
return Ok(pos);
}
continue; // reservation lost → re-anchor
} else if dif > 0 {
// Non-committing exit: consume the reservation before re-anchoring, so
// no live reservation leaks out of the window (see doc comment).
arch::current::atomic::kill_reservation();
continue; // stale anchor → re-anchor
} else {
arch::current::atomic::kill_reservation(); // consume, as above
return Err(ClaimUnavailable); // full / empty
}
}
}
3.1.2 PerProcess: Lazily-Allocated Per-Process Storage¶
PerProcess<T> is the per-process analog of PerCpu<T>: a slot of T keyed by
the current process, allocated on first write and reclaimed when the process
exits. It is for state that only a minority of processes ever touch — for
example the Windows NT handle table (Section 19.6), where
most processes issue no WEA syscall and must pay nothing.
/// Per-process lazily-allocated storage keyed by `ProcessId`.
///
/// **Zero cost when empty**: the backing store holds no entry until a process
/// first populates it via `get_or_insert_with`; a process that never touches
/// the storage has no slot and no memory footprint. Reads are lock-free under
/// an `RcuReadGuard`. The per-process entry is torn down by `reap_task()`
/// when the process's last thread exits, so nothing leaks across the 50-year
/// uptime envelope.
///
/// `T: Send + Sync` because a slot may be observed from any CPU the owning
/// process's threads run on.
pub struct PerProcess<T: Send + Sync> {
/// Sparse map of live per-process slots. Empty until first use. Values are
/// `Arc<T>` so a lookup can hand out a refcounted clone the caller keeps
/// past the RCU guard, and so `T` itself need not be `Clone` (the map's
/// `V: Clone` bound is satisfied by `Arc<T>`, which is always `Clone`).
slots: RcuHashMap<ProcessId, Arc<T>>,
}
impl<T: Send + Sync> PerProcess<T> {
/// Empty storage — no allocation. `const` so it can initialize a `static`
/// or a struct field directly (e.g. `NtObjectManager.handle_tables`).
pub const fn new() -> Self {
Self { slots: RcuHashMap::new() }
}
/// Return an `Arc` clone of the current process's slot, or `None` if the
/// process has never populated it. Lock-free hot path; the returned `Arc`
/// outlives `guard`.
pub fn get(&self, guard: &RcuReadGuard) -> Option<Arc<T>> {
self.slots.lookup(¤t_task().process.pid, guard)
}
/// Fetch-or-create the current process's slot, running `init` on first
/// access. Warm path: `RcuHashMap::get_or_insert_with` builds `Arc::new(init())`
/// outside the bucket lock, then inserts-if-absent under it; subsequent reads
/// are lock-free. `Err(KernelError::OutOfMemory)` if the slot cannot be
/// allocated.
pub fn get_or_insert_with(
&self,
init: impl FnOnce() -> T,
) -> Result<Arc<T>, KernelError> {
self.slots
.get_or_insert_with(current_task().process.pid, || Arc::new(init()))
}
}
3.1.3 RBTreeNode: Intrusive Red-Black-Tree Link¶
RBTreeNode is the ordered-tree counterpart to IntrusiveListNode: an
intrusive red-black-tree link embedded directly in a container struct (Linux
struct rb_node), so the tree is formed by these embedded links rather than by
separately heap-allocated tree nodes. It is the primitive for hot-path ordered
trees that must avoid a per-insert allocation by embedding the link in an object
the caller already allocates. It is a distinct tool from RBTree<K, V> in
Section 7.1, the non-intrusive ordered map that owns its entries by value:
that map is what the epoll interest tree uses
(Mutex<RBTree<EpollKey, EpollItem>> Section 19.1, a cold-path
structure — epoll_ctl is not a hot path), NOT this intrusive link.
/// Intrusive red-black-tree link, embedded in a node struct (Linux
/// `struct rb_node`). Zero-allocation: the link lives inside the container.
///
/// The parent pointer and node color are packed into one word
/// (`rb_parent_color`) exactly as Linux does — RB nodes are word-aligned, so
/// the low bit is free to carry the red/black color. All fields are mutated
/// only under the owning tree's lock; the raw pointers carry no lifetime and
/// are never dereferenced without that lock held (`Cell` provides the interior
/// mutability, not thread-safe sharing). Kernel-internal — not a KABI or wire
/// type.
pub struct RBTreeNode {
/// Parent pointer with the color bit in bit 0 (0 = red, 1 = black); the
/// remaining bits are the word-aligned parent address. Null-parent
/// (`0`) marks the tree root. Accessed via the tree helpers, never directly.
rb_parent_color: Cell<usize>,
/// Right child link, or `None`.
rb_right: Cell<Option<NonNull<RBTreeNode>>>,
/// Left child link, or `None`.
rb_left: Cell<Option<NonNull<RBTreeNode>>>,
}
3.1.4 PerCpuCounter: Batched Per-CPU Counter for Warm Paths¶
For warm-path counters where approximate reads are acceptable (dirty page counts,
free block counts, superblock writer counts), PerCpuCounter<T> provides batched
per-CPU accumulation with an approximate global view. Each CPU maintains a local
counter that periodically folds into a global sum when a batch threshold is exceeded.
This is NOT for hot-path counters like RSS — use PerCpu<AtomicI64> for those
(zero batch overhead, one fetch_add per update). See design decision AI-036
Option C in Section 4.8 (MmStruct.rss documentation) for
the full rationale distinguishing hot-path vs warm-path per-CPU counters.
| Counter type | Abstraction | Update cost | Read cost | Use case |
|---|---|---|---|---|
| Hot path | PerCpu<AtomicI64> |
~1-3 cycles (one fetch_add) |
O(num_cpus) sum | RSS (per page fault) |
| Warm path | PerCpuCounter<i64> |
~3-8 cycles (local add + threshold check) | ~1 cycle (read global) | dirty pages, free blocks, SbWriters |
Linux equivalent: struct percpu_counter (include/linux/percpu_counter.h,
lib/percpu_counter.c). Linux uses s32 __percpu *counters + s64 count + raw_spinlock_t.
UmkaOS uses PerCpu<AtomicI64> for the local slots (avoiding the need for IRQ
disable on the increment path) and a SpinLock for the global fold.
/// Batched per-CPU counter for warm-path statistics.
///
/// Each CPU accumulates increments/decrements in a local `AtomicI64` slot.
/// When the local value's absolute magnitude reaches `batch`, the local
/// value is folded into the global `count` under a spinlock and the local
/// slot is reset to zero. This amortises lock acquisition over `batch`
/// updates.
///
/// **Approximate reads** via `read_approximate()` return `count` without
/// summing per-CPU slots — O(1) but may drift by up to ±(batch * num_cpus).
/// **Exact reads** via `read_slow()` acquire the lock and sum all CPU slots
/// — O(num_cpus) but precise.
///
/// # When to use
///
/// Use `PerCpuCounter` for counters updated at moderate frequency (mount,
/// write, dirty-page tracking) where:
/// - Updates must be scalable (no cross-CPU contention on the common path).
/// - Reads are infrequent relative to updates (freeze/thaw, balance_dirty_pages).
/// - Approximate reads are acceptable for most consumers.
///
/// Do NOT use for:
/// - **Hot-path counters** (RSS, packet counts): use `PerCpu<AtomicI64>` —
/// zero batch overhead, one atomic op per update.
/// - **Counters read on every update**: the batch threshold check adds ~3-8
/// cycles that a raw atomic avoids.
///
/// # Batch threshold
///
/// Default: `max(32, num_online_cpus() * 2)` — matches Linux's
/// `compute_batch_value()`. Scaled dynamically on CPU hotplug. Callers
/// may override via `new_with_batch()` for counters with known access
/// patterns (e.g., superblock writer counts use a smaller batch because
/// freeze must drain quickly).
///
/// # Drift analysis
///
/// Maximum drift from true value when reading `count` without summing:
/// ±(`batch` * `num_possible_cpus()`). With default batch=64 on a 256-CPU
/// system: ±16384. For dirty page tracking (typical system has millions of
/// dirty pages), this drift is negligible. For counters where drift matters
/// (e.g., "is this counter exactly zero?"), use `read_slow()`.
///
/// # CPU hotplug
///
/// On CPU offline, the dying CPU's local slot is folded into `count` under
/// the lock (registered via the CPU hotplug callback in
/// [Section 3.2](#cpulocal-register-based-per-cpu-fast-path)). On CPU online, the
/// new CPU's slot starts at zero. The `batch` value is recomputed via
/// `max(32, num_online_cpus() * 2)`.
///
/// # 32-bit architecture note (no native 64-bit atomics)
///
/// `AtomicI64` requires native 64-bit atomic support. On 32-bit legs without
/// it (`not(target_has_atomic = "64")` — PPC32; ARMv7-A has LDREXD/STREXD),
/// the implementation compile-time selects a fallback, same policy as FIX-030
/// ([Section 22.2](22-accelerators.md#accelerator-scheduler)):
///
/// - `count` becomes a plain `i64` in an `UnsafeCell`, written under `lock`
/// (which already serializes every fold). `read_approximate()` also takes
/// `lock` on these legs — the lockless ~1-cycle read is a 64-bit-atomics
/// luxury; PerCpuCounter is a warm-path structure and PPC32 systems have
/// few cores, so the uncontended lock cost is acceptable.
/// - `local` slots become `AtomicI32`. Safe: a slot's magnitude between folds
/// is bounded by `batch + |amount|` (every `add` folds at the threshold),
/// and `batch = max(32, num_online_cpus() * 2)` — orders of magnitude below
/// `i32::MAX`. Same-CPU interrupt-handler races remain benign because the
/// slot is still a single atomic word. Precondition on these legs: a single
/// `add(amount)` must satisfy `|amount| <= i32::MAX` — all existing users
/// (dirty pages, free blocks, `SbWriters`) pass per-object deltas.
///
/// Debug-only lock contention statistics use a different 32-bit shape
/// (torn-tolerant `DebugStatU64`,
/// [Section 3.5](#locking-strategy--lock-contention-tracking)) because they update on
/// the contended lock path where a nested spinlock is unacceptable;
/// `PerCpuCounter` values have correctness consumers (freeze/thaw zero
/// checks) and therefore use the lock-protected fallback, never the
/// torn-tolerant cell.
pub struct PerCpuCounter {
/// Global approximate count. Updated when a per-CPU slot overflows
/// the batch threshold. Protected by `lock` for writes; may be read
/// without the lock for approximate reads (Relaxed load).
count: AtomicI64,
/// Per-CPU local accumulators. Each slot is independently updated
/// with `fetch_add` (no IRQ disable needed — `AtomicI64` is safe
/// against interrupt-handler races on the same CPU because the
/// worst case is a slightly delayed fold, not data corruption).
local: PerCpu<AtomicI64>,
/// Lock protecting `count` during fold operations and `read_slow()`.
/// Also serialises concurrent folds from different CPUs (rare — only
/// happens when two CPUs hit the batch threshold simultaneously).
lock: SpinLock<()>,
/// Current batch threshold. Recomputed on CPU hotplug.
/// Read with `Relaxed` on the increment path (stale value is benign —
/// a slightly larger or smaller batch just shifts the fold timing).
batch: AtomicI32,
}
impl PerCpuCounter {
/// Create a new counter initialised to `initial_value`, with the
/// default batch threshold `max(32, num_online_cpus() * 2)`.
///
/// # Safety
/// Must be called after the per-CPU allocator is available (boot phase 8+).
pub unsafe fn new(initial_value: i64) -> Self {
let nr_cpus = num_online_cpus() as i32;
Self {
count: AtomicI64::new(initial_value),
local: PerCpu::new(num_possible_cpus(), || AtomicI64::new(0)),
lock: SpinLock::new(()),
batch: AtomicI32::new(core::cmp::max(32, nr_cpus * 2)),
}
}
/// Create with a caller-specified batch threshold.
pub unsafe fn new_with_batch(initial_value: i64, batch: i32) -> Self {
Self {
count: AtomicI64::new(initial_value),
local: PerCpu::new(num_possible_cpus(), || AtomicI64::new(0)),
lock: SpinLock::new(()),
batch: AtomicI32::new(batch),
}
}
/// Increment by 1. Common-case wrapper for `add(1)`.
#[inline(always)]
pub fn inc(&self) {
self.add(1);
}
/// Decrement by 1. Common-case wrapper for `add(-1)`.
#[inline(always)]
pub fn dec(&self) {
self.add(-1);
}
/// Add `amount` to the counter (may be negative).
///
/// **Fast path** (~3-8 cycles): preempt_disable, fetch_add on the
/// local CPU's slot, check if |local| >= batch. If not, return.
///
/// **Slow path** (batch overflow): acquire `lock`, fold local slot
/// into `count`, reset local to zero, release lock.
///
/// On architectures with `cmpxchg` (x86-64, AArch64 via CASAL),
/// the local read-add-check can use a single `fetch_add` followed
/// by a branch on the result. On architectures without (PPC32),
/// the local slot is read and written under preempt_disable.
pub fn add(&self, amount: i64) {
let guard = preempt_disable();
let local_ref = self.local.get(&guard);
let new_local = local_ref.fetch_add(amount, Ordering::Relaxed) + amount;
let batch = self.batch.load(Ordering::Relaxed) as i64;
if new_local.abs() >= batch {
// Fold into global count.
let _lock = self.lock.lock();
// Re-read local: another fold may have raced (rare, requires
// interrupt handler also hitting the batch on this CPU).
let current = local_ref.swap(0, Ordering::Relaxed);
self.count.fetch_add(current, Ordering::Relaxed);
}
// guard dropped — preemption re-enabled.
}
/// Read the approximate global value. O(1), no lock, no CPU iteration.
/// May drift from the true value by up to ±(batch * num_cpus).
///
/// Suitable for: dirty page ratio estimation, free-block heuristics,
/// cgroup memory.stat reporting where exactness is not required.
#[inline]
pub fn read_approximate(&self) -> i64 {
self.count.load(Ordering::Relaxed)
}
/// Read the exact global value. O(num_cpus), acquires lock.
/// Folds all per-CPU slots into `count` and returns the result.
///
/// Suitable for: freeze/thaw drain checks ("is writer count exactly
/// zero?"), OOM scoring, `/proc` reporting where precision matters.
pub fn read_slow(&self) -> i64 {
let _lock = self.lock.lock();
let mut total = self.count.load(Ordering::Relaxed);
for cpu in 0..num_possible_cpus() {
// SAFETY: iterating all possible CPUs — the sound denominator for
// per-CPU counter sums (a CPU may offline with a nonzero residual,
// so ONLINE-only iteration would drop it). Non-online contribution:
// - a never-onlined (never-populated) CPU contributes ZERO: its
// slot was zero-initialized at construction and never written.
// Under the spine + populate-once model, per-CPU BULK stays
// unpopulated for such a CPU
// ([Section 3.2](#cpulocal-register-based-per-cpu-fast-path--initialization-sequence)),
// but this counter's local slots are a small dense scalar array
// (they stay dense), so the slot is present and reads zero.
// - an offlined CPU: its slot was folded during hotplug-down, so
// it reads zero.
// Online CPUs may be concurrently incrementing — the race is
// benign (we read a slightly stale value for that CPU).
let slot = unsafe { self.local.get_cpu(cpu) };
total += slot.load(Ordering::Relaxed);
}
total
}
/// Read the exact value, clamped to non-negative. Equivalent to
/// `max(0, read_slow())`. Used by filesystem free-block reporting
/// where a transiently-negative value (due to concurrent decrements)
/// must not be exposed to userspace.
pub fn read_positive_slow(&self) -> i64 {
core::cmp::max(0, self.read_slow())
}
/// Called from CPU hotplug callback when a CPU goes offline.
/// Folds the dying CPU's local slot into the global count.
pub fn cpu_dead(&self, cpu: usize) {
let _lock = self.lock.lock();
let slot = unsafe { self.local.get_cpu(cpu) };
let val = slot.swap(0, Ordering::Relaxed);
self.count.fetch_add(val, Ordering::Relaxed);
// Recompute batch for new CPU count.
let nr = num_online_cpus() as i32;
self.batch.store(core::cmp::max(32, nr * 2), Ordering::Relaxed);
}
}
get_cpu()note:PerCpu::get_cpu(cpu)is a raw slot accessor used only byread_slow()andcpu_dead()— it bypasses thePreemptGuardrequirement because the caller is either iterating all CPUs under a lock or handling a hotplug event for a specific CPU. This is the only escape hatch from the proof-token discipline, and it isunsafeto reflect that the caller must ensure the access is sound (no concurrent mutable borrow on that slot).
impl<T> PerCpu<T> {
/// Raw slot accessor for a specific CPU. Bypasses the `PreemptGuard`
/// requirement — the caller is responsible for ensuring that no concurrent
/// mutable borrow exists on the returned slot.
///
/// # Intended callers
/// - `PerCpuCounter::read_slow()` (iterating all CPUs under a SpinLock).
/// - `PerCpuCounter::cpu_dead()` (hotplug callback for a specific CPU).
/// - `mm_sum_rss()` in the OOM scorer ([Section 4.5](04-memory.md#oom-killer)): iterating all
/// possible CPUs to sum RSS counters (read-only, `Relaxed` loads).
///
/// # Safety
/// - `cpu` must be < `self.count` (num_possible_cpus()).
/// - The caller must ensure no concurrent `get_mut()` borrow exists on
/// the same `cpu` slot. Typically satisfied by: (a) holding a lock
/// that serializes all writers, (b) the CPU being offline (no code
/// running on it), or (c) the access being read-only on an Atomic type
/// (no `&mut T` aliasing concern).
pub unsafe fn get_cpu(&self, cpu: usize) -> &T {
debug_assert!(cpu < self.count, "PerCpu::get_cpu: cpu {} >= count {}", cpu, self.count);
&*(*self.data.add(cpu)).get()
}
}
/// Per-CPU locks: type-safe sharded locking without array-indexing aliasing hazards.
///
/// A common pattern in scalable kernels is to give each CPU its own lock protecting
/// a shard of data, avoiding contention on a single global lock. Naively implementing
/// this as `[SpinLock<T>; N]` creates a type-safety hole: Rust's aliasing rules permit
/// holding `&mut T` from `locks[i]` and `&mut T` from `locks[j]` simultaneously if
/// `i != j`, but nothing in the type system prevents accessing `locks[other_cpu]`
/// when the caller intended to access only `locks[this_cpu]`. Furthermore, if two
/// CPUs simultaneously hold their respective locks, the `&mut T` references are
/// derived from the same array allocation, which can violate LLVM's noalias assumptions
/// in ways that are difficult to reason about.
///
/// UmkaOS provides `PerCpuLock<T>` to enforce the per-CPU locking invariant at the
/// type level:
///
/// 1. **Separate allocations per CPU**: Each CPU's lock+data is a separate slab
/// allocation (one `PerCpuLockSlot<T>` per CPU), NOT an array element. This ensures
/// that `&mut T` references from different CPUs are derived from disjoint
/// allocations, satisfying Rust's aliasing rules without relying on the
/// "different array indices" reasoning that LLVM may not honor.
///
/// 2. **Access restricted to current CPU**: `PerCpuLock::lock()` takes a
/// `&PreemptGuard` (the same proof token used by `PerCpu<T>`) and only returns
/// a guard for the current CPU's lock. There is no API to access another CPU's
/// lock — the type system makes it impossible.
///
/// 3. **Cache-line aligned**: Each `PerCpuLockSlot<T>` is `#[repr(align(64))]` to
/// prevent false sharing. The alignment is part of the type, not a runtime hint.
///
/// 4. **Safe composition with `PerCpu<T>`**: For cases where per-CPU data needs
/// both a lock and lock-free access, use `PerCpu<UnsafeCell<T>>` with explicit
/// `PerCpuMutGuard` for writes, or wrap the data in `PerCpu<SpinLock<T>>` where
/// the lock itself is per-CPU. The key invariant is that `PerCpuLock<T>` never
/// exposes `&mut T` from one CPU while another CPU's lock is held — each CPU's
/// lock guards only that CPU's data shard.
///
/// **Use case**: Per-CPU statistics counters that need occasional atomic updates
/// across all CPUs (e.g., network Rx packet counts). Each CPU locks its own slot
/// for local updates; aggregating across all CPUs requires iterating without holding
/// any locks (the counters are `AtomicU64`, so reads are consistent).
pub struct PerCpuLock<T> {
/// Array of pointers to independently-allocated lock slots.
/// Each pointer points to a separately-allocated `PerCpuLockSlot<T>`.
/// Length = num_possible_cpus(), discovered at boot.
/// The indirection ensures each slot is a separate allocation for aliasing purposes.
///
/// **Allocation timing**: `PerCpuLock<T>` is initialized during phase 2 of boot
/// (after the slab allocator is available, Section 4.3). Each slot is allocated
/// via the slab allocator (not `Box`/global allocator). The pointer array itself
/// is allocated from the boot bump allocator during early init.
slots: *mut *mut PerCpuLockSlot<T>,
/// Number of CPU slots (set once at boot, never changes).
count: usize,
}
/// A single CPU's lock slot, cache-line aligned to prevent false sharing.
/// This is allocated independently (via boot allocator) for each CPU at boot.
///
/// `#[repr(align(64))]` guarantees:
/// - The struct's starting address is 64-byte aligned.
/// - The compiler pads the struct to a multiple of 64 bytes automatically.
/// - `size_of::<PerCpuLockSlot<T>>()` >= 64 regardless of `SpinLock<T>` size.
///
/// No explicit `_pad` field is needed — the compiler handles padding.
/// This works correctly for any `SpinLock<T>` size: small types get padded
/// up to 64 bytes, large types round up to the next 64-byte boundary.
#[repr(align(64))]
struct PerCpuLockSlot<T> {
/// The spinlock protecting this CPU's data shard.
lock: SpinLock<T>,
}
impl<T> PerCpuLock<T> {
/// Lock the current CPU's data shard.
///
/// # Safety contract
///
/// - Takes `&PreemptGuard` to prove the caller is pinned to a specific CPU.
/// - Returns `PerCpuLockGuard<'_, T>` that derefs to `&T` and `&mut T`.
/// - The guard holds the spinlock for this CPU's slot.
/// - No API exists to lock another CPU's shard — the only access path is
/// through the current CPU, enforced by the `PreemptGuard` proof token.
///
/// **Aliasing safety**: Because each slot is a separate heap allocation,
/// holding `&mut T` on CPU 0 and `&mut T` on CPU 1 simultaneously is sound —
/// the references are derived from disjoint allocations, not from different
/// indices of the same array. This satisfies Rust's aliasing rules and
/// LLVM's noalias semantics without subtle reasoning about array indexing.
///
/// **Interrupt safety**: `SpinLock::lock()` disables preemption for the
/// critical section. If the caller needs to also disable interrupts
/// (to prevent interrupt handlers from deadlocking on the same lock),
/// they must wrap the call in `local_irq_save()`/`local_irq_restore()`.
/// For per-CPU locks, this is rarely needed because interrupt handlers
/// typically access different data or use lock-free patterns.
pub fn lock<'g>(&self, guard: &'g PreemptGuard) -> PerCpuLockGuard<'g, T> {
let cpu = guard.cpu_id();
assert!(cpu < self.count, "PerCpuLock: cpu_id {} out of range", cpu);
// SAFETY: slots was allocated with count elements at boot. cpu is in bounds.
let slot_ptr = unsafe { *self.slots.add(cpu) };
// SAFETY: slot_ptr was obtained from slab allocation at boot and is never
// freed during kernel operation. It points to a valid PerCpuLockSlot<T>.
let slot = unsafe { &*slot_ptr };
PerCpuLockGuard {
inner: slot.lock.lock(),
_cpu_pin: PhantomData,
}
}
/// Try to lock the current CPU's data shard without blocking.
///
/// Returns `Some(PerCpuLockGuard)` if the lock was acquired, `None` if
/// the lock is currently held (e.g., by an interrupt handler on this CPU).
///
/// Same safety contract as `lock()`, but non-blocking.
pub fn try_lock<'g>(&self, guard: &'g PreemptGuard) -> Option<PerCpuLockGuard<'g, T>> {
let cpu = guard.cpu_id();
assert!(cpu < self.count, "PerCpuLock: cpu_id {} out of range", cpu);
let slot_ptr = unsafe { *self.slots.add(cpu) };
let slot = unsafe { &*slot_ptr };
slot.lock.try_lock().map(|inner| PerCpuLockGuard {
inner,
_cpu_pin: PhantomData,
})
}
/// Access all slots for cross-CPU aggregation (read-only, no locks held).
///
/// This is the ONLY way to access another CPU's slot, and it only provides
/// read-only access to the lock structure itself — NOT to the protected data.
/// The typical use case is iterating over all CPUs' atomic counters.
///
/// # Safety
///
/// The caller must ensure no CPU is currently mutating its data shard through
/// a `PerCpuLockGuard`. For atomic counter aggregation, this is safe because
/// the counters are read atomically without holding the lock. For non-atomic
/// data, the caller must use external synchronization (e.g., pause all CPUs
/// via IPI) before calling this method.
///
/// Returns an iterator over `&SpinLock<T>` for each CPU. The caller can
/// read the lock state or use `try_lock()` on each, but cannot obtain `&mut T`
/// through this path without holding the lock.
pub unsafe fn iter_slots(&self) -> impl Iterator<Item = &'_ SpinLock<T>> {
(0..self.count).map(move |i| {
let slot_ptr = *self.slots.add(i);
&(*slot_ptr).lock
})
}
}
/// Guard for a per-CPU lock. Implements `Deref`/`DerefMut` to access the protected data.
///
/// The guard holds a `SpinLockGuard` derived from the current CPU's lock slot.
/// The `PhantomData<&'a PreemptGuard>` ties the guard's lifetime to the CPU pin,
/// preventing use-after-migrate if the caller drops the `PreemptGuard` while
/// still holding the lock guard.
pub struct PerCpuLockGuard<'a, T> {
inner: SpinLockGuard<'a, T>,
_cpu_pin: PhantomData<&'a PreemptGuard>,
}
impl<'a, T> core::ops::Deref for PerCpuLockGuard<'a, T> {
type Target = T;
fn deref(&self) -> &T { &*self.inner }
}
impl<'a, T> core::ops::DerefMut for PerCpuLockGuard<'a, T> {
fn deref_mut(&mut self) -> &mut T { &mut *self.inner }
}
/// `PerCpuLockGuard` must NOT be sent to another CPU/thread.
/// The guard holds a `SpinLockGuard` from a specific CPU's lock slot.
/// Sending it to another thread would allow that thread to access a lock
/// that may be concurrently acquired by the original CPU (e.g., in an
/// interrupt handler), causing deadlock or data corruption.
impl<T> !Send for PerCpuLockGuard<'_, T> {}
Why separate allocations matter for type soundness:
The naive [SpinLock<T>; N] approach has a subtle aliasing problem. When CPU 0
holds locks[0].lock() and CPU 1 holds locks[1].lock(), both CPUs have
derived their &mut T references from the same array allocation. While Rust's
reference rules permit disjoint array element access, LLVM's noalias attribute
and the optimizer's alias analysis may not distinguish between "different indices
of the same array" and "same allocation." In practice, this is unlikely to cause
miscompilation with current LLVM, but the UmkaOS architecture takes a conservative
approach: each CPU's lock+data is a separate heap allocation, guaranteeing that
the &mut T references are truly disjoint at the allocation level.
This design also simplifies reasoning about memory reclamation: if a per-CPU lock slot needs to be freed (e.g., during CPU hot-unplug), the individual slab allocation can be returned without affecting other CPUs' slots.
3.1.5 ArcSwap — Lock-Free Atomic Arc<T> Replacement¶
ArcSwap<T> provides lock-free read access and atomic swap of Arc<T> values.
It is the kernel-internal equivalent of the userspace arc-swap crate, designed
for cases where a shared resource is read frequently on hot paths and replaced
rarely on cold paths (credential updates, cgroup migration, mm replacement during
exec, namespace changes via setns/unshare).
Key properties:
- Read path (load()): Returns an ArcSwapGuard<T> that derefs to &T and
extends the lifetime of the inner Arc<T> for the guard's duration. No locks,
no atomic increment on the Arc refcount, and — deliberately — no
StoreLoad fence: the classic hazard-pointer publish/verify fence is
replaced by a writer-side IPI rendezvous (asymmetric fencing, below).
Cost: two AtomicPtr loads + one per-CPU slot store (~5-10 cycles on
x86-64; all three accesses hit hot cache lines). On the scheduler hot path
this is significantly cheaper than Arc::clone() (a fetch_add on the
shared refcount — ~15-25 cycles on contended cache lines, and a
contention point by construction). The guard pins the CPU (it embeds a
PreemptGuard) — its hazard slot is per-CPU state.
- Write path (store(), swap()): Atomically replaces the inner Arc<T>,
then BLOCKS until no ArcSwapGuard still protects the old value: one IPI
rendezvous across all CPUs plus a bounded hazard-slot scan. The returned
(or dropped) old Arc<T> is then immediately reclaimable. Hazard pointers
(not RCU) because ArcSwap swap sites are not always within RCU read-side
critical sections and the old value must not linger for a grace period.
Cost: one AtomicPtr::swap(AcqRel) + IPI broadcast (~microseconds) —
acceptable because every swap site is a warm/cold path (exec, setns,
install_credentials, cgroup migration). Task context only.
- Interior mutability: store() and swap() take &self, not &mut self.
This is the primary reason ArcSwap exists — it allows atomic replacement of
Arc<T> through shared references (e.g., Task.namespace_set, Task.cgroup,
Process.mm). The caller provides synchronization for write-side serialization
(typically an external lock or single-writer guarantee).
/// Atomic `Arc<T>` container — lock-free reads, atomic swap.
///
/// Generic replacement for patterns where `Option<Arc<T>>` or `Arc<T>` must
/// be mutated through a shared reference. All methods take `&self`.
///
/// # Usage in UmkaOS
///
/// | Field | Read path | Write path | Serialization |
/// |---|---|---|---|
/// | `Task.namespace_set` | syscall dispatch (every syscall) | `setns(2)`, `unshare(2)` | per-task namespace-set serialization |
/// | `Task.cgroup` | scheduler tick, resource charge | cgroup migration | per-`Process` `threadgroup_rwsem` |
/// | `Process.mm` | page fault, /proc reads | `exec()` | single-thread at PNR |
/// | `Process.cred` | capability checks | `install_credentials()` | RCU CoW — no lock; `install_credentials()` atomic publish (see [Section 9.9](09-security.md#credential-model-and-capabilities)) |
/// | `Task.files` | fd operations | `close_on_exec` in exec | FdTable internal lock |
/// | `Task.fs` | path resolution | `chdir()`, `chroot()`, `unshare(CLONE_FS)` | FsStruct internal RwLock |
///
/// # Memory ordering
///
/// - `load()`: `Acquire` on the internal `AtomicPtr` (both the initial load
/// and the verify re-load). Pairs with the `Release` in `store()`/`swap()`.
/// Ensures all writes to the `T` inside the `Arc` are visible to the
/// reader after loading the pointer. The hazard-slot publication is a
/// `Release` store with **no reader-side StoreLoad fence** — see the
/// asymmetric-fence argument in the protocol section below.
/// - `store()`/`swap()`: `AcqRel` on the internal `AtomicPtr`. The `Release`
/// side ensures writes to the new `Arc<T>`'s contents are visible to future
/// `load()` callers. The `Acquire` side ensures the old value is fully read
/// before the pointer is replaced.
///
/// # Reclamation (normative protocol below)
///
/// The old `Arc<T>` returned by `swap()` (or implicitly dropped by `store()`)
/// must not be freed while any `ArcSwapGuard` from a prior `load()` still
/// references it. Tracking uses the per-CPU hazard-slot array
/// `CpuLocalBlock.arc_hazard` — `ARC_SWAP_HAZARD_SLOTS` (= 8) slots per CPU
/// ([Section 3.2](#cpulocal-register-based-per-cpu-fast-path)), one GLOBAL type-erased
/// array shared by every `ArcSwap<T>` instance, NOT one slot per CPU: a
/// single slot could not express nested protection (a `load()` on ArcSwap A
/// while a guard on ArcSwap B is live on the same CPU, or a hardirq-context
/// `load()` over an interrupted task-context guard).
///
/// The reclamation model is **blocking-in-swap**: `swap()` publishes the new
/// pointer, executes one all-CPU IPI rendezvous (the asymmetric fence), then
/// spins until no slot equals the old pointer. The spin is bounded by the
/// longest guard scope — guards disable preemption, forbid blocking, and are
/// short-lived by contract — so the writer never waits on a scheduling
/// latency (a preempted guard-holder is impossible by construction). A
/// deferred-free model (per-CPU retirement lists + an asynchronous reaper)
/// was rejected: it removes the IPI from the swap path but adds unbounded
/// deferred memory, a reaper to schedule and account, and a second
/// reclamation regime to verify — for writers that are all warm/cold paths.
///
/// # Comparison with alternatives
///
/// | Alternative | Problem |
/// |---|---|
/// | `RwLock<Arc<T>>` | Read lock on hot path (~20-40 cycles, contention under load) |
/// | `Mutex<Arc<T>>` | Exclusive lock on reads (unacceptable for concurrent readers) |
/// | `RcuCell<Arc<T>>` | Requires RCU read-side critical section; ArcSwap is usable outside RCU context |
/// | `AtomicPtr<T>` | No lifetime management; caller must manually manage Arc refcounts |
/// | Bare `Arc<T>` + `Arc::clone()` | `fetch_add` on refcount per read (~15-25 cycles contended) |
pub struct ArcSwap<T> {
/// Internal pointer to the `ArcInner<T>` (same layout as `Arc<T>`'s internal
/// pointer). Loaded with `Acquire`, stored with `Release`.
ptr: AtomicPtr<T>,
}
impl<T> ArcSwap<T> {
/// Create a new `ArcSwap` holding the given `Arc<T>`.
///
/// Consumes the `Arc` (incrementing its strong count is not needed — the
/// `ArcSwap` takes ownership of the reference count).
pub fn new(val: Arc<T>) -> Self {
let ptr = Arc::into_raw(val) as *mut T;
Self { ptr: AtomicPtr::new(ptr) }
}
/// Construct from a value directly (allocates a new `Arc`).
pub fn from_pointee(val: T) -> Self {
Self::new(Arc::new(val))
}
/// Load the current value. Returns an `ArcSwapGuard` that derefs to `&T`.
///
/// Lock-free and fence-free. The guard claims a hazard slot on the
/// current CPU (preemption disabled for the guard's lifetime — the slot
/// is per-CPU state) and publishes the loaded pointer into it; the slot
/// is cleared on guard drop.
///
/// # Contexts
/// Task, softirq, and hardirq context. **NMI-forbidden** (debug-asserted;
/// consistent with the PerCpu NMI rules — an NMI interleaving with a
/// same-slot claim would break the strict-nesting argument below). The
/// caller must not block while the guard is live (preemption is
/// disabled), and a guard created inside an interrupt handler MUST be
/// dropped before the handler returns.
///
/// # Performance
/// ~5-10 cycles on x86-64: two `Acquire` pointer loads + one slot store,
/// all on hot cache lines, no fence, no shared-cacheline RMW. See the
/// per-architecture cost note in the protocol section.
pub fn load(&self) -> ArcSwapGuard<'_, T> {
debug_assert!(!in_nmi(), "ArcSwap::load is NMI-forbidden");
let preempt = preempt_disable();
let hazard = hazard_claim(&preempt);
// Publish/verify loop. NO StoreLoad fence between the publish and
// the verify re-load — the writer-side IPI rendezvous substitutes
// for it (asymmetric fencing; correctness argument in the protocol
// section). If a swap raced, the verify fails and we republish the
// new pointer; retries are bounded by swap frequency (rare).
let ptr = loop {
let p = self.ptr.load(Ordering::Acquire);
hazard.publish(p as usize);
if self.ptr.load(Ordering::Acquire) == p {
break p;
}
};
// SAFETY: `ptr` originates from `Arc::into_raw` and the published
// hazard slot keeps its ArcInner alive until the guard drops; the
// Acquire loads make T's contents visible.
ArcSwapGuard {
value: unsafe { &*ptr },
_hazard: hazard,
_preempt: preempt,
}
}
/// Atomically replace the stored `Arc<T>` with `new_val`.
///
/// The old `Arc<T>` is dropped after the hazard quiesce+scan proves no
/// guard still references it (blocking — see `swap()`). Takes `&self` —
/// this is the interior mutability entry point.
///
/// # Write serialization
/// Multiple concurrent `store()` calls are serialized by the atomic swap
/// on `ptr`. However, the caller should provide external serialization
/// (a lock or single-writer guarantee) to ensure that the "old" vs "new"
/// value has a well-defined order. Without external serialization, two
/// concurrent stores may complete in either order.
pub fn store(&self, new_val: Arc<T>) {
let old = self.swap(new_val);
drop(old); // quiesced by swap(): decrements refcount, frees if last.
}
/// Atomically swap and return the old `Arc<T>`.
///
/// BLOCKING: the returned `Arc<T>` is safe to drop immediately because
/// `swap()` does not return until the hazard quiesce+scan
/// (`hazard_release_wait`, protocol section below) proves no
/// `ArcSwapGuard` still references the old value.
///
/// # Contexts
/// Task context ONLY, preemption and IRQs enabled, and NO spinlock held
/// (the quiesce is an all-CPU IPI rendezvous — the standard
/// cross-call deadlock rules apply). These restrictions are free in
/// practice: every swap site is a warm/cold path (exec, setns,
/// install_credentials, cgroup migration, writeback-context teardown).
pub fn swap(&self, new_val: Arc<T>) -> Arc<T> {
let new_ptr = Arc::into_raw(new_val) as *mut T;
let old_ptr = self.ptr.swap(new_ptr, Ordering::AcqRel);
// Quiesce + scan: on return, no hazard slot names old_ptr and none
// can newly name it (post-swap readers verify against the NEW ptr).
hazard_release_wait(old_ptr as usize);
// SAFETY: old_ptr was previously stored via Arc::into_raw, and the
// scan above proved every guard released it — the strong count this
// ArcSwap held returns to the caller.
unsafe { Arc::from_raw(old_ptr) }
}
/// Load the current value as an **owned** `Arc<T>` (one strong-count bump).
///
/// Unlike [`load`](Self::load), the returned `Arc` is not tied to a hazard
/// guard's lifetime, so it can outlive the current statement and be stored
/// or moved. Use this when a caller must hold an inner lock of `T` across a
/// critical section (e.g. `Process.sighand: ArcSwap<SignalHandlers>`, where
/// the `SignalHandlers` lock guard would otherwise dangle if the `ArcSwap`
/// were swapped by a concurrent `exec`). Costs one atomic increment vs
/// `load`'s zero, so prefer `load` for pure field reads.
///
/// # Performance
/// ~15-25 cycles (hazard-protected load + one `fetch_add` on the strong count).
pub fn load_full(&self) -> Arc<T> {
let guard = self.load();
let ptr = guard.value as *const T;
// SAFETY: the guard's hazard slot prevents reclamation between the
// load and the strong-count increment; after the increment the Arc
// is self-owning.
let arc = unsafe {
Arc::increment_strong_count(ptr);
Arc::from_raw(ptr)
};
drop(guard); // clears the hazard slot, re-enables preemption
arc
}
}
3.1.5.1 Hazard-Slot Protocol (normative)¶
/// Hazard slots per CPU (`CpuLocalBlock.arc_hazard`,
/// [Section 3.2](#cpulocal-register-based-per-cpu-fast-path)). Sized from a design
/// allocation of worst-case nesting on one CPU: task context ≤ 4
/// simultaneous guards (distinct ArcSwap fields read in one expression;
/// the ArcSwap fields in play are enumerated in the usage table above)
/// + softirq ≤ 2 + hardirq ≤ 2. NMI-context
/// loads are forbidden, so three contexts bound the sum. The bound is
/// ENFORCED by the overflow panic below, not trusted from an audit:
/// slot OVERFLOW PANICS in ALL builds (like a PerCpu borrow violation) —
/// it is a kernel bug — and a new nesting site must be reviewed and, if
/// legitimate, this constant grown (one cache line holds 8 slots; growth
/// is cheap).
pub const ARC_SWAP_HAZARD_SLOTS: usize = 8;
/// Claim the first free slot (value 0) on the current CPU. The
/// `&PreemptGuard` witnesses that the caller is pinned.
///
/// The find-then-claim sequence needs NO atomic RMW: the only competing
/// claimants are same-CPU interrupt contexts, and guard lifetimes on one
/// CPU are STRICTLY NESTED — an interrupt that claims a slot drops its
/// guard before the interrupted claimant resumes (normative guard contract:
/// a guard never escapes the context that created it — never stored, never
/// held across a handler return, never held across blocking). So a slot
/// observed free is either still free at the claiming store, or was
/// claimed AND released in between — both make the store correct.
fn hazard_claim(_preempt: &PreemptGuard) -> HazardSlot {
// SAFETY: preemption disabled — this is the executing CPU's block.
let slots = CpuLocal::arc_hazard();
for i in 0..ARC_SWAP_HAZARD_SLOTS {
if slots[i].load(Ordering::Relaxed) == 0 {
return HazardSlot { slot: i as u32 };
}
}
panic!("ArcSwap: hazard-slot overflow (> {} nested guards)", ARC_SWAP_HAZARD_SLOTS);
}
/// RAII handle to a claimed per-CPU hazard slot.
///
/// Held inside an [`ArcSwapGuard`]; the guard's embedded `PreemptGuard`
/// keeps the CPU pinned for this handle's whole lifetime (field order in
/// `ArcSwapGuard` drops the slot BEFORE re-enabling preemption). `!Send`:
/// the slot belongs to the CPU that claimed it.
struct HazardSlot {
/// Index of the owned slot within this CPU's hazard-pointer array.
slot: u32,
}
impl HazardSlot {
/// Publish `val` (a raw `ArcInner` pointer as usize) into the owned
/// slot. `Release` orders the publication before the verify re-load in
/// program order; cross-CPU visibility to the scanner is guaranteed by
/// the writer's IPI rendezvous, NOT by a reader-side fence.
fn publish(&self, val: usize) {
// SAFETY: the ArcSwapGuard's PreemptGuard is live — owning CPU.
let slots = CpuLocal::arc_hazard();
slots[self.slot as usize].store(val, Ordering::Release);
}
}
impl Drop for HazardSlot {
/// Clear the owned slot (store 0, `Release`), ending protection. The
/// Release pairs with the Acquire in the reclaimer's scan, so the scan
/// observes either the protected value (and keeps waiting) or the clear
/// ordered AFTER the guard's last read of `T`.
fn drop(&mut self) {
// SAFETY: the ArcSwapGuard's PreemptGuard is still live (declared
// after this field, dropped after it) — owning CPU.
let slots = CpuLocal::arc_hazard();
slots[self.slot as usize].store(0, Ordering::Release);
}
}
// Registered on the current CPU's hazard array; must not move to another CPU.
impl !Send for HazardSlot {}
/// Remote (any-context) view of CPU `cpu`'s hazard slots — a sanctioned
/// cross-CPU READ of `CpuLocalBlock` (precedent: the RCU GP kthread's
/// `is_idle` reads). All slots are atomics, so remote loads are tear-free.
/// Cross-CPU WRITES remain forbidden.
fn arc_hazard_of(cpu: u32) -> &'static [AtomicUsize; ARC_SWAP_HAZARD_SLOTS];
/// Writer-side quiesce + scan, called by `swap()`/`store()` AFTER the
/// pointer swap is globally visible. On return, no hazard slot names `old`
/// and none can newly come to name it durably (a post-swap reader's verify
/// fails and republishes the new pointer).
///
/// Task context only; no spinlocks held; preemption + IRQs enabled
/// (all-CPU cross-call rules).
fn hazard_release_wait(old: usize) {
// 1. Asymmetric fence: one IPI rendezvous across all online CPUs
// whose closure executes an EXPLICIT full fence. The correctness
// argument (below) rests on that fence plus the cross-call
// handshake ordering (the smp_call_function_all ordering contract)
// — NOT on any blanket "interrupt entry is a barrier" property,
// which the eight targets do not all provide (trap/exception
// entry is context-synchronizing on several of them, which is not
// a memory fence). This is Linux's membarrier structure: its IPI
// handler runs an explicit full fence rather than trusting the
// interrupt itself; Linux `kernel/sched/membarrier.c::ipi_mb()`
// makes the same conservative choice.
// The rendezvous substitutes for the per-load StoreLoad fence
// classic hazard pointers require.
smp_call_function_all(|| fence(Ordering::SeqCst));
// 2. Scan all POSSIBLE CPUs (offline CPUs have empty slots — guards
// pin preemption, so a CPU cannot go offline with a live guard).
// Spin while any slot equals `old`: bounded by the longest guard
// scope (~hundreds of cycles — guards are preemption-pinned and
// non-blocking, so this is a critical-section wait, never a
// scheduling-latency wait).
for cpu in 0..num_possible_cpus() {
let slots = arc_hazard_of(cpu as u32);
for i in 0..ARC_SWAP_HAZARD_SLOTS {
while slots[i].load(Ordering::Acquire) == old {
cpu_relax();
}
}
}
}
Why no reader-side fence is needed (asymmetric fencing). The classic
hazard-pointer bug is the reader's publish (store) passing its verify
(load) — requiring a SeqCst/StoreLoad fence per load(). UmkaOS moves that
cost to the rare writer: the load-bearing barrier is the explicit
fence(SeqCst) in the rendezvous closure (Linux membarrier's ipi_mb()
analog), combined with the cross-call handshake ordering
(Section 3.8). For any reader
CPU, the closure's fence executes at some instant t on that CPU, and two
orderings hold there: (1) the writer dispatched the cross-call AFTER the
pointer swap and the closure's execution acquires that dispatch, so the
swap is visible on the reader CPU no later than t, and the fence orders
every reader access after t behind it; (2) the closure's completion ack
is a release that the writer's rendezvous wait acquires, and the fence
orders every reader access before t ahead of it, so everything the
reader executed before t is visible to the writer's scan (which begins
only after every CPU has acked).
Case (a): the reader's verify re-load executed before t and saw the OLD
pointer — then its publish (program-order earlier, thus also before t)
is visible to the scan by (2), which waits for the guard to drop.
Case (b): the verify executes after t — by (1) it sees the NEW pointer,
fails, and the loop republishes the new value; any TRANSIENT publication
of the old pointer self-clears within the reader's next two instructions,
which the scan's spin absorbs. No guard for the old value can exist
unseen; no new one can form. The reader thus pays zero fences on all
eight architectures.
Per-architecture reader cost (two Acquire loads + one Release store, same cache lines):
| Architecture | Expansion | Approx. cost |
|---|---|---|
| x86-64, s390x | plain loads/stores (strong ordering) | ~5-10 cycles |
| AArch64 | LDAR ×2 + STLR | ~6-15 cycles |
| PPC64LE, PPC32 | lwsync-bracketed loads/store | ~10-25 cycles |
| ARMv7 | dmb-bracketed | ~15-30 cycles |
| RISC-V 64, LoongArch64 | fence r,rw / fence rw,w | ~10-25 cycles |
On the weakly-ordered architectures the barrier cost is comparable to what
Arc::clone() already pays for its refcount RMW — the win there is the
absence of a SHARED contended cache line, not barrier elimination. The
writer adds one IPI broadcast (~microseconds) per swap; every swap site in
the usage table is a warm/cold path.
/// Guard returned by `ArcSwap::load()`. Derefs to `&T`.
///
/// Holds a claimed hazard slot that prevents the referenced `Arc<T>` from
/// being reclaimed, and a `PreemptGuard` pinning the CPU that owns the
/// slot.
///
/// # Normative guard contract (load-bearing for slot-claim correctness)
/// - Short-lived: hold for a field access or computation, never across a
/// blocking point (preemption is disabled — blocking is illegal anyway).
/// - Never escapes its creating context: not stored in long-lived
/// structures, and a guard created in an interrupt handler is dropped
/// before the handler returns.
pub struct ArcSwapGuard<'a, T> {
/// Reference to the inner T, valid for the guard's lifetime.
value: &'a T,
/// Claimed hazard slot. Declared BEFORE `_preempt` so it drops first —
/// the slot is cleared while the CPU pin is still held.
_hazard: HazardSlot,
/// CPU pin for the slot's lifetime. Dropped last.
_preempt: PreemptGuard,
}
impl<T> core::ops::Deref for ArcSwapGuard<'_, T> {
type Target = T;
fn deref(&self) -> &T { self.value }
}
// ArcSwapGuard must not be sent across threads — the hazard pointer is
// registered on the current CPU's slot. Sending to another thread would
// leave the hazard pointer on the wrong CPU, failing to protect the
// referenced Arc from reclamation during a swap on the original CPU.
impl<T> !Send for ArcSwapGuard<'_, T> {}
/// Nullable variant of [`ArcSwap`]: atomically-swappable `Option<Arc<T>>`.
///
/// `None` is represented by a null internal pointer (niche), so `is_none()` is a
/// single atomic load with no allocation. Used for fields that are absent until
/// first set, e.g. `MmStruct.exe_file` (no backing file for kernel-thread mms or
/// before the first `exec` completes).
pub struct ArcSwapOption<T> {
/// Null = `None`; non-null = `Some(Arc<T>)` (raw `ArcInner<T>` pointer).
ptr: AtomicPtr<T>,
}
impl<T> ArcSwapOption<T> {
/// Empty (`None`).
pub const fn empty() -> Self {
Self { ptr: AtomicPtr::new(core::ptr::null_mut()) }
}
/// Construct from an initial `Option<Arc<T>>`.
pub fn new(val: Option<Arc<T>>) -> Self {
let ptr = match val {
Some(a) => Arc::into_raw(a) as *mut T,
None => core::ptr::null_mut(),
};
Self { ptr: AtomicPtr::new(ptr) }
}
/// Load an owned `Option<Arc<T>>` (one strong-count bump when `Some`).
/// Semantics, contexts, and cost mirror [`ArcSwap::load_full`]; a `None`
/// result publishes nothing (a null pointer needs no protection — it is
/// never dereferenced and never reclaimed).
pub fn load_full(&self) -> Option<Arc<T>> {
debug_assert!(!in_nmi(), "ArcSwapOption::load_full is NMI-forbidden");
let preempt = preempt_disable();
let hazard = hazard_claim(&preempt);
loop {
let p = self.ptr.load(Ordering::Acquire);
if p.is_null() {
return None; // hazard + preempt guards drop here (slot untouched or cleared)
}
hazard.publish(p as usize);
if self.ptr.load(Ordering::Acquire) == p {
// SAFETY: the published slot keeps the ArcInner alive across
// the strong-count bump; afterwards the Arc is self-owning.
let arc = unsafe {
Arc::increment_strong_count(p as *const T);
Arc::from_raw(p as *const T)
};
return Some(arc); // guards drop here, clearing the slot
}
// Raced with a store(): republish the current pointer.
}
}
/// Atomically replace with `new_val` (`None` stores null).
///
/// BLOCKING when the OLD value was `Some`: same quiesce+scan and same
/// context rules as [`ArcSwap::swap`] (task context only, no spinlocks
/// held). A `None`→anything transition returns immediately.
pub fn store(&self, new_val: Option<Arc<T>>) {
let new_ptr = match new_val {
Some(a) => Arc::into_raw(a) as *mut T,
None => core::ptr::null_mut(),
};
let old_ptr = self.ptr.swap(new_ptr, Ordering::AcqRel);
if !old_ptr.is_null() {
hazard_release_wait(old_ptr as usize);
// SAFETY: old_ptr came from Arc::into_raw and no hazard slot
// references it anymore — reclaim the strong count.
drop(unsafe { Arc::from_raw(old_ptr) });
}
}
/// True if currently `None` (single atomic load, no allocation).
pub fn is_none(&self) -> bool {
self.ptr.load(Ordering::Acquire).is_null()
}
}
ArcSwap<T> vs RcuCell<Arc<T>>: Both provide lock-free reads with deferred
reclamation. The key difference is the reclamation scope:
RcuCelldefers reclamation to the next RCU grace period. Requires the reader to be in an RCU read-side critical section (implicit in preempt-disabled kernel code). Best for data that is always read on fast kernel paths where RCU context is guaranteed (credential checks, routing table lookups, dentry cache).ArcSwapreclaims via hazard slots (per-CPU, cleared on guard drop): the writer blocks for one IPI rendezvous plus a bounded scan and the old value is freed IMMEDIATELY after — no grace-period lingering and no load on the RCU callback machinery. Does NOT require an RCU read-side context (the guard pins preemption for slot ownership, but readers need no RCU bookkeeping). Best for data that may be read outside RCU context (e.g., from userspace-triggered paths likesetns()that swap namespace_set, or from/procreads that are not in RCU context).
Both are correct; the choice depends on whether the read site is guaranteed to be
in RCU context. Task.cred uses RcuCell (reads are always in syscall fast path
= RCU context). Task.namespace_set and Task.cgroup use ArcSwap (reads may occur
from cgroup migration paths that are not in RCU context).
3.2 CpuLocal: Register-Based Per-CPU Fast Path¶
PerCpu<T> is the correct abstraction for general per-CPU data, but its
indirection (CPU ID lookup + array indexing + borrow-state checking) is
too expensive for the kernel's hottest paths — scheduler pick_next_task,
slab magazine alloc/free, NAPI poll, RCU quiescent-state reporting. These
paths execute millions of times per second and every cycle matters.
Linux solves this with architecture-specific per-CPU registers that allow single- or two-instruction access to critical per-CPU fields. UmkaOS adopts the same approach as a two-tier per-CPU model:
- Tier 1 —
CpuLocal: A fixed-layout struct pointed to by the architecture's dedicated per-CPU register. Access is 1-4 instructions with no function-call overhead. Used for ~10 of the hottest fields. - Tier 2 —
PerCpu<T>: The existing generic abstraction. Used for everything else (statistics counters, per-CPU caches, driver state).
Per-architecture register assignment:
| Architecture | Register | Instruction | Cycles | Notes |
|---|---|---|---|---|
| x86-64 | GS segment | mov %gs:OFFSET, %reg |
~1 | Segment prefix encodes offset in instruction. Set via MSR_GS_BASE per CPU at boot. Pointer materialization (when a real *CpuLocalBlock is needed): one gs-relative load of the block's self_ptr field — same cost shape as Linux's this_cpu_off. |
| AArch64 | TPIDR_EL1 |
mrs reg, tpidr_el1 + ldr |
~2-4 | System register, kernel-only (EL1). VHE kernels use TPIDR_EL2. |
| ARMv7 | TPIDRPRW |
mrc p15, 0, reg, c13, c0, 4 + ldr |
~3-5 | Privileged thread ID register (PL1 only). Requires ARMv6K+. |
| PPC64 | r13 (PACA) |
ld reg, OFFSET(r13) |
~1-3 | r13 permanently points to Per-processor Area (PACA). Matches Linux. |
| PPC32 | SPRG3 |
mfspr reg, SPRG3 + lwz |
~3-6 | SPRG3 is designated for OS use. Linux PPC32 does not optimize this; UmkaOS does. |
| RISC-V | tp (x4) |
mv reg, tp + ld |
~2-4 | Matches Linux RISC-V: tp holds per-CPU base in kernel mode; sscratch holds user tp (U-mode) or 0 (S-mode). On trap entry: csrrw tp, sscratch, tp swaps the two; sscratch == 0 distinguishes kernel re-entrant traps from user traps. |
| s390x | PREFIX page |
lg reg, LC_CPU_LOCAL_BASE (lowcore) |
~2-4 | s390x PREFIX register remaps the low 8 KiB to a per-CPU "lowcore" area. The per-CPU base pointer lives at the fixed lowcore offset LC_CPU_LOCAL_BASE (0x340); a single lg from that slot reaches the CpuLocalBlock. The slot is a prefixed-absolute address below 8 KiB — the PREFIX register remaps it to this CPU's lowcore — valid pre-DAT (real mode) and post-DAT provided the lowcore stays identity-mapped in the kernel address space. |
| LoongArch64 | $r21 ($u0) |
ld.d reg, $r21, OFFSET |
~1-3 | Kernel-reserved GPR: the kernel is built with $r21 fixed (same reservation class as RISC-V tp / PPC64 r13), so the per-CPU base lives in a pinned register and there is no base-fetch instruction. CSR.KS3 (0x33, KSave3; Linux name PERCPU_BASE_KS) holds an identical copy written once at init, consumed only at trap entry: user-origin entries reload $r21 from KS3, kernel-origin entries trust the live $r21 (see Section 2.13 for the trap-entry reload contract). KS0–KS2 (EXCEPTION_KS0–EXCEPTION_KS2) are exception-entry scratch, NOT per-CPU base. |
Design rationale: x86-64 is unique in encoding the per-CPU base within
the access instruction (the segment register carries the offset) without
consuming a general-purpose register. The pinned-GPR architectures — PPC64
r13, RISC-V tp, LoongArch64 $r21 — reserve one GPR for the base, so
per-CPU access is likewise a single load (ld / ld.d reg, base, OFFSET) with
no separate base-fetch. The remaining architectures are the system-register
legs: they read the base from a system register or lowcore slot first (AArch64
TPIDR_EL1, ARMv7 TPIDRPRW, PPC32 SPRG3, s390x lowcore), then load from
base+offset — two instructions. Every arrangement is
an order of magnitude faster than the PerCpu<T> generic path (~1-5 cycles vs
~20-30 cycles with CAS borrow checking). On LoongArch64 the per-CPU base is
$r21 on every kernel-mode access; CSR.KS3 is read (csrrd) only on the
trap-entry path to re-establish $r21 after a user-origin exception — a
csrrd KS3 on any non-entry path is a defect. Because stable Rust has no
gs-relative address space, the x86-64 leg realizes single-field accesses as
asm! %gs:offset forms (offsets via offset_of! const operands) and
materializes the block pointer — only where a genuine pointer is needed —
from the self_ptr mirror field.
s390x lowcore slot (normative): s390x has no dedicated per-CPU register — its base is a fixed doubleword in the prefixed lowcore. The offset is frozen behind a named constant, defined ONCE here and cited everywhere else (the register-table row above, the Section 3.8 s390x lowcore-offset table, and the Section 2.12 bring-up steps):
// code home: arch/s390x/cpulocal.rs
/// Lowcore offset of this CPU's `CpuLocalBlock` base pointer. Read by
/// `arch::current::cpu::cpu_local_block()` as `lg reg, LC_CPU_LOCAL_BASE(0)`
/// and written once per CPU at bring-up (BSP init / AP lowcore-image pre-store;
/// see the Initialization Sequence below). 0x340 is a doubleword-aligned slot in
/// the z/Architecture *program-available* lowcore window (0x0200-0x11AF): no
/// hardware-assigned location touches it (assigned storage is 0x0-0x1FF, then
/// 0x11B0 upward), it sits clear of the spec's own 0x200-0x23F entry save area
/// (`STMG r8,r15,0x0200` spans only 0x200-0x23F) and immediately below the
/// 0x348 / 0x350 / 0x368 stack-pointer cluster, so the per-CPU base and all
/// three stack anchors share one 256-byte lowcore cache line. NOT a Linux
/// transplant: Linux master happens to keep `current_task` at its own 0x340,
/// but nothing external observes lowcore layout above the architectural fields,
/// so the coincidence imposes no mirror obligation and a later Linux move is
/// never spec drift. Our slot holds the CpuLocalBlock BASE; `current_task` is
/// field 0 *inside* the block
/// (`const_assert!(offset_of!(CpuLocalBlock, current_task) == 0)`), so
/// `lg r,0x340; lg r,0(r)` reaches it within the row's two-instruction budget.
pub const LC_CPU_LOCAL_BASE: usize = 0x340;
Module home (normative): this section's implementation units split into a generic half and a per-architecture register-accessor half.
- Generic half —
umka-nucleus/src/sync/cpulocal.rs(generic: one body serving all architectures). Contents: theCpuLocalBlock/CpuLocalExport/CpuLocalTransittype definitions with theirconst_assert!offset/size blocks, theCpuLocalBundlelayout type and theCPU_LOCAL_PAGE/CPU_LOCAL_EXPORT_OFFSET/CPU_LOCAL_TRANSIT_OFFSET/CPU_LOCAL_BUNDLE_STRIDEconstants,MagazinePairand its re-exports, theJmpBufandDomainPanicHooktypes, the arch-independent convenience accessors (current_task(),this_rq(), thecpu_localsingle-field accessor module (thin wrappers over the arch offset-primitive family)), the typed field-scoped accessor surface (ESC-0431 — generic-half) and the fused window-transition operation family's three public operations (current_window_domain()/exit_current_window_to_core()/enter_current_window()— ESC-0432), the portable reference-semantics bodies of the offset-primitive family (reference semantics only — never the shipped realization; see the realization contract in the Arch Offset-Primitive Family section), the portabletransit()/export()bodies, and the allocation/handoff orchestration of the Initialization Sequence below. It homes insync/because that directory already hosts the locking primitives whose hot paths are CpuLocal's tightest consumers (preempt_count, the qspinlock per-CPU qnode array, the contention statistics), so the per-CPU fast path sits beside them rather than opening a new one-file top-level module. It is Nucleus because these are non-replaceable data-structure definitions withentry.Shardcoded-offset dependencies plus fixed (never policy-dispatched) accessors. - Per-architecture half — a DEDICATED
arch/<arch>/cpulocal.rs, eight legs (one per architecture). Contents: the per-CPU base register read/write for that architecture (cpu_local_block()/cpu_local_block_mut()(crate-internal), thetransit()/transit_of()/export()/export_of()register-relative addressing (the typed field-scoped accessor surface is GENERIC-half — ESC-0432 resolves the former double listing; the per-arch half supplies only the primitives beneath it; the whole-blockget()/get_mut()accessors and the reference-returningslab_magazine_pair_mut()projection remain DELETED), the arch offset-primitive family (cpu_local_read_*/cpu_local_write_*/cpu_local_add_*/cpu_local_sub_return_*— per-leg asm realization per the realization contract in the Arch Offset-Primitive Family section; NO leg re-exports the portable reference-semantics body), the fused window-transition raw operation pairwindow_exit_to_core_raw()/window_enter_raw()(pub(crate), facade-only — ESC-0432), thesetjmp/longjmpnaked-asm primitives, and the register-init step of the BSP/AP bring-up sequences (cpu_local_install_base). Deliberately NOT an addition toarch/<arch>/cpu.rs: that file is already the module home of the feature-detection and boot-handoff seams, and the register-accessor family is a cohesive, separately-implementable unit — a dedicated file keeps the two implementation surfaces independent. The glue that makes thearch::current::cpu::cpu_local_block()spelling used throughout the spec resolve unchanged is a single re-export line INSIDEarch/<arch>/cpu.rs—pub use super::cpulocal::*;— because a parent module cannot inject names into a child module's surface;arch/<arch>/mod.rscarries only themod cpulocal;registration line. Packet note (normative): that onepub use super::cpulocal::*;line inarch/<arch>/cpu.rsMUST be part of the CL stream's declaredcode_targetsat packet-cut time. The SCAFFOLDING-DESIGN §7 rule 14 registration grant enumerates exactly the registration line, the cfg-gatedpub use <arch> as current;alias, and the module doc header — "no otheruses" — so it will NOT auto-grant this re-export; without the explicit declaration the CL implementer would trespass on the E4 stream'scpu.rs.
Leg structure: per-arch register-accessor half = 8 legs; generic half = generic (single body).
/// Per-size-class free-object magazine for lock-free CPU-local slab allocation.
/// A magazine holds up to `MAGAZINE_SIZE` pre-freed objects of one size class.
/// All slots are pointers to objects of identical size; the size class is
/// implicit from which `MagazinePair` entry this magazine lives in.
///
/// # Allocation fast path
/// 1. Caller checks `loaded.count > 0`.
/// 2. If true, returns `loaded.objects[--count]` (no lock, no atomic).
/// 3. If false, swaps `loaded` ↔ `spare`; if spare was non-empty, retry step 1.
/// 4. If both are empty, refills `loaded` from the global per-size-class slab
/// under a short spinlock (typically 64 objects at a time).
///
/// # Free fast path
/// 1. If `loaded.count < MAGAZINE_SIZE`, store freed pointer at `loaded.objects[count++]`.
/// 2. If loaded is full, swap loaded ↔ spare; if spare was full, drain spare
/// to the global slab under a spinlock, then retry.
///
/// # Memory layout
/// `objects` is a fixed-size array of `MAGAZINE_SIZE` pointers. At
/// MAGAZINE_SIZE = 64, `size_of::<SlabMagazine>()` = 4 (count: u32) +
/// 4 (explicit `_pad`) + 64 × 8 (ptrs) = 520 bytes on 64-bit targets — nine cache
/// lines, contiguous and prefetch-friendly (260 bytes on 32-bit). Only the
/// POINTER to a magazine lives in the hot `CpuLocalBlock`; the magazine
/// body is a `MAGAZINE_SLAB` object.
///
/// **Canonical definition lives in [Section 4.3](04-memory.md#slab-allocator)** (`pub struct
/// SlabMagazine`, `count: u32`) together with `MAGAZINE_SIZE = 64` — this
/// file re-exports both (`pub use`) and deliberately does NOT redeclare
/// them: two full definitions previously disagreed on the `count` type
/// (u32 vs usize), which is a duplicate-symbol/type-mismatch compile
/// error and a 520-vs-260-byte layout ambiguity on 32-bit targets.
pub use crate::mm::slab::{SlabMagazine, MAGAZINE_SIZE};
/// Two-magazine pair per CPU per size class.
///
/// The two-magazine design avoids the "thrash" case where a tight alloc/free
/// loop on the same CPU would otherwise bounce between the global slab and the
/// per-CPU cache. With two magazines, the free path can fill the spare magazine
/// before touching the global slab, and the alloc path can drain the loaded
/// magazine fully before swapping in the spare.
///
/// # Type representation
///
/// Both fields use `Option<NonNull<SlabMagazine>>`:
/// - `NonNull<SlabMagazine>` encodes the invariant that the pointer is non-null
/// and properly aligned, providing a safety contract for `unsafe` dereference.
/// - `Option<NonNull<SlabMagazine>>` uses niche optimisation (same size as
/// `*mut SlabMagazine` — 8 bytes on 64-bit, 4 bytes on 32-bit) so there is
/// zero space overhead.
/// - On the **fast path**, both fields are always `Some`. Accessing the magazine
/// is `pair.loaded.unwrap().as_ref()` (or `as_mut()`), which compiles to a
/// single pointer dereference when the compiler can prove the `Some` invariant
/// (the `unwrap()` is elided in release builds on the fast path because the
/// alloc/free hot path only executes when `magazine_active == true`, and
/// initialisation guarantees `Some`).
/// - The `None` state occurs ONLY in the **CPU-hotplug offline drain
/// window**: the offline path clears `magazine_active` FIRST, then
/// `.take()`s both fields for depot return; the CPU re-onlines with
/// fresh magazines installed before the flag is set back to `true`.
/// The GC IPI drain (`drain_all_cpu_magazines()`) NEVER touches these
/// pairs at all — only DESTROYABLE dedicated caches are GC'd, and their
/// pairs live in `PERCPU_CUSTOM_MAGAZINES` (removed outright by the
/// drain, lazily re-created). In `slab_free_slow()`, the no-empties path
/// copies objects to a stack buffer and zeroes the magazine in-place,
/// keeping `pair.spare` as `Some`. Every slow-path access re-checks via
/// `magazine_pair_lookup()` ([Section 4.3](04-memory.md#slab-allocator)), which returns `None`
/// while `magazine_active == false` — so no code path can dereference a
/// taken pair.
/// - `core::mem::take()` on `*mut T` would produce null (`Default` for raw
/// pointers) -- the root cause of SLAB-13. With `Option<NonNull<T>>`,
/// `.take()` produces `None` (an explicit sentinel, not a null pointer),
/// and `None` is handled by `if let Some`. This eliminates the SLAB-13
/// null-deref bug by construction.
///
/// # Invariants
/// - `magazine_active == true` ⟺ both `loaded` and `spare` are `Some` —
/// they always point to valid, `MAGAZINE_SLAB`-allocated `SlabMagazine`
/// instances. This biconditional is THE load-bearing fast-path invariant:
/// `pair.loaded.unwrap()` is sound exactly when the flag is true.
/// - `None` is permitted only during the CPU-hotplug offline drain window
/// (where `magazine_active` was cleared first). The `slab_free_slow()`
/// no-empties path keeps both fields `Some` by zeroing the magazine
/// in-place; the GC never touches these pairs (see above). Code that
/// encounters `None` must handle it explicitly (the `None` state means
/// "magazine extracted for depot return, CPU magazines inactive").
/// - Both point to magazines for the same size class.
/// - Access to `loaded` and `spare` requires preemption disabled on the current
/// CPU (the containing `CpuLocalBlock` is only accessed with preemption off).
///
/// **Live evolution**: `MagazinePair` is a NON-REPLACEABLE data structure (part of
/// the allocator data layer, see [Section 4.3](04-memory.md#slab-allocator) for the full slab allocator
/// design including depot and partial-list interactions). The replaceable `SlabAllocPolicy` trait
/// ([Section 4.3](04-memory.md#slab-allocator))
/// controls how magazines are refilled and drained (batch size, NUMA node
/// selection), but the magazine pop/push hot path is fixed code that never goes
/// through the policy trait.
pub struct MagazinePair {
/// Currently active magazine: alloc pops from here, free pushes here first.
/// `Some` whenever `magazine_active == true` (boot init through any
/// hotplug offline drain); the offline drain `.take()`s it to `None`
/// only AFTER clearing the flag ([Section 4.3](04-memory.md#slab-allocator), §CPU hotplug
/// slab quiescence), so no flag-gated code path ever observes `None`.
/// (`Option` is retained for the pre-init/offline windows and for
/// niche-optimized pointer layout — see the const_assert below.)
pub loaded: Option<NonNull<SlabMagazine>>,
/// Backup magazine: swapped in when `loaded` is empty (alloc) or full (free).
/// Same `Some`-iff-`magazine_active` invariant and drain discipline as
/// `loaded`.
pub spare: Option<NonNull<SlabMagazine>>,
}
// Layout pin: Option<NonNull<T>> is niche-optimized to pointer size, so a
// MagazinePair is exactly two pointers with no padding. The slab fast path's
// per-class stride in `CpuLocalBlock.slab_magazines` depends on this — a
// change (e.g., adding a field) would silently grow the CpuLocalBlock and
// shift every field after `slab_magazines`.
const_assert!(core::mem::size_of::<MagazinePair>()
== 2 * core::mem::size_of::<usize>());
/// Number of slab size classes. Each size class has its own per-CPU magazine
/// pointer in CpuLocalBlock. Covers allocations from 8 bytes (class 0) to
/// 16384 bytes (class 25). Includes non-power-of-two classes 96 and 192
/// to reduce internal fragmentation for common 3-pointer structs.
/// Classes 0-3: powers of two (8..64). Class 4: 96. Class 5: 128.
/// Class 6: 192. Classes 7-9: 256, 512, 1024. Classes 10-25: step region.
/// Canonical definition in [Section 4.3](04-memory.md#slab-allocator); re-exported here (single
/// definition — no duplicate constant to drift).
pub use crate::mm::slab::SLAB_SIZE_CLASSES;
/// Fixed-layout per-CPU data block. Accessed via the architecture's
/// dedicated per-CPU register (x86-64 GS, AArch64 TPIDR_EL1, etc.).
/// Contains only the hottest fields — those accessed on every syscall,
/// every interrupt, or every scheduler tick.
///
/// # Invariants
///
/// - One `CpuLocalBlock` is allocated per CPU at boot (boot allocator).
/// - The per-CPU register is initialized to point to this block during
/// `CpuLocal::bsp_handoff()` (BSP) and `secondary_cpu_init()` (APs).
/// - The register value NEVER changes after init for a given CPU.
/// - Access requires preemption to be disabled (caller holds a
/// `PreemptGuard`), ensuring the CPU cannot change between register
/// read and field access. No runtime borrow check is needed — the
/// structural invariant (preemption disabled + dedicated register)
/// guarantees single-writer access.
///
/// # Field selection criteria
///
/// A field belongs in `CpuLocalBlock` only if ALL of the following hold:
/// 1. It is accessed on a hot path (syscall, IRQ, scheduler, slab, NAPI).
/// 2. It is per-CPU (not per-task, not global).
/// 3. It is a scalar or fixed-size pointer (no dynamically-sized data).
/// 4. It changes infrequently relative to how often it is read.
/// 5. It is NEVER accessed while a non-Core isolation image is live on
/// the CPU. This page carries the Core key; on the register-gated
/// architectures (x86-64 MPK, AArch64 POE, ARMv7 DACR) an access from
/// a domain-image context faults. Fields that must be touched from
/// such contexts belong on the `CpuLocalTransit` page (below);
/// cross-domain-readable scalars belong on `CpuLocalExport`.
/// All other per-CPU data belongs in `PerCpu<T>`.
/// # ABI Stability
///
/// Assembly code in per-architecture entry paths (`entry.S`)
/// accesses `CpuLocalBlock` fields at hardcoded offsets. Any reordering or
/// removal of existing fields will silently break all architectures.
///
/// **Evolution protocol**:
/// - New fields MUST be appended before `_pad` (and `_pad` shrunk accordingly).
/// - Never reorder or remove existing fields.
/// - Every assembly-accessed field must have a compile-time offset assertion
/// (see `const_assert!` block below the struct definition).
/// - When adding a field, update the `const_assert!` block AND all per-arch
/// assembly files that reference `CpuLocalBlock` offsets.
// kernel-internal, not KABI
#[repr(C, align(64))]
pub struct CpuLocalBlock {
/// Pointer to the currently executing task.
/// Read on every syscall entry, every interrupt, every context switch.
pub current_task: *mut Task,
/// Pointer to this CPU's runqueue.
/// Read by the scheduler on every tick and every wakeup.
pub runqueue: *mut RunQueue,
/// Preemption nesting count.
/// Incremented/decremented on every preempt_disable/enable pair.
/// Must be in CpuLocal because preempt_disable itself needs to
/// access it without going through a preempt-disabled guard (circular).
///
/// Unlike Linux's packed `preempt_count` (which encodes preemption depth,
/// softirq count, hardirq count, and NMI state in a single u32 with
/// bit-field packing), UmkaOS uses separate fields for clarity and type
/// safety. The `in_interrupt()` check uses
/// `irq_count > 0 || softirq_count > 0`, not bitmask extraction.
/// The `preemptible()` check uses
/// `preempt_count == 0 && irq_count == 0 && softirq_count == 0`.
pub preempt_count: u32,
/// Hardirq nesting count. Incremented by `irq_enter()`, decremented by
/// `irq_exit()`. `in_hardirq()` = `irq_count > 0`. Does NOT include
/// softirq depth — UmkaOS uses separate typed fields for clarity instead
/// of Linux's packed `preempt_count` bitfield layout.
pub irq_count: u32,
/// Softirq (bottom-half) nesting count. Incremented by `local_bh_disable()`,
/// decremented by `local_bh_enable()`. `in_softirq()` = `softirq_count > 0`.
/// `in_interrupt()` = `irq_count > 0 || softirq_count > 0`.
pub softirq_count: u32,
/// Per-CPU slab magazine pairs (one per size class).
/// The slab fast path (alloc/free) reads and updates these pairs.
/// See `MagazinePair` and `SLAB_SIZE_CLASSES` above (= 26, covering 8 B to 16 KB).
/// Each entry is a `MagazinePair` containing a loaded and a spare
/// `Option<NonNull<SlabMagazine>>`. On the fast path both are always `Some`.
pub slab_magazines: [MagazinePair; SLAB_SIZE_CLASSES],
/// NAPI poll budget remaining for the current poll cycle.
pub napi_budget: u32,
/// RCU nesting depth. Quiescent state is eligible for reporting when
/// this drops to 0 (see Section 3.3.1 for deferred reporting design).
pub rcu_nesting: u32,
/// Hint flag: set to `true` by `RcuReadGuard::drop()` when the outermost
/// RCU read-side critical section exits; cleared by
/// `rcu_check_callbacks()` (called from `timer_tick_handler()` step 3
/// — outside the runqueue lock — and from `finish_task_switch()` step
/// 1a after the rq lock is released), which propagates the quiescent
/// state up the `RcuNode` tree. NOT a reporting gate:
/// `rcu_check_callbacks()` reports whenever `rcu_nesting == 0` and a
/// GP needs it (`qs_pending`), regardless of this flag — a CPU that
/// never touches RCU still reports. The deferred model avoids
/// acquiring the leaf node's spinlock on every guard drop (~1 cycle
/// flag write vs. ~20-50 cycle lock acquisition).
///
/// `AtomicBool` (not plain `bool`) for NMI safety: `RcuReadGuard::drop()`
/// writes this field from task/softirq context, and an NMI could fire
/// during the write. While no current NMI handler reads this field,
/// `AtomicBool` with `Relaxed` ordering is defensive against future
/// diagnostic NMI handlers and costs zero extra cycles on x86 (same
/// codegen as plain `bool`) and ~1 cycle on ARM (store-release vs plain
/// store). The NMI write is visible within one tick boundary.
pub rcu_passed_quiesce: AtomicBool,
/// Timestamp of the last scheduler tick (nanoseconds, monotonic).
pub last_tick_ns: u64,
/// CPU index (redundant with register read, but avoids arch-specific
/// decoding of the register value to extract the CPU number).
pub cpu_id: u32,
// NOTE: the isolation raw-register shadow (`isolation_shadow`, PPC32
// `sr_shadow[16]`) and the domain-transition fields (`active_domain`,
// `domain_valid`, `domain_panic_jmpbuf`, `domain_panic_result`) do NOT
// live in this struct. They are written and read while a NON-Core
// isolation image is live on the CPU (the tail of `switch_domain()`,
// the consumer loop's post-switch stores, the in-domain panic hook,
// the NMI crash handler before image establishment), and this page
// carries the Core key — such accesses would fault on the
// register-gated architectures. They live on the per-CPU
// `CpuLocalTransit` page instead (see "CpuLocalTransit — Per-CPU
// Domain-Transition Page" below).
/// PPC64LE: MMIO barrier batching flag. Set by MMIO accessor macros,
/// drained on `SpinLock::unlock()`. Avoids redundant `sync` instructions
/// when multiple MMIO writes occur within a single critical section.
/// See [Section 3.5](#locking-strategy--mmio-barrier-batching-iosync-flag-ppc64le).
#[cfg(target_arch = "powerpc64")]
pub io_sync: IoSyncFlag,
/// Slab magazine validity flag for CPU hotplug. When `false`, slab
/// allocations on this CPU bypass the per-CPU magazine and fall through
/// to the depot slow path. Cleared before draining magazines when the CPU
/// goes offline; set to `true` when the CPU comes back online and fresh
/// magazines are allocated. Analogous to `pcp_valid` in the physical
/// allocator ([Section 4.2](04-memory.md#physical-memory-allocator)).
pub magazine_active: AtomicBool,
/// Nesting depth of `SimdKernelGuard` on this CPU (§3.10.2).
/// 0 = no kernel SIMD active. Incremented on guard acquire, decremented
/// on drop. Used by `SimdKernelGuard::is_active()` to detect re-entry
/// and by debug assertions to catch illegal nesting.
/// `AtomicU8` allows access via the field-scoped `CpuLocal::simd_kernel_depth()`
/// projection (a single-field `&'static AtomicU8`) without ever forming a
/// whole-block reference of either kind (`get()`/`get_mut()` are both
/// deleted — ESC-0431 — because NMI handlers may concurrently touch other
/// fields). Relaxed ordering compiles to plain loads/stores on x86-64 —
/// zero overhead vs plain u8.
pub simd_kernel_depth: AtomicU8,
/// Eager-reschedule fast-path MIRROR (the per-task
/// `TIF_NEED_RESCHED` bit is the authoritative request — see the
/// Need-Resched Delivery and Consumption Contract,
/// [Section 7.1](07-scheduling.md#scheduler--need-resched-delivery-and-consumption-contract)).
/// Checked on every `SpinLock::unlock()` / `preempt_enable()` (if set and
/// fully preemptible, `schedule()` is invoked) and on
/// interrupt-return-to-kernel.
///
/// **Writers — always the OWNING CPU** (CpuLocal blocks are never
/// written cross-CPU): `resched_curr(rq, Eager)` when `rq` is the
/// local runqueue, and `resched_ipi_handler()` (hardirq on the target)
/// on behalf of a remote `resched_curr(Eager)` or the RCU FQS
/// fallback. Lazy requests deliberately never set this flag.
/// **Cleared** only by `schedule()` step 2a (the single clear site).
///
/// `AtomicBool` (not plain `bool`) because the IPI handler writes it
/// from hardirq context while the interrupted code may be reading it
/// in `preempt_enable()` — a concurrent write + read on the same
/// memory, UB under the Rust memory model for non-atomic types.
/// `Relaxed` ordering suffices: all accesses are same-CPU, ordered by
/// interrupt serialization.
pub need_resched: AtomicBool,
/// True when this CPU is executing the idle loop (between
/// `cpu_idle_enter()` and `cpu_idle_exit()`). The RCU GP kthread
/// reads this to detect idle CPUs and report quiescent states
/// without sending an IPI. Set/cleared by the idle task:
/// `cpu_idle_enter()` stores true (Release); `cpu_idle_exit()`
/// stores false (Release). See [Section 7.1](07-scheduling.md#scheduler).
pub is_idle: AtomicBool,
/// NMI nesting flag. Set to `true` on NMI entry, cleared on NMI exit.
/// Checked by the locking strategy to prevent certain lock acquisitions
/// in NMI context (e.g., spinlocks that are not NMI-safe). Also used by
/// the RCU subsystem to detect NMI-within-RCU-read-side and by the perf
/// subsystem's PMI handler.
///
/// `AtomicBool` (not plain `bool`) because NMI fires asynchronously on
/// the same CPU. The NMI entry handler writes `true` while the
/// interrupted code may be reading `in_nmi` — two concurrent accesses
/// where one is a write constitutes UB on non-atomic types under the
/// Rust memory model. `Relaxed` ordering suffices (single-CPU access).
pub in_nmi: AtomicBool,
/// Per-CPU performance event context. Points to the `PerfEventContext`
/// for CPU-pinned events on this core. Set during perf subsystem init;
/// accessed from NMI handler and sampler kthread.
pub perf_ctx: *mut PerfEventContext,
/// Bitmask of pending softirq vectors (one bit per vector, bits 0-9
/// correspond to HI_SOFTIRQ through RCU_SOFTIRQ). Set by
/// `raise_softirq()` / `raise_softirq_irqoff()` via
/// `fetch_or(bit, Relaxed)`; consumed by `do_softirq()` on IRQ exit
/// via `swap(0, Relaxed)` for atomic snapshot-and-clear.
///
/// **AtomicU32 rationale**: Although `softirq_pending` is accessed only
/// by the local CPU (no cross-CPU writes), a hardirq can preempt
/// `do_softirq()` between the snapshot and clear steps. Under the Rust
/// abstract memory model, concurrent non-atomic access from different
/// execution contexts on the same CPU (hardirq preempting softirq) is
/// a data race — undefined behavior. `AtomicU32` with `Relaxed`
/// ordering eliminates the UB. On x86-64 (TSO), Relaxed ops compile to
/// plain loads/stores. On weakly-ordered architectures, Relaxed `swap`
/// is cheaper than the IRQ-disable/enable pair Linux uses (~5-10 cycles
/// vs ~20-30 cycles for DAIF manipulation on AArch64). AtomicU32 has
/// the same size and alignment as u32, so offset assertions are
/// unaffected.
/// See [Section 3.8](#interrupt-handling--softirq-deferred-interrupt-processing).
pub softirq_pending: AtomicU32,
/// ArcSwap hazard-pointer slots for this CPU
/// ([Section 3.1](#rust-ownership-for-lock-free-paths) — the ArcSwap section owns
/// the protocol and the `ARC_SWAP_HAZARD_SLOTS` constant). Each slot
/// holds the raw pointer (as usize) of an `Arc<T>` a live
/// `ArcSwapGuard` on this CPU is protecting; 0 = free. Type-erased and
/// GLOBAL: one array serves every `ArcSwap<T>` instance.
///
/// **Written ONLY by the owning CPU** (guard registration under a
/// `PreemptGuard`, cleared on guard drop) — the cross-CPU-write
/// invariant holds. **Read cross-CPU** by the swap-side reclamation
/// scan (atomic Acquire loads; a sanctioned remote read like the RCU GP
/// kthread's `is_idle` reads). Hot-path placement is the point: slot
/// registration is `ArcSwap::load()`'s only cost beyond two pointer
/// loads, and this line is already resident on every syscall.
/// Core-image-only access is preserved: `ArcSwap::load()` is a Core
/// service — in-domain Phase-3 code cannot call it
/// ([Section 12.8](12-kabi.md#kabi-domain-runtime)).
pub arc_hazard: [AtomicUsize; ARC_SWAP_HAZARD_SLOTS],
/// NUMA memory node id (raw `NumaNodeId.0` value,
/// [Section 4.11](04-memory.md#numa-topology-and-policy)) for this CPU — the node the
/// allocators draw from by default. Read by `numa_mem_id()`
/// ([Section 4.3](04-memory.md#slab-allocator)) on the slab and physical-allocator slow
/// paths as a single register-relative load (~1-2 cycles). Written
/// by the OWNING CPU only (CpuLocal blocks are never written
/// cross-CPU): set during the CPU's own online bring-up — before
/// the CPU runs any allocator path — and refreshed by the owning
/// CPU when it observes a NUMA memory-migration event (rare;
/// delivered as an IPI/tick-path refresh, never a cross-CPU store).
///
/// `AtomicU32` (not plain `u32`): the migration refresh runs in
/// hardirq context while a preempt-disabled task-context reader may
/// be mid-read — concurrent access from different execution
/// contexts on the same CPU requires atomics under the Rust memory
/// model (same rationale as `need_resched`/`softirq_pending`).
/// `Relaxed` suffices: same-CPU access, ordered by interrupt
/// serialization. Rust-only access — no assembly offset assertion
/// needed.
pub numa_mem_id: AtomicU32,
/// NUMA node id (raw `NumaNodeId.0` value) of this CPU's compute
/// affinity — usually equal to `numa_mem_id`; differs on systems
/// where a CPU's compute node and its preferred memory node
/// diverge (multi-chip modules, memory-less nodes). Read by
/// `current_numa_node()` ([Section 4.3](04-memory.md#slab-allocator)); same writer,
/// atomicity, and ordering contract as `numa_mem_id` above.
pub numa_node_id: AtomicU32,
/// Crash-rendezvous per-cycle ack dedup ([Section 12.8](12-kabi.md#kabi-domain-runtime)
/// recognition/once-only discipline). Written ONLY by this CPU's NMI
/// handler while a crash cycle is open; zero at boot; the first rendezvous
/// cycle is 1 so no slot spuriously matches. Cold — NOT one of the
/// register-hot fields (appended at the tail; no assembly offset assertion).
/// Typed `CpuLocalU64` for PPC32 cfg-portability of a logical u64 (no native
/// `AtomicU64` there); its tear-freedom is supplied by same-CPU access, not
/// by the cell — see the `CpuLocalU64` torn-tolerance note below.
pub nmi_crash_acked_cycle: CpuLocalU64,
/// Per-CPU LL/SC reservation-clearing scratch (`LlscDummy`,
/// [Section 7.3](07-scheduling.md#context-switch-and-register-state--llsc-reservation-clearing-arm-risc-v-powerpc)).
/// Target of the context-switch reservation clear on every ll/sc arch, and —
/// on PPC32 — of `arch::current::atomic::kill_reservation()` for the guarded
/// position claim
/// ([Section 3.1](#rust-ownership-for-lock-free-paths--guarded-position-claim)).
/// `LlscDummy` is itself `align(64)`, so it occupies its OWN cache line (the
/// compiler inserts alignment padding before it): the reservation-kill store
/// never invalidates a hot field's line. Appended at the tail; WRITE-ONLY
/// scratch, never read, so no assembly offset assertion is needed.
pub llsc_dummy: LlscDummy,
/// Maximum lock LEVEL currently held on this CPU — DEBUG builds ONLY
/// (lock-ordering diagnostics, [Section 3.4](#cumulative-performance-budget)). Read
/// via the field-scoped `CpuLocal::max_held_level()` accessor. `AtomicU32`
/// with `Relaxed` ordering: it has hardirq writers (a lock acquired in
/// interrupt context updates this on the same CPU), so a plain field would
/// race the interrupted updater under the Rust memory model; the atomic
/// makes field-scoped access sound. Rust-only and `#[cfg(debug_assertions)]`
/// — appended at the tail, no assembly offset assertion, and ABSENT from
/// release builds (zero release footprint; the `<= 4096` size bound is
/// evaluated per build config).
#[cfg(debug_assertions)]
pub max_held_level: AtomicU32,
/// Linear address of THIS block — the per-CPU base register's value
/// mirrored into the block itself. Written ONLY when the base register
/// is (re)pointed at the block (BSP init step 2, BSP handoff, AP slot
/// publication). Consumed on x86-64 (the base register cannot be read
/// cheaply in normal kernel code) to materialize the block pointer in
/// one gs-relative load; other architectures initialize it identically
/// (debug value) and read their base register directly. NEVER included
/// in field-copy operations — the BSP handoff excludes it explicitly.
/// Offset consumed only via `offset_of!` const operands (never a
/// hardcoded literal); entry.S does not reference it — no const_assert
/// (the `slab_magazines` precedent).
pub self_ptr: *const CpuLocalBlock,
// No explicit padding field needed — `#[repr(C, align(64))]` on the struct
// ensures cache-line alignment and prevents false sharing between adjacent
// CPUs' blocks. The compiler pads to a 64-byte boundary automatically.
}
// Total struct size bound (per-arch, defensive against accidental growth).
// Size is dominated by slab_magazines (SLAB_SIZE_CLASSES × MagazinePair) and
// varies by pointer width AND by the cfg-gated io_sync field on PPC64LE, so
// it is NOT a single constant — the enforced invariant is the 4096-byte
// page-size cap. On 64-bit, slab_magazines dominates: 26 ×
// MagazinePair (two Option<NonNull<SlabMagazine>> = 2 × 8 bytes via niche
// optimisation) = 416 bytes, + one pointer-sized `self_ptr` (8 B on 64-bit,
// 4 B on 32-bit). On 32-bit, pointers shrink to 4 bytes (magazines
// → 208 bytes; current_task/runqueue shrink too) — still well within 4096.
// Exceeding the 4096-byte bound would require multi-page allocation, breaking
// the register-based access pattern.
#[cfg(target_pointer_width = "64")]
const_assert!(core::mem::size_of::<CpuLocalBlock>() <= 4096);
#[cfg(target_pointer_width = "32")]
const_assert!(core::mem::size_of::<CpuLocalBlock>() <= 4096);
// Compile-time offset assertions for fields accessed from assembly.
// These MUST be updated when fields are added (append-only; see ABI Stability above).
// Offsets differ between 64-bit and 32-bit targets because pointer fields
// (current_task, runqueue) are 8 bytes on 64-bit and 4 bytes on 32-bit.
const_assert!(core::mem::offset_of!(CpuLocalBlock, current_task) == 0);
#[cfg(target_pointer_width = "64")]
mod offset_asserts_64 {
use super::*;
const_assert!(core::mem::offset_of!(CpuLocalBlock, runqueue) == 8);
const_assert!(core::mem::offset_of!(CpuLocalBlock, preempt_count) == 16);
const_assert!(core::mem::offset_of!(CpuLocalBlock, irq_count) == 20);
}
#[cfg(target_pointer_width = "32")]
mod offset_asserts_32 {
use super::*;
const_assert!(core::mem::offset_of!(CpuLocalBlock, runqueue) == 4);
const_assert!(core::mem::offset_of!(CpuLocalBlock, preempt_count) == 8);
const_assert!(core::mem::offset_of!(CpuLocalBlock, irq_count) == 12);
}
// slab_magazines: offset is compiler-computed (Rust struct access via CpuLocal
// register), NOT hardcoded in assembly. No const_assert needed. If
// SLAB_SIZE_CLASSES changes, the Rust compiler automatically adjusts all field
// accesses. The assertions above cover only fields referenced by hand-written
// assembly entry stubs (entry.S).
const_assert!(core::mem::align_of::<CpuLocalBlock>() == 64);
// (Total-size bound relocated to immediately below the struct definition so it
// sits within the const-assert gate's detection window.)
impl CpuLocalBlock {
/// All-zero boot value: every field at its const zero constructor
/// (null pointers, zero counters, `Atomic*::new(0)` /
/// `AtomicBool::new(false)`, `None` magazine slots,
/// `IoSyncFlag::new()` (cleared MMIO-sync flag, ppc64le leg),
/// zeroed scratch) —
/// exactly the state the BSS-zero convention relies on (the bring-up
/// invariant at the end of this section: zero bit-patterns are the
/// valid pre-init defaults). Because the value is all-zero-bits, the
/// `CPU0_BOOT_LOCAL_BLOCK` static it initializes stays in zero-fill
/// `.cpulocal` BSS.
pub const BOOT_ZERO: Self;
}
3.2.1.1 CpuLocalExport — Shared Read-Only Per-CPU Export Page¶
Per-CPU 64-bit cell (CpuLocalU64). Four fields use this cell. Three sit on
the export and transit pages (slice_remaining_ns here; isolation_shadow and
active_domain on the transit page) and are logically 64-bit atomics read
cross-CPU / cross-domain. The fourth, nmi_crash_acked_cycle, is an
owner-CPU-only field at the tail of CpuLocalBlock itself — never exported,
never read cross-CPU — that uses the cell for a DISTINCT reason: PPC32
cfg-portability of a logical u64, NOT cross-CPU tear tolerance (its tear-freedom
comes from same-CPU access, per the torn-tolerance note below).
AtomicU64 does not exist on PPC32 — the only supported leg without native 64-bit
atomics (max_atomic_width = Some(32) in rust-lang/rust
compiler/rustc_target/src/spec/targets/powerpc_unknown_linux_gnu.rs). These fields
therefore use a cfg-split cell type mirroring DebugStatU64
(Section 3.5) and EarlyLogByteCounter (Section 2.3): a
transparent AtomicU64 on every leg with target_has_atomic = "64" (all 64-bit
legs plus ARMv7-A via LDREXD/STREXD), and a lo/hi AtomicU32 pair on PPC32. Both
variants are exactly 8 bytes, so field offsets — including the entry-stub-visible
offsets on the transit page — are identical at both pointer widths. The exposed
surface is exactly load(Ordering) -> u64 / store(u64, Ordering), so every call
site that reads/writes these fields with an explicit Ordering is unchanged.
/// 64-bit per-CPU cell for `CpuLocalExport`/`CpuLocalTransit` fields that cross a
/// CPU or isolation-domain boundary. On legs with native 64-bit atomics this is a
/// transparent `AtomicU64`; on PPC32 (no 64-bit atomic) it is a lo/hi `AtomicU32`
/// pair (8 bytes) with a wait-free, torn-tolerant protocol.
///
/// **Family membership**: `CpuLocalU64` IS the `AtomicU64Cell` member of the
/// 64-bit-atomic semantic family
/// ([Section 3.5](#locking-strategy--64-bit-atomics-on-32-bit-legs-the-semantic-family)) —
/// wait-free, torn-tolerant load/store — with the added entry-stub constraint
/// that its lo half is placed FIRST (the word the transit-page assembly reads
/// for the bounded `active_domain` id).
///
/// **Torn-read tolerance (PPC32 leg)**: a reader in the store's carry window may
/// observe a value off by up to 2^32. Every consumer tolerates that:
/// `slice_remaining_ns` is a best-effort preemption-timer hint; `isolation_shadow`
/// is UNUSED on PPC32 (segment isolation uses `sr_shadow`); `core_image` is
/// likewise UNUSED on PPC32 (translation-gated — its entry stubs never read it)
/// and has no specified cross-CPU consumer, so no live PPC32 value is ever torn —
/// its same-CPU entry-stub reads are tear-free by program order, not by the cell
/// protocol, and any FUTURE remote diagnostic read of it must either tolerate the
/// PPC32 carry-window tear or be shown 32-bit-bounded; `active_domain` is a
/// bounded `DomainId` whose value always fits in the low 32 bits (the high half is
/// always 0, so it never tears); `nmi_crash_acked_cycle` is written and read only
/// by the owning CPU's NMI handler (single writer = single reader; NMIs do not
/// nest before return), so same-CPU program order guarantees every load observes
/// both halves of the last store — a torn lo/hi observation is not merely
/// tolerated here, it is structurally IMPOSSIBLE. Nothing relies on the cell's
/// own protocol for a tear-free 64-bit value: the ack-dedup word IS a correctness
/// value, but its tear-freedom is supplied by same-CPU access, not by the cell.
// kernel-internal, not KABI
#[cfg(target_has_atomic = "64")]
#[repr(transparent)]
pub struct CpuLocalU64(AtomicU64);
/// PPC32 variant: lo/hi `AtomicU32` pair (8 bytes, 4-aligned). See the 64-bit
/// variant's doc.
#[cfg(not(target_has_atomic = "64"))]
#[repr(C)]
pub struct CpuLocalU64 {
/// Low 32 bits of the logical u64 value. Placed first, so it is the word the
/// transit-page entry-stub assembly reads for the bounded `active_domain` id.
lo: AtomicU32,
/// High 32 bits of the logical u64 value.
hi: AtomicU32,
}
#[cfg(target_has_atomic = "64")]
impl CpuLocalU64 {
pub const fn new(v: u64) -> Self { Self(AtomicU64::new(v)) }
pub fn load(&self, order: Ordering) -> u64 { self.0.load(order) }
pub fn store(&self, v: u64, order: Ordering) { self.0.store(v, order) }
}
#[cfg(not(target_has_atomic = "64"))]
impl CpuLocalU64 {
pub const fn new(v: u64) -> Self {
Self { lo: AtomicU32::new(v as u32), hi: AtomicU32::new((v >> 32) as u32) }
}
/// Publish `v`: store `hi` (Relaxed), then `lo` with the caller's ordering as
/// the release point. Torn-tolerant per the type doc.
pub fn store(&self, v: u64, order: Ordering) {
self.hi.store((v >> 32) as u32, Ordering::Relaxed);
self.lo.store(v as u32, order);
}
/// Read the logical u64: `lo` with the caller's ordering, `hi` Relaxed.
/// Torn-tolerant per the type doc.
pub fn load(&self, order: Ordering) -> u64 {
let lo = self.lo.load(order) as u64;
let hi = self.hi.load(Ordering::Relaxed) as u64;
(hi << 32) | lo
}
}
// Both variants are 8 bytes: transparent `AtomicU64` (8, 8-aligned) or the lo/hi
// `AtomicU32` pair (8, 4-aligned). The size equality is what keeps the transit
// page's hardcoded entry-stub offsets valid at both pointer widths.
const_assert!(core::mem::size_of::<CpuLocalU64>() == 8);
CpuLocalBlock is Core-private data: its page carries the Core key, and no
driver domain can read it (current_task, runqueue pointers, and slab
magazines must never be disclosed to Tier 1 domains). But a small set of
per-CPU scalars are legitimately consumed ACROSS the isolation boundary —
the first is the scheduler's remaining-slice hint, which umka-kvm reads on
every VM entry to program the VMX preemption timer
(Section 18.1). Hardware memory keys are page-granular,
so exporting individual CpuLocalBlock fields is impossible; instead each
CPU gets a second, dedicated export page:
/// Per-CPU export page: per-CPU scalars readable by ALL driver domains.
///
/// **Isolation contract**: the page is tagged with the shared-read-only
/// key (PKEY 1 on x86-64 MPK; the equivalent shared-RO overlay/domain on
/// POE/DACR architectures; on architectures without fast
/// isolation the distinction is moot — everything is Tier 0). Driver
/// domains have READ permission for this key in every domain
/// configuration the domain service generates; WRITE is enabled only in
/// the Core domain's configuration. No grant negotiation, no per-domain
/// mapping step: the export pages are part of the standard kernel address
/// space that every Tier 1 domain already maps — the KEY is what gates
/// access, which is exactly the "shared read-only" key's purpose
/// ([Section 11.2](11-drivers.md#isolation-mechanisms-and-performance-modes)).
///
/// **Field policy**: only per-CPU scalars that (a) a cross-domain consumer
/// needs on a hot path (a ring round-trip would be absurd for one u64),
/// and (b) disclose no Core-private structure (values only, never
/// pointers). Every field is an atomic cell EXCEPT the pre-publication-
/// immutable `self_ptr` / `transit_ptr` mirrors, which are written inside
/// `populate_cpu_storage` before the bundle's `Release`-publication and
/// immutable thereafter (category (b) of the struct-evolution field
/// invariant, Publication and visibility contract below). The tearing
/// argument applies to the atomic class — readers are in other domains and
/// on 32-bit architectures a plain u64 read could tear against the owning
/// CPU's writer, so those fields are cells; the mirror class is safe by
/// immutability, not atomicity.
///
/// Allocated as the second page of this CPU's `CpuLocalBundle` (at first
/// online — eagerly at boot for boot-started CPUs; `populate_cpu_storage`)
/// (fixed offset `CPU_LOCAL_EXPORT_OFFSET` from the block, the
/// register-base target); reachable from the owning CPU via
/// `CpuLocal::export()` (register base + fixed offset, ~2-4 cycles) and
/// from any context via `CpuLocal::export_of(cpu_id)`.
// kernel-internal, not KABI
#[repr(C, align(64))]
pub struct CpuLocalExport {
/// Mirror of the running task's `EevdfTask.slice_remaining_ns`
/// ([Section 7.1](07-scheduling.md#scheduler)), stored by the scheduler's two existing write
/// points (tick step 3b and finish_task_switch's context-switch-in
/// refresh) with `Relaxed` — a best-effort hint (atomicity, not
/// ordering). Consumer: umka-kvm's `preemption_timer_ticks()`.
/// `CpuLocalU64` (transparent `AtomicU64`; lo/hi `AtomicU32` pair on
/// PPC32 — see the cell type above); a torn cross-domain read is
/// acceptable for this best-effort hint.
pub slice_remaining_ns: CpuLocalU64,
/// Linear address of THIS export page — the page's self-mirror,
/// consumed on x86-64 only (the per-CPU base register cannot be read
/// cheaply in normal kernel code) to materialize `CpuLocal::export()`
/// in one gs-relative load; the other seven legs form the address
/// arithmetically and treat this as a debug value. Written by
/// `populate_cpu_storage` step 3, before the step-4 `Release`-store
/// that publishes the bundle, and immutable thereafter — every reader
/// that observes a non-null slot (`Acquire`) observes it written;
/// sound as a plain field (not an atomic) under the covering shared
/// reference by that happens-before, the pre-publication-immutable
/// (category (b)) arm of the struct-evolution field invariant
/// (Publication and visibility contract below). Read-only to
/// driver domains like every byte of this page — corruption-immune.
/// Field-policy note: with `transit_ptr` below, the sanctioned
/// exceptions to "values only, never
/// pointers" — it discloses only this page's own linear address (every
/// Tier 1 domain already maps the export pages as part of the standard
/// kernel address space) plus, via the public fixed bundle offsets, the
/// addresses of its neighbors, whose protection is key enforcement,
/// not address secrecy.
pub self_ptr: *const CpuLocalExport,
/// Linear address of this CPU's transit page, consumed on x86-64 only
/// to materialize `CpuLocal::transit()` in one gs-relative load; the
/// other seven legs form the address arithmetically and treat this as
/// a debug value. Written by `populate_cpu_storage` step 3, before the
/// step-4 `Release`-store that publishes the bundle, at the same
/// instant as this page's `self_ptr`, and immutable thereafter — every
/// reader that observes a non-null slot (`Acquire`) observes it
/// written; sound as a plain field (not an atomic) under the covering
/// shared reference by that happens-before, the pre-publication-immutable
/// (category (b)) arm of the struct-evolution field invariant
/// (Publication and visibility contract below). Read-only to driver
/// domains like every byte of this page — corruption-immune. Offset consumed only via
/// `offset_of!` const operands, never a hardcoded literal — no offset
/// const_assert (the 0418-R2 / `slab_magazines` precedent).
pub transit_ptr: *const CpuLocalTransit,
}
// CpuLocalExport: one CpuLocalU64 (8) + two mirror pointers (self_ptr,
// transit_ptr — 8 each on 64-bit, 4 each on 32-bit; 24 or 16 bytes of fields
// total) in a repr(C, align(64)) struct — the cache-line alignment pads
// size_of up to 64, so the page-size bound holds unchanged. KVM reads
// slice_remaining_ns at offset 0 — a fixed cross-domain offset within this
// cache line; self_ptr and transit_ptr are appended AFTER it.
const_assert!(core::mem::size_of::<CpuLocalExport>() == 64);
New fields follow the CpuLocalBlock evolution protocol (append-only). The export page is why the KVM chapter's VM-entry path can read scheduler state with zero cross-domain overhead WITHOUT any "map CpuLocal read-only into the driver domain" special grant (no such mechanism exists) and without granting Task/tracked-storage visibility (a huge disclosure for one u64).
3.2.1.2 CpuLocalTransit — Per-CPU Domain-Transition Page¶
A domain switch is not instantaneous from the memory-key perspective: the instant the isolation register is written, the CPU's data-access rights change, but the transition machinery itself still has per-CPU state to finish updating. Concretely, the following accesses execute while a non-Core image is live on the CPU:
- the tail of
switch_domain()— the raw-shadow resync and the logical-domain-shadow store happen AFTER the hardware write, i.e. under the TARGET domain's image (and on the exit direction, its ENTRY reads of the shadow happen under the SOURCE domain's image); - the consumer loop's Phase 2/Phase 4 stores of
active_domainanddomain_valid, which deliberately bracket the hardware switch with a store-AFTER-write ordering for NMI visibility (Section 12.8); - the cross-domain trampoline's transit-word stores around its two
switch_domain()calls — the exit-prefix clears run under the SOURCE endpoint's image and the re-establishment stores under the TARGET's (Section 11.6); - the in-domain panic hook, which fires at an ARBITRARY instruction of
driver execution and must read
domain_panic_jmpbuf/ writedomain_panic_resultbeforelongjmp(); - the writer-#2 entry stubs' transit-word parking — the frame-save and
clear of
active_domain/domain_validon an asynchronous entry that interrupts an open window — which executes under the INTERRUPTED (non-Core) image, before the Core image is established (Section 11.2); - the virtualization world swap's internal raw-shadow resync
(
swap_isolation_image(), Section 11.2), executed while the CPU is in umka-kvm's domain image.
The CpuLocalBlock page carries the Core key, so on the register-gated
architectures every one of those accesses would fault. Hardware memory
keys are page-granular — the same constraint that produced
CpuLocalExport — so this state lives on a third dedicated per-CPU page:
/// Per-CPU domain-transition page: the ONLY per-CPU state that is
/// readable AND writable while a non-Core isolation image is live.
///
/// **Isolation contract**: on the register-gated architectures the page
/// is tagged with the shared read-write infrastructure key (PKEY 14 on
/// x86-64 MPK — the key class that also carries the shared DMA pool and
/// kernel stacks,
/// [Section 11.2](11-drivers.md#isolation-mechanisms-and-performance-modes--kernel-stack-key-tagging);
/// the shared-RW overlay index on AArch64 POE; DACR domain 14 on ARMv7),
/// so EVERY domain image grants read/write access. On the
/// translation-gated architectures (AArch64 page-table path,
/// PPC32) kernel translation reaches this page from every domain
/// context (kernel-half / PL1 translation is not swapped by the domain
/// switch), so no special tagging is needed; on riscv64/s390x/
/// loongarch64/ppc64le there is no fast isolation and the distinction
/// is moot.
///
/// **Security analysis (why all-domain WRITE is acceptable)**: a buggy
/// or malicious Tier 1 driver can scribble on this page. The blast
/// radius is confined to domain-transition state: corrupting
/// `isolation_shadow` or `core_image` causes a spurious or missing
/// hardware write on a later transition, which lands the CPU in a
/// wrong-image state whose first wrong access FAULTS into the domain
/// crash path; corrupting `active_domain`/`domain_valid` misroutes the
/// crash ejection protocol (a recovery-quality degradation, not a
/// disclosure). No pointer on this page targets Core data except
/// `domain_panic_jmpbuf`, which points into the consumer thread's own
/// stack — memory the executing domain can already reach (kernel stacks
/// carry the same shared-RW infrastructure tagging,
/// [Section 11.2](11-drivers.md#isolation-mechanisms-and-performance-modes--kernel-stack-key-tagging)). Compare the
/// alternative of granting domains the `CpuLocalBlock` key: that
/// discloses `current_task`/`runqueue` pointers and exposes the
/// scheduler's and slab allocator's hot state to stray writes, defeating
/// crash containment outright. On x86-64 this write exposure is also
/// strictly weaker than what a MALICIOUS driver can already do with the
/// unprivileged `WRPKRU` instruction
/// ([Section 11.2](11-drivers.md#isolation-mechanisms-and-performance-modes--wrpkru-threat-model-crash-containment-not-exploitation-prevention));
/// the threat model here is bugs, and a wild write to this one page
/// produces a contained fault, not silent Core corruption.
///
/// One transit word narrows this analysis: `self_ptr` is a diagnostic
/// mirror consumed by no materialization path on any leg — x86-64
/// materializes `transit()` via the driver-unwritable
/// `export.transit_ptr`; corrupting the transit mirror garbles
/// diagnostics only. No deflection vector exists.
///
/// **Access-model summary (the three per-CPU pages)**:
///
/// | Page | Core image | Driver-domain image |
/// |---|---|---|
/// | `CpuLocalBlock` | read/write | NO access (faults) |
/// | `CpuLocalExport` | read/write | read-only |
/// | `CpuLocalTransit` | read/write | read/write |
///
/// Allocated as the third page of this CPU's `CpuLocalBundle` (at first
/// online — eagerly at boot for boot-started CPUs; `populate_cpu_storage`)
/// (fixed offset `CPU_LOCAL_TRANSIT_OFFSET` from the block, the
/// register-base target). CPU bring-up initializes `core_image` here and — on
/// x86-64 — its sibling U=0 mirror on the per-CPU entry landing in the
/// same step (the CPL3-entry copy of the same value; see the
/// `core_image` field doc). Reachable from the owning CPU via
/// `CpuLocal::transit()` (register base + fixed offset, ~2-4 cycles) and
/// from any context via `CpuLocal::transit_of(cpu_id)` (diagnostic
/// remote scans of `active_domain` — see its contract). New fields
/// follow the CpuLocalBlock evolution protocol (append-only; the
/// leading fields are read by per-arch entry-stub assembly at hardcoded
/// offsets — see the const_assert block below).
// kernel-internal, not KABI
#[repr(C, align(64))]
pub struct CpuLocalTransit {
/// Isolation raw-register shadow — per-CPU cache of the current
/// hardware isolation register value. `switch_domain()`
/// ([Section 11.2](11-drivers.md#isolation-mechanisms-and-performance-modes--pkru-write-elision-mandatory))
/// compares the target image against this shadow and skips the
/// hardware write when they match; the four sanctioned writers
/// resync it ([Section 11.2](11-drivers.md#isolation-mechanisms-and-performance-modes--isolation-interface-contract)).
/// Arch-specific content: x86-64 PKRU (low 32 bits); AArch64 POR_EL1
/// (POE) or the full TTBR0_EL1 image, ASID in bits 63:48 (page-table
/// path — the context-switch restore loads `next.isolation_state`
/// into the hardware register and RESYNCS this shadow; the shadow is
/// a cache, never the restore source); ARMv7 DACR (low 32 bits);
/// unused on PPC32 (see
/// `sr_shadow`) and on riscv64/s390x/loongarch64/ppc64le (no fast
/// isolation). Accessed inside the arch module via
/// `arch::current::isolation::shadow_load()`/`shadow_store()` and
/// the `per_cpu::*_shadow()` views; `CpuLocalU64` (transparent
/// `AtomicU64` — Acquire loads, Release stores per the
/// cross-architecture ordering invariant — with the lo/hi `AtomicU32`
/// PPC32 variant of the cell type above) — an NMI can interrupt a
/// sanctioned writer between its hardware register write and the
/// paired shadow store (every writer is write-then-resync), so the
/// field must be race-free by type. Note this field is UNUSED on PPC32
/// (segment isolation uses `sr_shadow` below); its cell is a layout
/// placeholder there, so the PPC32 torn-tolerance of `CpuLocalU64`
/// never affects a live value.
pub isolation_shadow: CpuLocalU64,
/// Cached Core-domain register image for this CPU — the same value
/// `domain_register_image(CORE_DOMAIN_ID)` resolves, mirrored here
/// so the kernel-entry image establishment (the exception-nesting
/// half of sanctioned writer #2,
/// [Section 11.2](11-drivers.md#isolation-mechanisms-and-performance-modes--isolation-interface-contract))
/// can compare-and-install the Core image from entry-stub assembly
/// without walking the domain image table. The stub's compare
/// operand against this field is the LIVE isolation register, never
/// `isolation_shadow` — the shadow lags the register inside every
/// sanctioned writer's write-then-resync window (establishment
/// interleaving proof below). `CpuLocalU64` (transparent `AtomicU64`
/// on every leg with `target_has_atomic = "64"`; the lo/hi
/// `AtomicU32` pair on PPC32 — where this field is UNUSED: PPC32 is
/// translation-gated and its entry stubs never read `core_image`, so
/// the cell is a layout placeholder there exactly like
/// `isolation_shadow`). Armed ONCE per CPU AFTER the bundle is
/// published — by the owning CPU at AP protocol step 4, and by the
/// BSP at the named 'CpuLocal transit arming' boot step — via
/// `arm_transit_core_image()`, whose store is
/// `store(Ordering::Relaxed)`; read-only thereafter, including from
/// entry stubs under any KERNEL image. Cross-CPU soundness rests on
/// INTERIOR MUTABILITY, not on a write-once discipline: because the
/// field is a cell, the arming store is legal while any
/// `&'static CpuLocalTransit` handed out by `transit_of` is live (that
/// accessor projects only `active_domain` for its diagnostic scans,
/// and no specified consumer reads `core_image` cross-CPU). Same-CPU
/// entry-stub correctness rests on PROGRAM ORDER: the field is armed
/// before the first possible asynchronous entry under a non-Core image
/// on this CPU, so every stub that reads it — same-CPU and post-arming
/// by construction — observes the armed value; the stub's read remains
/// its existing native-width assembly load at offset 8, byte-for-byte
/// unchanged on all register-gated legs. On x86-64 the arming step
/// also initializes the SIBLING copy on the per-CPU entry landing (the
/// U=0 `core_image` mirror; the two are written together and never
/// diverge, and the sibling — reached only by its owning CPU on entry
/// landing, covered by no cross-CPU reference — stays a plain `u64`).
/// The one entry class that cannot read this field is the x86-64
/// USERSPACE-interrupted (CPL3) entry: this page is not reachable
/// until the Core image is established, so those stubs take the Core
/// value from the U=0 mirror instead and establish unconditionally
/// ([Section 11.2](11-drivers.md#isolation-mechanisms-and-performance-modes--x86-64-pku-mechanism-u-bit-smap-and-smep)
/// — the ordered CPL3 entry sequence).
pub core_image: CpuLocalU64,
/// Domain ID of the isolation domain currently executing on this
/// CPU. Set by the ring consumer loop (and the direct-call domain
/// entry shims) when execution enters a Tier 1 domain; cleared to 0
/// on exit back to Core. SECOND writer: writer #2's transit-word
/// half — an asynchronous kernel entry (IRQ, exception, NMI) that
/// interrupts an open window frame-saves this word (with
/// `domain_valid`) into the exception frame and clears it, and the
/// matching exception return restores it after re-resolving the
/// window's image through the domain image table
/// ([Section 11.2](11-drivers.md#isolation-mechanisms-and-performance-modes--isolation-interface-contract)).
/// Consequently the word names the domain of the code the CPU is
/// executing RIGHT NOW: handlers (and everything they run,
/// including `schedule()`) always observe 0, and a window survives
/// preemption/migration inside the exception frame on the task's
/// kernel stack, not in this per-CPU word.
///
/// Read by the synchronous domain-fault routing path
/// ([Section 11.9](11-drivers.md#crash-recovery-and-state-preservation)) to attribute a
/// fault to the executing domain, and by every Core-bracket
/// degeneration check (`producer_core_section()` /
/// `IrqRingHandle::post()`, [Section 12.8](12-kabi.md#kabi-domain-runtime)) — exact for
/// both, because of the parking rule above. The crash NMI handler
/// no longer consults it (it would read 0 by construction — its own
/// entry stub parked the words); crash ejection of interrupted
/// windows happens at frame-restore time instead.
///
/// `CpuLocalU64` for NMI-safe access (`Relaxed` suffices: same-CPU
/// writes; NMI delivery provides implicit ordering; the crash
/// coordinator's cross-CPU scan tolerates staleness by design). On
/// PPC32 the cell is a lo/hi `AtomicU32` pair (cell type above), which
/// does NOT violate the u64-counter rule: this word holds a BOUNDED
/// `DomainId` — a domain index (0 = Core, 1..N = Tier 1 domains, N
/// bounded by the live driver-domain count), NOT a monotonically
/// growing counter. The value is well within 32 bits, so on PPC32 the
/// high half is always 0 and the split never tears; storing it in the
/// 64-bit cell keeps this arch-neutral ID one uniform width across all
/// eight legs.
/// Value 0 = Core domain. Values 1-N = Tier 1 domains (logical
/// `DomainId` raw value — [Section 11.3](11-drivers.md#driver-isolation-tiers)).
///
/// **Relationship to `isolation_shadow`**: the shadow holds the raw
/// register image from which the domain COULD be derived by
/// arch-specific decoding; `active_domain` is the pre-extracted,
/// arch-neutral ID. `active_domain` is set AFTER the hardware
/// register write so the NMI handler never sees a value that leads
/// the actual hardware state (interleaving argument below).
pub active_domain: CpuLocalU64,
/// Pointer to the current domain's panic recovery `JmpBuf`. Used by
/// `catch_domain_panic()` ([Section 12.8](12-kabi.md#kabi-domain-runtime)) for the
/// recoverable-panic path: the pointer targets a `JmpBuf` on the
/// consumer thread's own stack. Read by the per-domain panic hook
/// WHILE THE DOMAIN IMAGE IS LIVE (a panic interrupts driver code at
/// an arbitrary instruction) — the defining reason this field is on
/// the transit page. `null` when no recovery point is active.
/// `AtomicPtr<()>`: the store(Release) in `catch_domain_panic()`
/// pairs with the panic hook's / NMI crash handler's load(Acquire).
pub domain_panic_jmpbuf: AtomicPtr<()>,
/// Result code from the last domain panic recovery. Written by the
/// domain's panic hook (under the domain image) before `longjmp()`;
/// read by `catch_domain_panic()` after `setjmp()` returns nonzero.
/// 0 = no panic, 1 = recoverable (per-request), 2 = fatal (full
/// domain teardown).
pub domain_panic_result: AtomicU8,
/// Per-CPU window-open flag — the second transit word, paired with
/// `active_domain` in every window protocol. Set to 1 as the LAST
/// store of every window entry (E3) and cleared as the FIRST store
/// of every window exit (X1) by the consumer loops, the Core-bracket
/// halves, and the cross-domain trampoline; frame-saved and cleared
/// (with `active_domain`) by writer #2's transit-word parking on
/// every asynchronous entry that interrupts an open window, and
/// restored by the matching unpark (remote CPUs' windows are parked
/// by the crash NMI's own entry stubs, so no remote clear exists or
/// is needed;
/// [Section 11.2](11-drivers.md#isolation-mechanisms-and-performance-modes--isolation-interface-contract)).
/// Its E3-last/X1-first position is what makes `domain_valid == 1`
/// certify a FULLY-established window (image installed AND
/// `active_domain` set) to any same-CPU observer, distinguishing an
/// open window from the benign mid-transition states.
///
/// (An earlier revision had the cross-domain trampoline read this
/// flag as a pre-switch "domain still valid" TOCTOU check; that
/// check is DELETED — under the CpuLocalTransit window protocol the
/// flag describes the CALLER's own window and is 0 for every Core
/// caller, and crash revocation rewrites the domain image table
/// entry to the deny-all image in place — never freeing it — so a
/// `switch_domain()` racing revocation installs deny-all and the
/// first access faults into ordinary crash handling. See
/// `trampoline_call` Step 2, [Section 11.6](11-drivers.md#device-services-and-boot). The
/// crash NMI handler likewise no longer clears it: parking replaced
/// the remote clear.) The writers span both bracketing images — set
/// under the freshly installed domain image, cleared under the
/// still-live domain image, parked under whatever image was
/// interrupted — the all-image access requirement that puts it on
/// this page. `AtomicU8`: 0 = no open window, 1 = window open.
pub domain_valid: AtomicU8,
/// PPC32 only: per-segment raw shadows (SR0–SR15) for the
/// segment-register isolation mechanism — the PPC32 counterpart of
/// `isolation_shadow`, reached via `per_cpu::sr_shadow(i)` /
/// `set_sr_shadow(i, v)` (the interface's documented PPC32 exception
/// to `shadow_load()`/`shadow_store()`). `AtomicU32` per segment for
/// the same NMI-interleaving reason as `isolation_shadow`. (PPC32 is
/// translation-gated — this page needs no special key there; the
/// array lives here anyway so ALL raw shadows share one home and one
/// contract.)
#[cfg(target_arch = "powerpc")]
pub sr_shadow: [AtomicU32; 16],
/// Linear address of THIS transit page — the page's self-mirror,
/// RETAINED as a debug/diagnostic mirror on all eight legs (the
/// uniform self-mirror idiom). Written by `populate_cpu_storage`
/// step 3, BEFORE the step-4 `Release`-store that publishes the
/// bundle, and immutable thereafter — every reader that observes a
/// non-null slot (`Acquire`) observes it written; sound as a plain
/// field (not an atomic) under the covering shared reference by that
/// happens-before, the pre-publication-immutable (category (b)) arm of
/// the struct-evolution field invariant (Publication and visibility
/// contract below). This page is shared-RW, so a
/// stray in-domain write CAN corrupt it; nothing on any leg consumes
/// it for materialization (x86-64 materializes `transit()` via the
/// driver-unwritable `export.transit_ptr`) — corruption garbles
/// diagnostics only.
pub self_ptr: *const CpuLocalTransit,
// `#[repr(C, align(64))]` pads to a cache-line boundary; the whole
// struct stays within one page (const_assert below).
}
// Entry-stub assembly on the register-gated architectures reads
// core_image (the establishment compare operand — the other operand is
// the LIVE isolation register, read directly) and stores isolation_shadow
// (the resync accompanying every stub register write) at hardcoded
// offsets (the writer-#2 exception-nesting establishment);
// active_domain's offset is pinned for
// the same reason (NMI-path assembly may pre-load it). All three
// offset-gated leading words (isolation_shadow, core_image, active_domain)
// are `CpuLocalU64`, which is exactly 8 bytes on every leg: a
// transparent `AtomicU64` on the 64-bit legs and the register-gated 32-bit
// target (ARMv7, LDREXD), and the lo/hi `AtomicU32` pair on PPC32 (which is
// translation-gated — PPC32 entry stubs never touch isolation_shadow or
// core_image, layout placeholders there). That uniform atomic prefix at
// offsets 0/8/16 — exactly 8 bytes each — is what holds the assembly-visible
// offsets at both pointer widths.
const_assert!(core::mem::offset_of!(CpuLocalTransit, isolation_shadow) == 0);
const_assert!(core::mem::offset_of!(CpuLocalTransit, core_image) == 8);
const_assert!(core::mem::offset_of!(CpuLocalTransit, active_domain) == 16);
const_assert!(core::mem::size_of::<CpuLocalTransit>() <= 4096);
const_assert!(core::mem::align_of::<CpuLocalTransit>() == 64);
NMI-never-sees-stale, restated and proved for the transit page. The
load-bearing ordering is unchanged from the pre-split design: the
consumer loop performs (E1) switch_domain(d) — hardware write — then
(E2) active_domain.store(d), (E3) domain_valid.store(1); on exit
(X1) domain_valid.store(0),
(X2) active_domain.store(0), then (X3) switch_domain(CORE). All
touches target pages legal under BOTH bracketing images, so the
ordering is implementable exactly as specified. The consumers of the
resulting invariant are the SYNCHRONOUS domain-fault routing (a fault
with active_domain != 0 is attributed to that domain — the exit order
guarantees the hardware really is in that domain, so a Core fault is
never misrouted), the Core-bracket degeneration checks, and writer #2's
entry-trigger rule register == Core image ⇒ words == (0, 0) — which
these orders are what make true (words are cleared BEFORE the hardware
write to Core and set only AFTER the hardware write to the domain).
Interleaving cases for an asynchronous observer of active_domain:
- NMI between E1 and E2: register = domain image,
active_domain= Core. The observer under-approximates (sees Core, treats the CPU as out-of-domain) — the benign entry-side race already analyzed in Section 12.8 (window ~1-2 instructions; a concurrently revoked domain faults on its next access instead). - NMI between X2 and X3: register = domain image,
active_domain= Core — same under-approximation, benign for the same reason (exit-side mirror analysis, Section 12.8). - At every other instant:
active_domain== the logical domain of the code the CPU is executing. This includes execution inside asynchronous handlers: writer #2's transit-word half (cases 7-9 below) parks the interrupted window's words in the exception frame and clears the live words, so a Core-image handler over an in-domain window correctly reads(0, 0)— the window it interrupted is not "currently executing" and is not attributed to this CPU.
No observer therefore ever OVER-approximates (never treats a CPU whose executing context is Core — a handler included — as in-domain) — the property the store-AFTER-write order and the parking rule jointly guarantee — and the stores themselves can no longer fault, which was the contradiction this page resolves.
Establishment inside writer windows (intra-writer interleavings).
Every sanctioned writer performs its hardware register write FIRST and
its shadow resync SECOND, so each transition opens a ~1-instruction
window in which isolation_shadow still names the PREVIOUS image.
Writer #2's exception-nesting stubs therefore decide from the LIVE
register, never the shadow
(Section 11.2
— writer #2's contract), and the frame-save source is that same live
read. The interleavings, continuing the case numbering above:
- IRQ or NMI inside a writer's window, entry direction — e.g. the
consumer's
switch_domain(d)has executed its hardware write (live register = domain image) but not yet its shadow store (shadow = Core); Phase 2 runs IRQ-enabled, so this window is reachable by any interrupt, not only NMI. The stub reads the LIVE register, sees non-Core, frame-saves it, installs Core, and resyncs the shadow to Core — the handler runs under Core and its Core-data accesses (irq_count, runqueues,NMI_CRASH_CTX) are legal. Had the stub compared the stale shadow againstcore_imageit would have seen equality, ELIDED establishment, and faulted on Core data under the driver image — withactive_domainstill 0, a false "Core fault" panic.active_domain= 0 in this window merely under-approximates for the crash NMI handler, exactly as in case 1. On exception return the stub sees frame ≠core_image, writes the frame-saved domain image back, and resyncs the shadow to it; the interrupted writer then resumes and executes its own — now redundant — shadow store of the same value. The pair converges to (register = image, shadow = image), so the writer's postcondition (shadow == live at writer return) holds on every interleaving. - Same window, exit direction — between
switch_domain(CORE)'s hardware write and its shadow store: live register = Core, shadow = domain image (stale). The live-register compare correctly ELIDES establishment (the CPU already runs Core); the frame saves Core; the return elides the restore; the interrupted writer's resync then completes. (A shadow-based compare would only have issued a redundant Core write here — this direction was never the hazard.) - NMI nested inside a stub's own window — between an entry stub's Core-image write and its shadow resync (or between a return stub's frame-image write and its resync): the nested NMI stub reads the live register, decides correctly (elides on the entry-stub case: live is already Core), and returns; the outer stub's resync completes afterwards. Because every decision reads the live register, nesting at ANY instruction converges the same way.
Parking and unparking the transit words (writer #2's word half —
Section 11.2).
An asynchronous entry that interrupts an open window frame-saves
active_domain/domain_valid and clears them in the X1/X2 order
BEFORE the Core-image write; the exception return re-resolves the
window's image through the domain image table, installs it, and
restores the words in the E1/E2/E3 order. Continuing the case
numbering:
- Async entry over an open window — the park sequence's
intermediate states (words cleared, register still the domain
image) are exactly the X1/X2→X3 windows of case 2: benign
under-approximation. The unpark sequence's intermediate states
(image installed, words still zero) are exactly the E1→E2 windows
of case 1. A nested NMI landing between the park's word-clears and
the Core-image write reads the live register (non-Core), parks the
(already-zero) words idempotently, establishes, and converges as in
case 6. At no instruction can any observer see
active_domain != 0with a Core-image register — an over-approximation is structurally impossible. - Crash NMI over a Core-established handler (the interrupted
window is one or more frames BENEATH the handler): the NMI's own
stub reads live register = Core, elides establishment and parking;
the handler observes words
(0, 0)— correct, the CPU's executing context is Core. The buried window is ejected when its own frame unparks: the restore's table re-resolve installs the post-revocation deny-all image, and the resumed window faults into ordinary domain crash handling on its first domain access (Section 11.9). Ejection of parked windows is DEFERRED by design; it is never lost. - Case-1/case-2 windows stretched by a handler — an async entry
landing exactly inside the ~1-2-instruction E1→E2 (or X2→X3) window
finds words
(0, 0)and parks nothing; the interrupted writer's raw frame image is restored VERBATIM on return (there is nosaved_active_domainto re-resolve by), and the writer completes its word stores afterwards. If the domain was revoked while the handler ran, the CPU briefly resumes under the pre-revocation image until its next state check or access fault — the same accepted under-approximation as cases 1/2, with the same trigger probability (the async entry must still land inside the same 1-2-instruction window; only the exposure DURATION grows by the handler's length). Containment is unchanged: the ring-stateDisconnectedstore precedes the crash NMI, so a consumer exits at its next Phase-1.5/section boundary, and all domain-private state it touched meanwhile is discarded wholesale by recovery Step 7.
Proof scope (ESC-0427). This interleaving proof covers COMPLETED
per-access word stores, parked words, and asynchronous OBSERVERS of the
words. It does NOT cover an address materialized (a &CpuLocalTransit
formed) BEFORE a migration and a store executed through it AFTER the
migration — the composition note below itself rests on per-access
re-derivation "against THAT CPU's transit page and register". That is why
unpinned transit-word access is operation-only (the fused window-transition
operation family, which re-derives per access) and the transit() reference
form is pinned-only.
Composition with the context-switch half is covered in the writer-#2
contract: preemption from case-4's or case-7's handler captures Core
into the preempted task's save slot (step 4b reads the LIVE register,
which the stub has set to Core) with transit words (0, 0), and the
frame-saved domain image and words are re-installed — on whichever CPU
the task resumes, against THAT CPU's transit page and register — only
when the task resumes through the same exception return. The transit
words are therefore (0, 0) at every schedule() invocation (voluntary
blocking clears them via the Core brackets; involuntary preemption
parks them via writer #2), which is why step 4b never saves or restores
them (Section 7.3).
Cost accounting. The transition path issues exactly the same
instruction sequence as before the split — one hardware register write
(elision unchanged) plus the same shadow/flag stores — and those stores
now share ONE transit cache line instead of dirtying the (Core-private)
CpuLocalBlock line, so Phase 2/Phase 4 touch strictly fewer distinct
lines. Zero additional register writes anywhere. The exception-nesting
stubs add one live-register READ per asynchronous entry (plus the
core_image load and compare — ~4-8 cycles per entry/exit pair, costed
in the writer-#2 contract); a read does not perturb the register/shadow
pair, so the zero-additional-writes claim is unaffected. The
transit-word half adds nothing to the all-Core common case on the
register-gated architectures (the word handling lives inside the
existing non-Core branch) and ~2-4 cycles per entry/exit pair on the
translation-gated Tier-1 architectures (one hot transit load + frame
store + predicted branch each way — costed in the writer-#2 contract);
the park/unpark work itself (a handful of transit-line and frame
stores plus one shared-RO image-table re-resolve on unpark) runs only
when an interrupt actually lands inside an open window — a path that
already pays two establishment register writes.
3.2.1.3 JmpBuf Type and setjmp/longjmp Primitives¶
The domain panic recovery mechanism (catch_domain_panic() in
Section 12.8) uses architecture-specific setjmp/longjmp
primitives to implement non-local returns from panicking domain code. These are
NOT the C library functions — they are kernel-internal primitives with restricted
semantics (no signal mask save/restore, no stack unwinding).
/// Architecture-specific jump buffer for setjmp/longjmp.
/// Stores callee-saved registers so that longjmp() can restore them
/// and resume execution at the setjmp() return point.
///
/// Each architecture saves a different set of callee-saved registers:
///
/// | Architecture | Callee-saved registers | JmpBuf size |
/// |---|---|---|
/// | x86-64 | rbx, rbp, r12-r15, rsp, rip | 64 bytes |
/// | AArch64 | x19-x28, x29(fp), x30(lr), sp | 104 bytes |
/// | ARMv7 | r4-r11, sp, lr | 40 bytes |
/// | RISC-V 64 | s0-s11, sp, ra | 112 bytes |
/// | PPC32 | r14-r31, sp, lr, cr | 88 bytes (84 padded) |
/// | PPC64LE | r14-r31, sp, lr, cr, toc(r2) | 176 bytes |
/// | s390x | r6-r13, r14(lr), r15(sp) | 80 bytes |
/// | LoongArch64 | s0-s8, sp, ra, fp | 96 bytes |
///
/// All sizes are padded to 8-byte alignment for consistent stack layout.
///
/// **No FP registers.** These rows deliberately save NO floating-point or
/// SIMD registers on any architecture: the UmkaOS kernel is compiled
/// FP/SIMD-free (general-regs-only codegen), so `setjmp`/`longjmp` — like
/// all general kernel code — never has live FP state to preserve. Issuing
/// an FP save instruction here would in fact TRAP on the architectures
/// whose FPU is disabled in kernel mode (AArch64/ARMv7/RISC-V/s390x/
/// LoongArch64). The only FP/SIMD instructions in the kernel are the
/// `#[target_feature]`-gated SIMD kernels run under `SimdKernelGuard` and
/// the scalar-FP functions of FPU-owning kthreads (lazy-trap path); see
/// the kernel-FP policy in
/// [Section 3.10](#algorithm-dispatch-and-in-kernel-simd--simdkernelguard-safe-in-kernel-simd-use).
// kernel-internal, not KABI
#[repr(C, align(8))]
pub struct JmpBuf {
/// Opaque register save area. Size is the maximum across all
/// architectures (176 bytes for PPC64LE: r14-r31, sp, lr, cr, toc(r2)
/// = 22 doublewords × 8). Architectures with smaller JmpBuf use only
/// the leading bytes.
regs: [u8; 176],
}
// JmpBuf: [u8; 176] register save area (max across all 8 arches, PPC64LE),
// repr(C, align(8)); 176 is already 8-aligned. Per-arch asm depends on this size.
const_assert!(core::mem::size_of::<JmpBuf>() == 176);
impl JmpBuf {
/// Create a zeroed JmpBuf. Must be initialized by `setjmp()` before
/// any call to `longjmp()`.
pub const fn new() -> Self {
Self { regs: [0u8; 176] }
}
}
/// Save the current execution context (callee-saved registers, stack
/// pointer, return address) into `buf`. Returns 0 on the initial call.
/// When `longjmp(buf, val)` is called, execution resumes at the
/// `setjmp()` return point with return value `val` (nonzero).
///
/// # Safety
///
/// - `buf` must be valid and aligned.
/// - The stack frame that called `setjmp()` must still be active when
/// `longjmp()` is called (no return from the calling function between
/// setjmp and longjmp). In the consumer loop, this is guaranteed because
/// `catch_domain_panic()` allocates `jmp_buf` on its own stack frame
/// and the consumer loop never returns while the domain is active.
/// - Must not be called from interrupt context (NMI, IRQ handlers).
///
/// # Architecture implementation
///
/// Each arch provides `arch::current::cpu::setjmp()` as a naked asm function
/// that saves callee-saved registers into `buf.regs` and returns 0.
pub unsafe fn setjmp(buf: &mut JmpBuf) -> i32;
/// Restore the execution context saved by `setjmp()` and resume
/// execution at the `setjmp()` return point with return value `val`.
/// Does not return.
///
/// # Safety
///
/// - `buf` must have been initialized by a prior `setjmp()` call.
/// - The stack frame from the `setjmp()` call must still be active.
/// - `val` must be nonzero (0 is reserved for the initial `setjmp()` return).
pub unsafe fn longjmp(buf: &JmpBuf, val: i32) -> !;
Per-domain panic hook installation: At domain creation, the domain service
installs a panic hook that writes domain_panic_result to the CpuLocalTransit
page and calls longjmp() on the current CPU's domain_panic_jmpbuf. The
hook runs WHILE THE DOMAIN IMAGE IS LIVE (the panic interrupts driver code),
which is why both fields live on the transit page, not in CpuLocalBlock.
The hook is a function pointer stored in the domain descriptor:
/// Panic hook installed by the domain service at domain creation.
/// Called by the Rust panic handler when a panic originates from code
/// executing within this domain (identified via CpuLocalTransit.active_domain).
///
/// The hook:
/// 1. Writes the panic classification to transit.domain_panic_result
/// (1 = recoverable, 2 = fatal — based on panic payload inspection).
/// 2. Loads the jmpbuf pointer from transit.domain_panic_jmpbuf.
/// 3. If non-null: calls longjmp() to return to catch_domain_panic().
/// 4. If null: falls through to the full domain teardown path (Path 2
/// in the crash recovery sequence).
type DomainPanicHook = fn(info: &core::panic::PanicInfo) -> !;
longjmp out of a live SimdKernelGuard — recovery contract (normative):
longjmp() performs NO Drop — it restores callee-saved registers and jumps to the
setjmp() return point, so any RAII guard held on the panicking domain's stack is
skipped without running its destructor. This matters for SimdKernelGuard
(Section 3.10):
kernel SIMD is enabled by the OUTERMOST guard and disabled only on its drop, so a domain
panic that longjmps past a live guard would otherwise leave the SIMD unit enabled and
simd_kernel_depth non-zero. After setjmp() returns nonzero, the
catch_domain_panic() recovery arm (Section 12.8) MUST therefore:
- Read
CpuLocal::simd_kernel_depth().load(Relaxed). If it is> 0: - Call
arch::current::cpu::simd_kernel_disable()UNCONDITIONALLY to return the unit to the baseline-disabled state. The outermost guard's savedarch_stateis lost with the skipped frames, but outside any guard the baseline IS the disabled state, so an unconditional disable is the correct restore (never a state-dependent one). CpuLocal::simd_kernel_depth().store(0, Relaxed).- Restore
preempt_count— which accounts for the skipped guards'PreemptGuards — to the value captured atsetjmp()time (the recovery arm's existing preempt-count snapshot, Section 12.8), NOT by counting individual dropped guards.
User FPU state needs NO action: the outermost guard already saved live user FPU state to
the task save area BEFORE enabling the unit (save_task_fpu_if_live,
Section 3.10), and
the unconditional disable makes the next user FP access trap-and-restore — the lazy-FPU
model self-heals.
AP initialization: On the BSP, the CpuLocalBlock is initialized during
early boot before SMP bringup. For APs, initialization occurs in
secondary_cpu_init() (each arch boot sequence file documents the specific
AP entry flow — see Section 2.2, Section 2.5,
Section 2.7, etc.). APs spin on a boot_flag until the
BSP signals readiness; all CpuLocalBlock fields are zero-initialized
before the dedicated register (GS base, TPIDR_EL1, tp) is set. The one
exception is the self-mirrors: when CpuLocal::populate_cpu_storage(i)
allocates CPU i's bundle (eagerly at boot for each AP actually woken, before
that AP runs), it sets the bundle's block.self_ptr, export.self_ptr, and
transit.self_ptr to the bundle's own three pages — and, at the same
write-once instant, export.transit_ptr to the bundle's transit page — and
Release-stores the bundle pointer into CPU_LOCAL_BUNDLES[i]. The
AP_BOOT_CELLS handshake then hands the AP its block base via
CpuLocal::block_base_of(i), so all the mirrors are
valid before any AP instruction can perform a CpuLocal access.
Architecture accessor functions (provided by arch::current::cpu):
/// Returns a raw pointer to the current CPU's CpuLocalBlock.
/// On x86-64: a single gs-relative load of the block's own `self_ptr`
/// field (`mov %gs:SELF_PTR_OFF, %reg`, 1 instruction). The GS base is
/// not directly readable in stable kernel Rust (rdgsbase rejected:
/// CR4.FSGSBASE-gated, #UD on pre-Ivy-Bridge Intel and pre-Zen AMD), so
/// the block mirrors its own linear address in `self_ptr`, written only
/// when the base register is pointed at the block. Hot single-field
/// accessors below do NOT call this function — they compile to direct
/// `%gs:offset` accesses via the arch offset-primitive family and never
/// materialize the base.
/// On AArch64: `mrs x0, tpidr_el1` (1 instruction).
/// On RISC-V: `mv x0, tp` (1 instruction — tp holds per-CPU base in kernel mode).
///
/// # Safety
///
/// Caller must have preemption disabled. The returned pointer is valid
/// only on the current CPU. If preemption is re-enabled, the pointer
/// may refer to another CPU's block after migration.
#[inline(always)]
pub unsafe fn cpu_local_block() -> *const CpuLocalBlock;
/// Mutable-pointer form of `cpu_local_block()`. CRATE-INTERNAL realization
/// seam of the no-reference-escape slab magazine operation family
/// ([Section 4.3](04-memory.md#slab-allocator) — `standard_magazine_try_pop`/`_try_push`/the hotplug
/// ops) — it has LEFT the sanctioned spec-pseudocode vocabulary: spec
/// pseudocode outside the CpuLocal module and that op family MUST NOT call it.
/// Its sole legitimate use is inside one of those operations, forming a
/// PRIVATE short-lived `&raw mut (*cpu_local_block_mut()).<field>` that never
/// outlives the operation and never escapes as a returned reference — with NO
/// intermediate whole-block reference. `&mut *cpu_local_block_mut()` (a
/// whole-block `&mut CpuLocalBlock`) is NAMED-AND-REJECTED — see the `get_mut`
/// deletion note below. `IrqDisabledGuard` is NEVER a uniqueness token
/// (ESC-0431): it evidences IRQ state and CPU pinning only, so it can NOT
/// serialize `&mut` loans — `local_irq_save()` is a safe nestable factory of
/// independent guards ([Section 3.8](#interrupt-handling)), and two guards would mint
/// two aliasing `&mut`. Uniqueness comes only from the op forming its `&mut`
/// internally and returning none.
///
/// # Safety
///
/// Same base contract as `cpu_local_block()`: preemption (or IRQs) must be
/// disabled, and the pointer is valid only on the current CPU. Forming a
/// per-FIELD `&mut` through this raw base additionally requires ALL THREE
/// premises: single-CPU (pinned) + IRQs-off + the projected field is in the
/// exclusively-owned class (NOT NMI/PMI-accessed and with NO sanctioned
/// cross-CPU accessor). IRQs-off does NOT exclude NMI, so IRQs-off alone is
/// insufficient; and NO precondition licenses a whole-block `&mut`, because
/// the block carries NMI/PMI-written fields (`in_nmi`,
/// `nmi_crash_acked_cycle`, `perf_ctx`) and sanctioned cross-CPU readers
/// (`is_idle`, `arc_hazard`) whose presence makes whole-block exclusivity
/// unsatisfiable on every architecture.
#[inline(always)]
pub(crate) unsafe fn cpu_local_block_mut() -> *mut CpuLocalBlock {
cpu_local_block() as *mut CpuLocalBlock
}
// `get()` — DELETED (ESC-0431). There is NO whole-block SHARED accessor of any
// shape, and NONE may be reintroduced. `&CpuLocalBlock` may NEVER be formed by
// post-publication runtime code, under ANY precondition — symmetric with the
// `get_mut()` deletion below and for the SAME clause-2 reason in its amended
// (lifetime-overlap) form: a whole-block SHARED reference asserts read-validity
// over the entire span for its whole lifetime, and the block carries plain
// async-written fields (`irq_count`/`preempt_count`/`softirq_count`/
// `napi_budget`/`rcu_nesting`/`last_tick_ns`, `irq_count` hardirq-written as
// the exemplar) whose writers the reference's lifetime can OVERLAP regardless
// of which field a caller reads — and the corpus already contained the
// prohibited pattern LIVE in NMI context (crash-NMI `get()` at
// [Section 11.9](11-drivers.md#crash-recovery-and-state-preservation) and the PMU NMI at
// [Section 20.8](20-observability.md#performance-monitoring-unit)). Pinning does NOT cure this (clause 2):
// an NMI is unmaskable and the block also has sanctioned cross-CPU readers
// (`is_idle`, `arc_hazard`). Post-publication runtime code reaches block fields
// ONLY through the typed field-scoped accessor surface below (value-returning
// for scalar/pointer fields; single-field `&'static AtomicXX` projections for
// atomic fields) or the offset-primitive family — never a whole-block
// reference of either kind. `cpu_local_block()`/`cpu_local_block_mut()` remain
// crate-private raw seams only. The one-off-accessor economy rule (below) is
// WAIVED for this surface: a one-off field accessor is ALWAYS preferable to a
// whole-block reference.
// `get_mut()` — DELETED (ESC-0428). There is NO whole-block mutable accessor
// of any shape, and NONE may be reintroduced. `&mut CpuLocalBlock` may NEVER
// be formed, under ANY precondition: the block carries NMI/PMI-written fields
// (`in_nmi`, `nmi_crash_acked_cycle`, `perf_ctx`) and sanctioned cross-CPU
// readers (`is_idle`, `arc_hazard`), so whole-block `&mut` exclusivity is
// unsatisfiable on every architecture (IRQs-off does not mask NMI, and the
// cross-CPU readers exist regardless of anything the owning CPU does). This
// deletion is the clause-2 (async / remote-aliasing axis) instance of the
// CpuLocal reference-formation doctrine (two clauses) below.
// `&mut *cpu_local_block_mut()` is NAMED-AND-REJECTED. The spec already
// records this accessor as unsound at
// [Section 3.8](#interrupt-handling--raising-a-softirq) (`raise_softirq`)
// and
// [Section 3.10](#algorithm-dispatch-and-in-kernel-simd--simdkernelguard-safe-in-kernel-simd-use).
// Mutation of a legitimately-exclusive block field goes through a
// guard-bounded typed projection accessor (below); atomic / async-accessed
// fields are touched ONLY by field-scoped atomic operations or the
// offset-primitive family, never promoted to `&mut`, in any context.
// `slab_magazine_pair_mut()` — DELETED (ESC-0431). The reference-returning
// guard-bounded projection is RETRACTED: its stated serialization basis was
// FALSE. `local_irq_save()` is a safe NESTABLE factory of independent guards
// ([Section 3.8](#interrupt-handling)), so `&mut IrqDisabledGuard` cannot serialize the
// loans — two guards mint two live aliasing `&mut MagazinePair` to the same
// pair. IrqDisabledGuard is NEVER a uniqueness token; it evidences IRQ state
// and CPU pinning only (recorded as a standing rule on `cpu_local_block_mut`
// above and at the op-family definition). It is replaced by the
// NO-REFERENCE-ESCAPE slab magazine operation family
// ([Section 4.3](04-memory.md#slab-allocator) — `standard_magazine_try_pop` /
// `standard_magazine_try_push` / the hotplug extract-and-refill ops), each of
// which takes `&IrqDisabledGuard` + `sc`, forms its OWN short-lived
// field-scoped `&mut MagazinePair` off the crate-private
// `cpu_local_block_mut()` seam, calls no function and takes no callback while
// that `&mut` is live, completes the whole mutation internally, and returns NO
// reference to the pair or its magazines — so no two `&mut` can coexist in ANY
// build. A reference-returning API may return ONLY via a future ruling that
// supplies a genuinely unique all-build token whose hot-path cost passes the
// negative-overhead gate. STATUS: the STANDARD fast paths use the op family
// (`standard_magazine_try_pop`/`_try_push`); the STANDARD slow-path magazine
// arms + the unified standard/dedicated resolver split are HELD (ESC-0431
// FINDING 2 slow-path decomposition — see the HELD note at
// [Section 4.3](04-memory.md#slab-allocator)), where `magazine_pair_lookup` retains an inline
// crate-private single-field `&mut MagazinePair` projection (the deleted
// accessor's former body — no whole-block reference) pending that design.
/// Shared reference to this CPU's `CpuLocalTransit` page (register base
/// + fixed offset, ~2-4 cycles — same addressing pattern as
/// `CpuLocal::export()`). Unlike `get()`, the returned page is legal to
/// access under ANY isolation image (its key/tagging contract is the
/// point of the type) — this is the accessor the domain-transition
/// machinery, the panic hook, and the NMI crash handler use. Used by
/// pseudocode as `CpuLocal::transit()`.
///
/// # Safety
///
/// Caller must be PINNED to this CPU for the reference's ENTIRE lifetime —
/// preemption disabled, OR in an interrupt/NMI context pinned to this CPU,
/// OR a CPU-affinity-bound thread (the consumer kthreads). The returned
/// reference is a FROZEN address: it names the CPU the caller was on when
/// `transit()` executed and does NOT re-target on migration. There is NO
/// migration-tolerant exception — an unpinned, preemptible caller performing
/// a domain-window transition uses the fused window-transition operation
/// family (`current_window_domain()` / `exit_current_window_to_core()` /
/// `enter_current_window()`, defined below), which re-derives the page at
/// each access, and NEVER this reference. This pinned-only rule is the
/// clause-1 (migration axis) instance of the CpuLocal reference-formation
/// doctrine (two clauses) below. Every field is an atomic cell
/// except the pre-publication-immutable `self_ptr`, so a shared reference is
/// sound for every operation a pinned caller performs (the struct-evolution
/// field invariant — Publication and visibility contract below).
///
/// **Realization**: on the seven non-x86-64 legs this is pure arithmetic —
/// per-CPU base register + `CPU_LOCAL_TRANSIT_OFFSET` — touching no memory
/// to form the address (in particular never the Core-keyed block page,
/// which a non-Core image cannot read; see the access-model table). On
/// x86-64 the base register is not readable in normal kernel code (see
/// `cpu_local_block()`): the accessor is ONE gs-relative load of the
/// export page's `transit_ptr` mirror through the arch offset-primitive
/// family
/// (`cpu_local_read_ptr::<{ CPU_LOCAL_EXPORT_OFFSET + offset_of!(CpuLocalExport, transit_ptr) }>()`)
/// — the export page is readable (shared-RO) under every image by its own
/// key contract, so materialization never touches the block;
/// materialization reads only the driver-unwritable export page, and the
/// returned `&'static`'s validity rests on the write-once bring-up
/// invariant, never on any driver-writable word. Same cost shape
/// as Linux's `this_cpu_off` materialization.
#[inline(always)]
pub unsafe fn transit() -> &'static CpuLocalTransit;
/// Any-context accessor for CPU `cpu_id`'s transit page (mirrors
/// `CpuLocal::export_of()`). Used for cross-CPU DIAGNOSTIC scans of
/// `active_domain` (crash-report annotation, `/ukfs` introspection) —
/// never for a correctness decision: a remote snapshot cannot see
/// windows parked in exception frames (writer #2's transit-word half),
/// and the crash NMI rendezvous counts acknowledgments from ALL CPUs
/// rather than pre-counting in-domain CPUs
/// ([Section 11.9](11-drivers.md#crash-recovery-and-state-preservation)).
/// Remote reads are inherently racy snapshots. Every post-publication-mutable
/// field of `CpuLocalTransit` is atomic (`core_image` included — `CpuLocalU64`,
/// armed once at bring-up via `arm_transit_core_image()`), making each snapshot
/// read tear-free per the field's cell contract (on PPC32, per the
/// `CpuLocalU64` torn-tolerance enumeration); `self_ptr` is written before the
/// bundle's publication and immutable thereafter, so a plain read through the
/// returned reference is sound. Projects through the spine: one `Acquire` load of
/// `CPU_LOCAL_BUNDLES.get().expect(..)[cpu_id as usize].bundle`, then
/// `&(*bundle).transit` when the pointer is non-null. Returns `None` for a
/// null slot — a possible-but-never-onlined CPU — which is the DEFINED
/// state a diagnostic scan skips (`BootOnceCell::get()` itself is `Some`
/// after the boot publication of the spine, [Section 2.3](02-boot-hardware.md#boot-init-cross-arch);
/// only the per-CPU `bundle` pointer can still be null). Wait-free, O(1).
pub fn transit_of(cpu_id: u32) -> Option<&'static CpuLocalTransit>;
/// Shared reference to this CPU's `CpuLocalExport` page (register base
/// + fixed offset, ~2-4 cycles — the same addressing pattern as
/// `transit()`). Used by pseudocode as `CpuLocal::export()`. Writers are
/// the owning CPU's scheduler paths (tick step 3b and
/// finish_task_switch's context-switch-in refresh, [Section 7.1](07-scheduling.md#scheduler)) —
/// both run with preemption disabled. A shared reference suffices for
/// every operation because every field is either an atomic cell or
/// pre-publication-immutable (the `self_ptr` / `transit_ptr` mirrors) —
/// the struct-evolution field invariant (Publication and visibility
/// contract below).
///
/// # Safety
///
/// Caller must have preemption disabled: the reference names the
/// current CPU only. Unlike `transit()` historically, `export()` carries NO
/// migration-tolerant exception — every caller is pinned for the reference's
/// lifetime (its writers are the owning CPU's preempt-disabled scheduler
/// paths; its only current-CPU reader runs in a preempt-disabled region).
/// Like `transit()`, this pinned-only requirement is a clause-1 (migration
/// axis) instance of the CpuLocal reference-formation doctrine (two clauses)
/// below.
///
/// **Realization**: on the seven non-x86-64 legs this is pure arithmetic —
/// per-CPU base register + `CPU_LOCAL_EXPORT_OFFSET` — touching no memory
/// to form the address (in particular never the Core-keyed block page,
/// which a non-Core image cannot read; see the access-model table). On
/// x86-64 the base register is not readable in normal kernel code (see
/// `cpu_local_block()`): the accessor is ONE gs-relative load of this
/// page's own `self_ptr` mirror through the arch offset-primitive family
/// (`cpu_local_read_ptr::<{ CPU_LOCAL_EXPORT_OFFSET + offset_of!(CpuLocalExport, self_ptr) }>()`)
/// — the export page is readable (shared-RO) under every image by its own
/// key contract, so materialization never touches the block. Same cost
/// shape as Linux's `this_cpu_off` materialization.
#[inline(always)]
pub unsafe fn export() -> &'static CpuLocalExport;
/// Any-context accessor for CPU `cpu_id`'s export page (mirrors
/// `transit_of()`). Used by pseudocode as `CpuLocal::export_of()`.
/// Used for cross-CPU DIAGNOSTIC reads of the exported scalars
/// (crash-report annotation, `/ukfs` introspection) — the any-context
/// reachability the `CpuLocalExport` doc promises. Current-CPU
/// consumers read via `export()` instead (umka-kvm's
/// `preemption_timer_ticks()` runs inside the VM-entry
/// preempt-disabled region, [Section 18.1](18-virtualization.md#host-and-guest-integration)).
/// Remote reads are best-effort hint snapshots — inherently racy. The exported
/// scalar `slice_remaining_ns` is atomic (`CpuLocalU64`), making each snapshot
/// read tear-free per the field's cell contract (on PPC32, per the
/// `CpuLocalU64` torn-tolerance enumeration); the `self_ptr` / `transit_ptr`
/// mirrors are written before the bundle's publication and immutable
/// thereafter, so a plain read through the returned reference is sound.
/// Projects through the spine:
/// one `Acquire` load of
/// `CPU_LOCAL_BUNDLES.get().expect(..)[cpu_id as usize].bundle`, then
/// `&(*bundle).export` when the pointer is non-null. Returns `None` for a
/// null slot — a possible-but-never-onlined CPU — which is the DEFINED
/// state a diagnostic read skips (`BootOnceCell::get()` itself is `Some`
/// after the boot publication of the spine, [Section 2.3](02-boot-hardware.md#boot-init-cross-arch);
/// only the per-CPU `bundle` pointer can still be null). Wait-free, O(1).
pub fn export_of(cpu_id: u32) -> Option<&'static CpuLocalExport>;
// ─── Fused window-transition operation family (ESC-0427) ───
// The domain-window transition protocol as executed from UNPINNED,
// preemptible producer/trampoline contexts. Each operation derives the
// transit page from the per-CPU base register AT EACH access (realized per
// the extended Arch Offset-Primitive Family realization contract below), so
// the ordered exit (X1→X2→X3) and entry (E1→E2→E3) protocol halves are
// unrepresentable-to-violate at the call site — no `&CpuLocalTransit` binding
// (a frozen address) is ever formed to be corrupted by a post-migration
// store. Blocking is FORBIDDEN inside an operation; a legally-blocking `f()`
// runs OUTSIDE them and the caller keeps `own` as a stack value across it.
//
// Module home (normative, ESC-0432): these three operations are GENERIC-half
// free functions in `umka-nucleus/src/sync/cpulocal.rs`, beside
// `current_task()`/`this_rq()` — NOT members of `pub mod cpu_local` (that
// module is scoped to one wrapper per hot `CpuLocalBlock` field operation; the
// fused ops touch `CpuLocalTransit` words plus `switch_domain`), and NOT
// members of the ESC-0431 typed field-scoped accessor surface (`active_domain`
// is a `CpuLocalTransit` word, not a block field — a page-axis category error).
// Call convention is BARE (all consumer call sites call them unqualified). The
// `// SPEC` provenance line sits on each of the three PUBLIC signatures below
// (the builder's A5 items); the per-arch write-raw pair (declared after
// `enter_current_window`) and `cpu_local_read_u64` are separately-lifted items.
/// Migration-tolerant read of THIS CPU's `active_domain` raw value — for
/// degeneration checks and diagnostics only. The value names the CPU the
/// caller was on AT THE INSTANT of the read and is documented stale the
/// moment preemption re-enables; never cache it across a preemption or
/// blocking point. (The migration-tolerant read counterpart of `this_rq()`.)
///
/// **Pinned-if-actionable** (ESC-0430): any caller that ACTS on the value —
/// a degeneration check whose result changes control flow — MUST be pinned
/// (preemption disabled) across the read. Under pinning the base-captured
/// read on a bracket-realized system-register leg is current-CPU-EXACT: the
/// captured base names the CPU the pinned caller is, and stays, on. Unpinned
/// use is diagnostic-only under the documented staleness (an unpinned caller
/// would observe a stale-but-well-formed value the next cycle even from a
/// single-instruction read, so the realization choice is immaterial there —
/// exactly the `this_rq()` staleness contract, which likewise absorbs the
/// base-capture window).
#[inline(always)]
pub fn current_window_domain() -> u64 {
// SAFETY: migration-tolerant read; per-leg realization per the TRANSACTION-clause
// predicate — IF branch on live-base legs where OFF encodes; ELSE branch (a) on
// system-register and displacement-limited legs, per the matrix of record; no
// per-arch current_window_domain item exists.
unsafe { arch::current::cpu::cpu_local_read_u64::<{ CPU_LOCAL_TRANSIT_OFFSET + offset_of!(CpuLocalTransit, active_domain) }>() }
}
/// Exit THIS CPU's open domain window to Core, returning the PREVIOUS
/// `active_domain` raw value (the window identity, kept on the caller's
/// stack for a later `enter_current_window`). If `active_domain == 0` the CPU
/// is already in Core: returns 0 with ZERO stores and zero register writes
/// (the degenerate Tier-0 path). Otherwise performs the FULL exit half in
/// order — X1 `domain_valid.store(0, Release)` → X2
/// `active_domain.store(0, Relaxed)` → X3 `switch_domain(CORE)` — each word
/// access derived register-relatively at the access. Blocking inside is
/// forbidden.
///
/// Migration-window closure (FAMILY RULE, per leg class, per the TRANSACTION
/// clause): single-instruction per-access on IF-branch legs (migration between
/// completed stores converges via writer #2 parking — the Migration-between-completed-word-stores paragraph of the matrix of record (interleaving proof cases 7-9)); the
/// branch-(b) one-IRQ-mask bracket on ELSE legs.
#[inline(always)]
pub unsafe fn exit_current_window_to_core() -> u64 {
arch::current::cpu::window_exit_to_core_raw()
}
/// Re-enter THIS CPU's domain window for `own` (the raw `DomainId` value a
/// prior `exit_current_window_to_core()` returned). Performs the FULL entry
/// half in order — E1 `switch_domain(DriverDomainId(own))` → E2
/// `active_domain.store(own, Relaxed)` → E3 `domain_valid.store(1, Release)`
/// — each word access derived register-relatively at the access. `own` MUST
/// be a non-Core domain; the caller guards the degenerate `own == 0` Core
/// case (which needs no re-entry). Blocking inside is forbidden.
///
/// Migration-window closure (FAMILY RULE, per leg class, per the TRANSACTION
/// clause): single-instruction per-access on IF-branch legs (migration between
/// completed stores converges via writer #2 parking — the Migration-between-completed-word-stores paragraph of the matrix of record (interleaving proof cases 7-9)); the
/// branch-(b) one-IRQ-mask bracket on ELSE legs.
#[inline(always)]
pub unsafe fn enter_current_window(own: u64) {
arch::current::cpu::window_enter_raw(own)
}
/// Per-arch write-raw realization seam for the two fused-window WRITE
/// operations (ESC-0432). One pair per arch leg in `arch/<arch>/cpulocal.rs`,
/// all 8 legs in the SAME commit. Each leg realizes the ENTIRE operation —
/// including exit's degenerate `active_domain == 0` zero-store branch — per the
/// TRANSACTION clause and the matrix of record. Facade-only: the two public
/// operations above (`exit_current_window_to_core`/`enter_current_window`) are
/// the ONLY callers; consumer code NEVER calls the raws. They are reachable
/// crate-internally via the existing `pub use super::cpulocal::*;` re-export
/// with visibility capped at `pub(crate)` — no public bypass surface.
///
/// SAFETY (for both): the public op's protocol obligations plus
/// post-bundle-handoff base validity. The IRQ-mask bracket is realization,
/// never a uniqueness token (standing rule, ESC-0431). Each leg's impl carries
/// its own `// SPEC` line against this declaration block.
pub(crate) unsafe fn window_exit_to_core_raw() -> u64;
pub(crate) unsafe fn window_enter_raw(own: u64);
/// Read the current task pointer. Single instruction on x86-64.
#[inline(always)]
pub fn current_task() -> *mut Task {
// SAFETY: Safe from any kernel context. The CpuLocal register
// (GS on x86-64, TPIDR_EL1 on AArch64, etc.) is always valid in
// kernel mode. The read is migration-atomic by realization (single
// instruction on x86-64 and the pinned-GPR legs; IRQ-masked
// base-fetch+load, or a leg-private current register, on the
// system-register legs — see the realization contract below) — a
// non-atomic base-captured read could return a DIFFERENT task
// entirely, not a stale self. The Task is reference-counted and
// cannot be freed while the task is executing. Preemption does NOT
// need to be disabled: post-read migration only makes the pointer
// refer to a task no longer current here; it remains valid.
unsafe { arch::current::cpu::cpu_local_read_ptr::<{ core::mem::offset_of!(CpuLocalBlock, current_task) }>() as *mut Task }
}
/// Read this CPU's runqueue pointer.
///
/// **Note on preemption**: Unlike `current_task()`, the runqueue pointer
/// is meaningful only on the CPU that owns it. If the caller is preempted
/// and migrated to another CPU after reading this pointer, the returned
/// pointer refers to the *original* CPU's runqueue, not the new CPU's.
/// Callers that need a stable runqueue reference must hold a preemption
/// guard or accept reading a potentially-stale-but-valid pointer. This
/// staleness contract also absorbs the base-capture window of an unmasked
/// (or portable-body) realization on the system-register legs: a captured
/// base yields the ORIGINAL CPU's runqueue pointer — exactly the
/// documented stale-but-valid outcome — so `this_rq()` carries NO context
/// requirement.
#[inline(always)]
pub fn this_rq() -> *mut RunQueue {
// SAFETY: Same as current_task() — CpuLocal register is valid in
// kernel mode; pointer-sized load is atomic on all architectures.
// The RunQueue is statically allocated and never freed.
unsafe { arch::current::cpu::cpu_local_read_ptr::<{ core::mem::offset_of!(CpuLocalBlock, runqueue) }>() as *mut RunQueue }
}
// ─── Typed field-scoped accessor surface (ESC-0431) ───
// Post-`get()` replacement: every block field a hot path reaches has a
// dedicated accessor, so NO whole-block `&CpuLocalBlock` is ever formed
// (clause 2). SCALAR/POINTER fields get value-returning accessors via the
// offset-primitive family. ATOMIC fields get a single-field `&'static AtomicXX`
// projection — a single-field span over an interior-mutable type is sound in
// EVERY context (including NMI/PMI), because the reference names only that
// atomic and cannot alias a sibling field an async writer touches. Each carries
// its own writer-exclusion note. This extends the `current_task()`/`this_rq()`
// precedent.
/// Read this CPU's per-CPU `PerfEventContext` pointer. Value-returning via the
/// offset-primitive family — no whole-block reference. `perf_ctx` is a plain
/// pointer written ONCE at perf-subsystem init on the owning CPU and read from
/// task context and the PMU NMI/PMI sampler; a single-instruction pointer read
/// is tear-free on every leg, so no exclusion beyond the write-once init
/// contract is needed.
#[inline(always)]
pub fn perf_ctx() -> *mut PerfEventContext {
// SAFETY: single-CPU pointer read; the CpuLocal register is valid in kernel
// mode. Migration only re-points to another (equally valid) CPU's context.
unsafe { arch::current::cpu::cpu_local_read_ptr::<{ core::mem::offset_of!(CpuLocalBlock, perf_ctx) }>() as *mut PerfEventContext }
}
/// Single-field `&'static AtomicBool` projection of this CPU's `need_resched`
/// mirror. Writers: the owning CPU (`resched_curr(rq, Eager)`) and the
/// `resched_ipi_handler()` hardirq — all field-scoped atomic stores. Sound in
/// any context: the span is one interior-mutable field.
#[inline(always)]
pub fn need_resched() -> &'static AtomicBool {
// SAFETY: single-field borrow off the raw base — NOT a whole-block
// reference. Preemption/IRQ pinning is the caller's correctness contract;
// the reference itself is sound in any context (one atomic field).
unsafe { &(*arch::current::cpu::cpu_local_block()).need_resched }
}
/// Single-field `&'static AtomicU32` projection of this CPU's `softirq_pending`
/// bitmask. Writers: the owning CPU (`raise_softirq` `fetch_or`, `do_softirq`
/// `swap`), possibly across a hardirq that preempts `do_softirq` — all
/// field-scoped atomic ops. Sound in any context.
#[inline(always)]
pub fn softirq_pending() -> &'static AtomicU32 {
// SAFETY: single-field borrow off the raw base — one interior-mutable field.
unsafe { &(*arch::current::cpu::cpu_local_block()).softirq_pending }
}
/// Single-field `&'static AtomicU8` projection of this CPU's
/// `simd_kernel_depth`. Writer: the owning CPU's `SimdKernelGuard`
/// acquire/drop (`fetch_add`/`fetch_sub`), under a `PreemptGuard`. Sound in any
/// context (one atomic field); an NMI is prohibited from SIMD, so no async
/// writer exists, but the projection would be sound even if one did.
#[inline(always)]
pub fn simd_kernel_depth() -> &'static AtomicU8 {
// SAFETY: single-field borrow off the raw base — one interior-mutable field.
unsafe { &(*arch::current::cpu::cpu_local_block()).simd_kernel_depth }
}
/// Single-field `&'static AtomicBool` projection of this CPU's `in_nmi` flag.
/// Writer: this CPU's NMI entry/exit (async, unmaskable). Sound precisely
/// because the span is one atomic field — a whole-block reference here would be
/// the clause-2 violation `get()` was deleted for.
#[inline(always)]
pub fn in_nmi() -> &'static AtomicBool {
// SAFETY: single-field borrow off the raw base — one interior-mutable field.
unsafe { &(*arch::current::cpu::cpu_local_block()).in_nmi }
}
/// Single-field `&'static AtomicU32` projection of this CPU's `numa_mem_id`.
/// Writer: the owning CPU (online bring-up, and a hardirq NUMA-migration
/// refresh) — field-scoped atomic stores. Sound in any context. `numa_node_id`
/// below is identical.
#[inline(always)]
pub fn numa_mem_id() -> &'static AtomicU32 {
// SAFETY: single-field borrow off the raw base — one interior-mutable field.
unsafe { &(*arch::current::cpu::cpu_local_block()).numa_mem_id }
}
/// Single-field `&'static AtomicU32` projection of this CPU's `numa_node_id`
/// (compute-affinity node). Same writer/atomicity contract as `numa_mem_id`.
#[inline(always)]
pub fn numa_node_id() -> &'static AtomicU32 {
// SAFETY: single-field borrow off the raw base — one interior-mutable field.
unsafe { &(*arch::current::cpu::cpu_local_block()).numa_node_id }
}
/// Single-field `&'static` projection of this CPU's `arc_hazard` slot array
/// (the CURRENT-CPU counterpart of `arc_hazard_of()`). Writer: the owning CPU
/// only (guard claim/publish/clear); read cross-CPU by the reclamation scan —
/// all slots are atomics, so every access is field-scoped. Sound in any
/// context: the span is one array-of-atomics field.
#[inline(always)]
pub fn arc_hazard() -> &'static [AtomicUsize; ARC_SWAP_HAZARD_SLOTS] {
// SAFETY: single-field borrow off the raw base — one interior-mutable field.
unsafe { &(*arch::current::cpu::cpu_local_block()).arc_hazard }
}
/// Single-field `&'static CpuLocalU64` projection of this CPU's
/// `nmi_crash_acked_cycle` cell (crash-NMI ack dedup + PMU NMI). Writer: this
/// CPU's NMI handler (async) — `CpuLocalU64` is an interior-mutable cell, so
/// the single-field projection is sound in NMI context (the exemplar clause-2
/// case). Load/store via the returned cell reference.
#[inline(always)]
pub fn nmi_crash_acked_cycle() -> &'static CpuLocalU64 {
// SAFETY: single-field borrow off the raw base — one interior-mutable cell.
unsafe { &(*arch::current::cpu::cpu_local_block()).nmi_crash_acked_cycle }
}
/// Single-field `&'static AtomicU32` projection of this CPU's `max_held_level`
/// (DEBUG builds only — the field exists only under `debug_assertions`; see the
/// struct definition). Writer: the owning CPU's lock-acquire debug path
/// (hardirq writers possible), Relaxed. Sound in any context.
#[cfg(debug_assertions)]
#[inline(always)]
pub fn max_held_level() -> &'static AtomicU32 {
// SAFETY: single-field borrow off the raw base — one interior-mutable field.
unsafe { &(*arch::current::cpu::cpu_local_block()).max_held_level }
}
On x86-64, current_task() compiles to a single gs-relative load —
mov %gs:0, %reg (offset 0 is const_asserted below CpuLocalBlock) —
the same single-instruction shape as Linux's current macro.
3.2.1.4 CpuLocal reference-formation doctrine (two clauses)¶
Two orthogonal hazards govern when a Rust reference to a per-CPU page may be
formed. A returned reference freezes an assertion the reference form cannot
uphold against an asynchronous event — address currency (migration) OR
exclusivity (aliasing) — so each hazard has its own clause. Pinning cures the
first but NOT the second. Both transit()/export() (ESC-0427) and the
deleted get_mut() (ESC-0428) are instances; this doctrine is stated once
and both accessor contracts above cross-reference it.
-
Clause 1 — migration axis. Owning-CPU per-CPU access from a migratable (unpinned, preemptible) context goes ONLY through migration-atomic operations: the arch offset-primitive family and the fused window-transition operation family (
current_window_domain/exit_current_window_to_core/enter_current_window), each realized by live-register addressing or an IRQ-mask transaction per the realization contract below. A returned&'static [mut]reference to a per-CPU page is legal ONLY for a caller pinned for the reference's ENTIRE lifetime. (Hencetransit()andexport()are pinned-only.) -
Clause 2 — async / remote-aliasing axis (pinning does NOT cure it), LIFETIME-OVERLAP formulation (ESC-0431). No reference, shared OR exclusive, may be formed whose span includes a field whenever the reference's LIFETIME can OVERLAP an unexcluded asynchronous access to that spanned field — NOT merely when the reference is formed inside an async handler. The unexcludable accesses are the NMI/PMI writers ALWAYS (an NMI is unmaskable, so its write can land at any point a whole-block reference is live), the hardirq writers unless IRQs are off for the reference's whole lifetime, and the sanctioned cross-CPU accessors (
is_idle,arc_hazard) which exist regardless of anything the owning CPU does. Such fields are reached ONLY via field-scoped atomic/primitive operations, in EVERY context including fully pinned ones — pinning bounds migration (clause 1), never the async/remote overlap. The shared-side corollary: a whole-page SHARED reference is itself a read access over its whole span for its whole lifetime, whose overlap with an async write to any spanned field is the same defect (and it retags/invalidates any live&mutsub-loan in an interrupted frame) — so clause 2 restricts BOTH reference kinds to field-scoped access. A whole-page shared reference is legal only where a struct-level field invariant guarantees every post-publication field is atomic-or-pre-publication-immutable (so no lifetime can overlap an unexcluded plain-field write): the struct-evolution field invariant (Publication and visibility contract below) supplies this forCpuLocalTransit/CpuLocalExport, butCpuLocalBlockdoes NOT satisfy it (plain mutable fields —preempt_count,slab_magazines,current_task,irq_count, …). Therefore, withget()andget_mut()both deleted,CpuLocalBlockNEVER gets a whole-block reference of EITHER kind, in ANY context, post-publication: block-field reads go through the typed field-scoped accessor surface or the offset-primitive family, and block-field mutation of the standard slab slot goes through the no-reference-escape magazine operation family (Section 4.3).
3.2.1.5 Arch Offset-Primitive Family (single-field fast path)¶
The single-field hot accessors do NOT project through the block pointer. Each
arch leg provides width-typed primitives parameterized by a const-generic
offset; generic accessors pass offset_of!(CpuLocalBlock, field).
// Provided by arch::current::cpu. The generic half defines the portable
// REFERENCE SEMANTICS (plain field projection over cpu_local_block()) —
// never the shipped realization: Rust captures the base pointer as a
// value, and a captured-base window breaks invariants (i)/(ii) under
// migration. Every leg realizes the primitives per the realization
// contract below.
pub unsafe fn cpu_local_read_ptr<const OFF: usize>() -> *mut ();
pub unsafe fn cpu_local_write_ptr<const OFF: usize>(p: *mut ());
pub unsafe fn cpu_local_read_u32<const OFF: usize>() -> u32;
pub unsafe fn cpu_local_write_u32<const OFF: usize>(v: u32);
pub unsafe fn cpu_local_add_u32<const OFF: usize>(delta: u32); // single-instruction RMW on x86-64 (addl)
pub unsafe fn cpu_local_sub_return_u32<const OFF: usize>(delta: u32) -> u32; // single memory-RMW on x86-64 (xadd, negated delta; returns new value = old.wrapping_sub(delta))
pub unsafe fn cpu_local_read_u8<const OFF: usize>() -> u8;
pub unsafe fn cpu_local_write_u8<const OFF: usize>(v: u8);
// cpu_local_read_u64 added per invariant (v) for current_window_domain
// (ESC-0432); classification per use — the current_window_domain use is a
// migration-TOLERANT read (unmasked realization on system-register legs,
// branch (a)). The cell realization is keyed on the COMPILER PREDICATE, never
// on pointer width: the sole not(target_has_atomic = "64") leg (ppc32) realizes
// as the CpuLocalU64 lo/hi pair per the matrix of record; armv7 has native
// 64-bit atomics (LDREXD/STREXD) and takes the transparent-AtomicU64 arm like
// the 64-bit legs.
pub unsafe fn cpu_local_read_u64<const OFF: usize>() -> u64;
Realization contract (normative, per leg class; no leg ships the portable reference-semantics body):
- x86-64: ONE
%gs:offset-prefixed instruction viaasm!constoperand (unchanged). - Pinned-GPR legs (ppc64le
r13/ riscv64tp/ loongarch64$r21): every read/write primitive is ONEasm!instruction addressing through the LIVE pinned register (ld rd, OFF(r13)/ld rd, OFF(tp)/ld.d rd, $r21, OFF); RMW primitives areasm!load/modify/store sequences in which BOTH memory accesses address through the live pinned register — the base is never copied into a general register. - System-register legs (aarch64
TPIDR_EL1/ armv7TPIDRPRW/ ppc32SPRG3/ s390x lowcore): base-fetch + access. MIGRATION-ATOMIC primitives (all RMWs; all plain writes callable from unpinned context; wrong-value-intolerant reads — currently exactly thecpu_local_read_ptruse behindcurrent_task()) are realized as oneasm!sequence with IRQs masked across fetch+access(es). Migration-TOLERANT reads (this_rq,cpu_id— contracts documenting staleness) may use the unmasked pair, realized as asm or as the portable body. A system-register leg MAY instead realize thecurrent_taskread via a leg-private dedicated-register mirror maintained at context switch (the block field remains the canonical store, written for the incoming task before the outgoing task's context can be resumed elsewhere — Section 7.3 step 8); the accessor contract is unchanged either way.
Window-transition operation TRANSACTION clause (normative,
PROPERTY-PREDICATED per leg class (P1 live-addressable base AND P2
OFF-encodability), ESC-0430 + ESC-0433; extends this contract to
current_window_domain/exit_current_window_to_core/enter_current_window;
all 8 legs land in the SAME commit per invariant (v)). Each operation touches
transit-page fields at bundle-window offsets
(CPU_LOCAL_TRANSIT_OFFSET + offset_of!(CpuLocalTransit, <word>), sanctioned
by invariant (vi)) plus one switch_domain register write.
The predicate (stated AS a predicate over two leg PROPERTIES, NEVER a leg list, so the multi-granule milestone re-selects realizations and a new leg classifies itself with no re-escalation). Both properties are evaluated per access, at THAT access's memory WIDTH and required memory ORDERING —
- (P1) live-addressable base: the leg's per-CPU base is LIVE in a location a
memory instruction can address through directly — a segment base folded into
the access, or a base pinned in a general-purpose register (examples: the
x86-64
%gssegment; ppc64ler13, riscv64tp, loongarch64$r21). P1 FAILS wherever the base lives in a system register or an architected memory slot that no ISA in that class admits as a load/store base operand (examples: aarch64TPIDR_EL1, armv7TPIDRPRW, ppc32SPRG3, s390x lowcore) — A64LDR(immediate), for one, definesRnas the general-purpose base register or stack pointer, so the base MUST first be materialized into a general register, which is exactly what the IF branch's no-copy clause forbids. -
(P2) OFF-encodability at width and ordering: a SINGLE instruction of that width and that required ordering encodes the bundle-window OFF (
CPU_LOCAL_TRANSIT_OFFSET + offset_of!(CpuLocalTransit, <word>)) in its displacement field. -
IF P1 AND P2: that single-instruction form is MANDATED and the base is NEVER copied into a general register.
- ELSE, by operation class:
- (a) migration-TOLERANT reads (
current_window_domain): unmasked scratch-register base MATERIALIZATION + access — materialization meaning a base fetch out of the system register (P1 failure) or base+offset arithmetic on a live-base leg whose displacement cannot reach (P2 failure). The captured base yields the value at the instant of the read — exactly the documented staleness of a migration-tolerant read. - (b) migration-ATOMIC ops (
exit_current_window_to_core/enter_current_windowword stores +switch_domain): ONE IRQ-mask bracket per operation covering the single base materialization + ALL of that operation's word accesses + theswitch_domainregister write, with NO materialized base surviving the bracket — the identical mechanism the system-register bullet of the primitive realization contract above already mandates. Folding the page-offset arithmetic into the materialized base inside the bracket is register arithmetic; the memory shape stays base-fetch + access, contract-legal.
P2's subordinate role INSIDE the ELSE branch: once a leg has taken ELSE, P2 no longer decides the branch — it decides only whether the POST-materialization access is one instruction or needs folded offset arithmetic on top of the materialized base. P2 NEVER lifts a P1-failing leg into IF.
Single-instruction access forms (P2 evidence — each is the form that encodes OFF at its width/ordering; whether it yields the IF branch depends on P1):
- On P1-SATISFYING legs, a form here satisfies the IF branch at its
width/ordering: x86-64
%gs-prefixed disp32; ppc64le DS-form (ld/std, si14<<2, ±32764, 4-aligned dword) and D-form (byte / non-scaled, ±32767); loongarch64LDPTR.D/STPTR.D(si14<<2, ±32764, 4-aligned word/dword). A P1-satisfying leg with NO form at the required width/ordering falls to the ELSE branch for that access. - On P1-FAILING (system-register) legs, the corresponding forms — armv7 plain
ldr/str; ppc32 D-form; s390x RXY; aarch64 scaledimm12— qualify NOTHING for IF. They specify the access-instruction shape used INSIDE the ELSE realization, after the base has been materialized: where the form encodes OFF, the post-materialization access is one instruction; where it does not, folded offset arithmetic is added. Their reaches are stated numerically per width below, in the same form as the P1-satisfying legs above, so P2 is decidable per access on every leg with no threshold left to invent: - armv7 (
TPIDRPRW): plainldr/str(word — the pointer width on this leg) andldrb/strb(byte) take an unsignedimm12displacement, 0..4095 (Arm DDI 0597,LDR(immediate) /LDRB(immediate); the U-bit subtract direction is irrelevant to the unsigned OFF domain of invariant (vi)). The A32LDREXform (u32 exclusive) admits an immediate of 0 only (Arm DDI 0597,LDREX) — reach {0}. The u64 forms are as already stated in the canonical per-leg shape table below:LDREXDhas no displacement field ([Rn]only) andLDRD'simm8reaches ±255. - aarch64 (
TPIDR_EL1): the unsigned-offset forms scaleimm12by the access size (pimm = imm12 × size), so the reach differs per width —ldrb/strb(byte) 0..4095;ldr/str(word) 0..16380, 4-aligned;ldr/str(dword, the pointer width) 0..32760, 8-aligned (Arm DUI 0801 and the Arm A64 ISA specification,LDR/LDRB(immediate, unsigned offset)). The transit-windowactive_domainOFF of 8208 = 8 × 1026 is inside the dword reach at 4K; the 16K-granule OFF of 32784 is outside it — see the multi-granule milestone below. - ppc32 (
SPRG3): D-form (lwz/stwword — the pointer width on this leg — andlbz/stbbyte) takes a 16-bit signed displacement, -32768..+32767 (Power ISA, D-form). TheCpuLocalU64cell on this leg is the lo/hiAtomicU32pair, so a u64 access is two D-form accesses, each with that reach. - s390x (lowcore): the RXY forms (
lg/stgdword — the pointer width on this leg — and the byte and word Y-forms) take a 20-bit signedDH:DLdisplacement, -524288..+524287 (z/Architecture Principles of Operation, long-displacement facility).
Reach {0} is not a special case (general P2 rule, all legs): an instruction
form with NO displacement field at all, or one whose immediate is architecturally
constrained to 0 — the A32 LDREX family above, and equally the no-offset
aarch64 STLR/STLRB Release stores and armv7 LDREXD — has reach {0} at its
width and ordering. It satisfies P2 at OFF = 0 and fails it at every other OFF,
decided by the same two-property predicate on the same terms. A missing
displacement field is an ordinary P2 failure, never an exception to the
predicate.
How the predicate resolves at the 4K granule (gavel-verified matrix of
record; active_domain is CpuLocalU64 at transit offset 16 — OFF = 8208 at
4K; domain_valid is AtomicU8, byte-width, at its unaligned transit offset):
- x86-64: every transit-word access is ONE
%gs:{const}instruction — no base is materialized and NOtransit_ptrmirror load appears anywhere in these operations (this is why the producer path IMPROVES over the former reference idiom, which paid a per-access mirror load). Shape:exit=movb $0, %gs:{T+domain_valid}(X1) ·movq $0, %gs:{T+active_domain}(X2) ·switch_domain(CORE)(X3);enter=switch_domain(dom)(E1) ·movq {own}, %gs:{T+active_domain}(E2) ·movb $1, %gs:{T+domain_valid}(E3);current_window_domain= onemovq %gs:{T+active_domain}, %reg. (domain_validis a 1-byteAtomicU8: X1/E3 aremovb— amovlwould be a 4-byte store spilling past the field. This is the ESC-0430 width correction.) - ppc64le (
r13):active_domain(dword) takes the IF branch via DS-formld/std(si14<<2 reaches 8208, 4-aligned);domain_valid(byte) via D-form (±32767). Both single-instruction at 4K, addressing through the LIVEr13; the base is never copied. - loongarch64 (
$r21):active_domain(4-aligned dword) takes the IF branch viaLDPTR.D/STPTR.D(si14<<2 reaches 8208) through the LIVE$r21;domain_valid(byte) has NO si14-scaled byte form (LD.D/ST.D/ST.Bare si12, ±2047, which does not reach 8208), so its access falls to ELSE branch (b)'s per-operation bracket. - riscv64 (
tp): I/S-type displacements are ±2047 — NEITHER field reaches at 4K, so every exit/enter access falls to ELSE branch (b)'s per-operation bracket, addressing through the LIVEtpinside the mask. (A STANDALONE migration-tolerantcurrent_window_domainread takes branch (a) instead — unmasked folded offset arithmetic off the livetp, never a bracket; see the canonical per-leg shape table below.) - aarch64 (
TPIDR_EL1, system-register leg): thedomain_validRelease stores X1/E3 have NO immediate-offset form in base A64 at any offset (STLR/STLRBare no-offset;STLUR/STLURBare FEAT_LRCPC2/v8.4-only), so the exit/enter operation is governed by ELSE branch (b)'s IRQ-mask bracket with the page-offset arithmetic folded into the materialized base. (active_domain's Relaxed scaled-imm12access satisfies P2 at 8208 — 8208 = 8 × 1026, inside the dword form's 8-aligned 0..32760 reach — but aarch64 FAILS P1 — the base must be materialized out ofTPIDR_EL1— so no access on this leg takes the IF branch, and the operation commits as ONE bracketed transaction per invariant (v) either way. A STANDALONEcurrent_window_domainRelaxed load is likewise ELSE branch (a): unmaskedmrs Xt, TPIDR_EL1+ldr Xd, [Xt, #8208]— two instructions, the base captured in a scratch GPR, P2's satisfaction deciding only that the post-materialization access is ONELDRwith no folded offset arithmetic. The capture window is absorbed by the documented migration-tolerant staleness; pinned-if-actionable already governs actionable callers.) - armv7 (
TPIDRPRW, system-register leg): governed by ELSE branch (b)'s bracket like the other system-register legs — the escalation'sassert_off!single-primitive realization was a mis-choice, not a contract defect. - ppc32 (
SPRG3, system-register leg): D-form reaches both fields, but ppc32 is a system-register leg — its base is fetched fromSPRG3, so the exit/enter accesses fall under ELSE branch (b)'s per-operation bracket (base-fetch + D-form access inside one IRQ mask). The PPC32CpuLocalU64lo/hiAtomicU32pair stores BOTH fall inside the bracket. - s390x (lowcore, system-register leg): RXY's 20-bit signed displacement (-524288..+524287) reaches both; accesses fall under ELSE branch (b)'s bracket (lowcore base + RXY access inside one mask).
Migration between completed word stores (IF-branch legs — x86-64, pinned-GPR): migration BETWEEN an operation's completed word stores is legal and converges via writer #2's transit-word parking exactly per the interleaving proof's cases 7-9 (valid for COMPLETED per-access stores — precisely their subject). On an ELSE-branch (b) bracketed leg the migration window is closed for the operation's duration; either way, same-CPU NMI interleavings over partial word states remain covered by the existing cases 1-9 analysis the operation text cites, NOT by the mask.
Multi-granule milestone (16K/64K images): because the predicate keys on leg
PROPERTIES — P1 unchanged by page granule, P2 re-evaluated at the new OFF —
never a fixed leg list, the realizations re-select automatically. At ≥16K the
transit-page OFF (8208 grows to 32784 at 16K) passes the displacement reach of
the D/DS-form and si14<<2 legs, so P2 fails there. active_domain on ppc64le
(and loongarch64's LDPTR.D) drops out of IF and re-selects BY OPERATION CLASS,
exactly like any other ELSE case: a migration-TOLERANT read takes branch (a) —
unmasked folded offset arithmetic off the still-LIVE r13 / $r21, since P1
holds at every page granule — and NEVER acquires a bracket; the exit/enter
transactions take branch (b)'s per-operation bracket. On the already-ELSE
system-register legs nothing re-classifies; only the reach of the in-ELSE
post-materialization access changes, and it is computed from the stated numbers
rather than assumed: 32784 exceeds both ppc32's D-form (-32768..+32767) and
aarch64's 8-aligned dword form (0..32760), so each of those accesses gains
folded offset arithmetic. All of it with NO re-escalation and NO ISA constant
entering generic layout (s390x RXY's 20-bit reach and x86-64 disp32 still
encode).
- Ordering (all legs): each store keeps its documented
Ordering(Releaseon X1 and E3;Relaxedon X2 and E2); the realization emits the fence per invariant (iv). No ordering-neutral raw-movdegradation of theReleasestores is permitted, and P2 is evaluated at the store's REQUIRED ordering — a P1-satisfying leg lacking a single displacement-encoded instruction AT that ordering takes the ELSE branch even where a weaker-ordered instruction would encode the same offset.
Canonical per-leg shapes for the migration-tolerant cpu_local_read_u64
(normative; the STANDALONE current_window_domain read of active_domain at
OFF = 8208, 4K granule — the certification target of the all-8-leg disassembly
gate below):
| Leg | Branch | Shape |
|---|---|---|
| x86-64 | IF | one %gs-relative mov |
| ppc64le | IF | one ld OFF(r13) (DS-form, si14<<2 reaches 8208) |
| loongarch64 | IF | one ldptr.d (si14<<2 reaches 8208) |
| riscv64 | ELSE (a) | folded offset arithmetic off the LIVE tp + ld — P1 holds, P2 fails (I/S-type imm12 is ±2047 and does not reach the transit window) |
| aarch64 | ELSE (a) | mrs Xt, TPIDR_EL1 + ldr Xd, [Xt, #8208] — P1 fails, P2 holds (the dword form's scaled imm12 reaches 0..32760, 8-aligned, and 8208 = 8 × 1026), so ONE access instruction after the base fetch |
| armv7 | ELSE (a) | mrc p15, 0, Rt, c13, c0, 4 + folded offset arithmetic + ONE native 64-bit atomic load — the arithmetic is NOT optional: LDREXD has no displacement field at all ([Rn] only) and LDRD's imm8 (±255) cannot reach 8208, so OFF must be folded into the materialized base. ARMv7-A satisfies target_has_atomic = "64", so the CpuLocalU64 cell is the transparent AtomicU64 arm, NOT a lo/hi pair |
| ppc32 | ELSE (a) | mfspr reg, SPRG3 + the CpuLocalU64 lo/hi pair loads per the cell protocol — the ONLY leg without a native 64-bit atomic |
| s390x | ELSE (a) | lg of the base from LC_CPU_LOCAL_BASE + lg (RXY, 20-bit signed DH:DL displacement, -524288..+524287, reaches) |
Verification rider (binding, ESC-0427 item 6): the builder records the
per-leg instruction shape (disassembly-level sketch, as above — for the
migration-tolerant read, against the canonical per-leg shape table immediately
above) for all THREE
operations on all 8 legs, and this unit's acceptance includes a producer
microbenchmark gate covering the degenerate Core path
(exit_current_window_to_core returning 0 with zero stores) and the
non-degenerate exit+re-entry, on at least x86-64 plus one system-register leg
— the fused ops must NOT regress producer_core_section().
The rider additionally binds, on all 8 legs, a pair of compile-time domain probes certifying that the OFF-domain guard of invariant (vi) actually evaluates:
- NEGATIVE probe: a compile-fail check instantiating a guarded offset
primitive at an out-of-domain OFF (
OFF >= CPU_LOCAL_BUNDLE_STRIDE) in a context that FORCES monomorphization and codegen — for example taking the function pointer into a#[used]or exported item. The build MUST fail. A bare instantiation is NOT sufficient: aconstblock inside a never-codegen'd#[inline(always)]generic is never evaluated, so an unforced instantiation builds clean and certifies nothing. - POSITIVE control: the largest serveable in-domain OFF for that leg and that access width MUST build.
The probe code lives in builder test infrastructure, but the probes' existence
and their pass/fail semantics are rider-bound acceptance criteria of this
primitive family, not generic test hygiene. The probe target is the permanent
domain boundary — OFF >= CPU_LOCAL_BUNDLE_STRIDE, the edge of invariant
(vi) — because a guard that never evaluates makes that boundary fiction.
Representative x86-64 realization (normative shape):
#[inline(always)]
pub unsafe fn cpu_local_read_ptr<const OFF: usize>() -> *mut () {
let out: *mut ();
// Non-`pure` asm: every call is a fresh gs-relative access. The
// compiler may not cache, merge, or reorder these across calls — the
// migration-tolerance contracts (value at the instant of the read)
// depend on per-call re-reads.
asm!("mov %gs:{off}, {out}",
off = const OFF, out = lateout(reg) out,
options(att_syntax, nostack, preserves_flags));
out
}
Invariants (all architectures): (i) each read/write primitive is a single
register-relative memory instruction on x86-64 (always) and on any leg
satisfying BOTH P1 (live-addressable per-CPU base) and P2 (a single instruction
encodes OFF at the access's width and ordering) — the two-property PREDICATE of
the window-transition transaction clause above, applied across the
full OFF domain of invariant (vi), governs this per-access. Where the leg fails
EITHER property — a system-register
leg (aarch64/armv7/ppc32/s390x fails P1, its base fetch never elided), or a
live-base leg at an OFF its displacement field cannot reach (fails P2:
e.g. loongarch64 byte access, riscv64 at the transit window, or the
pinned-GPR dword legs at a ≥16K page granule) — the realization is at most
base-fetch + access: unmasked for migration-tolerant reads, IRQ-mask-bracketed
for migration-atomic primitives (the bracket wraps register writes, not memory
accesses; the memory shape stays base-fetch + access); (ii) RMW
primitives touch
only the calling CPU's block — a same-CPU interrupt observes only before/after
states. On x86-64, mutate-only RMW primitives are a single memory-RMW
instruction; value-returning RMW primitives use a single memory-RMW
instruction that returns the prior value (xadd with negated delta,
caller-side register adjustment to the new value) — never a separate read-back
of the mutated slot. Elsewhere the load/modify/store window is safe against
SAME-CPU nesting because nesting IRQ handlers restore counters pairwise, and
against migration because every RMW accessor's window is closed by one of the
family-rule arguments below (pinning-counter, IRQ-masked realization, or
caller-pinned — named in each accessor's doc); the returned value is
computed in a register before the store, so no read-back exists on any leg;
(iii) no primitive is pure/readonly, and a flag-writing RMW realization
(x86-64 addl, xadd) must NOT claim preserves_flags in its asm! options —
those instructions write OF/SF/ZF/AF/CF/PF, so the claim would be a false unsafe
contract (UB); only flag-neutral bodies (mov) may carry preserves_flags;
(iv) memory ordering is the caller's job —
the generic accessor issues its documented fence, the primitive is
ordering-neutral; (v) the family is closed over the operations the generic
accessors actually use — a new accessor needing a new width/op extends the
family in the same commit, all 8 legs; (vi) the OFF domain is the per-CPU
bundle, not merely the block: 0 ≤ OFF < CPU_LOCAL_BUNDLE_STRIDE
(Section 3.2);
bundle-window constants (page offset + offset_of! within the page struct)
are sanctioned family uses — this is how the x86-64 transit()/export()
mirrors are read. Across this full unsigned domain, whether a given OFF admits
the single-instruction form at a given access width/ordering is decided by the
two-property predicate (invariant (i) and the window-transition transaction
clause), NOT assumed: a bundle-window OFF a leg's displacement field cannot
reach at 4K, or that grows out of reach at a larger page granule, takes the
base-fetch + access realization (unmasked or per-operation bracketed by class)
with NO change to this OFF domain and NO change to the layout.
FAMILY RULE (migration-window closure, invariant (ii) companion): the
pinning-counter argument (spelled out per class in preempt_count_inc's doc)
is valid ONLY for counters whose nonzero value forbids preemption and whose
value at any switch-in is 0; every RMW accessor's doc MUST name which argument
covers it — single-instruction, pinning-counter, IRQ-masked realization, or
caller-pinned.
NAMED-AND-REJECTED (value-returning RMW): subl-then-reload. Realizing
cpu_local_sub_return_u32 (and any value-returning primitive) as a memory
mutation followed by a separate re-read of the mutated slot is rejected for
the whole primitive family, at every width. The decrement becomes globally
visible at the subl; for the principal consumer
preempt_count_dec_and_test_resched the count can reach 0 there, an interrupt
in the two-instruction window may preempt and migrate the task, and the reload
then reads a different CPU's block — violating invariant (ii)'s "touch only
the calling CPU's block". The sanctioned x86-64 shape (invariant (ii)) computes
the returned value from the xadd-captured prior value in a register, so no
post-mutation re-read exists on any leg. The plain load/modify/store reading is
safe (the returned value comes from this CPU; same-CPU pairwise IRQ restore
holds, and migration mid-window is closed for its consumers by the family-rule
arguments — pinning-counter for the preempt counters, caller-pinned for the RCU
nesting counters),
but it forfeits the structural no-window property without compensation and is
NOT the sanctioned shape; xadd with a negated delta is normative.
Representative x86-64 realization of the value-returning shape (normative):
#[inline(always)]
pub unsafe fn cpu_local_sub_return_u32<const OFF: usize>(delta: u32) -> u32 {
let neg = delta.wrapping_neg(); // subtract == add the two's complement
let old: u32;
// Single memory-touching instruction: %gs:OFF += neg, register := old value.
// No window in which the mutated slot is globally visible before the
// primitive has captured everything it returns; no read-back of the slot.
asm!("xadd {v:e}, %gs:{off}",
off = const OFF, v = inout(reg) neg => old,
options(att_syntax, nostack));
old.wrapping_add(neg) // new value = old - delta
}
3.2.1.6 Generic cpu_local Field Accessors¶
Hot-path pseudocode throughout the spec uses cpu_local::…() single-field
accessors — one #[inline(always)] wrapper per hot CpuLocalBlock field
operation. Each compiles to ~1-3 register-relative instructions via the arch
offset-primitive family above (on x86-64: direct %gs:offset forms — the
block pointer is never materialized on these paths), but names
the OPERATION (increment, decrement-and-test, flag set) instead of exposing the
raw block pointer, so call sites carry no unsafe blocks and no field-projection
boilerplate. These are generic (arch-independent) wrappers — the only
arch-specific part is the per-CPU base register access they inline.
/// Generic single-field CpuLocal accessors. All of them:
/// - operate on the CALLING CPU's `CpuLocalBlock` only;
/// - are safe to call from any kernel context (each either tolerates
/// migration or itself pins the task — see per-function contracts);
/// - compile to ~1-3 instructions (register-relative load/store/RMW).
pub mod cpu_local {
/// `preempt_count += 1` — disable preemption. Safe from any kernel
/// context: the increment itself is what pins the task to this CPU.
/// Migration-window closure (the FAMILY RULE's pinning-counter
/// argument, per leg class): x86-64: the single memory-RMW is
/// interrupt-atomic. Pinned-GPR legs: preemption can interleave only
/// while the loaded value is 0 (IRQ-return preempts only at count==0,
/// and the store has not landed); a preemptible task is only switched
/// in with count 0, so the migrated store — addressing through the
/// LIVE pinned register — writes 1 on the NEW CPU, which is correct
/// there; if the loaded value is >0, preemption is impossible
/// mid-window; same-CPU IRQ nesting restores pairwise.
/// System-register legs: the realization is IRQ-masked; no window.
/// Plain u32 RMW.
pub fn preempt_count_inc();
/// `preempt_count -= 1`; if the count reaches 0, the CPU is fully
/// preemptible (`irq_count == 0 && softirq_count == 0`) and the
/// per-CPU `need_resched` flag is set, invoke `schedule()`
/// ([Section 7.1](07-scheduling.md#scheduler--schedule-the-dispatch-loop)). This is
/// the standard `preempt_enable()` epilogue used by guard drops
/// (RcuReadGuard, PreemptGuard). Debug builds assert the count does
/// not underflow. Migration-window closure: pinning-counter argument —
/// mid-window preemption is impossible (the count is >= 1 throughout
/// the load/modify/store window).
pub fn preempt_count_dec_and_test_resched();
/// `rcu_nesting += 1`; returns the NEW nesting depth (1 = outermost
/// read-side critical section). Caller must already have preemption
/// disabled (`rcu_read_lock()` increments `preempt_count` first).
/// Migration-window closure: caller-pinned.
pub fn rcu_nesting_inc() -> u32;
/// `rcu_nesting -= 1`; returns the NEW nesting depth (0 = fully
/// exited — the caller then marks the quiescent hint). Debug builds
/// assert the count does not underflow. Caller still holds the
/// preemption pin taken at `rcu_read_lock()`. Migration-window
/// closure: caller-pinned.
pub fn rcu_nesting_dec() -> u32;
/// `rcu_passed_quiesce.store(v)` — set/clear the deferred quiescent
/// hint flag ([Section 3.4](#cumulative-performance-budget)). Release ordering
/// (free on x86-64 TSO; one fence on weakly-ordered architectures)
/// so RCU-protected reads cannot be reordered past the guard drop.
/// Context: preemption disabled (called from the RCU guard drop
/// before preemption is re-enabled); the store therefore targets the
/// owning CPU on every leg, and the unmasked base-fetch+store pair is
/// sufficient on the system-register legs.
pub fn set_rcu_passed_quiesce(v: bool);
/// Read this CPU's `CpuLocalBlock.cpu_id` (~1-3 cycles, register-
/// relative load). Safe from any kernel context.
///
/// **Migration-tolerance contract**: the value names the CPU the caller
/// was executing on AT THE INSTANT of the read; unless the caller is
/// pinned (holds a `PreemptGuard`, runs in IRQ context, or is a
/// CPU-bound kthread), it may be stale the moment preemption re-enables
/// — the same caveat pattern as `this_rq()` above. Callers that must
/// ACT on the CPU the id names hold a `PreemptGuard` and use
/// `PreemptGuard::cpu_id()` ([Section 3.1](#rust-ownership-for-lock-free-paths)),
/// which delegates here with the pin as a type-level witness.
///
/// This is the CANONICAL current-CPU-id seam. The two sanctioned
/// wrappers are `PreemptGuard::cpu_id()` (pinned, returns a table
/// index) and the scheduler chapter's `current_cpu() -> CpuId`
/// ([Section 7.1](07-scheduling.md#scheduler)). Spec pseudocode that calls an unqualified
/// `cpu_id()` refers to this accessor.
pub fn cpu_id() -> u32;
}
The accessor set is deliberately small: a field earns a cpu_local:: wrapper
only when a hot path performs the same compound operation at ≥2 call sites.
One-off field reads use the typed field-scoped accessor surface (value-returning
for scalar/pointer fields, &'static AtomicXX projections for atomic fields) or
the offset-primitive family directly — NEVER a whole-block reference (get() is
deleted, ESC-0431). The economy rule is WAIVED for that surface: a one-off field
accessor is always preferable to a whole-block reference.
CpuLocal vs PerCpu — when to use which:
| Criterion | CpuLocal | PerCpu\<T> |
|---|---|---|
| Access cost | ~1-10 cycles (arch-dependent) | ~3-5 cycles (release) / ~20-30 cycles (debug) |
| Borrow checking | None (structural safety) | Debug-only CAS (see Section 3.3) |
| Data types | Fixed scalars/pointers only | Any T |
| Number of fields | ~10 (fixed at compile time) | Unlimited (one PerCpu<T> per data type) |
| Adding new fields | Requires CpuLocalBlock layout change |
Just declare a new PerCpu<T> |
| Interrupt safety | Inherent (no borrow state to corrupt) | get_mut() disables IRQs |
| Use case | current_task, runqueue, preempt_count, slab magazines | Per-CPU page pools (Section 4.2), stats, driver state |
Slab magazines use CpuLocal (~1-10 cycles) because slab is the hottest allocation
path. PCP page lists use PerCpu<T> because page allocation is less frequent and
PcpPagePool is too large for CpuLocalBlock (see Section 4.2).
3.2.2 Initialization Sequence¶
Allocation — runtime-discovered, no compile-time MAX_CPUS: storage falls into exactly TWO lifetimes:
- Bootstrap block (link time):
/// Bootstrap per-CPU block: used by the BSP from the first Rust instruction
/// until `bsp_handoff()`. ONE block — a fixed cost, not a CPU-count assumption.
/// `#[unsafe(no_mangle)]` is REQUIRED: entry stubs address this symbol from
/// assembly (ppc64le step 2c forms r13 via CPU0_BOOT_LOCAL_BLOCK@toc@ha/@toc@l);
/// a mangled private static cannot be named there — the symbol name is part of
/// the entry.S<->Rust boot ABI seam, and any leg's entry stub may reference it.
#[unsafe(no_mangle)]
#[link_section = ".cpulocal"]
pub static mut CPU0_BOOT_LOCAL_BLOCK: CpuLocalBlock = CpuLocalBlock::BOOT_ZERO;
The bootstrap block is a bare CpuLocalBlock — no export or transit pages
exist before the bundle handoff (bsp_handoff() below), and the owning-CPU
page accessors are
illegal until it completes (every specified caller is Phase-1+ by
construction: the earliest is the transit.core_image bring-up write —
arm_transit_core_image() — which targets the bundle).
2. Per-CPU bundle spine (boot time): after ACPI MADT / device-tree CPU
enumeration, the boot allocator provides ONE dense, page-aligned,
zero-filled slice of num_possible_cpus() CpuLocalSlot elements — the
spine — published through a single BootOnceCell in the leaked-boot
allocation discipline (never freed or moved):
CPU_LOCAL_BUNDLES: BootOnceCell<&'static mut [CpuLocalSlot]>
/// One spine slot per POSSIBLE CPU. The `bundle` pointer is null until
/// that CPU is first brought online — or, for boot-started CPUs, populated
/// eagerly at boot — at which point `populate_cpu_storage(cpu_id)`
/// Release-stores the node-local bundle pointer. Populate-once: a slot is
/// written exactly once and never cleared, so offline and every re-online
/// are pure state-machine transitions, never allocation events. A slot
/// that stays null forever (a possible-but-never-onlined CPU) is a DEFINED
/// state — the any-context projections return `None`.
// kernel-internal, not KABI
#[repr(C)]
pub struct CpuLocalSlot {
/// Node-local `CpuLocalBundle` for this CPU, or null before first
/// online. Written once via `AtomicPtr::store(_, Release)` by
/// `populate_cpu_storage`; read `Acquire` by the any-context
/// projections so the store publishes the fully-initialized bundle.
bundle: AtomicPtr<CpuLocalBundle>,
}
const_assert!(core::mem::size_of::<CpuLocalSlot>() == core::mem::size_of::<*mut CpuLocalBundle>());
The spine costs one pointer per possible CPU (32 KiB at 4096 possible),
never the full bundle footprint: the bundles themselves are allocated
lazily and node-locally by populate_cpu_storage (below), so a wide
maxcpus= possible mask on an 8-vCPU guest or an edge device pays only the
pointer spine, not the tens-to-hundreds of MiB of unpopulated per-CPU bulk
a dense-at-boot bundle array would waste. Footprint adapts to the
populated topology while num_possible_cpus() stays the mandatory spine
length (ESC-0412 SEAM-B, as amended by Gavel-B g5-enum-seam: the count
resolves at canonical Phase 0.65, the Possible-CPU inventory, which runs
before this Phase-0.7 spine is sized — Section 2.3). Each
populated slot points at a CpuLocalBundle — this CPU's
three per-CPU pages, layout unchanged from ESC-0417:
/// Per-CPU page bundle: this CPU's three per-CPU pages, contiguous at
/// fixed offsets from the per-CPU base register (which points at
/// `block`, offset 0). The owning-CPU accessors depend ONLY on the
/// intra-bundle offsets: address formation is register + constant
/// arithmetic — never a memory read of the Core-keyed block page and
/// never a `BootOnceCell` load — so it is legal under any isolation
/// image. The any-context path reaches each bundle through the spine's
/// per-CPU `AtomicPtr` — the pointer table IS the model (no longer a
/// future republish): `populate_cpu_storage` allocates each CPU's bundle
/// NODE-LOCAL at first online, so bundles are never inter-bundle
/// contiguous and no huge single-node contiguous allocation is required.
/// The owning-CPU accessors are unaffected — they never load the spine,
/// only register base + fixed intra-bundle offset.
// kernel-internal, not KABI
#[repr(C, align(4096))] // alignment FLOOR; member page-alignment derives
// from the allocator's page-aligned base plus the
// page-multiple stride (const_asserted below)
pub struct CpuLocalBundle {
pub block: CpuLocalBlock, // Core key
_pad_block: [u8; CPU_LOCAL_PAGE - size_of::<CpuLocalBlock>()],
pub export: CpuLocalExport, // shared-RO key
_pad_export: [u8; CPU_LOCAL_PAGE - size_of::<CpuLocalExport>()],
pub transit: CpuLocalTransit, // shared-RW infra key
_pad_transit: [u8; CPU_LOCAL_PAGE - size_of::<CpuLocalTransit>()],
}
/// Arch page size for bundle layout (compile-time per arch).
pub const CPU_LOCAL_PAGE: usize = 1 << arch::current::mm::PAGE_SHIFT;
pub const CPU_LOCAL_EXPORT_OFFSET: usize = CPU_LOCAL_PAGE;
pub const CPU_LOCAL_TRANSIT_OFFSET: usize = 2 * CPU_LOCAL_PAGE;
pub const CPU_LOCAL_BUNDLE_STRIDE: usize = 3 * CPU_LOCAL_PAGE;
const_assert!(core::mem::offset_of!(CpuLocalBundle, export) == CPU_LOCAL_EXPORT_OFFSET);
const_assert!(core::mem::offset_of!(CpuLocalBundle, transit) == CPU_LOCAL_TRANSIT_OFFSET);
const_assert!(core::mem::size_of::<CpuLocalBundle>() == CPU_LOCAL_BUNDLE_STRIDE);
const_assert!(CPU_LOCAL_BUNDLE_STRIDE % CPU_LOCAL_PAGE == 0);
The pad arrays underflow at compile time if a member outgrows its page —
the same guard the per-struct size_of ≤ 4096 asserts provide, now
page-size-exact. (If the corpus already defines a canonical page-size
constant, bind CPU_LOCAL_PAGE to it rather than redefining; umka-lint
arbitrates.)
This is a BootOnceCell, not a bare OnceCell: the ambient
core::cell::OnceCell is !Sync for every T, so a bare static OnceCell
does not satisfy the Sync bound on statics and will not compile;
BootOnceCell (§Write-Once Boot Publication Cell in Section 2.3)
is the single-writer boot-publication primitive whose inline
UnsafeCell<MaybeUninit<T>> storage carries the fat &'static mut [CpuLocalSlot]
spine — the exclusive-ownership token surrendered into the leaked-boot cell.
No MAX_CPUS constant exists — consistent with the scheduler's RUNQUEUES
(Section 7.1) and the
num_possible_cpus() per-CPU-array rule in
Section 7.3. The spine
cell is published via CpuLocal::publish_bundles(spine), which surrenders
the &'static mut [CpuLocalSlot] through BootOnceCell::set() in this same
boot step — the slots start null; each CPU's bundle pointer is
Release-stored into its slot later by populate_cpu_storage (§Publication
seam below), always before any per-CPU register is repointed at that
bundle's block. This publication is the seam the any-context accessors read:
CpuLocal::transit_of(cpu_id) and CpuLocal::export_of(cpu_id) perform one
Acquire load of CPU_LOCAL_BUNDLES[cpu_id].bundle and, when it is
non-null, project the bundle's .transit / .export page — a null slot (a
possible-but-never-onlined CPU) yields None. The owning-CPU transit() /
export() accessors reach the same per-CPU page via the register base plus
the fixed intra-bundle offsets (CPU_LOCAL_TRANSIT_OFFSET /
CPU_LOCAL_EXPORT_OFFSET), not through the BootOnceCell. Each CPU arms its own transit.core_image via
arm_transit_core_image() during its bring-up (the BSP once between
domain-image-table init and the first possible non-Core execution on the
BSP — the named boot step "CpuLocal transit arming"; each AP for itself in
its init step 4) — before any code on that CPU can execute under a non-Core
image. Its same-CPU functional reader (the entry-stub establishment
compare) therefore observes the armed value by program order; a cross-CPU
diagnostic reader may observe the unarmed 0 during the boot window and must
treat 0 as unarmed (core_image soundness rests on interior mutability,
not a write-once discipline — see the field doc). Key
tagging (shared-RO for export, shared-RW for transit) is applied when the
isolation subsystem initializes its keys, before the first Tier 1 domain is
created.
Publication seam. The module-private CPU_LOCAL_BUNDLES cell is written,
its slots populated, and its per-CPU block addresses formed, through exactly
three functions — never a pub static:
/// Surrender exclusive ownership of the per-CPU bundle *spine* into the
/// module-private `CPU_LOCAL_BUNDLES` cell. Called EXACTLY ONCE, in the
/// CpuLocal BSP Init boot step (canonical Phase 0.7). The spine slots are
/// null at publication — each CPU's bundle is materialized later by
/// `populate_cpu_storage`. Moving the `&'static mut [CpuLocalSlot]` through
/// `BootOnceCell::set()` IS the soundness argument for the transfer: the
/// exclusive-ownership token is consumed, after which the spine is reachable
/// only through the module-private accessors, and the slots' `AtomicPtr`
/// interior mutability carries every subsequent populate. A second call is a
/// boot bug — `BootOnceCell`'s single-writer discipline is debug-asserted
/// (`set()` on an already-set cell panics in debug builds).
///
/// (Reconciliation note: the ESC-0419 seam is preserved — same function name,
/// same one-shot private-cell publication role, same `&'static mut [_]`
/// ownership-token shape; only the element type moves from `CpuLocalBundle`
/// to the spine slot `CpuLocalSlot`, because the cell now stores the spine.)
pub fn publish_bundles(spine: &'static mut [CpuLocalSlot]);
/// Materialize CPU `cpu_id`'s per-CPU storage on its FIRST online, and no
/// other time. Runs under `CPU_HOTPLUG_LOCK`, BEFORE the CPU's online bit is
/// set ([Section 7.3](07-scheduling.md#context-switch-and-register-state--cpu-hotplug-integration)); it
/// is also called eagerly at boot for every boot-started CPU (BSP plus the
/// APs actually woken). Steps, in order:
/// 1. If `CPU_LOCAL_BUNDLES.get()[cpu_id].bundle.load(Acquire)` is non-null
/// the CPU was already populated (a re-online) — return immediately; no
/// re-allocation.
/// 2. Allocate this CPU's 3-page `CpuLocalBundle` NODE-LOCAL (on the NUMA
/// node the CPU belongs to), plus its bulk per-CPU storage — runqueue
/// (the scheduler's `RUNQUEUES` spine slot, [Section 7.1](07-scheduling.md#scheduler)), slab
/// magazine depot, escalation-envelope slots — all node-local. These are
/// leaked-boot / leaked-hotplug allocations: never freed (populate-once).
/// 3. Initialize the bundle's block/export/transit fields and self-mirrors
/// — including `export.transit_ptr` (= the bundle's transit page,
/// written at the same write-once instant as the page's `self_ptr`) —
/// and set `block.runqueue` to the freshly allocated runqueue; apply the
/// isolation key tagging (shared-RO export, shared-RW transit).
/// `transit.core_image` is NOT armed here — at BSP Phase 0.7 the domain
/// image table does not yet exist; arming is the separate, later
/// `arm_transit_core_image()` seam by construction.
/// 4. `Release`-store the bundle pointer into
/// `CPU_LOCAL_BUNDLES.get()[cpu_id].bundle`, publishing the fully
/// initialized bundle to any-context readers.
/// The Release/Acquire pair (store here, load in `block_base_of` /
/// `transit_of` / `export_of`) is the visibility argument: a reader that sees
/// a non-null pointer sees a fully-initialized bundle.
pub fn populate_cpu_storage(cpu_id: u32);
/// Raw base address of CPU `cpu_id`'s `CpuLocalBlock`, obtained by one
/// `Acquire` load of the spine slot's bundle pointer and offset 0 (the block
/// is the bundle's first page). Returns a `*mut CpuLocalBlock` — NEVER a
/// reference: forming the pointer does not read the block page itself, so it
/// is legal in ANY Core context, including AP bring-up and the s390x
/// lowcore-image pre-store (target CPU not yet running). PRECONDITION:
/// `populate_cpu_storage(cpu_id)` has run (the slot is non-null) — this
/// accessor is called only to program that CPU's own base register during its
/// bring-up, which the online sequence sequences after populate. FOR per-CPU
/// base-register / lowcore-slot programming ONLY; the returned pointer is
/// never dereferenced cross-CPU (see the Publication and visibility contract
/// below).
pub fn block_base_of(cpu_id: u32) -> *mut CpuLocalBlock;
// SAFETY: sound ONLY under the Publication and visibility contract below — the
// cell stays module-private and no `&CpuLocalBundle` ever escapes, so the only
// cross-CPU sharing reaches the atomic-only transit/export pages while every
// non-atomic block field is mutated solely by its owning CPU through the
// register base. Each spine slot holds an `AtomicPtr<CpuLocalBundle>` (itself
// `Send + Sync`), and `CpuLocalBundle` holds raw pointers (the pages'
// `self_ptr` mirrors) that are neither `Send` nor `Sync` by default; the
// bundle is shared cross-CPU by publication through that `AtomicPtr`, so it
// still requires these impls to be shared soundly. This rests on the
// struct-evolution field invariant (Publication and visibility contract
// below): every `CpuLocalTransit` / `CpuLocalExport` field is either an
// atomic/interior-mutable cell or written before the bundle's
// `Release`-publication and immutable thereafter, so the `self_ptr` mirrors
// (category (b)) are never mutated once any reader can observe the slot and
// the atomic-only cross-CPU sharing stated above holds as written.
unsafe impl Send for CpuLocalBundle {}
unsafe impl Sync for CpuLocalBundle {}
Publication and visibility contract (normative):
CPU_LOCAL_BUNDLESMUST stay module-private.publish_bundlesis its only writer;populate_cpu_storageis the only writer of a slot'sbundlepointer;block_base_of,transit_of, andexport_ofare its only readers.- No API may EVER return
&CpuLocalSlot(which would expose the rawbundleAtomicPtrfor cross-CPU dereference), nor&CpuLocalBundle, nor a&CpuLocalBlockfor any CPU other than the caller's own. A bundle'sblockpage is reachable exactly two ways: through the owning CPU's per-CPU base register (the owning-CPU accessors), or as the rawblock_base_ofpointer used solely to program a base register / lowcore slot. The cross-CPU projectionstransit_of/export_ofreach exclusively the atomic-only transit / export pages — never a cross-CPU&block. - Those two rules are precisely the SAFETY obligation discharged by the
mandatory
unsafe impl Send for CpuLocalBundleandunsafe impl Sync for CpuLocalBundleabove. The impls are sound ONLY while (1) the cell stays private and (2) no&CpuLocalBundleescapes; under those conditions the only shared cross-CPU access is to the atomic-only transit/export pages, every non-atomic block field is mutated solely by its owning CPU through the register base, and no aliasing&/&mutpair over a block ever exists. pub static CPU_LOCAL_BUNDLESis NAMED-AND-REJECTED. A public static hands any caller&CPU_LOCAL_BUNDLES.get()[i](a&CpuLocalSlot), from which.bundle.load(Acquire)yields the*mut CpuLocalBundleand hence a cross-CPU&CpuLocalBlock: (a) aliasing UB against the owning CPU's register-based mutation of that block; (b) it invalidates the mandatoryunsafe impl Send/Sync, whose soundness rests on rule (2); and (c) per the ESC-0417 Core-keyed block-page ruling, dereferencing that reference under a non-Core (driver) image FAULTS (Section 11.2). The function seam makes the unsound access unrepresentable; a public static would make it the default.- Struct-evolution field invariant. Every field of
CpuLocalTransitandCpuLocalExportMUST be either (a) an atomic / interior-mutable cell, or (b) written only BEFORE the bundle'sRelease-publication (populate_cpu_storagestep 3, published by the step-4Release-store) and immutable thereafter (pre-publication-immutable). Appending a field under the CpuLocalBlock evolution protocol that satisfies neither is a defect. This invariant is the soundness basis of BOTHtransit()'s andtransit_of()'s&'staticreturns (and theirexport()/export_of()counterparts): category (a) fields may be mutated through a live shared reference — e.g.core_imagearmed post-publication viaarm_transit_core_image(), whose interior-mutablestore(Relaxed)is legal while a cross-CPU&'static CpuLocalTransitis live; category (b) fields — the pages'self_ptrmirrors andexport.transit_ptr— are never mutated after any reader can observe the slot, so a plain read through the returned reference races nothing. A whole-struct shared reference over the page is therefore sound regardless of which fields a given consumer touches.
Stream ownership. publish_bundles, populate_cpu_storage,
block_base_of, bsp_handoff, arm_transit_core_image, and the
AP_BOOT_CELLS handshake cells are CL-stream implementation units (code home:
the generic sync/cpulocal half); cpu_local_install_base is a CL-stream
unit in the per-architecture arch/<arch>/cpulocal.rs half (all 8 legs). The
Phase 0.7 CALL SITE — the canonical three-call sequence
CpuLocal::publish_bundles(spine) → CpuLocal::populate_cpu_storage(0) →
CpuLocal::bsp_handoff() — is a
boot/S0-stream unit, specified per-arch in the boot-sequence sections. The
hotplug CALL SITE of populate_cpu_storage(N) (first online of a
beyond-boot-present CPU) lives in the scheduler online sequence
(Section 7.3).
BSP handoff (bootstrap block → CPU 0 bundle): the tail of the canonical
Phase 0.7 three-call sequence every boot file's CpuLocal BSP Init phase
expresses (Section 2.3) —
CpuLocal::publish_bundles(spine) → CpuLocal::populate_cpu_storage(0) →
CpuLocal::bsp_handoff(). publish_bundles(spine) surrenders the spine into
the private cell (§Publication seam above); populate_cpu_storage(0)
allocates CPU 0's bundle node-local, initializes its
export.self_ptr = &bundle0.export, export.transit_ptr = &bundle0.transit,
and transit.self_ptr = &bundle0.transit write-once
mirrors, and Release-stores the bundle pointer into CPU_LOCAL_BUNDLES[0].
The handoff itself is a formal unit:
/// Bootstrap->bundle handoff for the BSP (tail of canonical Phase 0.7).
/// PRECONDITIONS: `publish_bundles` and `populate_cpu_storage(0)` have run;
/// IRQs disabled; no CpuLocal-derived reference or pointer is live. Copies the
/// bootstrap block's fields into CPU 0's block page (via `block_base_of(0)`),
/// excluding `self_ptr`, which is set to that block's own address; then installs
/// the base via `arch::current::cpu::cpu_local_install_base(block_base_of(0))`.
/// The bootstrap block is dead after return (debug builds poison it).
pub unsafe fn bsp_handoff();
The base install inside bsp_handoff() — and in every later bring-up that
programs a per-CPU base — is the arch primitive:
/// Program THIS CPU's per-CPU base mechanism (boot/bring-up only; never hot).
/// One realization per leg: wrmsr IA32_GS_BASE / msr TPIDR_EL1 / mcr TPIDRPRW /
/// mv tp / mtspr SPRG3 / mr r13 / move $r21 / store to own lowcore
/// LC_CPU_LOCAL_BASE. s390x AP note: the AP-side install is a no-op — SIGP
/// SET_PREFIX already installed the pre-stored slot; only the BSP calls this leg.
pub unsafe fn cpu_local_install_base(base: *mut CpuLocalBlock);
APs never use the bootstrap block — they are started only
after the spine exists and their bundles are populated, receiving their block
base via the AP_BOOT_CELLS handshake (below), so no AP ever observes the
handoff.
BSP initialization (Phase 0, before heap):
- Zero
CPU0_BOOT_LOCAL_BLOCK(BSS guarantees this; explicitly documented for clarity). - Point the architecture register at
&CPU0_BOOT_LOCAL_BLOCK: - x86-64: store
&CPU0_BOOT_LOCAL_BLOCKintoCPU0_BOOT_LOCAL_BLOCK.self_ptr, thenWRMSR MSR_GS_BASEwith the same address (the block is self-describing the instant GS points at it) - AArch64:
MSR TPIDR_EL1, <addr> - ARMv7:
MCR p15, 0, <addr>, c13, c0, 4(TPIDRPRW) - RISC-V:
mv tp, <addr>thenCSRW sscratch, 0(tp = per-CPU base, sscratch = 0 = kernel mode) - PPC64:
r13is the dedicated thread pointer; load&CPU0_BOOT_LOCAL_BLOCKinto r13 before any kernel Rust code executes - PPC32:
mtspr SPRG3, <addr>— SPRG3 (SPR 275; SPR 259 is the user-mode read-only alias USPRG3/SPRG3R, never anmtsprtarget) is the PPC32 per-CPU data pointer that the fast-pathmfspr reg, SPRG3reads, so the bootstrap block's base is installed into the SAME register the runtime accessor uses (the x86-64WRMSR MSR_GS_BASEanalogue — no separate scratch register, so no bootstrap-window access can deref an uninitialized base).r2is reserved-unused andr13is an ordinary nonvolatile register — NEVER the per-CPU pointer (Section 2.10);SPRG0-SPRG2are reserved for exception-handler scratch and are never used for CpuLocal. - s390x:
stgthe address ofCPU0_BOOT_LOCAL_BLOCKtoLC_CPU_LOCAL_BASE(0x340) in the BSP lowcore. The BSP's prefix is 0 from the entry stub'sspx, so the absolute store lands in the BSP lowcore;cpu_local_block()then reads it withlg reg, LC_CPU_LOCAL_BASE. - LoongArch64:
move $r21, <addr>(runtime per-CPU base), thencsrwr <scratch>, KS3(0x33) writing the identical address —$r21is the runtime base, KS3 the trap-entry copy consumed only on user-origin exception entry; both written once here, never rewritten after init.
On every architecture, this step also sets the block's self_ptr field to
the block's own address (consumed on x86-64; debug value elsewhere).
3. Set cpu_id = 0, preempt_count = 0, current_task = &idle_task[0].
4. runqueue, irq_count, and slab_magazines remain zeroed (valid default).
BSP Phase 2 (after slab init):
- Initialize slab magazines for CPU 0: call
slab_init_cpu_magazines(0).
AP initialization (during SMP bring-up, per AP n):
The handshake state is one definition, all legs — two formal units:
/// AP bring-up handshake cells — one per possible CPU. Provenance (stated once,
/// here): allocated from the buddy allocator at SMP-bringup setup (Phase-11
/// activation, after `BootAlloc` retires at the Phase-1.1 hand-off), sized to
/// the Phase-0.65 possible-CPU count (`num_possible_cpus()`), and published
/// before SMP bring-up through a `BootOnceCell` (the same publication primitive
/// as CPU_LOCAL_BUNDLES; never a compile-time CPU cap). Each cell carries ONLY
/// the BSP→AP CpuLocal delivery word — the online-completion signal is the AP's
/// atomic `CPU_HOTPLUG.online_mask` set + `online_count` increment
/// ([Section 2.3](02-boot-hardware.md#boot-init-cross-arch)), the sole online authority, so no separate
/// ready latch exists.
/// Kernel-internal, not KABI.
pub struct ApBootCell {
/// AP's CpuLocalBlock base: BSP `store(Release)` before wake (protocol
/// step 1); AP `load(Acquire)` in its trampoline (step 3). Null until
/// step 1. (s390x: unused — the lowcore pre-store replaces it.)
pub cpulocal: AtomicPtr<CpuLocalBlock>,
}
pub static AP_BOOT_CELLS: BootOnceCell<&'static [ApBootCell]>;
/// Arm THIS CPU's `transit.core_image` from the domain image table's Core
/// entry. The arming write is
/// `CpuLocal::transit().core_image.store(image, Ordering::Relaxed)` — a
/// `Relaxed` store to the `CpuLocalU64` cell THROUGH the covering shared
/// reference, never a raw-pointer write path; interior mutability makes it
/// legal even while a cross-CPU `&'static CpuLocalTransit` from `transit_of`
/// is live (see the `core_image` field doc). On x86-64 the same step also
/// writes the sibling U=0 entry-landing mirror (unchanged). PRECONDITIONS:
/// caller pinned (bring-up context); the Core entry exists. Call sites: each AP
/// for itself at protocol step 4, and the BSP once between domain-image-table
/// init and the first possible non-Core execution on the BSP (named boot step:
/// 'CpuLocal transit arming').
pub unsafe fn arm_transit_core_image();
(On x86-64 the arming step also writes the sibling U=0 entry-landing
core_image mirror at the same instant — the pair never diverges; see the
core_image field doc.)
Before step 1, the BSP calls
CpuLocal::populate_cpu_storage(n)(eager boot population — every AP actually woken is a boot-started CPU), which allocates APn's bundle node-local andRelease-stores the pointer intoCPU_LOCAL_BUNDLES[n]. This is what makesCpuLocal::block_base_of(n)in the steps below resolve to a non-null bundle.
- BSP writes the AP's
CpuLocalBlockbase intoAP_BOOT_CELLS[n].cpulocalwithstore(Release)ordering (a per-AP cell in the boot-allocatedAP_BOOT_CELLSslice, published before SMP bring-up; the arch trampoline relays as needed until the AP can address kernel virtual memory). (s390x does not use the cell'scpulocalword: the BSP instead pre-storesCpuLocal::block_base_of(n)atLC_CPU_LOCAL_BASE(0x340) into the AP's lowcore IMAGE beforeSIGP SET_PREFIXin step 2 — the slot arrives with the prefix, so no AP-side store is needed. See Section 2.12.) - BSP sends the platform wake signal:
- x86-64: INIT-SIPI-SIPI sequence to the AP's local APIC ID
- AArch64/ARMv7:
PSCI CPU_ON(cpu_id, entry, context_id)via HVC/SMC - RISC-V:
SBI HSM_HART_START(hartid, start_addr, opaque) - PPC32 (e500): ePAPR spin-table release — the BSP writes the AP entry
point physical address to the AP's
cpu-release-addrslot (addr_lo; value 1 = held), thendcbf+syncto publish; the AP's firmware spin loop observes the store and branches to the entry point — a memory store, not an interrupt (see Section 2.10 for the spin-table format and release sequence) - PPC64:
RTAS start-cpuor direct OPAL call - s390x:
SIGP SET_PREFIX (0x0D)to install the AP's lowcore, thenSIGP RESTART (0x06)to start it at its restart PSW (see Section 2.12 for the full five-step sequence) - LoongArch64: IOCSR mailbox + IPI doorbell — the BSP writes the AP entry
point physical address to the AP's IOCSR mailbox, then raises the AP's
boot IPI via the IOCSR IPI-send doorbell (
IOCSRWR); the AP's firmware wait loop reads the mailbox and jumps to the entry point (see Section 2.13 for the full BSP/AP sequence) - AP trampoline:
load(Acquire)fromAP_BOOT_CELLS[n].cpulocalto get its block address; installs it as its own per-CPU base viaarch::current::cpu::cpu_local_install_base(the same per-leg install primitive the BSP handoff uses). (s390x: no load/store here, and the install leg is a no-op — the register-init is complete the instantSIGP SET_PREFIX(step 2) installs the lowcore; the AP's firstlg reg, LC_CPU_LOCAL_BASEafter RESTART reads the BSP-pre-stored pointer.) - AP writes
cpu_id = n,preempt_count = 0,current_task = &idle_task[n], and arms its owntransit.core_imageviaarm_transit_core_image()(resolved from the domain image table's Core entry, which exists before SMP bring-up) — the AP-side half of thetransit.core_imagebring-up arming specified in the allocation paragraph above, executed before any code on this CPU can run under a non-Core image. - AP initializes its own per-CPU slab magazines:
slab_init_cpu_magazines(n). (AP owns its CpuLocalBlock — BSP must never write to a remote AP's block.)
The CpuLocal handshake ends here; the AP continues its broader init and reports
online by the atomic CPU_HOTPLUG.online_mask set + online_count increment
(AP init step f, Section 2.3) — the single completion signal.
There is NO per-AP BSP spin: the BSP tracks bring-up through
CPU_HOTPLUG.online_count against the global 30 s deadline (the deadline-bounded
binary fan-out model, Section 2.3), and a CPU that misses the
deadline is marked offline (reduced-CPU operation is valid), never hangs boot.
Invariant: No kernel code may access CpuLocal::* before step 3 completes on that
CPU. The preempt_count field reads as zero by BSS convention even before step 3, but
only the owning CPU may write to its own block. Cross-CPU writes to another CPU's block
are never permitted; the only cross-CPU interaction is the AP_BOOT_CELLS[n].cpulocal
handshake (write by BSP before wakeup, read by AP during trampoline) — and, on
s390x, the equivalent BSP pre-store of the CpuLocalBlock base into the AP's
lowcore IMAGE at LC_CPU_LOCAL_BASE before SIGP SET_PREFIX, which is likewise a
write to the AP's slot/image before that CPU runs, not a write to a live CPU's
block (the same sanction class).
For the complete kernel init phase ordering across all subsystems, see the Kernel Init Phase Reference table in Section 2.3.
3.3 PerCpu Borrow Checking: Debug-Only in Release Builds¶
The PerCpu<T> borrow-state CAS (Section 3.1) serves as a runtime bug
detector, not a safety mechanism. The actual safety guarantee comes from
the structural invariants:
get()requires&PreemptGuard→ preemption disabled → CPU pinned.get_mut()requires&mut PreemptGuard+ disables IRQs → exclusive access.- Therefore: if the caller follows the API contract (one guard per critical section), aliased access is structurally impossible.
Note: PCP page pools (Section 4.2) require IRQs disabled (not just preemption disabled) because interrupt handlers may access PCP pools. This is why
get_mut()unconditionally callslocal_irq_save()— preemption-disable alone is insufficient for data structures accessed from hardirq context. (local_irq_save()/local_irq_restore()and theIrqDisabledGuardproof token are defined in Section 3.8.)
The CAS detects violations of rule (3) — e.g., creating two PreemptGuards
and using both to obtain &mut T. This is a logic error in the caller, not a
race condition. In release builds, this class of bug should have been caught
during development and testing.
Design decision: In release builds (cfg(not(debug_assertions))), the
borrow-state CAS is replaced with a no-op. The borrow_state array is still
allocated (for binary compatibility with debug modules), but get() and
get_mut() skip the atomic operations:
impl<T> PerCpu<T> {
pub fn get_mut<'g>(&self, guard: &'g mut PreemptGuard) -> PerCpuMutGuard<'g, T> {
let cpu = guard.cpu_id();
// SAFETY: local_irq_save() MUST be called BEFORE updating borrow_state.
// See Section 3.1.1 for the full rationale.
let saved_flags = local_irq_save();
#[cfg(debug_assertions)]
{
// Full CAS borrow-state checking — catches aliasing bugs.
let state = self.borrow_state(cpu);
if state.compare_exchange(0, u32::MAX, Ordering::Acquire, Ordering::Relaxed).is_err() {
local_irq_restore(saved_flags);
panic!("PerCpu: slot {} already borrowed", cpu);
}
}
// SAFETY: PreemptGuard proves CPU is pinned. local_irq_save()
// prevents interrupt handler interference. In debug builds, the
// CAS above additionally verifies no aliased borrows exist.
// In release builds, we trust the structural invariants.
unsafe {
PerCpuMutGuard {
value: &mut *self.data.add(cpu).as_ref().unwrap().get(),
saved_flags,
#[cfg(debug_assertions)]
borrow_state: self.borrow_state(cpu),
_guard: PhantomData,
}
}
}
}
impl<'a, T> Drop for PerCpuMutGuard<'a, T> {
fn drop(&mut self) {
#[cfg(debug_assertions)]
{
self.borrow_state.store(0, Ordering::Release);
}
local_irq_restore(self.saved_flags);
}
}
Release-mode cost: get_mut() = CPU ID lookup (~1-3 cycles) + array
index + local_irq_save/local_irq_restore (~5-10 cycles total).
Approximately ~3-8 cycles per access, down from ~20-30 with the CAS.
3.3.1 IRQ Save/Restore Elision: get_mut_nosave()¶
The get_mut() function unconditionally calls local_irq_save() and
local_irq_restore() to guarantee exclusive access. On x86-64, this costs
~5-10 cycles (pushfq/cli + popfq). However, many hot paths already
have IRQs disabled at the call site:
- Hardirq handlers: IRQs disabled by hardware on entry.
- Softirq context: IRQs disabled during
do_softirq()execution. - Spinlock holders:
SpinLock::lock()disables IRQs before acquiring. - Context switch path:
schedule()disables IRQs around runqueue locking.
In these contexts, the IRQ save/restore is redundant — IRQs are already off.
UmkaOS provides a proof-token variant that elides the redundant operation.
This is a core design decision, not a deferred optimization — the
IrqDisabledGuard token is woven into the type system from day one.
The two primitive operations — local_irq_save() / local_irq_restore() — and
the IrqDisabledGuard proof token they produce form the
arch::current::interrupts local-IRQ seam, now defined in
Section 3.8 (the
Local Interrupt Save/Restore subsection). This section only uses that seam:
get_mut_nosave() below consumes an IrqDisabledGuard produced there.
The PerCpu<T> variant that accepts the proof token:
/// Guard returned by `PerCpu::get_mut_nosave()`.
///
/// Holds a mutable reference to the per-CPU value. Unlike `PerCpuMutGuard<T>`,
/// this guard does NOT save/restore IRQ flags: the caller already holds an
/// `IrqDisabledGuard` proving IRQs are disabled, so saving/restoring them
/// again would be redundant (~5-10 cycles saved on x86-64 per access).
///
/// # Safety invariant
/// The caller must ensure `IrqDisabledGuard` remains live for the entire
/// lifetime `'a` of this guard. Dropping the `IrqDisabledGuard` while this
/// guard is alive would re-enable IRQs, allowing interrupt handlers to race
/// on the same per-CPU slot — undefined behaviour.
///
/// The `PhantomData<&'a IrqDisabledGuard>` field enforces this at the type
/// level: the borrow checker will not allow the `IrqDisabledGuard` to be
/// consumed (moved or dropped) while a `PerCpuMutRefNosave` derived from it
/// is still in scope.
///
/// # Drop behaviour
/// `Drop` only clears the debug-mode borrow state (sets the per-slot
/// `borrow_state` back to `0`). It does NOT call `local_irq_restore()` —
/// that is the caller's responsibility via their `IrqDisabledGuard`.
pub struct PerCpuMutRefNosave<'a, T: 'a> {
value: &'a mut T,
/// Proof that the caller holds an `IrqDisabledGuard` for lifetime `'a`.
_irq: PhantomData<&'a IrqDisabledGuard>,
/// Debug-builds only: reference to the per-slot borrow-state counter so
/// that `Drop` can reset it to `0` (free), allowing subsequent callers
/// to detect aliasing. Elided in release builds — the structural
/// invariants (IRQs disabled + preemption disabled) are sufficient.
#[cfg(debug_assertions)]
borrow_state: &'a AtomicU32,
/// Ties guard lifetime to the `PreemptGuard` (via `get_mut_nosave` signature).
_guard: PhantomData<&'a mut PreemptGuard>,
}
impl<'a, T> core::ops::Deref for PerCpuMutRefNosave<'a, T> {
type Target = T;
fn deref(&self) -> &T { self.value }
}
impl<'a, T> core::ops::DerefMut for PerCpuMutRefNosave<'a, T> {
fn deref_mut(&mut self) -> &mut T { self.value }
}
/// `PerCpuMutRefNosave` must NOT be sent to another CPU/thread.
/// The contained `&'a mut T` is a per-CPU slot reference; moving it to a
/// different thread would allow that thread to alias it without holding the
/// required `IrqDisabledGuard` or `PreemptGuard`.
impl<T> !Send for PerCpuMutRefNosave<'_, T> {}
impl<T> PerCpu<T> {
/// Like `get_mut()`, but skips `local_irq_save()`/`local_irq_restore()`.
/// The caller must prove IRQs are already disabled by providing an
/// `IrqDisabledGuard`. This saves ~5-10 cycles on x86-64 (no `pushfq`/
/// `cli`/`popfq`) and ~3-8 cycles on ARM (no `mrs`/`msr` DAIF).
///
/// # When to use
///
/// Use `get_mut_nosave()` instead of `get_mut()` when:
/// - Inside a hardirq or softirq handler (IRQs disabled by hardware).
/// - Holding a `SpinLockGuard` from `SpinLock::lock()`.
/// - In the context switch path (scheduler holds IRQ-disabling lock).
/// - In any code path where an `IrqDisabledGuard` is already available.
///
/// In all other contexts, use `get_mut()` which manages IRQ state itself.
///
/// # Why `&IrqDisabledGuard`, not `&PreemptGuard`
///
/// `IrqDisabledGuard` is a strictly stronger guarantee than `PreemptGuard`:
/// it subsumes preemption disabling (tick IRQ is masked) while also preventing
/// interrupt handlers from racing on the same per-CPU slot. `PreemptGuard`
/// alone would not be sufficient — an IRQ could fire between the CPU-id read
/// and the slot access, running a handler that mutably accesses the same slot.
pub fn get_mut_nosave<'g, 'irq>(
&self,
guard: &'g mut PreemptGuard,
_irq: &'irq IrqDisabledGuard,
) -> PerCpuMutRefNosave<'g, T>
where
'irq: 'g, // IrqDisabledGuard must outlive the returned reference
{
let cpu = guard.cpu_id();
#[cfg(debug_assertions)]
{
let state = self.borrow_state(cpu);
if state.compare_exchange(0, u32::MAX, Ordering::Acquire, Ordering::Relaxed).is_err() {
panic!("PerCpu: slot {} already borrowed", cpu);
}
}
// No local_irq_save() — the IrqDisabledGuard proves IRQs are off.
// SAFETY: PreemptGuard pins to this CPU. IrqDisabledGuard proves
// IRQs disabled. Together, these guarantee exclusive access.
unsafe {
PerCpuMutRefNosave {
value: &mut *self.data.add(cpu).as_ref().unwrap().get(),
_irq: PhantomData,
#[cfg(debug_assertions)]
borrow_state: self.borrow_state(cpu),
_guard: PhantomData,
}
}
}
}
/// Mutable reference guard that does NOT save/restore IRQ flags on drop.
/// Created by `get_mut_nosave()`. Drop only clears the debug borrow state.
impl<'a, T> Drop for PerCpuMutRefNosave<'a, T> {
fn drop(&mut self) {
#[cfg(debug_assertions)]
{
self.borrow_state.store(0, Ordering::Release);
}
// No local_irq_restore() — caller's IrqDisabledGuard manages IRQ state.
}
}
Cost comparison (x86-64, release builds):
| Variant | IRQ management | Total cycles | Use case |
|---|---|---|---|
get_mut() |
pushfq/cli + popfq |
~3-8 | General per-CPU mutation |
get_mut_nosave() |
None (proof token) | ~1-3 | Hardirq, softirq, spinlock, scheduler |
On hot paths (NVMe interrupt handler, scheduler tick, NAPI poll), the IRQ
elision saves ~5-10 cycles per PerCpu access. With 1-2 PerCpu accesses
per NVMe completion, this yields ~5-20 cycles saved per I/O operation.
3.3.2 NMI Safety¶
local_irq_save() does NOT prevent NMIs (non-maskable interrupts). An NMI
can arrive while a PerCpu<T> variable is mutably borrowed. This creates
a soundness hazard: if the NMI handler also accesses the same PerCpu<T>,
the mutable borrow invariant is violated.
NMI handler rules:
-
NMI handlers MUST NOT call
PerCpu<T>::get_mut()orget_mut_nosave()on any variable. In debug builds, the CAS check panics (correct — panic is better than data corruption). In release builds, this would be UB. -
NMI handlers MAY call
PerCpu<T>::get()(shared borrow) on variables that are never mutably borrowed during normal interrupt-enabled execution. In practice, this limits NMI-safe reads to a small set of immutable-after-init or structurally read-only variables. -
NMI handlers that need per-CPU mutable state MUST use dedicated NMI-safe buffers allocated outside the
PerCpu<T>mechanism — e.g., raw per-CPU arrays indexed bycpu_local::cpu_id()withAtomicU64fields, or theCpuLocalregister-based fast path (Section 3.2) which is inherently NMI-safe (single-instruction register read, no borrow tracking).
CpuLocal field NMI safety classification:
| Field | NMI-safe | Reason |
|---|---|---|
cpu_id |
Yes | Immutable after init, single register read |
current_task |
Read-only yes | Written only by context_switch() which cannot race with NMI on same CPU |
preempt_count |
Read-only yes | NMI can read; must not modify (would corrupt preemption state) |
in_nmi |
Yes (dedicated) | NMI entry sets this flag; NMI-specific counter |
slab_magazines |
No | Mutable under get_mut() during allocation |
runqueue |
No | Mutable during scheduler tick |
rcu_data |
No | Mutable during quiescent state reporting |
Enforcement: In debug builds, PerCpu<T>::get_mut() sets an atomic flag
that get() from NMI context checks — if the flag is set, panic with
"PerCpu<T> mutably borrowed during NMI". NMI context is detected by reading
CpuLocal.in_nmi. This catches violations during testing. In release builds,
the cost of this check (~3 cycles per NMI entry) is elided; the invariant is
enforced by code review and the /// # NMI Safety documentation convention.
Documentation convention: Any function that may be called from NMI context
MUST include a /// # NMI Safety doc section listing which per-CPU state it
accesses and why that access is safe. This is analogous to the /// # Safety
requirement for unsafe fn.
3.4 Cumulative Performance Budget¶
The complete set of UmkaOS overhead-reduction techniques — CpuLocal (Section 3.2),
debug-only PerCpu CAS (Section 3.3), IRQ elision (Section 3.3), RCU deferred
quiescent state (Section 3.1, RcuReadGuard::drop), PKRU shadow elision
(Section 11.2), capability amortization (Section 12.3), and doorbell
coalescing (Section 11.7) — are all core design decisions implemented
from day one. They are not deferred optimizations. Below is the cumulative
overhead analysis for the three hottest I/O paths with all techniques active.
Platform coverage: The 5% overhead budget applies to architectures with hardware-assisted Tier 1 isolation: x86-64 (MPK), AArch64 (POE, Cortex-X4+ / Neoverse V3+), ARMv7 (DACR), and PPC32 (segment registers). All comfortably meet the budget. On RISC-V 64, s390x, LoongArch64, and PPC64LE (Section 11.2), no fast hardware isolation mechanism exists — Tier 1 drivers run as Tier 0 (in-kernel, zero isolation overhead) or Tier 2 (Ring 3, full process isolation). The 5% Tier 1 budget is N/A on these platforms. Pre-POE AArch64 (page-table isolation) exceeds 5% and also falls back to Tier 0/Tier 2. See Section 11.2 for per-architecture isolation mechanism details.
NVMe 4KB random read (~10μs = ~25,000 cycles at 2.5 GHz):
| Source | Cycles | Notes |
|---|---|---|
| MPK domain switches (×4, shadow-elided) | ~47-92 | Shadow elides 1-2 of 4 WRPKRU on back-to-back transitions |
| CpuLocal slab magazine access (×2) | ~2-8 | alloc + free, arch-dependent |
| PerCpu (nosave) in IRQ context (×1) | ~1-3 | IRQ-disabled proof token, no pushfq/popfq |
| RCU quiescent flag (×1) | ~1 | CpuLocal bool write, deferred to tick |
| Capability validation (×1, amortized) | ~8-19 | ValidatedCap token + REVOKED_FLAG check, 3-4 sub-calls use cached |
| Doorbell write (×1, amortized over batch-32) | ~5 | Single MMIO write for entire batch |
| Total additional over Linux | ~64-128 | |
| % of 10μs operation | ~0.26-0.51% |
TCP RX packet (~5μs per-packet, NAPI batch-64):
| Source | Cycles | Notes |
|---|---|---|
| MPK domain switches (×4, shadow-elided, amortized/64) | ~0.7-1.4/pkt | Shadow + NAPI batching |
| CpuLocal NAPI budget + socket access | ~2-6/pkt | |
| RCU conntrack guard (×1) | ~0.02/pkt | CpuLocal flag write, amortized |
| Capability (amortized over NAPI batch) | ~0.1-0.2/pkt | One ValidatedCap per NAPI poll cycle |
| Total additional per packet | ~3-8 | |
| % of 5μs operation | ~0.02-0.06% | per packet, with batching |
| % without batching (batch=1) | ~1.5-2.2% | worst case, still improved |
Context switch (~2μs = ~5,000 cycles):
| Source | Cycles | Notes |
|---|---|---|
| CpuLocal runqueue access | ~1-4 | pick_next_task |
| Isolation shadow save/restore | ~2-4 | Memory writes only, no WRPKRU |
| RCU quiescent report (batched) | ~3-5 | Check flag + report, once per switch |
| Total additional | ~6-13 | integer workload, no FPU |
| % of ~2μs switch | ~0.1-0.3% |
Cumulative worst case (nginx-like: receive + read + send + switch):
Platform: x86-64 MPK:
TCP RX (NAPI-64): ~0.04%
NVMe read (batched): ~0.35%
TCP TX (batched): ~0.25%
Context switches: ~0.15%
──────────────────────────
Compound: ~0.8% (4.2% headroom under 5% budget)
Platform: AArch64 POE (Cortex-X4+, Neoverse V3+):
TCP RX (NAPI-64): ~0.4%
NVMe read (syscall): ~1.5%
TCP TX (batched): ~0.5%
Context switches: ~0.2%
──────────────────────────
Compound: ~1.5-2.0% (3.0-3.5% headroom under 5% budget)
NVMe 4KB random read — syscall path with LSM (~10μs):
The budget above assumes io_uring SQPOLL (zero domain crossings for the
read itself). A conventional read() syscall adds VFS and LSM overhead:
| Source | Cycles | Notes |
|---|---|---|
| All isolation overhead (as above) | ~63-123 | Same as SQPOLL path |
| VFS domain crossing (ring dispatch, amortized) | ~8-20 | Ring submit + consumer dispatch, amortized at N≥12 per domain-switch cycle |
LSM hooks (2-3 per read: file_security(FileSecurityOp::Permission), inode_security(InodeSecurityOp::Permission)) |
~30-90 | Static dispatch, single LSM: ~10-30 per hook. 3 hooks typical. |
| UserPtr validation (copy_to_user) | ~5-10 | Bounds check + potential page fault |
| Total additional over Linux | ~121-269 | |
| % of 10μs operation | ~0.5-1.1% | Still well within 5% budget |
TCP RX packet with NetBuf overhead (~5μs per-packet, NAPI batch-64):
| Source | Cycles | Notes |
|---|---|---|
| All isolation overhead (as above) | ~3-8/pkt | Same as base budget |
| NetBuf slab alloc in receiving domain (per-packet) | ~15-25/pkt | Magazine pop (hot), slab fallback (warm) |
| NetBuf ring entry serialization (128-byte write) | ~10-15/pkt | 2 cache lines, sender side |
| NetBuf ring entry deserialization + reconstruction | ~10-15/pkt | 2 cache lines read + field copy, receiver side |
| NetBuf slab free in sending domain (per-packet) | ~10-15/pkt | Magazine push (hot), slab return (warm) |
| Total additional per packet | ~48-78 | |
| % of 5μs operation | ~0.4-0.6% | per packet, with NAPI batching |
NetBuf lifecycle cost breakdown: The domain crossing copies the 128-byte
NetBufRingEntry wire format (2 cache lines), NOT the full 296-byte NetBuf
struct. The receiving domain reconstructs a full NetBuf from the ring entry
plus the shared DMA data handle. Data pages are shared zero-copy via the DMA
buffer pool — only metadata crosses the domain boundary.
Cumulative worst case with LSM (nginx-like: receive + read + send + switch):
Platform: x86-64 MPK (with LSM):
TCP RX (NAPI-64): ~0.4% (includes NetBuf lifecycle overhead)
NVMe read (syscall): ~0.9% (includes VFS + LSM)
TCP TX (batched): ~0.5% (includes NetBuf lifecycle + LSM)
Context switches: ~0.15%
LSM hooks (open/close/sendfile, 6 total): ~0.5% (static dispatch)
──────────────────────────
Compound: ~2.5% (2.5% headroom under 5% budget)
Platform: AArch64 POE (with LSM):
The ~2.5% headroom is tighter than the SQPOLL-only estimate (~4.2%). This is the realistic worst case for a typical server workload with security enabled.
NVMe write path additional overhead: The read path budget above is the simpler
case. The write path incurs additional costs beyond the read path:
(1) the writeback domain crossing pair (~23-46 cycles on x86-64),
(2) the bio dispatch domain crossing pair (~23-46 cycles), and
(3) LSM hooks (file_security(FileSecurityOp::Permission) + inode_security(InodeSecurityOp::Permission), ~20-60 cycles).
In total, the write path adds ~66-152 cycles (~0.3-0.7% of 10 us) over the read
path. The compound budget remains within the 5% envelope: ~2.8-3.2% worst case for
a mixed read/write workload. See Section 1.3 for the itemized breakdown.
3.4.1.1 Write Path Completion Latency (fsync/O_SYNC)¶
The bio completion callback defers end_page_writeback() to the blk-io
workqueue because page cache operations may acquire sleeping locks (required
for VFS crash recovery — see Section 11.9).
This deferral adds ~1-5us per write completion on the fsync/O_SYNC path.
| Metric | Value | Notes |
|---|---|---|
| Workqueue deferral latency | ~1-5us | Per write completion |
| Impact on fsync() at 100K IOPS | ~100-500ms/sec additional | Sequential, not pipelined |
| Impact on async writeback | ~0% throughput | Workqueue runs concurrently |
| Comparison with Linux | Linux: 0us (softirq context) | Linux lacks VFS crash recovery |
This is the price of crash containment: UmkaOS's VFS crash recovery requires sleeping locks on the writeback path, which cannot execute in softirq context. The throughput budget (cycles per operation at pipeline throughput) is unaffected because the workqueue runs asynchronously for non-sync I/O. Only O_SYNC/fsync callers observe the sequential latency penalty.
3.4.1.2 Metadata-Heavy Workload Note¶
Metadata-heavy workloads (stat, readdir, open/close): Individual metadata syscalls pay
~18ns overhead per call on x86-64 (~46 cycles for Core → VFS+FS → Core round-trip).
This is ~3.6-9% per individual stat() (base cost ~200-500ns). For throughput benchmarks,
the overhead is amortized across mixed I/O operations. For purely metadata-heavy workloads
(e.g., find traversal, package manager resolution), expect ~5-9% syscall-level overhead
per metadata operation, dominated by the VFS domain crossing cost. This is the design
tradeoff for VFS crash containment: the dentry/inode cache lives in the VFS domain,
enabling crash recovery via cache rebuild, at the cost of one domain crossing pair per
metadata syscall.
Per-architecture metadata overhead (single stat() call, L1-hot, dentry cache hit):
| Architecture | VFS round-trip | stat() base cost | Overhead |
|---|---|---|---|
| x86-64 (MPK) | ~46 cycles (~18ns) | ~200-500ns | ~3.6-9% |
| AArch64 (POE) | ~80-160 cycles (~32-64ns) | ~200-500ns | ~6.4-32% |
| ARMv7 (DACR) | ~60-80 cycles (~30-40ns) | ~200-500ns | ~6-20% |
| PPC64LE (Tier 1 unavailable) | N/A — no domain crossing (VFS runs Tier 0) | ~200-500ns | N/A |
| PPC32 (segments) | ~40-80 cycles (~20-40ns) | ~200-500ns | ~4-20% |
| RISC-V (page table) | ~400-1000 cycles (~160-400ns) | ~200-500ns | ~32-200% |
| AArch64 (page table) | ~150-300 cycles (~60-120ns) | ~200-500ns | ~12-60% |
On RISC-V and page-table-fallback AArch64, metadata-heavy workloads exceed the 5% budget per-operation. These architectures should run VFS as Tier 0 (no domain crossing) unless crash containment is specifically required, in which case the overhead is accepted as the cost of isolation.
Amortized metadata overhead: The raw per-call overhead above applies to isolated
stat() calls with no prefetch. For the dominant readdir+stat access pattern (e.g.,
find, ls -la, package managers), the VFS readdir-plus prefetch mechanism
(Section 14.1) reduces effective overhead
to ~0.3-0.5% on x86-64 by serving stat results from a per-task Core-memory buffer with
zero domain crossings (~95% hit rate). For io_uring batched statx workloads, coalescing
reduces overhead to ~0.05% per stat (one crossing amortized over 64 operations). These
improvements apply to Always and CacheAware filesystem policies; Never-policy
filesystems (DLM-based cluster FS) retain the unoptimized cost, which is negligible
relative to their ~50-500us distributed lock overhead.
3.4.1.3 Non-Batched / Latency-Sensitive Workloads¶
The budgets above assume batch amortization (NAPI batch-64, doorbell batch-32). These assumptions hold for sustained throughput workloads (web servers, streaming, bulk storage). They do not hold for workloads where each operation is independent and unbatched: database transactions, RPC microservices, interactive key-value stores.
Single-request RPC scenario (gRPC/Thrift service: recv 1 request → 2 DB reads → format → send 1 response, x86-64 MPK):
| Source | Cycles | Notes |
|---|---|---|
| TCP RX: NIC driver domain switch (×2, enter+exit) | ~46 | No NAPI batch — single packet wakes poll, NAPI completes with 1 packet |
| TCP RX: umka-net domain switch (×2) | ~46 | TCP → socket buffer → userspace |
| NetBuf lifecycle (×1: alloc + serialize + deserialize + free) | ~48-78 | Full ring entry copy + slab alloc/free |
| Capability validation (×1) | ~8-19 | Amortized across NIC+TCP sub-calls (includes REVOKED_FLAG check) |
LSM hooks: socket_security(SocketSecurityOp::Recvmsg) (×1) |
~10-30 | |
| NVMe read #1: VFS domain crossing (×2) | ~46-92 | syscall read → VFS → NVMe driver |
| NVMe read #1: doorbell (×1, no batching) | ~150 | Single MMIO write, no coalescing |
NVMe read #1: LSM (file_security(FileSecurityOp::Permission) + inode_security(InodeSecurityOp::Permission)) |
~20-60 | |
| NVMe read #1: capability (×1) | ~8-19 | Includes REVOKED_FLAG check |
| NVMe read #2: same as #1 | ~223-316 | |
| TCP TX: NIC domain switch (×2) | ~46 | |
| TCP TX: umka-net domain switch (×2) | ~46 | |
| TCP TX: NetBuf lifecycle + LSM | ~58-108 | NetBuf lifecycle ~48-78 + LSM ~10-30 |
| Total additional over Linux | ~755-1051 | |
| Typical RPC latency | ~25-50 μs (62,500-125,000 cycles at 2.5 GHz) | |
| Overhead (L1-hot) | ~0.8-1.7% | Well within 5% |
3.4.1.4 Cache-Cold Multiplier (L1I Displacement from Domain Working Set Switching)¶
The L1-hot assumption above is optimistic for mixed workloads. When control transfers between umka-nucleus and a Tier 1 driver, the driver's instruction cache footprint (hot loop, DMA descriptor handling, interrupt acknowledgment) displaces umka-nucleus's L1I entries. On return, umka-nucleus's code must be re-fetched from L2.
This L1I displacement tax is a structural overhead of the isolation model that Linux does not incur — in Linux, all kernel code shares one address space and one working set, so the NVMe driver's hot loop and the VFS dispatch code coexist in L1I without displacement pressure from domain switches. In UmkaOS, each domain switch is a working set boundary: the hardware prefetcher and branch predictor must re-warm after every transition. The cache-cold penalty is ~2-3x on the isolation-related cycles (domain switches, LSM hooks, capability checks), but NOT on device I/O latency (NVMe latency is device-bound, not cache-bound).
The L2-warm row in the table below captures this effect. L1-hot numbers are the optimistic case; L2-warm (~2x multiplier on isolation cycles) is the realistic steady-state for production workloads with mixed request types or multiple active Tier 1 drivers. The headroom calculation (4.2% on x86-64) uses the L2-warm estimate, ensuring the 5% budget holds under realistic cache conditions.
| Scenario | Isolation cycles | Total overhead | When this applies |
|---|---|---|---|
| L1-hot (sustained single-type RPC) | ~755-1,051 | ~0.8-1.7% | Tight loop hitting one driver repeatedly |
| L2-warm (mixed RPC types, ~2x isolation penalty) | ~1,200-1,700 | ~1.2-2.7% | Production steady-state: multiple drivers, varied request types |
| L3/cold (rare path, first request after idle, ~3x) | ~1,800-2,500 | ~1.9-4.0% | First request after idle, cold code path |
Tier 1 domain switch contribution: In all scenarios above, domain switch cycles (WRPKRU on x86-64, POR_EL1 on AArch64 POE, etc.) are the dominant isolation cost component. Per-architecture domain switch costs and the full breakdown are in the per-architecture table below (line "Isolation cost").
Key observations:
-
Single-request overhead is higher than batched but still within budget. The worst case (L3-cold, 2 DB reads per RPC) is ~3.5% on x86-64. This is under 5% but with less headroom than the nginx throughput scenario.
-
Doorbell coalescing is the biggest loss. Without batch-32, each NVMe submission costs ~150 cycles (raw MMIO write) instead of ~5 cycles/cmd. This is the single largest contributor to non-batched overhead. Mitigation: io_uring with
IORING_SETUP_SQPOLLrecovers batching even for RPC workloads by coalescing submissions from the poll thread. -
NAPI batch-1 is common for low-rate RPC. When a NIC interrupt fires for a single packet, NAPI polls once and exits. The 4 domain switches (NIC enter/exit + umka-net enter/exit) are NOT amortized. Per-packet isolation overhead: ~92 cycles on a ~5 μs packet = ~1.8%. This matches the "without batching" line in the TCP RX budget above (~1.5-2.2%).
-
Tail latency vs throughput. The 5% budget targets throughput on macro benchmarks (as scoped in Section 1.3). Individual request tail latency (p99, p99.9) may show higher percentage overhead because:
- Cache-cold paths on rare request types
- NAPI poll finding only 1 packet (no amortization)
- Coincidence with RCU grace period processing or FMA health check
- TLB pressure from domain switches on page-table-fallback architectures
For p99 tail latency on latency-sensitive services, we target <8% overhead on fast-isolation architectures (x86-64 MPK, AArch64 POE) and <15% on page-table-fallback architectures. These are realistic bounds that account for cache variability and non-batched paths. Per-operation (individual syscall or interrupt) tail latency budgets are not specified — consistent with industry practice where only workload-level latency targets are meaningful.
- Mitigation strategies for latency-sensitive services:
- io_uring SQPOLL: Recovers doorbell coalescing even for single requests by batching submissions in the poll thread. Eliminates the ~150 cycle/cmd penalty.
- Busy-polling (
SO_BUSY_POLL): Avoids NAPI interrupt path entirely; the application polls the NIC ring directly (still through Tier 1 domain crossing). Trades CPU for latency determinism. - Tier 0 promotion for latency-critical drivers: On architectures without
fast isolation, or for services where even MPK overhead matters, the admin can
set a trusted NIC or NVMe driver to Tier 0 via
echo 0 > /ukfs/kernel/drivers/<name>/tier. This eliminates domain switch overhead entirely at the cost of crash containment. - Adaptive NAPI coalescing: UmkaOS NAPI uses adaptive interrupt coalescing (Section 16.14). Under low load, packets are delivered with minimal batching (latency-optimized). Under high load, batching increases (throughput-optimized). This is the same tradeoff Linux makes.
Per-architecture non-batched RPC overhead (single gRPC request, 2 DB reads, L2-warm):
| Architecture | Isolation cost/crossing | ~10 crossings | Total RPC overhead |
|---|---|---|---|
| x86-64 (MPK) | ~23 cycles | ~460 (×2 warm) = ~920 | ~1.2-2.0% |
| AArch64 (POE) | ~40-80 cycles | ~1,200 (×2) = ~2,400 | ~2.5-4.0% |
| AArch64 (page table) | ~150-300 cycles | ~4,500 (×2) = ~9,000 | ~8-15% |
| ARMv7 (DACR + ISB) | ~30-40 cycles | ~700 (×2) = ~1,400 | ~1.8-3.0% |
| PPC64LE (Tier 1 unavailable) | N/A — no domain-switch cost | N/A | N/A — Tier 0 drivers: 0 isolation cycles |
| PPC32 (segments + isync) | ~20-40 cycles | ~600 (×2) = ~1,200 | ~1.5-2.5% |
| RISC-V (page table) | ~200-500 cycles | ~7,000 (×2) = ~14,000 | ~12-22% |
On RISC-V and pre-POE AArch64, non-batched RPC overhead exceeds the 5% throughput budget. This is expected and documented: on platforms without Tier 1 hardware isolation, drivers run as Tier 0 (no isolation overhead, no crash containment) or Tier 2 (Ring 3 + IOMMU, higher overhead but full crash containment). The placement depends on licensing requirements, driver default preference, and sysadmin configuration (see Section 1.1).
The ~2.5% headroom is sufficient because: (cumulative overhead, same nginx workload):
| Architecture | CpuLocal cost | PerCpu cost | Isolation cost | Shadow savings | Total |
|---|---|---|---|---|---|
| x86-64 | ~1 cycle | ~1-3 (nosave) | ~23 cycles/WRPKRU | ~23-46 elided | ~0.7-1.2% |
| AArch64 (POE) | ~2-4 cycles | ~3-6 (nosave) | ~40-80/MSR | ~40-80 elided | ~1.2-2.0% |
| AArch64 (page table) | ~2-4 cycles | ~3-6 (nosave) | ~150-300/switch | N/A (TLB) | ~3-6% |
| ARMv7 | ~3-5 cycles | ~4-8 (nosave) | ~30-40/MCR+ISB | ~30-40 elided | ~1.8-3.0% |
| RISC-V | ~5-10 cycles | ~6-12 (nosave) | ~200-500/PT | N/A (TLB) | ~4-10% |
| PPC64LE | ~1-3 cycles | ~2-5 (nosave) | N/A — Tier 1 unavailable | N/A | N/A — Tier 0: 0 isolation cycles |
| PPC32 | ~3-6 cycles | ~5-10 (nosave) | ~20-40/mtsr+isync | ~20-40 elided | ~1.5-3.0% |
Boundary crossing cycle cost reference table:
Individual crossing costs for the key operations that appear in budgets above. All values are per-crossing, measured or estimated per architecture:
| Crossing | x86-64 | AArch64 (POE) | ARMv7 | RISC-V | PPC64LE | PPC32 |
|---|---|---|---|---|---|---|
| MPK/POE domain switch (WRPKRU / POR_EL1) | ~23 | ~40-80 | ~30-40 | N/A (PT) | N/A (Tier 1 unavailable) | ~20-40 |
| Shadow elision (back-to-back same domain) | saves ~23 | saves ~40-80 | saves ~30-40 | N/A | N/A | saves ~20-40 |
| UserPtr validation (copy_to_user bounds) | ~5-8 | ~6-10 | ~8-12 | ~8-15 | ~5-10 | ~8-15 |
| CapHandle validation (cached ValidatedCap + REVOKED_FLAG) | ~8-19 | ~9-21 | ~11-23 | ~11-25 | ~9-20 | ~11-25 |
| LSM hook (static dispatch, single LSM) | ~10-30 | ~12-35 | ~15-40 | ~15-45 | ~12-35 | ~15-45 |
| NetBuf slab alloc (magazine pop, single) | ~15-25 | ~18-30 | ~20-35 | ~20-40 | ~18-30 | ~20-40 |
| NetBuf full lifecycle (alloc + ring ser/deser + free) | ~48-78 | ~56-95 | ~65-115 | ~65-130 | ~56-95 | ~65-130 |
| VFS domain crossing (full round-trip) | ~46-92 | ~80-160 | ~60-80 | ~400-1000 | N/A (VFS runs Tier 0) | ~40-80 |
Notes: - RISC-V VFS domain crossing uses page table switching (no MPK equivalent), hence ~400-1000 cycles including TLB flush. This is the primary overhead contributor on RISC-V. - All values assume L1-hot data (worst case for cache-cold paths is ~2-3x). - "Static dispatch, single LSM" means UmkaOS's default (one active LSM with direct function call, no indirect branch). Stacked LSMs (if supported) multiply per-hook cost by the stack depth. - RCU read-side cost (~1 cycle flag check) is included in isolation and capability validation paths. RCU write-side (grace period processing, callback invocation) is workload-dependent and not included in per-operation budgets — it runs on a dedicated kthread and amortizes across batched callbacks.
On x86-64, AArch64 with POE, ARMv7, and PPC32: comfortably within the 5% budget with substantial headroom. On AArch64 without POE and RISC-V: isolation cost dominates, and the per-CPU/shadow optimizations matter less — the bottleneck is page-table-based domain switching, not per-CPU access.
3.4.1.4.1 Cache-Cold Sensitivity Analysis¶
The compound overhead tables above assume L1 cache-hot access for all metadata
lookups (XArray node traversals, KABI vtable dereferences, CapEntry reads,
ValidatedCap token checks). Under realistic workloads, some fraction of these
accesses miss in L1 and hit in L2 or L3. This section quantifies the impact to
confirm the 5% budget holds under pessimistic assumptions.
Per-component cache-miss penalty model:
Each metadata access has a baseline L1-hot cost. On an L1 miss, the penalty depends on where the line is found. The weighted miss penalty uses the distribution below.
| Component | L1-hot cost | L2 hit (+ns) | L3 hit (+ns) | Per-miss weighted penalty |
|---|---|---|---|---|
| XArray node traversal (1-2 levels for typical radix depth) | 3-5 ns | +5-15 ns | +30-80 ns | +12.5-34.5 ns |
| KABI vtable dereference (single indirect load) | 2-3 ns | +5-15 ns | +30-80 ns | +12.5-34.5 ns |
| CapEntry read (RCU-protected, single cache line) | 3-5 ns | +5-15 ns | +30-80 ns | +12.5-34.5 ns |
| ValidatedCap REVOKED_FLAG check (atomic load) | 1-2 ns | +5-15 ns | +30-80 ns | +12.5-34.5 ns |
Compound overhead under varying L1 miss rates (x86-64 MPK, NVMe 4KB read path):
| L1 Miss Rate | XArray (ns) | Vtable Deref (ns) | CapEntry (ns) | REVOKED_FLAG (ns) | Compound Total (ns) | % of 10 μs NVMe | Within 5%? |
|---|---|---|---|---|---|---|---|
| 0% (baseline) | 3-5 | 2-3 | 3-5 | 1-2 | 9-15 | 0.09-0.15% | Yes |
| 10% (typical steady-state) | 4.3-8.5 | 3.3-6.5 | 4.3-8.5 | 2.3-5.5 | 14.2-29 | 0.14-0.29% | Yes |
| 20% (moderate contention) | 5.5-11.9 | 4.5-9.9 | 5.5-11.9 | 3.5-8.9 | 19-42.6 | 0.19-0.43% | Yes |
| 30% (heavy contention / cold restart) | 6.8-15.4 | 5.8-13.4 | 6.8-15.4 | 4.8-12.4 | 24.2-56.6 | 0.24-0.57% | Yes |
| 50% (worst case: first access after long idle) | 9.3-22.3 | 8.3-20.3 | 9.3-22.3 | 7.3-19.3 | 34.2-84.2 | 0.34-0.84% | Yes |
Cross-architecture compound overhead at 20% L1 miss rate (NVMe 4KB read):
| Architecture | Isolation hw cost (ns) | Metadata cache-cold (ns) | Combined (ns) | % of 10 μs NVMe | Within 5%? |
|---|---|---|---|---|---|
| x86-64 (MPK) | 18-37 | 19-42.6 | 37-80 | 0.37-0.80% | Yes |
| AArch64 (POE) | 32-64 | 19-42.6 | 51-107 | 0.51-1.07% | Yes |
| ARMv7 (DACR + ISB) | 24-32 | 19-42.6 | 43-75 | 0.43-0.75% | Yes |
| PPC64LE (Tier 1 unavailable) | N/A — no isolation switch | 19-42.6 | 19-42.6 | 0.19-0.43% | N/A |
| PPC32 (segments + isync) | 16-32 | 19-42.6 | 35-75 | 0.35-0.75% | Yes |
Cross-architecture compound at 20% miss rate for 1 μs syscall (stat/open/close):
| Architecture | Isolation hw cost (ns) | Metadata cache-cold (ns) | Combined (ns) | % of 1 μs syscall | Within 5%? |
|---|---|---|---|---|---|
| x86-64 (MPK) | 18-37 | 19-42.6 | 37-80 | 3.7-8.0% | Marginal |
| AArch64 (POE) | 32-64 | 19-42.6 | 51-107 | 5.1-10.7% | No |
| ARMv7 (DACR + ISB) | 24-32 | 19-42.6 | 43-75 | 4.3-7.5% | Marginal |
| PPC64LE (Tier 1 unavailable) | N/A — no isolation switch | 19-42.6 | 19-42.6 | 1.9-4.3% | N/A |
| PPC32 (segments + isync) | 16-32 | 19-42.6 | 35-75 | 3.5-7.5% | Marginal |
For sub-microsecond metadata syscalls under cache pressure, the 5% budget is exceeded on some architectures. This is consistent with the metadata-heavy workload analysis above and is mitigated by the VFS readdir-plus prefetch mechanism (Section 14.1), which achieves ~95% hit rate in Core-memory buffers, eliminating domain crossings (and their associated cache-cold metadata lookups) for the common case.
Assumptions:
- L2 hit latency: 5-15 ns (varies by microarchitecture: AMD Zen 4 ~5 ns, ARM Neoverse V2 ~8 ns, Intel Sapphire Rapids ~12 ns, IBM POWER10 ~10 ns)
- L3 hit latency: 30-80 ns (Zen 4 ~30 ns, Neoverse V2 ~40 ns, Sapphire Rapids ~50 ns, POWER10 ~60 ns)
- Miss distribution: 70% L2 hit / 30% L3 hit (conservative; real workloads show >85% L2 hit rate for kernel metadata due to temporal locality of XArray nodes, vtable pointers, and capability entries)
- Weighted per-miss penalty: 0.7 × L2_mid + 0.3 × L3_mid = 0.7 × 10 + 0.3 × 55 = 23.5 ns (used for interpolation; tables above use the full range, not just the midpoint)
- Metadata access count per NVMe read: 4 (XArray lookup, vtable dispatch, CapEntry validation, REVOKED_FLAG check). Additional accesses (LSM hook dispatch, VFS dentry lookup) are accounted separately in the LSM and metadata-heavy workload sections above.
Conclusion: Even under a pessimistic 30% L1 miss rate with 30% of misses
propagating to L3, the compound metadata overhead on a 10 μs NVMe read reaches
only ~0.57% of the operation cost — well within the 5% performance budget.
Combined with the hardware isolation overhead (domain switches, shadow elision),
the total stays under ~1.5% on all fast-isolation architectures. The per-VMA
lock optimization (Section 4.8) further reduces
cache pressure by eliminating mmap_lock contention, which is a primary source
of L1 cache pollution on multi-core systems.
Note: These numbers represent the UmkaOS isolation metadata overhead on top of both the base syscall cost and the hardware isolation cost (domain switches). They do not double-count: the boundary crossing cycle cost table above covers hardware isolation; this section covers the software metadata lookups that accompany each crossing. A 0.57% metadata overhead plus ~0.5% hardware isolation overhead yields ~1.1% total — imperceptible to applications.
Optimization summary (all implemented from day one, not deferred):
| Technique | Section | Cycles saved per I/O | Cumulative impact |
|---|---|---|---|
| CpuLocal register-based access | Section 3.2 | ~15-25 (vs old PerCpu CAS) | Major: hottest paths |
| Debug-only PerCpu CAS | Section 3.3 | ~15-20 (release elision) | Major: all PerCpu paths |
| IRQ save/restore elision | Section 3.3 | ~5-10 (per get_mut) | Moderate: IRQ/spinlock paths |
| RCU deferred quiescent state | Section 3.1 | ~4-9 (per outermost drop) | Moderate: all RCU paths |
| Isolation shadow elision | 03a, Section 11.2 | ~23-80 (per elided write) | Major: back-to-back switches |
| Capability amortization | Section 12.3 | ~8-36 (per KABI dispatch) | Moderate: all KABI calls |
| Doorbell coalescing | Section 11.7 | ~145/cmd (batch-32) | Major: batched NVMe/virtio |
RCU interaction:
KabiDispatchGuard(which scopesValidatedCap<'dispatch>) holds an RCU read-side lock for the duration of every KABI dispatch — see Section 12.3. This means every KABI call adds one RCU nesting level. The consequence for the concurrency model: capability revocations (rcu_callcallbacks) cannot complete while a KABI dispatch is in progress on any CPU. Long KABI dispatches therefore increase RCU grace period latency, which is bounded by the maximum KABI call duration (~100 μs worst case for NVMe completion). Designers adding new KABI interfaces must keep dispatch handlers short; blocking operations (sleeping, waiting on locks) inside aKabiDispatchGuardscope are prohibited.Revocation traversal cost: Capability revocation uses a two-phase breadth-first protocol (Section 9.1). Phase 1 is lock-free (~1-5 cycles, single
fetch_oronactive_ops). Phase 2 enqueuesCapRevocationWorkitems to the per-CPU workqueue, processing one delegation tree level per workqueue pass. The worst-case spinlock hold time per node is O(256) iterations (onechildrenlist scan), not O(children × depth) as a recursive traversal would require. Total revocation latency for deep trees increases slightly due to workqueue scheduling overhead (~50-200 ns per level, up to 16 levels = ~0.8-3.2 μs added), but worst-case interrupt latency improves dramatically: no single spinlock hold exceeds ~256 iterations regardless of tree depth.Tier 2 dispatch exception: Tier 2 drivers communicate via IPC syscalls, not domain ring buffers. The Tier 2 dispatch path does NOT hold a
KabiDispatchGuardor RCU read lock during the cross-address-space IPC. Instead, capability validation uses a two-phase approach: 1. Validate capability under a short RCU read lock (~10 cycles). CopyValidatedCapfields. 2. Drop RCU read lock before issuing the blocking IPC send/recv. 3. On IPC completion, re-validate if the response references new capabilities.This avoids RCU stalls from Tier 2 latency while maintaining revocation safety: a capability revoked between phases 1 and 3 is caught by re-validation in phase 3.
Spectre mitigation interaction: The KABI vtable dispatch is an indirect call, which incurs retpoline overhead (~15-25 cycles) on pre-eIBRS hardware. This cost applies equally to any indirect call in both Linux and UmkaOS; UmkaOS's differential cost is exactly one additional retpoline per domain crossing (the vtable dispatch itself). On eIBRS-capable hardware (Intel Ice Lake+, AMD Zen 3+), the indirect call is predicted at ~2-5 cycles and retpoline is not used. All cycle counts in this section assume identical Spectre mitigations on both Linux and UmkaOS. See Section 2.18 for the complete per-mitigation overhead analysis.
3.4.1.5 RCU-Protected Container Types¶
Pseudocode convention: Code in this section uses Rust syntax and follows Rust ownership, borrowing, and type rules.
&selfmethods use interior mutability for mutation. Atomic fields use.store()/.load(). See CLAUDE.md §Spec Pseudocode Quality Gates.
This is the canonical home for the four RCU-protected container types used
across the kernel — RcuCell, RcuPtr, RcuHashMap, and RcuVec. Other
chapters refer to these definitions; they do NOT re-specify them. All four
share the same substrate: an AtomicPtr published with Release, read under an
RcuReadGuard with a single Acquire load, and old values reclaimed after a
grace period via rcu_call/rcu_defer_free (both defined below with the
grace-period machinery). They differ only in shape and write discipline:
| Type | Value shape | Read returns | Writer discipline | Empty state |
|---|---|---|---|---|
RcuCell<T> |
one boxed T, always present |
&T |
external WriterProof guard |
n/a (non-null) |
RcuPtr<T> |
one boxed T, nullable |
Option<&T> |
external WriterProof guard |
null() (const) |
RcuHashMap<K,V> |
bucketed node chains | Option<V> (clone) |
per-bucket SpinLock (internal) |
new() (const, lazy) |
RcuVec<T> |
one flat snapshot array | &[T] |
external WriterProof guard |
new() (const, null snapshot) |
3.4.1.5.1 WriterProof — Sealed Writer-Exclusivity Proof¶
The three externally-serialized containers (RcuCell, RcuPtr, RcuVec)
take a writer-proof token on every update(): a borrowed lock guard that
demonstrates, at compile time, that the caller holds an exclusive lock for the
duration of the swap. The proof's semantic content is exactly "some exclusive
writer lock is held" — which lock kind provides the exclusivity is
irrelevant to the single-writer argument. The earlier signature demanded
specifically &MutexGuard<'_, ()>, which was over-narrow on both axes:
- Lock kind. Corpus writers legitimately guard hot-adjacent structural RCU
publishes with a
SpinLock— a short non-sleeping critical section is the RIGHT discipline for edits like the cgroup child-list swap (Cgroup.children_lock: SpinLock<()>, Section 17.2) or the cgroup-BPF effective-set republish (underCgroup.bpf_progs: SpinLock<…>, Section 17.2). Forcing a sleepingMutexonto those sites would be a regression, not a fix. - Guard payload. The natural writer serializer is often a lock that already
protects real data (
SpinLock<ArrayVec<BpfCgroupLink, 64>>,Mutex<ImaMeasurementLog>). Restricting the proof to unit-payload()guards would force a second, artificialMutex<()>next to the real lock — two locks where one suffices, plus a new "acquired the wrong one" failure mode.
WriterProof is therefore a sealed marker trait implemented for every
exclusive guard type in the kernel, generic over the guard's payload:
/// Marker for "the caller holds an exclusive (writer) lock".
///
/// `Rcu{Cell,Ptr,Vec}::update()` accept `&impl WriterProof` as their
/// serialization proof. Implementors are EXACTLY the kernel's exclusive
/// lock guards; the trait is SEALED (private supertrait below) so no code
/// outside this module — in particular no driver or Evolvable module — can
/// implement it on a forgeable token type and erase the compile-time
/// single-writer guarantee.
///
/// Deliberately NOT implemented for shared guards: `RwLockReadGuard` (shared
/// readers are not writer-exclusive) and `RcuReadGuard` (not a lock at all).
///
/// The proof establishes "AN exclusive lock is held", not "the RIGHT lock is
/// held" — that remains the structural co-location convention (the writer
/// lock lives in the same struct as the container it serializes; see the
/// implementation note after `RcuCell`). This is unchanged from the previous
/// `MutexGuard`-only token, which had the same property.
pub trait WriterProof: sealed::Sealed {}
mod sealed {
/// Private supertrait: only nameable inside this module, so external
/// `impl WriterProof for X` fails to compile (cannot implement `Sealed`).
pub trait Sealed {}
}
// Sleeping exclusive guard ([Section 3.5](#locking-strategy)).
impl<'a, T> sealed::Sealed for MutexGuard<'a, T> {}
impl<'a, T> WriterProof for MutexGuard<'a, T> {}
// Spinning exclusive guard, IRQ-saving ([Section 3.5](#locking-strategy)).
impl<'a, T> sealed::Sealed for SpinLockGuard<'a, T> {}
impl<'a, T> WriterProof for SpinLockGuard<'a, T> {}
// Write half of the sleeping reader-writer lock ([Section 3.5](#locking-strategy)).
// The READ guard is deliberately not an implementor.
impl<'a, T> sealed::Sealed for RwLockWriteGuard<'a, T> {}
impl<'a, T> WriterProof for RwLockWriteGuard<'a, T> {}
// Leveled-lock guard (`Lock<T, LEVEL>`, §Lock Ordering below) — wraps an
// exclusive `SpinLock`/`Mutex` acquisition, so it carries the same
// exclusivity guarantee at any level.
impl<const LEVEL: u32> sealed::Sealed for LockGuard<LEVEL> {}
impl<const LEVEL: u32> WriterProof for LockGuard<LEVEL> {}
Merit. One trait fixes the proof corpus-wide: every existing call site
passes whatever exclusive guard it already holds (&children_lock_guard from a
SpinLock<()>, a bpf_progs.lock() payload guard, a config_lock
MutexGuard) and type-checks, while the compile-time guarantee — two
unsynchronized writers cannot both present a proof for the same co-located
lock — is exactly as strong as before. Sealing keeps the token unforgeable;
payload-genericity keeps one lock per structure instead of two.
/// RCU in Rust: zero-lock read path, deferred reclamation.
/// Readers hold an RcuReadGuard (analogous to rcu_read_lock).
/// Writers swap the pointer atomically and defer freeing the old
/// value until all readers have exited their critical sections.
///
/// **Clone-and-swap write pattern**: The canonical way to make incremental
/// changes to RCU-protected state (e.g., adding a route to a routing table,
/// registering a service in a registry) is:
/// 1. Acquire the write-side lock (the exclusive writer lock co-located
/// with the `RcuCell` — a `Mutex<()>`, a `SpinLock<()>`, or the
/// enclosing structure's own exclusive lock; any `WriterProof` guard).
/// 2. Read the current value via `cell.read()`.
/// 3. Clone the current value: `let mut new_val = (*current).clone();`
/// 4. Modify `new_val` as needed.
/// 5. Publish: `cell.update(new_val, &guard)`.
/// The old value is freed after the RCU grace period (all readers that saw
/// the old pointer have exited their critical sections). This is not a
/// performance concern for infrequent writes (service registration, config
/// updates, interface addition) — the cost is acceptable because these
/// operations happen at driver-load time or in response to admin actions.
/// The read path is always lock-free.
pub struct RcuCell<T: Send + Sync> {
ptr: AtomicPtr<T>,
/// Address of a `'static`-storage initial value installed by
/// [`new_static`](Self::new_static), or null for heap-constructed cells.
/// This address is ONLY ever compared for identity against the value being
/// reclaimed (in `update()` and `Drop`) — it is never dereferenced through
/// this field. A `'static` initial value must never be handed to
/// `rcu_defer_free`/`rcu_call`: it was not produced by `Box::into_raw`, and
/// freeing rodata/static storage is undefined behavior. Heap cells store
/// null here, so every non-null replaced value is reclaimed normally.
static_ptr: *const T,
}
// SAFETY: all *data* access goes through the atomic `ptr`, whose `Send + Sync`
// for `T: Send + Sync` is what makes publishing the pointed-to value across
// threads sound. `static_ptr` is a bare address used only for identity
// comparison on the write/drop paths; it is never dereferenced through this
// field and points either at immortal `'static` storage or is null. It does
// not weaken the `T: Send + Sync` sharing guarantee. Explicit impls are needed
// only because the raw-pointer field suppresses the auto-derived ones.
unsafe impl<T: Send + Sync> Send for RcuCell<T> {}
// SAFETY: see the `Send` impl above.
unsafe impl<T: Send + Sync> Sync for RcuCell<T> {}
impl<T: Send + Sync> RcuCell<T> {
/// Create a new RcuCell with an initial value. The value is heap-allocated
/// via `Box::try_new` (fallible) and the RcuCell takes ownership of the
/// raw pointer. Returns `Err(KernelError::OutOfMemory)` if allocation
/// fails. The pointer is always non-null after successful construction.
///
/// All RcuCell allocation is fallible. Callers must handle OutOfMemory.
pub fn new(value: T) -> Result<Self, KernelError> {
let ptr = Box::try_new(value).map_err(|_| KernelError::OutOfMemory)?;
Ok(Self {
ptr: AtomicPtr::new(Box::into_raw(ptr)),
// Heap cell: no `'static` value to protect from reclamation.
static_ptr: core::ptr::null(),
})
}
/// Construct a cell in `const` context from a `'static`-storage initial
/// value. Unlike [`new`](Self::new), this performs NO heap allocation, so it
/// is infallible and usable in `static`/`const` initializers — where the
/// non-`const`, fallible `Box::try_new` inside `new()` cannot appear.
///
/// The initial value is stored by pointer and recorded in `static_ptr`, so
/// the reclamation paths (`update`, `Drop`) skip it by identity: `'static`
/// storage must never be passed to `rcu_call`/`rcu_defer_free`. Every value
/// installed by a subsequent `update()` IS heap-allocated (`Box::try_new`)
/// and IS reclaimed normally. The cell is **non-null from construction**, so
/// the non-null invariant that `read()` relies on (it dereferences the
/// pointer unconditionally) holds immediately: there is no uninitialized
/// window and no two-phase init contract for callers to honor.
///
/// **Merit**: this keeps `RcuCell` a plain machine word on the *read* path —
/// `static_ptr` is touched only on write/drop — while giving the read-mostly
/// policy/registry statics a correct `const` constructor. It replaces the
/// earlier ad-hoc `new_empty()` stub, whose null pointer would have made the
/// first `read()` a null dereference (UB) against the non-null invariant.
/// Consumers that require it:
/// - `RECLAIM_POLICY` ([Section 4.4](04-memory.md#page-cache)) and `VMM_POLICY`
/// ([Section 4.8](04-memory.md#virtual-memory-manager)) — `RcuCell<&'static dyn Policy>` swapped
/// only by live evolution;
/// - `CONG_CTL_REGISTRY` ([Section 16.10](16-networking.md#pluggable-tcp-congestion-control)) —
/// `RcuCell<[Option<CongCtlEntry>; MAX_CONG_CTLS]>` whose initial value is
/// the all-`None` table.
///
/// # Example
/// ```rust
/// // The initial value must have `'static` storage. For a trait-object
/// // policy, name the reference in a `static`, then point the cell at it:
/// static VMM_POLICY_INIT: &'static dyn VmmPolicy = &DefaultVmmPolicy;
/// pub static VMM_POLICY: RcuCell<&'static dyn VmmPolicy> =
/// RcuCell::new_static(&VMM_POLICY_INIT);
/// ```
pub const fn new_static(initial: &'static T) -> Self {
let p = initial as *const T;
Self {
// `AtomicPtr::new` is `const`; the `*const T -> *mut T` cast is a
// const-valid provenance-preserving cast (no int cast).
ptr: AtomicPtr::new(p as *mut T),
static_ptr: p,
}
}
pub fn read<'a>(&'a self, _guard: &'a RcuReadGuard) -> &'a T {
unsafe { &*self.ptr.load(Ordering::Acquire) }
}
/// Atomically replace the value. The old value is scheduled for deferred
/// freeing after all current RCU read-side critical sections complete
/// (grace period). The caller must NOT access the old value after this
/// call — RCU owns it and will free it asynchronously.
///
/// **Writer serialization**: Takes `&self` (not `&mut self`) plus a
/// `WriterProof` guard (sealed trait above) that demonstrates the caller
/// holds the co-located exclusive write-side lock. This is the standard
/// RCU pattern: readers are lock-free via `read()`, writers serialize
/// through an external exclusive lock — sleeping `Mutex` for
/// sleep-legal config paths, `SpinLock` for short non-sleeping
/// structural edits.
///
/// **Why `&self` instead of `&mut self`**: RCU cells are typically
/// stored in global/static data structures (routing tables, config
/// registries, module lists) accessed concurrently by many readers.
/// Requiring `&mut self` would force the caller to hold an exclusive
/// reference to the entire `RcuCell`, which is impractical for global
/// state — it would require wrapping the `RcuCell` in a `Mutex` or
/// `RwLock` that also blocks readers, defeating the purpose of RCU.
/// With `&self` + lock proof, the `RcuCell` can live in a shared
/// context (e.g., `static`, `Arc`, or behind `&`), readers access it
/// without any lock, and writers prove serialization by passing the
/// guard. The guard's lifetime ensures the lock is held for the
/// duration of the `update()` call but does not restrict access to
/// the `RcuCell` itself.
///
/// Concurrent writers without the lock would race on the swap: writer A
/// swaps old->new_A, writer B swaps new_A->new_B, then writer B defers
/// freeing new_A — which was just published and may have active readers.
/// The `WriterProof` guard prevents this at compile time.
///
/// All RcuCell allocation is fallible. Returns `Err(KernelError::OutOfMemory)`
/// if allocation fails; the existing value is left unchanged in that case.
pub fn update(
&self,
new_value: T,
_writer_lock: &impl WriterProof,
) -> Result<(), KernelError> {
let new_box = Box::try_new(new_value).map_err(|_| KernelError::OutOfMemory)?;
let old = self.ptr.swap(Box::into_raw(new_box), Ordering::Release);
// Schedule old value for deferred freeing after grace period — UNLESS
// it is the `'static` initial value installed by `new_static()`
// (identity-compared via `static_ptr`), which must never be freed.
// After the first `update()` on a `new_static` cell, every subsequent
// replaced value is a heap `Box` and IS reclaimed. For heap cells
// `static_ptr` is null, so this comparison never skips a real box.
if old as *const T != self.static_ptr {
// rcu_defer_free takes ownership of the raw pointer and will
// reconstruct the Box and drop it after the grace period elapses.
// SAFETY: `old` was created by a previous `Box::into_raw()` call
// (either in `new()` or a prior `update()`) and is not the
// `'static` initial value. The writer lock guarantees no concurrent
// `update()` can swap the same pointer twice. After this call, the
// caller must not access `old`.
unsafe { rcu_defer_free(old) };
}
Ok(())
}
}
// Implementation note on WriterProof tokens: The `_writer_lock` parameter
// ensures single-writer semantics at compile time. In the actual implementation,
// the writer lock (a `Mutex<()>`, `SpinLock<()>`, or the enclosing structure's
// own exclusive lock) must be embedded in the same struct that contains the
// `RcuCell`, or in a per-instance wrapper, so that each RcuCell has its own
// dedicated writer lock. Passing an unrelated exclusive guard would compile but
// violate the invariant. This is enforced structurally: the kernel's
// RCU-protected data structures always pair their `RcuCell` and its writer lock
// in the same struct (e.g., `struct RcuProtected<T> { cell: RcuCell<T>,
// writer_lock: SpinLock<()> }`), and the `update()` call site acquires the
// co-located lock. This pattern is standard in Rust kernel design (similar to how
// Linux's `struct rcu_head` is always embedded in the protected struct).
// Context discipline follows the LOCK, not the container: a writer holding a
// SpinLock guard is in atomic context for the duration of the update — the
// allocation inside `update()` (`Box::try_new`) must therefore use a
// non-sleeping allocation class on SpinLock-guarded sites, exactly as for
// any other allocation under a spinlock.
// Note: `RcuCell<T>` does NOT block in its `Drop` implementation.
// Calling `rcu_synchronize()` (a blocking wait) from `Drop` would be
// unsafe in contexts where blocking is illegal: interrupt handlers, code
// executing under a spinlock, NMI handlers, or any other atomic context.
// Because `RcuCell` values can be dropped from any of these contexts
// (e.g., a global `RcuCell` freed during module unload while holding a lock),
// `Drop` uses `rcu_call()` (deferred callback) instead. The current pointer
// is enqueued for deferred freeing via the RCU callback mechanism; the actual
// `Box::drop` runs in the RCU grace period worker thread, which executes in
// a fully schedulable, non-atomic context. This matches the pattern used by
// `update()`, which also defers old-value freeing via `rcu_defer_free()`.
impl<T: Send + Sync> Drop for RcuCell<T> {
fn drop(&mut self) {
// SAFETY: We have &mut self (exclusive access). This guarantees no
// concurrent writers (update() takes &self + a WriterProof guard, but &mut self
// is incompatible with any shared reference). Readers may still hold
// &T references obtained via read(); rcu_call() defers the actual
// Box::drop until all pre-existing RCU read-side critical sections
// complete (the grace period), ensuring no live references to the
// pointed-to value remain before it is freed. The pointer was created
// by Box::into_raw() in new() or update() and has not yet been passed
// to rcu_defer_free() (only old values replaced by update() are
// deferred there). This Drop path covers only the *current* (final)
// value still held by the RcuCell at destruction time.
//
// Using rcu_call() (non-blocking enqueue) rather than rcu_synchronize()
// (blocking wait) is mandatory here: Drop can be invoked from atomic
// contexts (interrupt handlers, spinlock-held paths, etc.) where
// blocking would deadlock or corrupt kernel state.
let ptr = self.ptr.load(Ordering::Relaxed);
// Skip a `'static` initial value from `new_static()` (identity via
// `static_ptr`): it is not a heap box and must not be freed. Heap cells
// have `static_ptr == null`, so only a genuine box reaches `rcu_call`.
if !ptr.is_null() && ptr as *const T != self.static_ptr {
// SAFETY: ptr was created by Box::into_raw() and is non-null.
// rcu_call takes ownership of the raw pointer and will reconstruct
// the Box and drop it after the grace period elapses in the RCU
// callback worker thread.
unsafe extern "C" fn drop_box<T>(ptr: *mut ()) {
// SAFETY: ptr was created by Box::into_raw::<T>() and is only
// passed to this callback once, after the RCU grace period.
drop(unsafe { Box::from_raw(ptr as *mut T) });
}
unsafe { rcu_call(drop_box::<T>, ptr as *mut ()) };
}
}
}
3.4.1.5.2 RcuPtr<T> — Nullable Single-Owner RCU Pointer¶
RcuPtr is the thin sibling of RcuCell. Where RcuCell<T> guarantees a
non-null value (read returns &T), RcuPtr<T> permits the NULL/absent state
and returns Option<&T>. It owns exactly one boxed T at a time (no internal
Arc, no shared refcount); a writer replaces the whole value under an external
mutex and the old box is reclaimed after a grace period. Reads are lock-free.
When to use which:
- RcuCell<T> — value always present (a policy, a registry snapshot).
- RcuPtr<T> — value may be absent (an empty tree, an uninitialized slot).
Example: the per-mm maple-tree root (Section 4.8), NULL for
an address space with no VMAs.
- To retain a reference PAST the RCU read section (e.g. across a copy_to_user
that may sleep), store an Arc: with RcuPtr<Arc<U>> a reader loads
&Arc<U> under the guard, clones the Arc (one refcount bump), drops the
guard, and holds the clone. This is exactly the read path of UTS strings
(RcuPtr<Arc<UtsStrings>>, Section 17.1) and IMA namespace
rules (RcuPtr<Arc<[ImaRule]>>, Section 9.5) — the
two dereferences those sections describe are box→Arc then Arc→payload.
(ArcSwap<U> in Section 3.1 is the alternative
when no nullable/boxed-owner semantics are wanted.)
/// A nullable, single-owner, RCU-protected pointer to a heap-allocated `T`.
/// Read path: one `Acquire` load of `ptr` (defined below alongside the
/// grace-period machinery — `RcuReadGuard`, `rcu_call`, `rcu_defer_free`).
pub struct RcuPtr<T: Send + Sync> {
ptr: AtomicPtr<T>,
}
impl<T: Send + Sync> RcuPtr<T> {
/// Construct an empty (NULL) `RcuPtr` in `const` context. No allocation —
/// usable directly in `static`/struct initializers (e.g. a fresh
/// `MapleTree` whose root is null until the first VMA is inserted).
pub const fn null() -> Self {
Self { ptr: AtomicPtr::new(core::ptr::null_mut()) }
}
/// Construct an `RcuPtr` owning `value` (heap-allocated via `Box::try_new`).
/// Fallible: returns `Err(OutOfMemory)` if allocation fails. Callers in
/// fallible constructors propagate with `?` (e.g.
/// `local_rules: RcuPtr::new(Arc::new([]))?`).
pub fn new(value: T) -> Result<Self, KernelError> {
let boxed = Box::try_new(value).map_err(|_| KernelError::OutOfMemory)?;
Ok(Self { ptr: AtomicPtr::new(Box::into_raw(boxed)) })
}
/// Load the current value under an active RCU read guard; `None` if NULL.
/// The returned borrow is bounded by the guard's lifetime.
///
/// **Single load**: `ptr` is loaded exactly once into a local, so a
/// concurrent `update()` cannot tear the observation.
pub fn read<'a>(&'a self, _guard: &'a RcuReadGuard) -> Option<&'a T> {
let p = self.ptr.load(Ordering::Acquire);
if p.is_null() {
None
} else {
// SAFETY: `p` was produced by `Box::into_raw`. The RCU guard keeps
// the target alive for `'a`: `update()`/`Drop` defer the box's free
// to after the grace period, so no reclaim can race this borrow.
Some(unsafe { &*p })
}
}
/// Atomically replace the value, or clear it with `value == None`. The
/// previous box (if any) is reclaimed after a grace period. Requires the
/// caller's external `WriterProof` guard, exactly like `RcuCell::update`:
/// two unsynchronized writers would swap the same pointer and double-free.
pub fn update(
&self,
value: Option<T>,
_writer_lock: &impl WriterProof,
) -> Result<(), KernelError> {
let new_raw = match value {
Some(v) => {
let boxed = Box::try_new(v).map_err(|_| KernelError::OutOfMemory)?;
Box::into_raw(boxed)
}
None => core::ptr::null_mut(),
};
let old = self.ptr.swap(new_raw, Ordering::Release);
if !old.is_null() {
// SAFETY: `old` came from a prior `Box::into_raw`; the writer lock
// serializes writers so it is swapped out exactly once. Deferred
// free waits out readers that may still borrow it via `read`.
unsafe { rcu_defer_free(old) };
}
Ok(())
}
}
impl<T: Send + Sync> Drop for RcuPtr<T> {
fn drop(&mut self) {
// SAFETY: exclusive `&mut self` ⇒ no concurrent writers. A non-null box
// is reclaimed via `rcu_call` (deferred, non-blocking — legal even
// under a spinlock / in IRQ context), matching `RcuCell::drop`.
let p = self.ptr.load(Ordering::Relaxed);
if !p.is_null() {
unsafe extern "C" fn drop_box<T>(ptr: *mut ()) {
// SAFETY: `ptr` came from `Box::into_raw::<T>()`, freed once.
drop(unsafe { Box::from_raw(ptr as *mut T) });
}
unsafe { rcu_call(drop_box::<T>, p as *mut ()) };
}
}
}
3.4.1.5.3 RcuHashMap<K, V> — RCU Reads, Per-Bucket-Locked Writes¶
An RCU-protected hash map: lock-free reads under an RcuReadGuard, writes
serialized per bucket (not by cloning the whole table). Reads never write
shared state (no cache-line bouncing); a writer touches only the one bucket it
mutates. This is the discipline the largest consumers already assume —
NetNamespace.tcp_ehash demultiplexes on the per-packet RX path and
inserts/removes on every connection establish/teardown
(Section 16.2), the bridge FDB churns per learned MAC
(Section 16.13). Those tables are read-hot AND
write-frequent, so the earlier "clone the bucket list and swap atomically /
written rarely" contract was wrong and is replaced here.
Other consumers: the KABI version registry (Section 13.18),
UserNamespace.users (Section 8.8), NFS exports —
all obtain the same API.
/// Initial bucket-array order used on the first `insert` (16 buckets).
const INITIAL_ORDER: u32 = 4;
/// Per-boot random 128-bit key `[k0, k1]` for the map hasher, installed once
/// during early init (before any network-facing table is populated) from the
/// CSPRNG. Keying `SipHasher13` (and `siphash_1_3`) with a boot-random key
/// defeats algorithmic-complexity collision attacks on attacker-influenced keys
/// (4-tuples, MAC addresses). Both words are published before any reader runs
/// (boot happens-before), so the `Relaxed` loads in `hash_key` observe the
/// installed key, never the zero placeholder. Two `AtomicU64`s rather than a
/// single word so the key is a full 128-bit SipHash key, matching the
/// `[u64; 2]` convention of `siphash_1_3` and `SipHasher13::new_keyed`.
static HASH_SEED: [AtomicU64; 2] = [AtomicU64::new(0), AtomicU64::new(0)];
/// Keyed SipHash-1-3 over exactly two 64-bit message words — the shared
/// keyed-hash utility for kernel hash tables that must resist
/// algorithmic-complexity collision attacks. Consumers: the mount hash
/// ([Section 14.6](14-vfs.md#mount-tree-data-structures-and-operations)), the default VFS dentry
/// hash, and the IPVS connection hash all call this with their two identifying
/// words.
///
/// `key` is a 128-bit key `[k0, k1]` established once at boot from a CSPRNG;
/// `w0`/`w1` are the two message words (e.g. `(parent_mount_id,
/// mountpoint_inode)`). SipHash-1-3 runs one `SipRound` per message block
/// (c = 1) and three at finalization (d = 3) — the fast variant Linux uses for
/// its fixed-width `siphash_*` helpers. Deterministic, allocation-free, ~20
/// cycles; safe on hot and IRQ paths.
/// One SipHash compression/finalization round (the ARX permutation over the
/// four internal words). Shared by the fixed-width `siphash_1_3` and the
/// streaming `SipHasher13` so the keyed core is defined exactly once.
#[inline]
fn siphash_sipround(v0: &mut u64, v1: &mut u64, v2: &mut u64, v3: &mut u64) {
*v0 = v0.wrapping_add(*v1);
*v1 = v1.rotate_left(13);
*v1 ^= *v0;
*v0 = v0.rotate_left(32);
*v2 = v2.wrapping_add(*v3);
*v3 = v3.rotate_left(16);
*v3 ^= *v2;
*v0 = v0.wrapping_add(*v3);
*v3 = v3.rotate_left(21);
*v3 ^= *v0;
*v2 = v2.wrapping_add(*v1);
*v1 = v1.rotate_left(17);
*v1 ^= *v2;
*v2 = v2.rotate_left(32);
}
#[inline]
pub fn siphash_1_3(key: [u64; 2], w0: u64, w1: u64) -> u64 {
// Standard SipHash IV, keyed with the 128-bit key.
let mut v0 = key[0] ^ 0x736f_6d65_7073_6575;
let mut v1 = key[1] ^ 0x646f_7261_6e64_6f6d;
let mut v2 = key[0] ^ 0x6c79_6765_6e65_7261;
let mut v3 = key[1] ^ 0x7465_6462_7974_6573;
// Two full 8-byte message blocks (c = 1 round each).
for m in [w0, w1] {
v3 ^= m;
siphash_sipround(&mut v0, &mut v1, &mut v2, &mut v3);
v0 ^= m;
}
// Final block: length (16 bytes) in the top byte, no trailing bytes.
let b: u64 = 16 << 56;
v3 ^= b;
siphash_sipround(&mut v0, &mut v1, &mut v2, &mut v3);
v0 ^= b;
// Finalization: d = 3 rounds.
v2 ^= 0xff;
siphash_sipround(&mut v0, &mut v1, &mut v2, &mut v3);
siphash_sipround(&mut v0, &mut v1, &mut v2, &mut v3);
siphash_sipround(&mut v0, &mut v1, &mut v2, &mut v3);
v0 ^ v1 ^ v2 ^ v3
}
/// Streaming keyed SipHash-1-3 — the `core::hash::Hasher` form of
/// `siphash_1_3`. Used where the message is not a fixed pair of 64-bit words
/// but an arbitrary byte stream produced by a `Hash` impl: `RcuHashMap::hash_key`
/// (generic `K: Hash`, below) and `FutexKey::hash`
/// ([Section 19.4](19-sysapi.md#futex-and-userspace-synchronization)). Keyed once at boot from a CSPRNG
/// so bucket placement is unpredictable (algorithmic-complexity DoS defence).
/// Compression uses c = 1 round per 8-byte block and d = 3 finalization rounds —
/// identical parameters to `siphash_1_3`, so both agree on the keyed core
/// (`siphash_sipround`).
///
/// Kernel-internal, allocation-free, `#![no_std]`. Not an ABI or wire type;
/// the digest is used only for in-memory bucket selection and is never
/// serialized, so its stability across compiler versions is irrelevant.
pub struct SipHasher13 {
/// The four SipHash state words (keyed at construction).
v0: u64,
v1: u64,
v2: u64,
v3: u64,
/// Total number of bytes consumed. Only the low 8 bits reach the final
/// block, but the full count is kept as `u64` so `write` cannot silently
/// wrap on very large inputs.
length: u64,
/// Partial (not-yet-compressed) message block, filled little-endian.
tail: u64,
/// Number of bytes currently buffered in `tail` (0..8).
ntail: usize,
}
impl SipHasher13 {
/// Construct keyed with a 128-bit key `[k0, k1]` (the same key convention as
/// `siphash_1_3`). The key is established once at boot from a CSPRNG; the
/// caller supplies the two 64-bit words.
#[inline]
pub fn new_keyed(key: [u64; 2]) -> Self {
Self {
v0: key[0] ^ 0x736f_6d65_7073_6575,
v1: key[1] ^ 0x646f_7261_6e64_6f6d,
v2: key[0] ^ 0x6c79_6765_6e65_7261,
v3: key[1] ^ 0x7465_6462_7974_6573,
length: 0,
tail: 0,
ntail: 0,
}
}
/// Compress one full 8-byte little-endian message block (c = 1 round).
#[inline]
fn compress_block(&mut self, m: u64) {
self.v3 ^= m;
siphash_sipround(&mut self.v0, &mut self.v1, &mut self.v2, &mut self.v3);
self.v0 ^= m;
}
}
impl core::hash::Hasher for SipHasher13 {
#[inline]
fn write(&mut self, mut bytes: &[u8]) {
self.length = self.length.wrapping_add(bytes.len() as u64);
// Top up an existing partial block first.
if self.ntail != 0 {
let take = (8 - self.ntail).min(bytes.len());
for (i, &b) in bytes[..take].iter().enumerate() {
self.tail |= (b as u64) << (8 * (self.ntail + i));
}
self.ntail += take;
bytes = &bytes[take..];
if self.ntail < 8 {
return; // block still incomplete
}
let m = self.tail;
self.compress_block(m);
self.tail = 0;
self.ntail = 0;
}
// Consume whole 8-byte blocks.
let mut chunks = bytes.chunks_exact(8);
for c in &mut chunks {
// `c` is exactly 8 bytes, so the conversion cannot fail.
let m = u64::from_le_bytes(c.try_into().unwrap());
self.compress_block(m);
}
// Buffer the leftover (< 8 bytes) for the next `write`/`finish`.
let rem = chunks.remainder();
for (i, &b) in rem.iter().enumerate() {
self.tail |= (b as u64) << (8 * i);
}
self.ntail = rem.len();
}
#[inline]
fn finish(&self) -> u64 {
// `finish` must not mutate `self` (it takes `&self`); work on copies.
let mut v0 = self.v0;
let mut v1 = self.v1;
let mut v2 = self.v2;
let mut v3 = self.v3;
// Final block: buffered tail bytes plus the total length (mod 256) in
// the top byte.
let b: u64 = ((self.length & 0xff) << 56) | self.tail;
v3 ^= b;
siphash_sipround(&mut v0, &mut v1, &mut v2, &mut v3);
v0 ^= b;
// Finalization: d = 3 rounds.
v2 ^= 0xff;
siphash_sipround(&mut v0, &mut v1, &mut v2, &mut v3);
siphash_sipround(&mut v0, &mut v1, &mut v2, &mut v3);
siphash_sipround(&mut v0, &mut v1, &mut v2, &mut v3);
v0 ^ v1 ^ v2 ^ v3
}
}
/// RCU-protected hash map. Reads are lock-free under an `RcuReadGuard`; writers
/// take the target bucket's `SpinLock`. Removed nodes and replaced bucket
/// arrays are reclaimed via `rcu_call` after a grace period.
pub struct RcuHashMap<K: Eq + Hash + Send + Sync, V: Clone + Send + Sync> {
/// Published bucket array (power-of-two length), or null before the first
/// insert (lazy allocation — keeps `new()` `const` and infallible so it can
/// appear in struct/`static` initializers).
table: AtomicPtr<RcuBucketArray<K, V>>,
/// Live entry count, used only to trigger a resize. Approximate under
/// concurrent writers to different buckets; only compared to the threshold.
count: AtomicUsize,
/// Serializes bucket-array replacement (resize) against other resizes and
/// against the lazy first-array allocation. Per-bucket locks continue to
/// serialize entry insert/remove within a given array generation.
resize_lock: Mutex<()>,
}
/// A bucket array of `1 << order` buckets — one heap allocation (DST tail).
/// Allocated by the same length-prefixed protocol as `RcuVecInner` (below):
/// `Layout::new::<usize>()` (the `mask` header) extended by
/// `Layout::array::<Bucket<K, V>>(1 << order)`, then the fat pointer is
/// synthesized from the thin base and the element count.
struct RcuBucketArray<K, V> {
/// `mask == (1 << order) - 1`; bucket index = `hash & mask`.
mask: usize,
buckets: [Bucket<K, V>], // exactly `mask + 1` buckets
}
/// One RCU hash bucket: a writer lock plus an RCU-walkable chain head.
struct Bucket<K, V> {
lock: SpinLock<()>,
head: AtomicPtr<HashNode<K, V>>,
}
/// A single map entry, reclaimed via `rcu_call` after it is unlinked.
struct HashNode<K, V> {
hash: u64,
key: K,
value: V,
/// Next node in the bucket chain (RCU-walked by readers).
next: AtomicPtr<HashNode<K, V>>,
}
impl<K: Eq + Hash + Send + Sync, V: Clone + Send + Sync> RcuHashMap<K, V> {
/// Empty map, no allocation; the first bucket array is allocated lazily on
/// the first `insert`. `const` for use in initializers.
pub const fn new() -> Self {
Self {
table: AtomicPtr::new(core::ptr::null_mut()),
count: AtomicUsize::new(0),
resize_lock: Mutex::new(()),
}
}
/// The map's hash function: SipHash-1-3 keyed once at boot (defeats
/// algorithmic-complexity collision attacks on network-facing tables). A
/// consumer whose key already carries a good hash (e.g. `FourTuple` with
/// Linux's `inet_ehashfn`) may override by supplying it as the `hash` field
/// of a pre-hashed node; the table stores `node.hash` and compares it
/// before the full-key compare so the hasher runs at most once per op.
fn hash_key(&self, key: &K) -> u64 {
let seed = [
HASH_SEED[0].load(Ordering::Relaxed),
HASH_SEED[1].load(Ordering::Relaxed),
];
let mut h = SipHasher13::new_keyed(seed);
key.hash(&mut h);
h.finish()
}
/// Look up `key` under the caller's RCU guard. Returns a CLONE of the value
/// (for `V = Arc<TcpCb>` this is an `Arc` clone — a refcount bump, so the
/// caller may hold it past the guard). `None` if absent or table empty.
pub fn lookup(&self, key: &K, _guard: &RcuReadGuard) -> Option<V> {
// One `Acquire` load of the array — a concurrent resize is observed
// whole (old or new generation), never torn.
let t = self.table.load(Ordering::Acquire);
if t.is_null() {
return None;
}
// SAFETY: `t` is one consistent observation; the RCU guard keeps this
// array and its nodes alive for the walk (resize/remove defer their
// frees past the grace period).
let arr = unsafe { &*t };
let h = self.hash_key(key);
let bucket = &arr.buckets[(h as usize) & arr.mask];
let mut node = bucket.head.load(Ordering::Acquire);
while !node.is_null() {
// SAFETY: node kept alive by the guard (see above).
let n = unsafe { &*node };
if n.hash == h && n.key == *key {
return Some(n.value.clone());
}
node = n.next.load(Ordering::Acquire);
}
None
}
/// Insert or overwrite `key`. Fallible (allocates a node, and — on the
/// first insert — the initial bucket array).
pub fn insert(&self, key: K, value: V) -> Result<(), KernelError> {
let h = self.hash_key(&key);
// Step 1: ensure a bucket array exists (lazy first allocation).
let arr = self.ensure_table()?;
let bucket = &arr.buckets[(h as usize) & arr.mask];
// Step 2: serialize writers to this bucket.
let _bg = bucket.lock.lock();
// Step 3: overwrite existing, or prepend new.
let mut prev = &bucket.head;
let mut cur = prev.load(Ordering::Acquire);
while !cur.is_null() {
// SAFETY: bucket lock held; node not yet reclaimed.
let n = unsafe { &*cur };
if n.hash == h && n.key == key {
let repl = HashNode::alloc(h, key, value, n.next.load(Ordering::Acquire))?;
prev.store(repl, Ordering::Release);
// SAFETY: `cur` unlinked; readers mid-walk drain by grace period.
unsafe { rcu_defer_free(cur) };
return Ok(());
}
prev = &n.next;
cur = n.next.load(Ordering::Acquire);
}
let node = HashNode::alloc(h, key, value, bucket.head.load(Ordering::Acquire))?;
bucket.head.store(node, Ordering::Release);
drop(_bg);
// Step 4: resize if the load factor is exceeded.
if self.count.fetch_add(1, Ordering::Relaxed) + 1 > (arr.mask + 1) * 3 / 4 {
self.grow();
}
Ok(())
}
/// Remove `key` (no-op if absent). Takes the bucket lock, unlinks the node
/// with a single `next`/`head` store, decrements `count`, then
/// `rcu_call`-frees the unlinked node (readers mid-walk keep their borrow
/// valid until the grace period ends).
pub fn remove(&self, key: &K) {
let t = self.table.load(Ordering::Acquire);
if t.is_null() {
return;
}
// SAFETY: array kept alive — no resize frees it while this writer runs
// (resize takes `resize_lock` and republishes; the old array is freed
// only after a grace period, and this thread is in no RCU section that
// matters because it re-reads through the bucket lock below).
let arr = unsafe { &*t };
let h = self.hash_key(key);
let bucket = &arr.buckets[(h as usize) & arr.mask];
let _bg = bucket.lock.lock();
let mut prev = &bucket.head;
let mut cur = prev.load(Ordering::Acquire);
while !cur.is_null() {
// SAFETY: bucket lock held.
let n = unsafe { &*cur };
if n.hash == h && n.key == *key {
prev.store(n.next.load(Ordering::Acquire), Ordering::Release);
self.count.fetch_sub(1, Ordering::Relaxed);
// SAFETY: `cur` unlinked; deferred free waits out readers.
unsafe { rcu_defer_free(cur) };
return;
}
prev = &n.next;
cur = n.next.load(Ordering::Acquire);
}
}
/// Visit every entry (warm/cold: iteration, teardown). Holds an internal
/// RCU read section for the walk; `f` must not sleep.
pub fn for_each<F: FnMut(&K, &V)>(&self, mut f: F) {
let guard = rcu_read_lock();
let t = self.table.load(Ordering::Acquire);
if t.is_null() {
return;
}
// SAFETY: guard held; array and nodes alive for the walk.
let arr = unsafe { &*t };
for bucket in arr.buckets.iter() {
let mut node = bucket.head.load(Ordering::Acquire);
while !node.is_null() {
// SAFETY: guard held.
let n = unsafe { &*node };
f(&n.key, &n.value);
node = n.next.load(Ordering::Acquire);
}
}
drop(guard);
}
/// Look up `key`, or insert `init()` if absent, returning a CLONE of the
/// resident value either way (for `V = Arc<T>`, a refcount bump the caller
/// may keep past any RCU guard). The check-and-insert is atomic under the
/// bucket lock, so concurrent callers racing on the same key converge on the
/// single published value. `init` runs OUTSIDE the bucket lock (it may
/// allocate a large object); a caller that loses the race drops its freshly
/// built value and returns the winner's. Warm path (first-touch per key).
pub fn get_or_insert_with(
&self,
key: K,
init: impl FnOnce() -> V,
) -> Result<V, KernelError> {
// Fast path: already present — a lock-free read, no `init` call.
{
let guard = rcu_read_lock();
if let Some(v) = self.lookup(&key, &guard) {
return Ok(v);
}
}
// Absent so far: build the value before taking the bucket lock, so a
// large `init` allocation does not run under the spinlock.
let value = init();
let h = self.hash_key(&key);
let arr = self.ensure_table()?;
let bucket = &arr.buckets[(h as usize) & arr.mask];
let _bg = bucket.lock.lock();
// Re-check under the lock: a concurrent caller may have inserted `key`
// between the fast-path read and here.
let mut cur = bucket.head.load(Ordering::Acquire);
while !cur.is_null() {
// SAFETY: bucket lock held; node not yet reclaimed.
let n = unsafe { &*cur };
if n.hash == h && n.key == key {
return Ok(n.value.clone()); // lost the race; `value` is dropped
}
cur = n.next.load(Ordering::Acquire);
}
let node = HashNode::alloc(h, key, value.clone(), bucket.head.load(Ordering::Acquire))?;
bucket.head.store(node, Ordering::Release);
drop(_bg);
if self.count.fetch_add(1, Ordering::Relaxed) + 1 > (arr.mask + 1) * 3 / 4 {
self.grow();
}
Ok(value)
}
/// Ensure `table` is non-null, allocating the initial `INITIAL_ORDER`
/// (16-bucket) array under `resize_lock` on first use (double-checked to
/// avoid a racing double allocation). Returns the published array.
fn ensure_table(&self) -> Result<&RcuBucketArray<K, V>, KernelError> {
let t = self.table.load(Ordering::Acquire);
if !t.is_null() {
// SAFETY: published array; lives until a resize replaces it, and a
// replacement is only freed after a grace period.
return Ok(unsafe { &*t });
}
let _rg = self.resize_lock.lock();
let t2 = self.table.load(Ordering::Acquire);
let arr = if t2.is_null() {
let fresh = RcuBucketArray::alloc(INITIAL_ORDER)?;
self.table.store(fresh, Ordering::Release);
fresh
} else {
t2
};
// SAFETY: just published (or observed) under `resize_lock`.
Ok(unsafe { &*arr })
}
/// Resize primitive — see the **Resize policy** paragraph below. Takes
/// `resize_lock`, allocates a `2×` array, rehashes every node into it,
/// publishes it (`Release`), and `rcu_call`-frees the old array.
fn grow(&self) {
// Body per the Resize policy paragraph: double the bucket count,
// move (not clone) nodes into `hash & new_mask` buckets, publish, and
// `rcu_defer_free` the old `RcuBucketArray`.
}
}
Resize policy (grow()): when a writer observes count > capacity * 3 / 4
it takes resize_lock, allocates a 2× bucket array, moves every node into its
new bucket (rehash by hash & new_mask — values are moved, never cloned),
publishes the new array (Release), and rcu_calls the old array. Bucket count
is discovered from live load, never a compile-time cap; at 10⁶ entries the table
self-sizes to ~2²¹ buckets (load factor < 1), so per-packet lookups stay O(1)
expected. Shrink is not performed — a table that spikes then drains retains its
peak bucket count (each bucket is one pointer + one lock; re-growth churn is
worse than the retained footprint). Concurrent readers observe either the old or
the new array (one Acquire load of table) and walk a consistent generation.
3.4.1.5.4 RcuVec<T> — RCU Immutable-Snapshot Vector¶
An RCU-protected immutable-snapshot vector: lock-free reads under an
RcuReadGuard, whole-array replacement under an external writer mutex. Each
published snapshot is a single heap allocation (a length-prefixed flat array);
an update allocates a fresh snapshot, copies the elements, swaps the pointer
(Release), and reclaims the old snapshot after a grace period. Suited to
read-mostly collections with rare STRUCTURAL change: KVM memslots
(Section 18.1), IMA's committed measurement log
(Section 9.5), the ksmbd share table
(Section 15.21).
/// RCU immutable-snapshot vector. Read: one `Acquire` load of `ptr`.
pub struct RcuVec<T: Clone + Send + Sync> {
/// Published snapshot (`RcuVecInner`), or null when empty.
ptr: AtomicPtr<RcuVecInner<T>>,
}
/// One immutable published snapshot: a length followed by exactly `len`
/// contiguous `T`. There is deliberately **no spare-capacity field** — the
/// vector is replaced wholesale on every update, so unused capacity could never
/// be consumed; each snapshot is allocated to its exact length. Construction
/// (`new_inner(elems)`):
/// 1. `layout = Layout::new::<usize>().extend(Layout::array::<T>(elems.len())?)?.0.pad_to_align()`
/// — the `len` header word followed by the element array;
/// 2. `base = alloc(layout)` (fallible → `OutOfMemory`);
/// 3. write `elems.len()` into the header word, then `clone` each element
/// into `data[i]`;
/// 4. synthesize the fat pointer `*mut RcuVecInner<T>` from the thin `base`
/// and the element count as DST metadata
/// (`core::ptr::slice_from_raw_parts_mut(base, len)` cast to the inner
/// type). Reclamation (`rcu_defer_free_vec`) recomputes the identical
/// `layout` from the snapshot's stored `len`, drops the elements in place,
/// and `dealloc`s.
struct RcuVecInner<T> {
len: usize,
data: [T], // trailing unsized array, allocated to exactly `len`
}
impl<T: Clone + Send + Sync> RcuVec<T> {
/// Empty vector, no allocation. `const` for use in initializers
/// (e.g. `committed: RcuVec::new()`).
pub const fn new() -> Self {
Self { ptr: AtomicPtr::new(core::ptr::null_mut()) }
}
/// Read the current snapshot under an RCU read guard.
///
/// **Single load**: `ptr` is loaded EXACTLY ONCE into a local; both the
/// length and the element data are read from that one observation. This
/// closes the torn-read race a two-load form would have — a concurrent
/// `update()` publishing a longer snapshot between two loads could
/// otherwise splice the new `len` onto the old (shorter) `data`, producing
/// an out-of-bounds slice.
pub fn load<'g>(&self, _guard: &'g RcuReadGuard) -> &'g [T] {
let inner = self.ptr.load(Ordering::Acquire);
if inner.is_null() {
return &[];
}
// SAFETY: `inner` is one consistent observation produced by a prior
// `new_inner()`; the RCU guard keeps it alive for `'g` (reclamation is
// deferred past the grace period). `len` and `data` therefore belong to
// the same snapshot — no torn read.
unsafe { &(*inner).data[..(*inner).len] }
}
/// Replace the whole vector with `new_elements`. The caller MUST hold the
/// vector's dedicated external writer lock and pass its guard as the
/// `WriterProof` — two concurrent `update()`s would each clone from the
/// same old snapshot, and one writer's elements would be silently lost.
/// The old snapshot is reclaimed after a grace period. Fallible
/// (allocates the new snapshot).
///
/// Each consumer names its writer lock: KVM `Vm::memslots_update_lock`
/// ([Section 18.1](18-virtualization.md#host-and-guest-integration)); IMA the enclosing
/// `measurement_log: Mutex<ImaMeasurementLog>`
/// ([Section 9.5](09-security.md#runtime-integrity-measurement)) — a payload guard,
/// `MutexGuard<'_, ImaMeasurementLog>`, satisfies `WriterProof` directly.
/// A consumer that has no natural enclosing lock (e.g. ksmbd's genl
/// share-config path, [Section 15.21](15-storage.md#smb-server-ksmbd)) MUST declare a per-table
/// `Mutex<()>` and hold it here — single-writer-by-protocol is not
/// sufficient on its own and must be made explicit in the type.
pub fn update(
&self,
new_elements: &[T],
_writer_lock: &impl WriterProof,
) -> Result<(), KernelError> {
let new_inner = Self::new_inner(new_elements)?; // Layout protocol above
let old = self.ptr.swap(new_inner, Ordering::Release);
if !old.is_null() {
// SAFETY: `old` came from a prior `new_inner`; the writer lock
// serializes writers so it is swapped out once. Deferred free waits
// out readers still borrowing it via `load`.
unsafe { rcu_defer_free_vec(old) };
}
Ok(())
}
}
impl<T: Clone + Send + Sync> Drop for RcuVec<T> {
fn drop(&mut self) {
// SAFETY: exclusive `&mut self`. A non-null snapshot is reclaimed via
// `rcu_call` with the `rcu_defer_free_vec` reclaimer (deferred,
// non-blocking), which drops the elements and deallocs the DST.
let inner = self.ptr.load(Ordering::Relaxed);
if !inner.is_null() {
unsafe { rcu_defer_free_vec(inner) };
}
}
}
/// Reclaim an `RcuVecInner<T>` snapshot after a grace period. The sibling of
/// `rcu_defer_free` for the length-prefixed DST: it enqueues an `rcu_call`
/// whose callback recomputes the allocation `Layout` from the snapshot's stored
/// `len`, drops each element in place, and `dealloc`s. The fat
/// `*mut RcuVecInner<T>` degrades to its thin base address across the
/// `*mut ()` callback boundary; `len` is recovered from the header word, and
/// the fat pointer is re-synthesized (`core::ptr::from_raw_parts`) before drop.
///
/// # Safety
/// `inner` must have been produced by `RcuVec::new_inner` and not yet reclaimed.
unsafe fn rcu_defer_free_vec<T>(inner: *mut RcuVecInner<T>) {
unsafe extern "C" fn reclaim<T>(base: *mut ()) {
// SAFETY: `base` is the thin address of one `RcuVecInner<T>`; the header
// word holds `len`, from which the DST layout and element count are
// recovered exactly once, after the grace period.
let len = unsafe { *(base as *const usize) };
let fat: *mut RcuVecInner<T> =
core::ptr::from_raw_parts_mut(base, len);
drop(unsafe { Box::from_raw(fat) });
}
// The fat->thin cast keeps the base address; `reclaim` re-derives `len`.
unsafe { rcu_call(reclaim::<T>, inner as *mut ()) };
}
RCU Grace Period Detection — Hierarchical Tree-RCU:
The grace period mechanism determines when all pre-existing RCU read-side critical
sections have completed, making it safe to execute deferred callbacks (such as freeing
old values swapped out by RcuCell::update()).
UmkaOS uses hierarchical tree-RCU — a multi-level tree of RcuNode structures
that aggregates per-CPU quiescent state reports bottom-up. This is the same
fundamental design as Linux's Tree RCU (introduced in 2.6.29), adapted for
UmkaOS's non-preemptible model.
Why a tree, not flat per-CPU polling? A flat array of per-CPU quiescent state
flags requires the GP kthread to sequentially poll every online CPU. On a 256-CPU
system with 8 NUMA nodes, even with per-node threads, each thread polls 32 CPUs
sequentially — 32 remote cache line reads per GP. With a tree (fan-out 64), a
single root node covers 64 leaf nodes, each covering 64 CPUs. Quiescent state
propagation is O(log_{fanout}(nr_cpus)) per CPU, and the GP kthread only monitors
the root node's qsmask. On 4096 CPUs this is 2 levels instead of 4096 polls.
Data structures:
/// `WaitQueue` is an alias for `WaitQueueHead` — the type used for blocking
/// waiters on grace period completion. Using the alias keeps RCU-specific
/// code readable without importing internal queue type names.
pub type WaitQueue = WaitQueueHead;
// ---------------------------------------------------------------------------
// Hierarchical RCU Node Tree
// ---------------------------------------------------------------------------
/// One node in the hierarchical RCU tree.
///
/// The tree is built at boot time based on the actual CPU topology discovered
/// from ACPI/DT (no compile-time MAX_CPUS constant). Leaf nodes cover contiguous
/// ranges of CPUs; interior nodes aggregate their children's quiescent state.
///
/// **Cache alignment**: Each `RcuNode` is 64-byte aligned to prevent false sharing
/// between nodes on different NUMA domains. The `lock` field and `qsmask` field are
/// co-located within the same cache line because they are always accessed together.
/// Lock level for `RcuNode.lock` — row 52 in the master lock table below.
///
/// Above `RQ_LOCK(50)`: the QS-propagation call sites (`timer_tick_handler()`
/// step 3, `finish_task_switch()` step 1a, GP-kthread FQS) hold NO runqueue
/// lock — that is a documented PRECONDITION of `rcu_report_qs_leaf()` — and
/// the propagation may end in `scheduler::unblock(gp_kthread)` → RQ_LOCK(50),
/// which is legal only because every RcuNode lock has been RELEASED before
/// the unblock (two-phase discipline; see `rcu_report_qs_up()`). RcuNode
/// locks are never nested with EACH OTHER either: leaf and each parent are
/// acquired sequentially, each released before the next is taken.
pub const RCU_NODE_LOCK_LEVEL: u32 = 52;
// kernel-internal, not KABI — RcuNode contains SpinLock and raw pointers.
#[repr(C, align(64))]
pub struct RcuNode {
/// Spinlock protecting `qsmask`, `gp_seq`, and QS-related fields in this node.
/// Held briefly during quiescent state propagation (one atomic bit-clear + check).
/// IRQ-saving: QS reporting can occur from the timer-tick hardirq path.
/// Ordered at `RCU_NODE_LOCK_LEVEL` (52) — see the constant's doc.
pub lock: SpinLock<RcuNodeInner, RCU_NODE_LOCK_LEVEL>,
/// Parent node in the tree. `None` for the root node.
/// Set once at boot during `rcu_build_tree()`, never modified after.
// SAFETY: Points into the `RcuState::nodes` Box<[RcuNode]> which is
// allocated at boot and never freed or reallocated. The pointer is
// valid for the kernel's lifetime. Only dereferenced in
// `rcu_report_qs_up()` with the parent node's spinlock held.
pub parent: Option<*const RcuNode>,
/// Index of this node in the parent's children array. Used to compute the
/// bit position to clear in the parent's `qsmask`.
/// Set once at boot, never modified.
pub parent_idx: u16,
/// Level in the tree (0 = root, increasing toward leaves).
/// Set once at boot, never modified.
pub level: u8,
/// For leaf nodes: index of the first CPU covered by this node.
/// For interior nodes: index of the first CPU covered by any descendant leaf.
/// Set once at boot, never modified.
pub cpu_lo: u32,
/// For leaf nodes: index of the last CPU (inclusive) covered by this node.
/// For interior nodes: index of the last CPU covered by any descendant leaf.
/// Set once at boot, never modified.
pub cpu_hi: u32,
/// NUMA node affinity hint. The GP kthread for this NUMA node monitors
/// the corresponding subtree. For interior nodes that span multiple NUMA
/// nodes, this is the NUMA node of the first child.
pub numa_node: u32,
}
// RcuNode: repr(C, align(64)) single cache line. SpinLock<RcuNodeInner>(4 lock
// word + 4 pad + 32 inner = 40) + parent Option<*const>(8) + parent_idx(2)
// + level(1) + pad(1) + cpu_lo(4) + cpu_hi(4) + numa_node(4) = 64.
const_assert!(core::mem::size_of::<RcuNode>() == 64);
/// Mutable state within an `RcuNode`, protected by `RcuNode::lock`.
pub struct RcuNodeInner {
/// Bitmask of children (leaf CPUs or child nodes) that have **not yet**
/// reported a quiescent state for the current grace period.
///
/// - For a **leaf node**: bit N corresponds to the (N + self.cpu_lo)-th CPU.
/// When CPU C passes a quiescent state, bit (C - cpu_lo) is cleared.
/// - For an **interior node**: bit N corresponds to child node index N.
/// When child node N's `qsmask` reaches zero, bit N in this node is cleared.
///
/// When `qsmask == 0`, all descendants have reported quiescent states.
/// For interior nodes, this triggers propagation to the parent.
/// For the root node, `qsmask == 0` means the grace period is complete.
///
/// `u64` supports up to 64 children per node (matching Linux's RCU_FANOUT).
/// On systems with more than 64 CPUs per leaf group, increase tree depth.
pub qsmask: u64,
/// Grace period sequence number last seen by this node. Used to detect
/// stale QS reports from a previous grace period (if a CPU reports a QS
/// for an already-completed GP, the report is silently discarded).
pub gp_seq: u64,
/// Number of children that this node covers.
/// For leaf nodes: number of online CPUs in range [cpu_lo, cpu_hi].
/// For interior nodes: number of child `RcuNode` entries.
/// Set at boot, updated on CPU hotplug.
pub n_children: u16,
/// Bitmask of children that are online/active. Used during GP initialization
/// to set `qsmask` to only the bits corresponding to active children.
/// Updated on CPU hotplug (online/offline).
pub online_mask: u64,
}
/// Global RCU state — one instance, allocated and initialized at boot.
pub struct RcuState {
/// Monotonically increasing grace period sequence number.
///
/// **Encoding**: The low 2 bits encode the grace period phase:
/// - `0b00` (phase 0): idle — no grace period in progress.
/// - `0b01` (phase 1): GP started — `qsmask` initialized on all nodes,
/// waiting for all CPUs to report quiescent states.
/// - `0b10` (phase 2): GP completing — all QS reported, callbacks being
/// advanced, `gp_seq` about to be incremented to the next idle phase.
/// - `0b11`: reserved (never used; provides detection of corruption).
///
/// Each complete grace period advances `gp_seq` by 4 (one full cycle through
/// idle → started → completing → idle). The upper 62 bits form the grace
/// period number proper.
///
/// *Wraparound*: at the theoretical maximum rate of one grace period per 10 μs,
/// a `u64` counter wraps after approximately 5.8 million years. No special
/// wraparound handling is required for `u64`.
///
/// *Comparison semantics*: any code that compares two `gp_seq` values to decide
/// whether grace period A completed before grace period B **must** use wrapping
/// subtraction rather than a direct `<` comparison. Direct comparison is
/// incorrect near the wraparound boundary (irrelevant for `u64` in practice,
/// but required by design so that a future space-saving change to `u32` — where
/// wraparound occurs in ~42 seconds at 100 K GP/s — cannot silently introduce
/// a correctness bug):
///
/// ```rust
/// /// GP sequence phase constants.
/// pub const RCU_GP_IDLE: u64 = 0;
/// pub const RCU_GP_STARTED: u64 = 1;
/// pub const RCU_GP_COMPLETING: u64 = 2;
/// pub const RCU_GP_PHASE_MASK: u64 = 0x3;
///
/// /// Returns true if grace period `a` completed strictly before `b`.
/// #[inline]
/// fn gp_before(a: u64, b: u64) -> bool {
/// (b.wrapping_sub(a) as i64) > 0
/// }
///
/// /// Extract the phase from a gp_seq value.
/// #[inline]
/// fn gp_phase(seq: u64) -> u64 {
/// seq & RCU_GP_PHASE_MASK
/// }
/// ```
///
/// All callers that compare grace period sequence numbers (e.g.,
/// `rcu_synchronize` checking whether its target sequence has been reached,
/// `rcu_gp_kthread` waking waiters) must use `gp_before` or an equivalent
/// wrapping form. Linux's `time_before` / `time_after` macros use the
/// same pattern for the same reason.
///
/// 64-bit-atomic family member: `AtomicU64Exact` — a tear-free CORRECTNESS
/// value (the wrapping `gp_before` comparisons require exact reads; a torn
/// half-read would corrupt them, so neither the torn-tolerant `AtomicU64Cell`
/// nor the add-only `AtomicU64Counter` is admissible). Single-writer: only the
/// GP kthread stores it, as `load(Relaxed) + 1` then `store(Release)` at GP
/// start/completion — a WARM/COLD writer at GP boundaries, so it stays clear
/// of `AtomicU64Exact`'s 32-bit-leg leaf-`SpinLock` cost gate (on a 32-bit leg
/// every access — readers included — takes the lock, so HOT-PATH use there
/// requires an explicit justification; GP boundaries are warm/cold)
/// ([Section 3.5](#locking-strategy--64-bit-atomics-on-32-bit-legs-the-semantic-family)).
pub gp_seq: AtomicU64Exact,
/// Flat array of all `RcuNode` entries in the tree, level by level.
/// Index 0 = root. Indices [1..1+fan_out) = level-1 nodes. Etc.
/// Dynamically allocated at boot based on discovered CPU count and fan-out.
/// Never reallocated after boot (CPU hotplug updates `online_mask` and
/// `n_children` within existing nodes but does not grow the tree).
pub nodes: Box<[RcuNode]>,
/// Number of levels in the tree (1 = root only, for systems with <= fan_out CPUs).
pub num_levels: u8,
/// Fan-out of the tree (children per interior node). Default: 64.
/// Configurable at boot via `rcu.fanout=N` kernel parameter.
/// Must be in range [2, 64] (constrained by `qsmask: u64`).
pub fan_out: u8,
/// Fan-out of leaf nodes (CPUs per leaf). May differ from `fan_out` on
/// architectures where leaf fan-out is constrained by cache topology.
/// Default: same as `fan_out`. Configurable via `rcu.leaf_fanout=N`.
pub leaf_fan_out: u8,
/// Array of pointers from CPU ID → leaf `RcuNode` that covers that CPU.
/// Length = `num_possible_cpus()`. Index = CPU logical ID.
/// Dynamically allocated at boot; never reallocated.
// SAFETY: Each element points into the `nodes` array (same Box allocation).
// Valid for the kernel's lifetime. Read-only after boot except during
// CPU hotplug (which updates under the global CPU hotplug lock).
pub cpu_to_leaf: Box<[*const RcuNode]>,
/// Dedicated kernel thread that drives grace period progression.
/// One per system (pinned to NUMA node 0's first online CPU).
/// Scheduled at `SCHED_FIFO` priority 10.
pub gp_kthread: *mut Task,
/// Wait queue for the GP kthread. The GP kthread sleeps here between
/// grace periods, waiting for `gp_requested` to become true. Separated
/// from `gp_completion_wq` to prevent missed-wakeup races: the kthread
/// uses `scheduler::unblock()` semantics, while `rcu_synchronize()`
/// callers use `wait_event()` with a condition predicate. Conflating
/// both on a single wait queue risks the kthread consuming a wakeup
/// intended for a synchronize caller, or vice versa.
pub gp_kthread_wq: WaitQueue,
/// Wait queue for `rcu_synchronize()` callers. Tasks blocked in
/// `rcu_synchronize()` sleep here until the GP kthread wakes them
/// after `gp_seq` advances past their target sequence number.
/// Wakeup: GP kthread broadcasts to all waiters whose
/// `wait_gp_seq <= complete_seq` at GP completion (step 13).
pub gp_completion_wq: WaitQueue,
/// Set to `true` by `rcu_call()` or `rcu_synchronize()` when a new GP
/// is needed. The GP kthread checks this on wakeup.
pub gp_requested: AtomicBool,
/// Force-quiescent-state interval in nanoseconds. Default: 1 ms.
/// After this interval without a QS report from a CPU, the GP kthread
/// sends that CPU a reschedule IPI.
/// Configurable at boot via `rcu.fqs_interval_ns=N`.
pub fqs_interval_ns: u64,
}
impl RcuState {
/// Root of the hierarchical `RcuNode` tree. The grace period is complete
/// when `root().lock().qsmask == 0`.
///
/// Derives the root as `&self.nodes[0]` — no separate raw pointer field needed.
/// The `nodes` array is allocated at boot and never freed or reallocated, so
/// the returned reference is valid for the kernel's lifetime. This eliminates
/// the redundant raw pointer (previously `pub root: *const RcuNode`) that
/// created a dangling pointer risk if `nodes` were ever dropped.
#[inline(always)]
pub fn root(&self) -> &RcuNode {
&self.nodes[0]
}
}
/// Tree construction at boot time.
///
/// Called once from `rcu_init()`, after the CPU topology is discovered from
/// ACPI MADT / device tree / platform enumeration. The tree geometry is
/// computed from the actual number of possible CPUs and the configured fan-out.
///
/// **Algorithm** (`rcu_build_tree`):
/// ```
/// 1. nr_cpus = num_possible_cpus() // discovered from ACPI/DT, not hardcoded
/// 2. fan_out = boot_param("rcu.fanout", default=64), clamped to [2, 64]
/// 3. leaf_fan_out = boot_param("rcu.leaf_fanout", default=fan_out)
/// 4. Compute tree geometry:
/// - num_leaf_nodes = ceil(nr_cpus / leaf_fan_out)
/// - For each level L from leaves toward root:
/// nodes_at_L = ceil(nodes_at_(L+1) / fan_out)
/// until nodes_at_L == 1 (root level).
/// - num_levels = number of levels computed.
/// - total_nodes = sum of nodes at all levels.
/// 5. Allocate nodes: Box<[RcuNode]> with `total_nodes` entries (boot allocator).
/// 6. Initialize each node:
/// - Set level, parent pointer, parent_idx, cpu_lo, cpu_hi.
/// - Leaf nodes: cpu_lo = first CPU index, cpu_hi = last CPU index in range.
/// - Interior nodes: cpu_lo/cpu_hi span the union of children's ranges.
/// - Set numa_node based on ACPI SRAT proximity domain of first covered CPU.
/// 7. Allocate cpu_to_leaf: Box<[*const RcuNode]> with nr_cpus entries.
/// 8. For each CPU C, set cpu_to_leaf[C] = pointer to its covering leaf node.
/// 9. Initialize all qsmask = 0, online_mask = 0 (CPUs are offline at this point).
/// 10. Start gp_kthread on NUMA node 0.
/// ```
///
/// **Example**: 256-CPU system, fan_out=64:
/// - Leaf level: ceil(256/64) = 4 leaf nodes, each covering 64 CPUs.
/// - Root level: 1 root node with 4 children.
/// - Total: 5 nodes, 2 levels. Tree depth = 2.
///
/// **Example**: 4096-CPU system, fan_out=64:
/// - Leaf level: ceil(4096/64) = 64 leaf nodes.
/// - Level 1: ceil(64/64) = 1 interior node.
/// - Root = that 1 node. Total: 65 nodes, 2 levels.
///
/// **Example**: 16384-CPU system, fan_out=64:
/// - Leaf: ceil(16384/64) = 256 leaf nodes.
/// - Level 1: ceil(256/64) = 4 interior nodes.
/// - Root: 1 node. Total: 261 nodes, 3 levels.
pub fn rcu_build_tree(nr_cpus: u32, fan_out: u8, leaf_fan_out: u8) -> RcuState;
// NOTE: RCU callback rings are **per-CPU**, not global. Each CPU maintains its
// own 4-segment callback pipeline in `RcuPerCpu` (below). This eliminates global
// serialization on the callback-enqueue path — `rcu_call()` only touches the
// calling CPU's `next` segment under a short preemption-disabled section,
// with no cross-CPU lock contention.
/// Fixed-capacity ring of RCU callbacks. **Per-CPU** — each CPU has four
/// segment rings (done/wait/next_ready/next), so `rcu_call()` never contends
/// with other CPUs on the enqueue path.
///
/// Capacity = 4096 entries per CPU (typical system drains well under 256 per GP).
/// Pre-allocated at boot during per-CPU initialization (no runtime allocation).
///
/// Design: UmkaOS uses a typed, pre-allocated ring rather than Linux's intrusive
/// linked-list (`rcu_head` embedded in objects). This eliminates the need for
/// `container_of` pointer arithmetic and gives a predictable allocation profile.
/// Callers pass a closure-style `(fn(*mut ()), *mut ())` pair; the receiving side
/// calls `func(data)` after the grace period. Objects being freed pre-register their
/// cleanup function at `call_rcu()` time rather than embedding a list node.
///
/// **Overflow policy**: If a CPU's ring is full when `rcu_call()` is invoked (4096
/// callbacks pending), the behavior depends on the calling context:
///
/// - **Task context** (preempt_count == 0, IRQs enabled): `rcu_call()` calls
/// `rcu_synchronize()` to block until the current grace period completes, then
/// invokes `func(data)` directly. A warning is logged to flag the condition.
///
/// - **Atomic context** (IRQs disabled or preempt_count > 0): Blocking is forbidden.
/// `rcu_call()` writes the callback into `RcuPerCpu::overflow_buf`, a
/// `RCU_OVERFLOW_BUF_CAPACITY`-slot pre-allocated emergency buffer. The buffer is
/// drained at the next timer tick by `rcu_tick_drain_overflow()` once ring space
/// is available. If both ring and overflow buffer are full, the callback is dropped
/// and an error is logged — this is a catastrophic condition indicating a persistent
/// RCU stall, logged at `log::error!` severity.
///
/// A warning is logged whenever the main ring is full, as it indicates either a grace
/// period stall or an unusually high callback production rate that may need tuning
/// (e.g., increasing the grace period thread priority or reducing batch sizes).
/// Per-segment capacity. Each RCU callback segment holds up to 4096 entries.
/// 4 segments × 4096 = 16384 outstanding callbacks per CPU. At peak callback
/// rates of ~100K/sec, this provides ~160ms of headroom per grace period.
pub const RCU_RING_CAPACITY: usize = 4096;
pub struct RcuCallbackRing {
/// Inline callback storage, allocated at boot (no runtime allocation).
// SAFETY: Backing memory is allocated from the boot allocator (static kernel
// lifetime). Never freed. Use raw pointer instead of Box to avoid UB on drop.
entries: *mut [RcuCallback; RCU_RING_CAPACITY],
head: usize,
tail: usize,
}
/// A single RCU callback entry.
pub struct RcuCallback {
/// Cleanup function. Called with `data` after the grace period.
func: unsafe fn(*mut ()),
/// Opaque pointer to the object being cleaned up (e.g., raw pointer to Box contents).
data: *mut (),
}
/// Per-CPU RCU state (stored in the per-CPU data region, zero-allocation access).
///
/// Each CPU has its own independent callback segments, quiescent state counter, and
/// nesting tracker. This per-CPU design eliminates global serialization on the
/// callback-enqueue path — `rcu_call()` only touches the local CPU's state.
pub struct RcuPerCpu {
/// Pointer to the leaf `RcuNode` covering this CPU.
/// Set once at boot during tree construction (`rcu_build_tree` step 8).
/// Never modified after boot.
// SAFETY: Points into `RcuState::nodes` Box<[RcuNode]>, allocated at
// boot and never freed or reallocated. Valid for the kernel's lifetime.
// Dereferenced in `rcu_check_callbacks()` and `rcu_report_qs_leaf()`
// with the leaf node's spinlock held.
pub leaf_node: *const RcuNode,
/// Bit position of this CPU within the leaf node's `qsmask`.
/// Equal to `(cpu_id - leaf_node.cpu_lo)`. Set once at boot.
pub leaf_bit: u8,
/// Grace period sequence number last acknowledged by this CPU.
/// Compared against `RcuState::gp_seq` to determine whether a new
/// GP has started since this CPU last reported a quiescent state.
/// When `gp_seq_local != rcu_state.gp_seq`, this CPU owes a QS report.
pub gp_seq_local: u64,
/// True when this CPU still needs to report a quiescent state for the
/// current grace period. Set to `true` (Release) by the GP kthread when
/// a new GP starts (step 5a). Read (Acquire) and cleared (Relaxed) by
/// the local CPU's `rcu_check_callbacks()` (steps 2 and 4).
///
/// **AtomicBool rationale**: The GP kthread (running on a different CPU)
/// writes this field during GP initialization (step 5a), while the local
/// CPU reads it during `rcu_check_callbacks()`. A plain `bool` would be
/// a data race (undefined behavior) on weakly-ordered architectures.
/// AtomicBool with Release/Acquire ordering ensures the GP kthread's
/// write is visible to the local CPU before it checks the flag.
pub qs_pending: AtomicBool,
/// Context tracking enabled for this CPU (nohz_full / CG cores).
///
/// When `true`, the CPU's kernel⇄user transitions increment `eqs_seq`
/// (below) so the GP kthread can detect user-mode extended quiescent
/// states remotely. `false` on ordinary ticking CPUs — they report via
/// the tick, and the per-transition increments would be pure overhead.
/// Set at boot from `nohz_full=` and updated by CG-core
/// provision/deprovision ([Section 7.11](07-scheduling.md#core-provisioning-and-workload-partitioning));
/// `AtomicBool` because provisioning writes it from another CPU.
pub ct_active: AtomicBool,
/// Extended-quiescent-state sequence counter (dyntick-style context
/// tracking — Linux equivalent: the RCU-watching counter in
/// `struct context_tracking` (`ct->state`), kernel/context_tracking.c,
/// sampled by the FQS scan in kernel/rcu/tree.c).
///
/// **Parity convention: ODD = the CPU is in an EQS** (executing in
/// user mode); **EVEN = in kernel**. Incremented ONLY when `ct_active`
/// (isolated cores), at exactly two transition classes:
/// - kernel→user (syscall/interrupt/exception return to user):
/// `fetch_add(1, AcqRel)` — even→odd. AcqRel orders all prior
/// kernel-side RCU reads before the EQS becomes observable.
/// - user→kernel (syscall entry, interrupt/exception entry from
/// user): `fetch_add(1, AcqRel)` — odd→even, before any RCU-
/// protected access in the entry path.
/// (Idle needs no increments: `CpuLocal::is_idle` already gives the
/// GP kthread an idle-EQS signal; guest mode counts as user mode —
/// the VM-enter/exit path performs the same increments.)
///
/// The GP kthread samples this REMOTELY during FQS: an odd value
/// means "in EQS right now"; a value that CHANGED between two scans
/// means "passed through an EQS since the last scan". Either way the
/// kthread reports the QS on the CPU's behalf — zero IPIs to
/// user-mode isolated cores. Memory-resident (NOT a register-based
/// CpuLocal field, which cannot be read from another CPU — see the note
/// at GP-start step 6a) precisely BECAUSE it must be remotely readable.
///
/// **Width — native `AtomicU32`, NOT a 64-bit-atomic family member**:
/// this counter is consumed ONLY as (a) parity (`snap is odd`) and (b)
/// changed-since (`snap != eqs_snap`) — never as an absolute magnitude and
/// never in a before/after ordering (contrast `gp_seq`, which uses wrapping
/// `gp_before` MAGNITUDE comparisons over long spans and therefore stays
/// `AtomicU64Exact`). Parity is exact under any wrap (low bit). The only
/// wrap hazard is a "changed-since" false negative: the counter advancing
/// by exactly a multiple of 2^32 between two consecutive FQS scans and
/// landing on the identical value. FQS scans are milliseconds apart, and
/// each kernel⇄user transition costs hundreds of cycles on cores that make
/// few entries by design, so even at an unrealistic 10^7 transitions/s/CPU
/// a full 2^32 wrap takes ≈ 429 s — four-plus orders of magnitude longer
/// than one FQS interval. Unreachable. `AtomicU32` is a native
/// single-instruction atomic on all 8 legs (including PPC32, which has no
/// native 64-bit atomic), so the `fetch_add` writer path below is wait-free
/// everywhere and eqs_seq needs none of the 64-bit-atomic family machinery
/// ([Section 3.5](#locking-strategy--64-bit-atomics-on-32-bit-legs-the-semantic-family)).
pub eqs_seq: AtomicU32,
/// GP-kthread-private FQS snapshot of `eqs_seq` for this CPU, taken
/// on the first FQS scan that finds the CPU holding out; compared on
/// subsequent scans (changed ⇒ EQS passage ⇒ report on behalf).
/// Plain `u32` (matches `eqs_seq`'s width): written and read ONLY by the
/// single GP kthread.
pub eqs_snap: u32,
/// Nesting depth of active RcuReadGuards on this CPU.
pub nesting: u32,
/// 4-segment callback list partitioned by grace period generation.
///
/// The four segments form a pipeline that advances as grace periods complete:
/// - `done`: Callbacks whose grace period has completed. Ready for immediate
/// execution. Drained in softirq context by `rcu_process_callbacks()`.
/// - `wait`: Callbacks waiting for the current grace period to complete.
/// When the GP completes, `wait` → `done`.
/// - `next_ready`: Callbacks registered during the current GP. They cannot
/// be satisfied by the current GP (they need at least one more full GP).
/// When the GP completes, `next_ready` → `wait`.
/// - `next`: Callbacks being actively registered by `rcu_call()` right now.
/// When the GP completes, `next` → `next_ready`.
///
/// This 4-segment design ensures that callbacks registered during a GP
/// are never freed too early — they must wait for the *next* GP after the
/// one in which they were registered.
///
/// Each segment is a pre-allocated `RcuCallbackRing` (4096 slots per segment,
/// allocated at boot). The 4-segment advancement is O(1) — just pointer swaps.
/// **Lock ordering exemption**: `cb_segments` spinlocks are exempt from the
/// global lock level table ([Section 3.4](#cumulative-performance-budget--lock-ordering)).
/// Rationale: they are per-CPU (never acquired cross-CPU), held with IRQs
/// disabled (SpinLock does this automatically), and serialize only single-CPU
/// access to the callback pipeline. Since `rcu_call()` can be called from
/// `Drop` implementations under arbitrary lock contexts, these locks cannot
/// be assigned a single global level — they must be below ALL other locks.
/// The per-CPU + IRQ-disabled invariant ensures no deadlock: no other CPU
/// can hold this lock, and no interrupt on this CPU can preempt and re-acquire.
pub cb_segments: [SpinLock<RcuCallbackRing>; 4],
/// Emergency overflow buffer: used when the `next` ring is full AND the caller
/// is in atomic context (IRQs disabled or preempt_count > 0). Blocking
/// (`rcu_synchronize`) is forbidden in atomic context, so callbacks are staged
/// here instead. Drained at the next timer tick via `rcu_tick_drain_overflow()`.
/// Pre-allocated at boot; never heap-allocates on the enqueue path.
///
/// `u8` is sufficient for the length field: `RCU_OVERFLOW_BUF_CAPACITY` (64) < 256.
pub overflow_buf: [MaybeUninit<RcuCallback>; RCU_OVERFLOW_BUF_CAPACITY],
pub overflow_len: u8,
}
/// Callback segment indices for `RcuPerCpu::cb_segments`.
pub const RCU_DONE: usize = 0;
pub const RCU_WAIT: usize = 1;
pub const RCU_NEXT_READY: usize = 2;
pub const RCU_NEXT: usize = 3;
Memory ordering requirements:
On RcuReadGuard acquisition (rcu_read_lock()):
No barrier needed. The read-side section is a pure software contract.
Hardware TSO (x86) and load-acquire semantics (ARM, RISC-V) ensure that
loads within the critical section see all stores that completed before
rcu_read_lock() was called.
On RcuReadGuard drop (rcu_read_unlock()):
Relaxed store to rcu_passed_quiesce (CpuLocal, AtomicBool).
rcu_passed_quiesce uses Relaxed ordering because the flag is consumed
only by the local CPU's rcu_check_callbacks() at the next tick — no
cross-CPU visibility is needed for this flag itself. The cross-CPU
ordering guarantee comes from the leaf node lock acquisition in
rcu_report_qs_leaf(), not from this store. The lock acquisition
(implicit Acquire barrier) ensures that all RCU-protected loads within
the critical section are visible to other CPUs before the qsmask
bit-clear is observed by the GP kthread.
On quiescent state propagation (rcu_report_qs_leaf → parent):
Acquire load on leaf node's lock acquisition (implicit in SpinLock::lock()).
The lock acquisition ensures the reporting CPU's RCU-protected stores are
visible before the qsmask bit-clear is observed by the GP kthread or
any parent node.
On grace period start (gp_kthread initializes tree):
Release store on gp_seq (via AtomicU64::store(_, Release)).
All per-node qsmask initializations must be visible before gp_seq transitions
to the "started" phase. The GP kthread acquires each node's lock to set
qsmask, providing the necessary ordering per node.
On grace period completion (root qsmask reaches 0):
Acquire load on root node's qsmask (under lock).
Full fence (`fence(SeqCst)`) before executing callbacks.
This ensures callbacks see all memory stores made by RCU-protected writers
before the grace period started. The fence is issued by the GP kthread
after confirming root.qsmask == 0 and before draining the `done` segments.
On gp_seq reads by rcu_synchronize() callers:
Acquire load on rcu_state.gp_seq.
Ensures the caller sees all memory stores that were ordered before the
GP kthread's Release store to gp_seq at GP completion.
Quiescent state identification:
- Quiescent points occur at: (1) context switch (the outgoing task releases all
RcuReadGuards before being descheduled — preempt_count == 0 implies no active RCU read-side critical section), (2) idle entry (cpu_idle_enter()— a CPU entering the idle loop has no active critical sections), (3) return to userspace (user code never holds kernel RCU references), and (4) explicitrcu_quiescent_state()calls in long-running kernel loops that do not hold RCU references. - KABI boundary crossing constitutes an RCU quiescent state: every KABI vtable call
entry and return is treated as a quiescent point for the calling CPU. This ensures
that Tier 1 drivers that return from KABI calls within bounded time (enforced by the
per-call timeout watchdog, Section 11.4) cannot block RCU grace period completion
indefinitely. Drivers that perform long-polling loops must call
rcu_quiescent_state()at each poll iteration, or use the KABI polling helperkabi_poll_wait()which includes an implicit quiescent state. - Grace period detection is batched: multiple
rcu_defer_free()/rcu_call()invocations are coalesced into the same grace period to amortize the per-CPU reporting overhead.
/// RCU read-side guard. Obtained via `rcu_read_lock()`, released via Drop.
///
/// This is a zero-cost marker type — it does not perform any atomic operations
/// or memory barriers on acquisition. The RCU read-side critical section is
/// purely a contract with the grace period detection mechanism: as long as any
/// CPU holds an `RcuReadGuard`, the current grace period cannot complete.
///
/// The guard is `!Send` because RCU read-side sections are per-CPU — the quiescent
/// state tracking (Section 3.1.1, "RCU Grace Period Detection") is CPU-local.
/// Sending an `RcuReadGuard` to another thread would allow the grace period
/// detection to miss an active reader.
///
/// # Example
/// ```rust
/// let guard = rcu_read_lock();
/// // Within this scope, any RCU-protected data can be safely read.
/// // The grace period will not complete until this guard is dropped.
/// let value = rcu_cell.read(&guard);
/// // guard dropped here; this CPU may now pass through a quiescent point
/// ```
pub struct RcuReadGuard {
/// CPU ID on which this guard was acquired. Used for debug assertions
/// and the KRL timeout mitigation (Section 9.2.8). Not used for
/// grace period tracking — that is handled by per-CPU quiescent state counters.
_cpu_id: u32,
/// Nesting depth snapshot at acquisition time. Used only in debug builds
/// (`debug_assert!` in `Drop`) to verify that the per-CPU nesting counter
/// has not been corrupted between lock and unlock. In release builds, the
/// field is retained for layout stability but is not read.
///
/// The actual nesting tracking is in `CpuLocal.rcu_nesting` — the Drop
/// impl reads that counter, not this field. This field exists solely as
/// a cross-check.
_nesting: u32,
/// Marker to prevent Send/Sync auto-traits.
_not_send: PhantomData<*const ()>,
}
impl Drop for RcuReadGuard {
fn drop(&mut self) {
// Decrement the per-CPU nesting counter via CpuLocal (Section 3.1.2).
// Only the outermost guard (nesting reaches 0) needs further action.
// Nested RCU read sections work correctly — an inner guard's drop
// does NOT affect quiescent state while the outer section is active.
let nesting = cpu_local::rcu_nesting_dec();
if nesting > 0 {
return; // Still inside an outer RCU read-side critical section.
}
// Outermost guard dropped — set the per-CPU "passed quiescent point"
// flag. This is a CpuLocal boolean write (~1 cycle), NOT an immediate
// report to the grace period machinery. The actual quiescent state
// report is deferred to the next scheduler tick or context switch,
// which are the natural quiescent checkpoints (see below).
//
// On architectures with weak memory ordering, a Release store is
// used to prevent RCU-protected accesses from being reordered past
// the guard's drop point.
cpu_local::set_rcu_passed_quiesce(true);
// === Design rationale: deferred quiescent state reporting ===
//
// **Why NOT report immediately on every outermost drop**:
// The previous design called `rcu_note_quiescent_state()` here — a
// function call + per-CPU atomic store (~5-10 cycles). On NVMe paths
// with frequent short RCU sections (conntrack lookup, routing table
// lookup), this adds ~5-10 cycles per I/O. Across millions of IOPS,
// the overhead is measurable.
//
// **How deferred reporting works**:
// 1. `RcuReadGuard::drop()` sets `cpu_local.rcu_passed_quiesce = true`
// (~1 cycle CpuLocal write, no function call, no atomic). This is
// a HINT for diagnostics — not a reporting gate (see below).
// 2. `timer_tick_handler()` step 3 (HZ=1000, BEFORE scheduler_tick
// takes rq.lock) calls `rcu_check_callbacks()`: if the
// interrupted context holds no RcuReadGuard (rcu_nesting == 0),
// that instant IS a quiescent state and is propagated up the
// RcuNode tree — whether or not any guard was dropped since the
// last GP. This batches QS reporting to once per GP per CPU
// (the qs_pending gate) and guarantees a ticking CPU never
// stalls a GP, including CPU-bound threads that never use RCU.
// 3. `finish_task_switch()` step 1a also calls
// `rcu_check_callbacks()` — after the rq lock is released,
// before preemption is re-enabled. Every voluntary or
// involuntary context switch is a quiescent point, and on
// TICKLESS CPUs this is the only local propagation site.
// 4. `cpu_idle_enter()` reports unconditionally — idle is always
// quiescent (and FQS reports on behalf of idle CPUs remotely).
//
// For `nohz_full` / CG CPUs (tickless), a CPU-bound USER task
// generates neither ticks nor context switches. Quiescence is
// DETECTED remotely instead of reported locally: the kernel⇄user
// transition paths on these CPUs (ct_active) increment the
// memory-resident `RcuPerCpu.eqs_seq` counter (odd = in user
// mode), and the GP kthread's FQS scan samples it and reports on
// the CPU's behalf — zero IPIs to user-mode isolated cores. See
// the Force-Quiescent-State (FQS) Scan section. (`rcu_nocbs`-style
// callback OFFLOAD is a separate, complementary mechanism: it
// moves callback EXECUTION off the isolated core; it does not
// detect quiescent states. Linux separates these the same way:
// context tracking / RCU-watching counter for QS detection,
// rcu_nocbs for callback invocation.)
//
// **Comparison with Linux**: In non-preemptible Linux kernels,
// `rcu_read_unlock()` generates zero code — quiescent states are
// inferred entirely from context switches, idle, and usermode return.
// UmkaOS's approach now matches Linux's model: the outermost drop sets
// a lightweight per-CPU flag, and the actual report is deferred to
// tick/switch checkpoints. The per-flag-write cost (~1 cycle) is
// effectively zero compared to the prior ~5-10 cycle function call,
// while maintaining the stall-freedom guarantee via the tick handler.
// Re-enable preemption (matching the preempt_count_inc in rcu_read_lock).
// If preempt_count reaches 0 and need_resched is set, invoke the scheduler.
cpu_local::preempt_count_dec_and_test_resched();
}
}
impl !Send for RcuReadGuard {}
impl !Sync for RcuReadGuard {}
/// Acquire an RCU read-side critical section guard.
///
/// The returned guard prevents the current RCU grace period from completing
/// until it is dropped. On drop, the outermost guard re-enables preemption
/// and sets a per-CPU `rcu_passed_quiesce` flag (CpuLocal write, ~1 cycle).
/// The actual quiescent state report to the grace period machinery is deferred
/// to the next scheduler tick or context switch — see `RcuReadGuard::drop()`
/// for the full design rationale.
///
/// **Cost**: near-zero-cost (~2 instructions: increment `preempt_count` via
/// CpuLocal register, no memory barriers, no cache-line bouncing). Preemption
/// is disabled for the duration of the RCU read-side critical section
/// (non-preemptible RCU model). This means RCU readers must not sleep or
/// block — any context switch is a quiescent state by definition.
///
/// # Safety invariants
/// - Must be paired with a `Drop` (RAII pattern — cannot be leaked).
/// - Must not be held across a blocking operation (sleep, mutex acquisition)
/// unless the holder is prepared for an extended grace period latency.
/// - The KRL timeout mitigation (Section 9.2.8) enforces a maximum critical
/// section duration for KRL access to prevent DoS.
pub fn rcu_read_lock() -> RcuReadGuard {
// Non-preemptible RCU: disable preemption by incrementing preempt_count.
// This is ~2 instructions via CpuLocal register (read-modify-write on
// per-CPU preempt_count), no memory barriers, no cache-line bouncing.
// A context switch while preempt_count > 0 is prevented, so any context
// switch is a quiescent state — the core of non-preemptible RCU.
cpu_local::preempt_count_inc();
let nesting = cpu_local::rcu_nesting_inc();
RcuReadGuard {
// Canonical current-CPU-id seam; pinned by the preempt_count_inc
// above ([Section 3.2](#cpulocal-register-based-per-cpu-fast-path--generic-cpulocal-field-accessors)).
_cpu_id: cpu_local::cpu_id(),
_nesting: nesting,
_not_send: PhantomData,
}
}
/// A slice wrapper that can only be dereferenced within an RCU read-side
/// critical section. Used for RCU-protected arrays (e.g., KRL revoked_keys).
///
/// This is a zero-cost newtype around a raw pointer. The `Deref` implementation
/// is gated on an `RcuReadGuard` reference, ensuring that the pointed-to data
/// cannot be accessed after its backing memory is freed by the RCU callback.
///
/// # Type parameter
/// - `T`: The element type of the slice. Typically `[u8; 32]` for hash arrays.
///
/// # Safety contract
/// - The pointer must have been obtained from memory that will remain valid
/// until at least the next RCU grace period.
/// - The `RcuReadGuard` passed to `deref()` must have been acquired after the
/// last RCU update that could have freed this memory.
/// - Callers must not hold the `Deref` result across an RCU grace period
/// boundary (e.g., must not call `rcu_synchronize()` while holding a
/// reference derived from this slice).
///
/// # Example
/// ```rust
/// pub struct KeyRevocationList {
/// pub revoked_keys: RcuSlice<[u8; 32]>,
/// pub revoked_count: u32,
/// }
///
/// fn is_revoked(krl: &RcuCell<KeyRevocationList>, fingerprint: &[u8; 32]) -> bool {
/// let guard = rcu_read_lock();
/// let krl_ref = krl.read(&guard);
/// // Deref RcuSlice within the guard's lifetime
/// let keys: &[[u8; 32]] = krl_ref.revoked_keys.deref(&guard);
/// keys[..krl_ref.revoked_count as usize]
/// .binary_search(fingerprint)
/// .is_ok()
/// }
/// ```
// Kernel-internal, not an ABI/wire type: a thin (`*const T`, `usize`) pair that
// never crosses a KABI, wire, or userspace boundary, so it carries no `#[repr(C)]`.
// Its layout is private and its size is platform-dependent (the raw pointer is
// pointer-width), so plain `repr(Rust)` is the correct representation — a
// `const_assert!` size gate would be meaningless for a non-boundary generic type.
pub struct RcuSlice<T> {
ptr: *const T,
len: usize,
}
impl<T> RcuSlice<T> {
/// Create a new RcuSlice from a raw pointer and length.
///
/// # Safety
/// The caller must ensure that:
/// 1. The pointer is valid and properly aligned for type `T`.
/// 2. `ptr` points to `len` contiguous initialized elements of type `T`.
/// 3. The memory pointed to will remain valid until at least the next RCU
/// grace period after the last access via this slice.
pub unsafe fn from_raw(ptr: *const T, len: usize) -> Self {
Self { ptr, len }
}
}
impl<T> RcuSlice<T> {
/// Dereference the slice within an RCU read-side critical section.
///
/// The returned reference is valid only for the lifetime of the guard.
/// Accessing the reference after the guard is dropped is undefined behavior.
///
/// # Arguments
/// - `_guard`: A reference to an `RcuReadGuard`, proving that the caller
/// is within an RCU read-side critical section. The guard's lifetime
/// bounds the returned reference.
///
/// # Returns
/// A shared reference to the underlying slice of `T` elements.
/// For `RcuSlice<[u8; 32]>`, this returns `&[[u8; 32]]`.
pub fn deref<'a>(&self, _guard: &'a RcuReadGuard) -> &'a [T] {
// SAFETY: The caller has provided an RcuReadGuard, proving they are
// within an RCU read-side critical section. The memory pointed to by
// self.ptr was allocated by an RCU-protected update and will remain
// valid until at least the next grace period. Since the guard prevents
// grace period completion, the memory is valid for the guard's lifetime.
// The len field was set at construction time and is invariant.
unsafe { core::slice::from_raw_parts(self.ptr, self.len) }
}
}
// RcuSlice does NOT implement Send or Sync directly. It can only be accessed
// through an RcuReadGuard, which is itself !Send. The containing struct
// (e.g., KeyRevocationList) provides the necessary Send/Sync impls when
// accessed through RcuCell, which enforces the RCU lifetime contract.
/// Error returned by `rcu_call()` when a deferred callback cannot be queued.
/// Kernel-internal (not KABI). The sole failure mode is ring exhaustion.
pub enum RcuCallError {
/// Both the calling CPU's `next` callback segment ring AND its
/// `overflow_buf` are full. Only reachable from atomic context — task
/// context falls back to a synchronous `rcu_synchronize()` instead of
/// returning this error. Indicates a persistent RCU stall (catastrophic):
/// the callback is dropped and the condition is logged at error level.
RingFull,
}
/// Queue a callback to be invoked after the current RCU grace period.
/// Safe to call from any context (including interrupt context).
/// Does NOT block.
///
/// # Implementation
/// Adds `RcuCallback { func, data }` to the calling CPU's per-CPU
/// `next` callback segment (`RcuPerCpu::cb_segments[RCU_NEXT]`). Each CPU
/// has its own independent 4096-slot ring per segment, so there is no
/// global serialization bottleneck on the enqueue path.
///
/// **Overflow policy**: If the calling CPU's `next` segment ring is full, the
/// behavior depends on the calling context:
///
/// - **Task context** (preempt_count == 0, IRQs enabled): `rcu_call()` calls
/// `rcu_synchronize()` to block until the grace period completes, then invokes
/// `func(data)` directly. The callback is guaranteed to execute before return.
/// A warning is logged to flag the overflow condition.
///
/// - **Atomic context** (IRQs disabled or preempt_count > 0): `rcu_call()` writes
/// the callback into `RcuPerCpu::overflow_buf`. The buffer is drained at the next
/// timer tick by `rcu_tick_drain_overflow()`. If the overflow buffer is also full,
/// the callback is dropped and `Err(RcuCallError::RingFull)` is returned — this
/// is a catastrophic condition (persistent RCU stall) and is logged at error level.
///
/// # Ordering guarantee
/// After `rcu_call(func, data)` returns `Ok(())`, `func(data)` will be called at
/// some future point after a complete grace period has elapsed (i.e., after every
/// CPU has passed through a quiescent state). In the task-context overflow fallback
/// case, `func(data)` has already been called by the time `rcu_call()` returns.
pub fn rcu_call(func: unsafe fn(*mut ()), data: *mut ()) -> Result<(), RcuCallError>;
rcu_call(func, data) algorithm:
rcu_call(func, data):
1. Save caller's preemption state: was_atomic = CpuLocal::preempt_count() > 0 || !arch::current::interrupts::are_enabled().
2. Disable preemption (ensures we stay on this CPU for the duration).
3. Get this CPU's RcuPerCpu state.
Acquire lock: guard = cb_segments[RCU_NEXT].lock().
// The SpinLock serializes concurrent callers on the same CPU
// (e.g., an IRQ handler calling rcu_call while a task-context
// rcu_call is in progress). Without this lock, two callers
// could race on ring.push() and corrupt the ring's head pointer.
4. If guard.is_full():
a. Release lock: drop(guard).
b. Log warning: "RCU callback ring full on CPU N".
c. If was_atomic:
// Atomic context: cannot block. Use pre-allocated overflow buffer.
// Safe from IRQ re-entrancy: overflow_buf is per-CPU and accessed
// with preemption disabled. If an IRQ fires here, it will see
// its own rcu_call() path also observes local IRQs disabled and uses the
// same buffer — but only AFTER this store completes (single-CPU
// sequential execution). No concurrent access is possible.
if overflow_len < RCU_OVERFLOW_BUF_CAPACITY as u8:
overflow_buf[overflow_len as usize].write(RcuCallback { func, data })
overflow_len += 1
// Will be drained at next tick by rcu_tick_drain_overflow().
Re-enable preemption.
return Ok(()).
else:
// Both ring and overflow buffer full: catastrophic RCU stall.
log::error!("rcu_call: overflow buffer full in atomic context — callback dropped")
Re-enable preemption.
return Err(RcuCallError::RingFull).
d. Else:
// Task context: safe to block.
Re-enable preemption.
log::warn!("rcu_call: ring full, synchronizing (task context)")
rcu_synchronize(); // Block until current grace period completes.
unsafe { func(data) }; // Execute directly — GP elapsed, readers done.
return Ok(()).
5. guard.push(RcuCallback { func, data }).
6. let len = guard.len();
Release lock: drop(guard).
7. If len >= RCU_BATCH_DRAIN_THRESHOLD (default: 256):
Set rcu_state.gp_requested = true.
Wake rcu_state.gp_kthread via scheduler::unblock().
8. Re-enable preemption.
9. return Ok(()).
rcu_tick_drain_overflow() algorithm (called at each timer tick for CPUs with overflow_len > 0):
rcu_tick_drain_overflow():
1. Disable preemption.
2. Get this CPU's RcuPerCpu state. ring = cb_segments[RCU_NEXT].
3. While overflow_len > 0 && !ring.is_full():
a. overflow_len -= 1.
b. cb = overflow_buf[overflow_len].assume_init_read().
c. ring.push(cb).
4. If overflow_len == 0 && ring.len() >= RCU_BATCH_DRAIN_THRESHOLD:
Set rcu_state.gp_requested = true.
Wake rcu_state.gp_kthread via scheduler::unblock().
5. Re-enable preemption.
/// Wait for an RCU grace period to complete (blocking).
///
/// This function blocks the calling thread until all pre-existing RCU
/// read-side critical sections have completed. Use this when you need
/// to free memory that may be referenced by RCU readers.
///
/// **MUST NOT be called from atomic context** (interrupt handler, spinlock-held,
/// preempt-disabled, or NMI). In atomic contexts, use `rcu_call()` instead.
pub fn rcu_synchronize();
rcu_seq_snap() — Compute GP target sequence number:
/// Symbolic constants for RCU grace period sequence encoding.
///
/// The low 2 bits of `gp_seq` encode the grace period phase:
/// 0b00 = idle, 0b01 = started, 0b10 = completing, 0b11 = reserved.
/// Each complete GP advances `gp_seq` by 4.
///
/// `RCU_SEQ_STATE_MASK` matches Linux `kernel/rcu/tree.c` naming.
/// `RCU_GP_PHASE_MASK` is the alias used in the `gp_phase()` helper
/// (defined alongside `gp_before()` in the RcuState doc comment).
/// Both are the same value.
pub const RCU_SEQ_STATE_MASK: u64 = 0x3;
pub const RCU_GP_PHASE_MASK: u64 = RCU_SEQ_STATE_MASK;
/// Compute a snapshot value that is guaranteed to be past the end of any
/// grace period that is in progress at the time of the call, AND past the
/// end of at least one full grace period that has not yet started.
///
/// This is the correct target for `rcu_synchronize()`: the caller must wait
/// until `gp_seq` advances past the returned snapshot value.
///
/// The formula adds `2 * RCU_SEQ_STATE_MASK + 1` (= 7) to ensure that even
/// if `seq` is mid-GP (phase 1 or 2), the target rounds UP past the current
/// GP and requires one additional full GP. Masking off the low bits produces
/// a phase-0 (idle) target that can only be reached after both the current
/// in-progress GP and the next GP complete.
///
/// This matches Linux `kernel/rcu/tree.c` `rcu_seq_snap()` exactly.
///
/// Examples:
/// seq=0 (idle): snap = (0 + 7) & !3 = 4 → wait for GP 1 to complete
/// seq=1 (started): snap = (1 + 7) & !3 = 8 → wait for GP 1 AND GP 2
/// seq=2 (completing): snap = (2 + 7) & !3 = 8 → wait for GP 2
/// seq=4 (idle): snap = (4 + 7) & !3 = 8 → wait for GP 2
/// seq=5 (started): snap = (5 + 7) & !3 = 12 → wait for GP 2 AND GP 3
#[inline]
pub fn rcu_seq_snap(seq: u64) -> u64 {
(seq.wrapping_add(2 * RCU_SEQ_STATE_MASK + 1)) & !RCU_SEQ_STATE_MASK
}
rcu_synchronize() algorithm:
rcu_synchronize():
1. Snapshot seq = rcu_state.gp_seq.load(Acquire).
2. Compute target = rcu_seq_snap(seq).
// rcu_seq_snap() rounds UP past the current GP to ensure that all
// pre-existing RCU read-side critical sections have completed.
// When called mid-GP (seq is phase 1 or 2), the snapshot targets
// one GP beyond the current one — the current GP may have started
// before our snapshot, so readers from before our call may not have
// exited until the next GP completes.
3. Set rcu_state.gp_requested.store(true, Relaxed).
Wake rcu_state.gp_kthread via scheduler::unblock().
// ALWAYS request a GP, regardless of current GP phase. When called
// mid-GP, rcu_seq_snap() targets the end of the NEXT GP. If we only
// set gp_requested when idle, and no callbacks are pending (nobody
// called rcu_call()), the current GP completes without requesting
// another one. Our target requires TWO GPs but only one runs —
// deadlock. The GP kthread checks gp_requested at GP completion
// (step 14) and starts another if set. Linux `rcu_gp_init()` also
// always checks for pending work.
4. Add current task to rcu_state.gp_completion_wq with wait_gp_seq = target.
5. schedule() — task sleeps until woken by gp_kthread.
6. On wakeup: verify gp_before(target, rcu_state.gp_seq.load(Acquire) + 1).
// gp_before(a, b) returns true when (b - a) as i64 > 0, i.e., b > a
// (wrapping-aware signed comparison). Adding 1 to gp_seq converts the
// strict-before check into an at-or-after check:
// gp_before(target, gp_seq + 1) ≡ gp_seq + 1 > target ≡ gp_seq >= target.
// Without the + 1, we would check gp_seq > target, missing the case
// where gp_seq == target (GP completed exactly to our target).
If not satisfied (spurious wakeup), re-sleep (go to step 5).
7. Return (grace period completed).
3.4.2 Hierarchical Quiescent State Reporting¶
When a CPU passes through a quiescent state (context switch, idle entry, userspace
return, explicit rcu_quiescent_state() call), the report propagates bottom-up
through the RcuNode tree. This is the core mechanism that makes Tree RCU scale:
each CPU only touches its leaf node's lock, and propagation only climbs the tree
when a node's last child reports.
rcu_qs() — record quiescent state on current CPU (called from context switch,
idle entry, userspace return, KABI boundary):
rcu_qs():
1. Set CpuLocal::rcu_passed_quiesce = true.
// A cheap hint, not a correctness gate (see rcu_check_callbacks step 2
// note). The actual tree propagation is deferred to
// rcu_check_callbacks(), called from timer_tick_handler() step 3 and
// from finish_task_switch() step 1a (after the rq-lock release). This
// avoids acquiring the leaf node's spinlock on every quiescent point —
// batching QS reports from the 1 ms tick interval.
rcu_check_callbacks() — propagate QS up the tree. Call sites (exactly
two, both holding NO runqueue lock — the rcu_report_qs_leaf() precondition):
timer_tick_handler() step 3 (Section 7.1,
BEFORE scheduler_tick() takes rq.lock), and finish_task_switch() step 1a
(Section 7.3,
AFTER the rq lock is released, before preemption is re-enabled). The
context-switch site is what keeps tickless (nohz_full/CG) CPUs propagating:
rcu_check_callbacks():
1. If CpuLocal::rcu_nesting != 0: return. // Inside RCU read-side
// critical section — reporting QS now would allow callbacks
// (including Box::drop) to execute while readers hold references.
// The QS will be reported after the outermost RcuReadGuard drops.
2. If !rcu_percpu.qs_pending.load(Acquire): return. // No GP needs our report.
// NOTE — no `rcu_passed_quiesce` gate. Step 1 already established
// rcu_nesting == 0, i.e., THIS INSTANT the CPU holds no RCU read
// guard — that IS a quiescent state, whether or not any guard was
// dropped since the last GP. Gating on the drop-flag would let a
// CPU-bound thread that never uses RCU (flag never set) stall every
// grace period despite ticking at HZ. `rcu_passed_quiesce` remains
// a diagnostic hint written by RcuReadGuard::drop / rcu_qs(), not a
// gate. (Linux parity: rcu_sched_clock_irq() reports based on the
// interrupted context's nesting, not on a drop-event flag.)
3. CpuLocal::rcu_passed_quiesce = false. // Hint hygiene.
4. rcu_percpu.qs_pending.store(false, Relaxed).
// Relaxed is sufficient here because this is a local-CPU-only write.
// The GP kthread will not read this field until the next GP start,
// at which point it writes `true` (Release), providing the ordering.
5. rcu_percpu.gp_seq_local = rcu_state.gp_seq.load(Relaxed).
6. Call rcu_report_qs_leaf(rcu_percpu.leaf_node, rcu_percpu.leaf_bit).
rcu_report_qs_leaf() — clear bit in leaf node and propagate (called from
rcu_check_callbacks() and the GP kthread's on-behalf paths, with preemption
disabled). Precondition: the caller holds NO runqueue lock — the
propagation may terminate in scheduler::unblock(gp_kthread), which acquires
an RQ_LOCK(50); both call sites above satisfy this by construction:
rcu_report_qs_leaf(leaf: &RcuNode, bit: u8):
1. Acquire leaf.lock.
2. If leaf.inner.gp_seq != rcu_state.gp_seq.load(Relaxed):
// Stale report from a previous GP — discard silently.
Release leaf.lock.
return.
3. Clear bit `bit` in leaf.inner.qsmask:
leaf.inner.qsmask &= !(1u64 << bit).
4. mask = leaf.inner.qsmask.
5. Release leaf.lock.
6. If mask != 0: return. // Other CPUs in this leaf haven't reported yet.
7. // This leaf is fully quiescent — propagate to parent.
rcu_report_qs_up(leaf).
rcu_report_qs_up() — propagate zero-qsmask up toward root (called from
rcu_report_qs_leaf() when a leaf's qsmask reaches 0):
rcu_report_qs_up(child: &RcuNode):
node = child
loop:
parent = node.parent
If parent is None:
// We just cleared the root's qsmask to 0 — GP is complete.
// Wake the GP kthread (it sleeps on rcu_state.gp_kthread_wq).
//
// LOCK DISCIPLINE: at this point EVERY RcuNode lock is already
// released (each iteration releases the node's lock before
// descending into this branch or the next iteration).
// scheduler::unblock() acquires the gp-kthread's RQ_LOCK(50) —
// legal ONLY because no RCU_NODE_LOCK(52) is still held (52→50
// nested would be a compile-rejected descent). Combined with
// rcu_report_qs_leaf()'s "caller holds no RQ_LOCK" precondition,
// the full chain is: [rq-lock-free context] → 52 (released) → …
// → 52 (released) → 50. Two-phase, never nested.
scheduler::unblock(rcu_state.gp_kthread)
return.
idx = node.parent_idx
Acquire parent.lock.
If parent.inner.gp_seq != rcu_state.gp_seq.load(Relaxed):
// Stale — GP already advanced. Discard.
Release parent.lock.
return.
Clear bit `idx` in parent.inner.qsmask:
parent.inner.qsmask &= !(1u64 << idx).
mask = parent.inner.qsmask.
Release parent.lock.
If mask != 0: return. // Other children haven't reported yet.
node = parent
// Continue propagating upward.
Lock contention analysis: In the common case (many CPUs reporting QS for the
same GP), each CPU acquires only its leaf node's lock (contention limited to
leaf_fan_out CPUs per lock, default 64). Propagation to the parent occurs only
when the last CPU in a leaf group reports — so parent locks see at most one
acquisition per leaf node per GP. On a 4096-CPU system (64 leaves, 1 interior
node = root), the root lock sees at most 64 acquisitions per GP. This is
O(num_nodes) total lock acquisitions per GP, not O(num_cpus).
3.4.3 Grace Period State Machine (rcu_gp_kthread)¶
A single dedicated kernel thread (rcu_gp_kthread, SCHED_FIFO priority 10,
pinned to NUMA node 0's first online CPU) drives grace period progression. Unlike
the previous flat model with per-NUMA-node threads, the hierarchical tree makes a
single GP kthread sufficient — the tree structure distributes the QS collection
work to the CPUs themselves (bottom-up propagation), so the GP kthread only needs
to initialize the tree and wait for the root to clear.
Constants:
/// Starting wait time for force-quiescent-state (FQS) scan interval.
pub const RCU_FQS_INITIAL_MS: u64 = 1;
/// Maximum FQS scan interval cap.
pub const RCU_FQS_MAX_MS: u64 = 100;
/// FQS backoff multiplier between scans.
pub const RCU_FQS_BACKOFF_MULTIPLIER: u64 = 2;
/// Absolute maximum grace period wait before RCU stall warning.
/// After this duration, a warning is emitted (advisory, not fatal).
pub const RCU_STALL_WARN_MS: u64 = 10_000;
/// Default force-quiescent-state IPI interval in nanoseconds.
/// After this interval without a QS from a CPU, send it a reschedule IPI.
/// Configurable at boot via `rcu.fqs_interval_ns=N`.
pub const RCU_FQS_IPI_NS: u64 = 1_000_000; // 1 ms
/// Capacity of the per-CPU overflow buffer used by `rcu_call()` in atomic context
/// when the main `RcuCallbackRing` is full. Pre-allocated in `RcuPerCpu`; no
/// heap allocation occurs on the enqueue path. Sized to absorb bursts from short
/// interrupt storms; `rcu_tick_drain_overflow()` drains the buffer at each timer tick.
///
/// **Sizing rationale**: The overflow buffer is a last resort — only used when ALL
/// 4096 entries in the main ring are full AND the caller is in atomic context. In
/// practice, atomic-context `rcu_call()` bursts are bounded by the work done per
/// interrupt/softirq invocation: a single NAPI poll cycle processes at most ~64
/// packets, a single timer tick processes a bounded number of deferred items. The
/// 64-entry buffer covers the typical burst from any single softirq handler.
///
/// **Monitoring**: A per-CPU overflow counter is exposed via
/// `/sys/kernel/rcu/per_cpu/N/overflow_count` to detect systems approaching the
/// limit. If real-world profiling shows exhaustion, increase to 256.
pub const RCU_OVERFLOW_BUF_CAPACITY: usize = 64;
Algorithm:
rcu_gp_kthread (single kernel thread, SCHED_FIFO priority 10):
Loop:
╔══════════════════════════════════════════════════════════════════════╗
║ Phase 0: IDLE — wait for work ║
╚══════════════════════════════════════════════════════════════════════╝
1. Sleep on rcu_state.gp_kthread_wq until rcu_state.gp_requested == true
(or woken by rcu_call / rcu_synchronize / rcu_report_qs_up on root).
╔══════════════════════════════════════════════════════════════════════╗
║ Phase 1: GP START — initialize tree ║
╚══════════════════════════════════════════════════════════════════════╝
2. rcu_state.gp_requested.store(false, Relaxed).
3. new_seq = rcu_state.gp_seq.load(Relaxed) + 1.
// Advance from idle (phase 0) to started (phase 1).
assert!(gp_phase(new_seq) == RCU_GP_STARTED).
4. Initialize the tree — for each RcuNode (top-down, root first):
a. Acquire node.lock.
b. node.inner.qsmask = node.inner.online_mask.
// Set bits for all online children/CPUs. Offline children are
// already "quiescent" — they have no active readers.
c. node.inner.gp_seq = new_seq.
d. Release node.lock.
5. rcu_state.gp_seq.store(new_seq, Release).
// The Release store ensures all tree initialization (qsmask writes,
// qs_pending stores below) are ordered after gp_seq. This is critical
// for RCU-F06: rcu_report_qs_leaf() loads gp_seq with Relaxed and
// compares against the leaf node's gp_seq. If gp_seq were stored
// AFTER qs_pending, a CPU on a weakly-ordered architecture (ARM/RISC-V/PPC)
// could observe qs_pending=true, report QS, but see a stale gp_seq in the
// leaf node staleness check — discarding a valid QS report.
// By storing gp_seq FIRST (Release), the subsequent qs_pending stores
// (also Release) are ordered after gp_seq, and the qs_pending Acquire
// load in rcu_check_callbacks() transitively orders the gp_seq visibility.
6. For each online CPU C:
a. rcu_percpu[C].qs_pending.store(true, Release).
// Note: CpuLocal::rcu_passed_quiesce is NOT cleared here. The GP
// kthread cannot write to another CPU's register-based CpuLocal field.
// Stale rcu_passed_quiesce from a previous GP is harmless because
// rcu_check_callbacks() gates on rcu_nesting == 0 (step 1) and
// qs_pending == true (step 3). A stale flag causes the CPU to
// report QS "early" for the new GP, but this is safe: if rcu_nesting
// is 0, the CPU is genuinely outside all RCU read-side critical
// sections at the point of the report.
// Idle CPUs: check CpuLocal::is_idle[C]. If idle, immediately report
// QS for CPU C by calling rcu_report_qs_leaf(cpu_to_leaf[C], C - leaf.cpu_lo).
// Idle CPUs have no active RCU readers — idle entry is a quiescent state.
╔══════════════════════════════════════════════════════════════════════╗
║ Phase 2: WAIT — wait for root.qsmask == 0 ║
╚══════════════════════════════════════════════════════════════════════╝
7. fqs_wait = RCU_FQS_INITIAL_MS.
total_wait = 0.
8. Loop (force-quiescent-state scan loop):
a. Sleep for fqs_wait ms (with 10% jitter to spread wakeups:
jitter = rand_bounded(fqs_wait / 10 + 1), sleep fqs_wait + jitter).
b. total_wait += fqs_wait + jitter.
c. Check root: acquire rcu_state.root().lock.
If root.inner.qsmask == 0: release lock, goto step 11 (GP complete).
holdout_mask = root.inner.qsmask.
Release root.lock.
d. FQS scan — for each set bit B in holdout_mask:
Walk the subtree rooted at root.children[B] to find holdout CPUs.
For each holdout leaf node L:
For each set bit in L.inner.qsmask:
cpu = L.cpu_lo + bit_position.
If CpuLocal::is_idle[cpu]:
// Idle CPU — report QS on its behalf. (is_idle is a
// memory-resident AtomicBool, remotely readable.)
rcu_report_qs_leaf(L, bit_position).
Else if rcu_percpu[cpu].ct_active.load(Acquire):
// Isolated (nohz_full / CG) CPU: sample the EQS counter
// instead of interrupting it. See `RcuPerCpu.eqs_seq`.
snap = rcu_percpu[cpu].eqs_seq.load(Acquire)
If snap is odd:
// In user-mode EQS RIGHT NOW — quiescent by definition.
rcu_report_qs_leaf(L, bit_position).
Else if this is the first scan seeing this holdout:
rcu_percpu[cpu].eqs_snap = snap. // GP-kthread-private
Else if snap != rcu_percpu[cpu].eqs_snap:
// Counter moved: the CPU passed through at least one
// kernel⇄user transition — i.e., through an EQS.
rcu_report_qs_leaf(L, bit_position).
Else if total_wait >= RCU_FQS_IPI_NS / 1_000_000:
// Stuck IN KERNEL on an isolated core (even, unchanged):
// a legitimate IPI target — the noise budget protects
// user-mode execution, not overlong kernel sections.
send_resched_ipi(cpu).
Else if total_wait >= RCU_FQS_IPI_NS / 1_000_000:
// Ordinary ticking CPU stuck in the kernel without
// reporting — send the reschedule IPI.
send_resched_ipi(cpu).
// The IPI handler ([Section 7.1](07-scheduling.md#scheduler--need-resched-delivery-and-consumption-contract))
// sets the target's need_resched mirror; the target's IRQ
// exit / next preemption point runs schedule(), and
// finish_task_switch() step 1a calls rcu_check_callbacks()
// — propagation does NOT depend on a next tick existing.
e. If total_wait >= RCU_STALL_WARN_MS:
emit_rcu_stall_warning(total_wait, holdout_mask).
// Advisory, not fatal. Continue waiting.
f. fqs_wait = min(fqs_wait * RCU_FQS_BACKOFF_MULTIPLIER, RCU_FQS_MAX_MS).
g. Check root again (quick path — avoids sleeping if QS arrived during scan):
acquire root.lock, check qsmask, release.
If root.inner.qsmask == 0: goto step 11.
h. Continue loop (go to step 8a).
╔══════════════════════════════════════════════════════════════════════╗
║ Phase 3: GP COMPLETE — advance callbacks and wake waiters ║
╚══════════════════════════════════════════════════════════════════════╝
11. fence(SeqCst).
// Full memory barrier ensures all RCU-protected stores from all CPUs
// (which were ordered before their QS reports) are visible before
// callbacks execute.
12. Store complete_seq to gp_seq (Release). Broadcast RCU_SOFTIRQ to
ALL online CPUs for per-CPU softirq advancement:
complete_seq = (rcu_state.gp_seq.load(Relaxed) & !RCU_SEQ_STATE_MASK)
.wrapping_add(4).
// Relaxed ordering is sufficient here because the preceding fence(SeqCst) (step 11)
// already provides the necessary ordering fence. This load only reads the
// current gp_seq to compute the next completed sequence number; it does not
// need to synchronize with any concurrent writer (only this kthread writes gp_seq).
// Uses symbolic constant RCU_SEQ_STATE_MASK (= 0x3) instead of a magic number.
// The current gp_seq is at phase 1 (started). Masking off the phase bits
// and adding 4 advances to phase 0 (idle) of the next GP number.
// This is equivalent to the previous `(gp_seq | RCU_GP_PHASE_MASK) + 1`
// formulation but uses the canonical mask-and-add form for consistency
// with `rcu_seq_snap()`.
// wrapping_add(4) matches the wrapping_add discipline in rcu_seq_snap().
rcu_state.gp_seq.store(complete_seq, Release).
raise_softirq_on_all_cpus(SoftirqVec::Rcu).
// Broadcast to ALL online CPUs, not a filtered subset. This is O(1)
// (one IPI bitmap write to the APIC/GIC/etc.) and avoids an O(nr_cpus)
// scan of per-CPU cb_segments[RCU_WAIT] to determine which CPUs have
// pending callbacks. Each CPU's RCU_SOFTIRQ handler checks its own
// gp_seq_local < gp_seq condition locally (step 3 below) and returns
// immediately if there is no work — the cost of a no-op softirq is
// negligible compared to the cache-line bouncing of scanning remote
// per-CPU data.
//
// Note: a CPU's softirq may fire early from a stale pending flag
// (e.g., raised during a previous GP). This is harmless: the handler
// reads the *current* gp_seq (Acquire), so early firing either
// advances segments correctly (if gp_seq is already updated) or
// finds gp_seq_local == gp_seq and returns with no action.
// No additional synchronization is needed between the gp_seq store
// and the broadcast.
//
// Linux `rcu_gp_cleanup()` similarly wakes one waiter per leaf node
// and relies on each CPU to check locally.
13. Wake all tasks on rcu_state.gp_completion_wq whose wait_gp_seq <= complete_seq.
14. If rcu_state.gp_requested.load(Relaxed):
// More callbacks arrived during this GP — start another immediately.
Goto step 2.
15. Goto step 1 (sleep, wait for next request).
Callback execution (rcu_process_callbacks):
rcu_process_callbacks() — softirq handler (RCU_SOFTIRQ), per-CPU:
// Phase 1: Advance local callback segments if GP(s) have completed.
// This replaces the previous design where the GP kthread acquired
// remote locks on all CPUs (O(nr_cpus) remote lock acquisition).
// Now each CPU advances its own segments in softirq context — zero
// remote lock acquisition, proven scalable at 256+ CPUs. Matches
// Linux's per-CPU callback advancement in rcu_core().
//
// Multi-stage advancement: If multiple GPs completed between softirq
// invocations (e.g., the CPU was in a long IRQ-disabled section while
// two GPs completed), we advance segments by the number of completed GPs.
// Each completed GP advances the pipeline by one stage:
// 1 GP: WAIT→DONE, NEXT_READY→WAIT, NEXT→NEXT_READY
// 2 GPs: WAIT→DONE (then drain), NEXT_READY→DONE, NEXT→WAIT
// Without multi-stage advancement, callbacks in WAIT would stall until
// the next GP completes — unnecessary delay (RCU-F14).
1. Disable preemption.
2. local_gp_seq = rcu_state.gp_seq.load(Acquire).
3. gps_completed = (local_gp_seq - rcu_percpu.gp_seq_local) / 4.
// Each GP advances gp_seq by 4 (one full phase cycle).
// Division by 4 gives the number of complete GPs since last check.
4. If gps_completed >= 1:
// At least one GP completed. Advance segments.
// First advancement: WAIT→DONE, NEXT_READY→WAIT, NEXT→NEXT_READY.
a. Acquire rcu_percpu.cb_segments[RCU_DONE].lock.
b. Swap DONE ↔ WAIT (ring buffer pointer swap, O(1)).
// "Swap" means exchanging the (head, tail, data_ptr) triple of two
// RcuCallbackRing instances — 3 pointer-sized writes per swap.
// No individual callbacks are moved. The ring buffer backing memory
// is identity-swapped: what was the WAIT ring becomes the DONE ring.
// This is O(1) regardless of the number of queued callbacks.
c. Release lock.
d. Acquire rcu_percpu.cb_segments[RCU_WAIT].lock.
e. Swap WAIT ← NEXT_READY (ring buffer pointer swap, O(1)).
f. Release lock.
g. Acquire rcu_percpu.cb_segments[RCU_NEXT_READY].lock.
h. Swap NEXT_READY ← NEXT (ring buffer pointer swap, O(1)).
i. Release lock.
5. If gps_completed >= 2:
// Second GP also completed — advance again. The previous NEXT_READY
// (now WAIT) callbacks have also satisfied their GP requirement.
a. Drain DONE into local batch (to be executed in Phase 2).
b. Acquire rcu_percpu.cb_segments[RCU_DONE].lock.
c. Swap DONE ↔ WAIT (WAIT callbacks also done now).
d. Release lock.
e. Acquire rcu_percpu.cb_segments[RCU_WAIT].lock.
f. Swap WAIT ← NEXT_READY.
g. Release lock.
// For gps_completed >= 3: further advancement is a no-op because
// NEXT_READY and NEXT are both empty after two rounds (no callbacks
// could have been registered between GPs that both completed while
// this CPU was not running softirqs). Two advancement rounds is the
// maximum useful depth.
6. rcu_percpu.gp_seq_local = local_gp_seq.
// After advancement:
// done = callbacks ready for execution (from 1 or 2 completed GPs)
// wait = callbacks that need the *next* GP
// next_ready = callbacks registered during the most recent GP
// next = empty (ready for new rcu_call() registrations)
// Phase 2: Drain and execute done callbacks.
7. Acquire rcu_percpu.cb_segments[RCU_DONE].lock.
8. Drain all entries from the done ring, appending to local batch.
9. Release lock.
10. Re-enable preemption.
11. For each callback in the local batch:
unsafe { (cb.func)(cb.data) }; // Typically Box::drop or dealloc.
12. If batch_size > RCU_OFFLOAD_THRESHOLD (default: 64):
// Log advisory: high callback rate on this CPU.
// Consider offloading to a dedicated RCU callback-offload thread.
3.4.4 Force-Quiescent-State (FQS) Scan¶
The FQS mechanism handles CPUs that are slow to report quiescent states. It runs as part of the GP kthread's wait loop (step 8d above) and uses exponential backoff to balance latency against overhead.
EQS context tracking (the counter FQS samples): CPUs flagged
ct_active (populated at boot from nohz_full= and toggled by CG-core
provision/deprovision) bracket their user-mode intervals with two
increments of RcuPerCpu.eqs_seq (AtomicU32, memory-resident so the GP
kthread can sample it remotely — register-based CpuLocal fields cannot be
read cross-CPU):
// Return-to-user path (syscall exit / interrupt exit / exception exit),
// after the LAST kernel-side RCU-protected access:
rcu_user_enter(): // even → ODD (in EQS)
if rcu_percpu.ct_active.load(Relaxed):
rcu_percpu.eqs_seq.fetch_add(1, AcqRel)
// Kernel-entry path (syscall entry / interrupt or exception entry from
// user), before the FIRST kernel-side RCU-protected access:
rcu_user_exit(): // odd → EVEN (in kernel)
if rcu_percpu.ct_active.load(Relaxed):
rcu_percpu.eqs_seq.fetch_add(1, AcqRel)
The AcqRel RMWs order the CPU's RCU-protected accesses relative to the
counter value the GP kthread observes: a kthread that sees ODD (or a
change) knows every read-side critical section that began before the
transition has completed. Cost: one uncontended atomic RMW per
kernel⇄user crossing, on isolated cores only (ct_active is false — one
predicted-untaken branch — on ordinary CPUs); isolated cores make few
kernel entries by design, so this is well inside the CG noise budget.
The arch syscall/interrupt entry-exit trampolines call these hooks
(generic entry code — one call site per direction per architecture);
KVM's VM-enter/exit path calls the same pair (guest mode is an EQS).
Hotplug: rcu_cpu_online() resets eqs_seq to an even value.
FQS for idle CPUs: The GP kthread reports QS on behalf of idle CPUs directly.
An idle CPU is always in a quiescent state — cpu_idle_enter() guarantees no
active RCU read-side critical sections. The GP kthread reads CpuLocal::is_idle[C]
(an AtomicBool per CPU, set/cleared by cpu_idle_enter() / cpu_idle_exit())
and calls rcu_report_qs_leaf() for idle CPUs without sending an IPI. This avoids
waking idle CPUs unnecessarily (power savings on partially loaded systems).
FQS for user-mode isolated CPUs (EQS sampling — zero IPIs): For CPUs
with ct_active set (nohz_full / CG cores), the GP kthread never needs to
interrupt user-mode execution. It samples the memory-resident
RcuPerCpu.eqs_seq counter (odd = in user-mode EQS right now; changed
since the previous scan = passed through an EQS) and calls
rcu_report_qs_leaf() on the CPU's behalf — the same on-behalf mechanism
as idle CPUs. A CG core running a CPU-bound user task therefore receives
ZERO RCU IPIs and the grace period completes within one-to-two FQS scans,
preserving the <1 µs/sec CG noise budget
(Section 7.11). Linux equivalent: the
FQS dyntick sampling of the context-tracking RCU-watching counter
(kernel/rcu/tree.c snapshot/recheck of ct->state,
kernel/context_tracking.c).
FQS for kernel-mode holdout CPUs (reschedule IPI): After
fqs_interval_ns (default 1 ms) without a QS report from a CPU that is
neither idle nor in (or through) an EQS, the GP kthread sends the
reschedule IPI (send_resched_ipi(cpu)). The IPI HAS a — deliberately
minimal — handler, resched_ipi_handler()
(Section 7.1), which
sets the target's per-CPU need_resched mirror; UmkaOS's unpacked
preempt_count has no Linux-style PREEMPT_NEED_RESCHED bit, so the
mirror flag is the mechanism. The target's interrupt-return /
preempt_enable() path then calls schedule(), and
finish_task_switch() step 1a runs rcu_check_callbacks() after
releasing the rq lock — the QS propagates up the tree WITHOUT requiring
any future tick (holdouts may be tickless).
FQS tree walk: The GP kthread does not scan all CPUs sequentially. It reads the
root's qsmask to identify which top-level subtrees still have holdouts, then
descends only into those subtrees. On a 4096-CPU / 64-leaf system where 4000 CPUs
have already reported, the FQS scan touches only the 1-2 leaf nodes covering the
remaining holdout CPUs — not all 64 leaves.
| FQS scan # | Wait before scan | Cumulative wait | Action |
|---|---|---|---|
| 1 | 1 ms | 1 ms | Scan holdout leaves; report idle CPUs |
| 2 | 2 ms | 3 ms | Same + IPI to running holdouts (if > fqs_interval_ns) |
| 3 | 4 ms | 7 ms | Same |
| 4 | 8 ms | 15 ms | Same |
| 5 | 16 ms | 31 ms | Same |
| 6 | 32 ms | 63 ms | Same |
| 7 | 64 ms | 127 ms | Same |
| 8+ | 100 ms (capped) | 227+ ms | Same; stall warning at 10 s |
3.4.5 Expedited Grace Periods¶
synchronize_rcu_expedited() bypasses the normal tree-based wait and forces an
immediate quiescent state on all online CPUs via IPI. This is expensive (O(nr_cpus)
IPIs) but provides bounded GP latency.
/// Force an immediate grace period by IPI-ing all online CPUs.
///
/// **Cost**: One IPI per online CPU + one context switch per CPU.
/// On a 256-CPU system: ~256 IPIs, ~50 μs total latency.
///
/// **Use sparingly**: This is appropriate for emergency operations
/// (module unload, CPU hotplug, OOM kill cleanup) where waiting for
/// the normal GP (10-100 ms) is unacceptable. Normal `rcu_synchronize()`
/// is preferred for all other cases.
///
/// **MUST NOT be called from atomic context.**
pub fn synchronize_rcu_expedited();
Algorithm:
synchronize_rcu_expedited():
1. If only one CPU is online: return immediately (single CPU = always quiescent).
2. Snapshot seq = rcu_state.gp_seq.load(Acquire).
3. Initialize the tree (same as GP start, steps 2-6 above).
4. For each online CPU C (excluding self):
Send IPI_RCU_EXP to CPU C.
// IPI handler on target CPU:
// a. If CpuLocal::rcu_nesting > 0: set CpuLocal::rcu_passed_quiesce = true.
// (CPU is in an RCU read-side section — the handler sets the flag;
// the section's outermost RcuReadGuard::drop will call rcu_qs(),
// and the next tick's rcu_check_callbacks() propagates the QS.)
// b. If CpuLocal::rcu_nesting == 0: call rcu_check_callbacks() directly.
// (CPU is not in an RCU section — report QS immediately.)
5. Call rcu_check_callbacks() on self (report own QS).
6. Wait for root.qsmask == 0 (busy-wait with short pause loop, no backoff).
// Expedited GPs are rare; busy-wait is acceptable.
// Timeout: if root.qsmask != 0 after 1 second, emit stall warning
// and send a second round of IPIs.
7. fence(SeqCst).
8. Advance callback segments (same as GP complete, steps 12-15).
9. Return.
3.4.6 RCU Interaction with Live Kernel Evolution¶
During Phase B of live evolution (Section 13.18), all CPUs are halted via IPI for the atomic vtable pointer swap (~1-10 us). No RCU grace period can complete during this window because no CPU can pass through a quiescent state while halted. RCU callbacks queued before or during Phase B are processed after all CPUs resume normal execution. The Phase B window is short enough (bounded by the stop-the-world timeout, default 100 us) that RCU grace period stall detection (threshold: 21 seconds) is never triggered. The GP kthread resumes its normal polling loop after Phase B completes and detects quiescent states from the resumed CPUs within one scheduler tick (~1 ms).
Two evolution windows, one stall-safety argument. Live evolution opens the RCU
grace period to no quiescent states in TWO shapes, and the same bounded-window argument
covers both. (1) The STW IPI window above (generic/batch component swap): halted CPUs
pass no quiescent states, bounded by the stop-the-world timeout. (2) The cooperative
parked rendezvous window (scheduler evolution, Section 13.18): each CPU
acks only after completing its current scheduler transaction — including
finish_task_switch()'s rq.lock release — and then parks OUTSIDE scheduler code, so it
too passes no quiescent state while parked; the window is bounded by the rendezvous
deadline exactly as the STW window is bounded by its timeout. Additionally, the scheduler
evolution's Phase A'-drain uses a NORMAL process-context RCU grace period to drain
non-blocking policy readers BEFORE the parked window opens — the grace period never
overlaps the park, so a parked CPU is never waited on by an in-progress grace period.
KABI Direct RcuBounded stubs are classic RCU readers. A same-domain Direct call to
a nonblocking method takes the RcuBounded witness
(Section 12.8): a
preempt-disabled RCU read-side section that writes NO global metadata (only local
preempt-count accounting, maintained wherever the code runs). Because preempt accounting is
maintained on every CPU, ANY same-domain caller — Tier 0 Core or a Tier 1 same-domain
peer — takes this leg; this is the frame ADDENDUM of record. The binding-level LEASE
fallback is the stated EXCEPTION, only for configurations that genuinely cannot maintain
the preempt-disable/RCU bracket discipline — never the Tier-1-peer default. Component
quiescence for the RcuBounded cohort therefore depends on exactly ONE grace period after
the image gate closes: quiesce_direct_calls() closes the gate, then calls
synchronize_rcu(), and the old target becomes unreachable to RcuBounded readers when
that grace period ends (Section 12.8).
3.4.7 CPU Hotplug and the RCU Tree¶
When a CPU comes online or goes offline, the tree's online_mask fields must be
updated to prevent the GP kthread from waiting for a CPU that will never report.
CPU online (rcu_cpu_online(cpu)):
rcu_cpu_online(cpu):
1. leaf = rcu_state.cpu_to_leaf[cpu].
2. bit = cpu - leaf.cpu_lo.
3. Acquire leaf.lock.
4. leaf.inner.online_mask |= (1u64 << bit).
5. leaf.inner.n_children += 1.
6. Release leaf.lock.
7. // If a GP is in progress, this CPU's bit will be set in qsmask
// at the next GP start. For the current GP, the newly online CPU
// is not required to report (it wasn't online when the GP started).
8. Initialize rcu_percpu[cpu]: gp_seq_local = rcu_state.gp_seq,
qs_pending.store(false, Relaxed).
CPU offline (rcu_cpu_offline(cpu)):
rcu_cpu_offline(cpu):
1. leaf = rcu_state.cpu_to_leaf[cpu].
2. bit = cpu - leaf.cpu_lo.
3. Acquire leaf.lock.
4. leaf.inner.online_mask &= !(1u64 << bit).
5. leaf.inner.n_children -= 1.
6. If leaf.inner.qsmask & (1u64 << bit) != 0:
// This CPU owed a QS for the current GP. Clear its bit
// (offline = implicit quiescent state — no active readers).
leaf.inner.qsmask &= !(1u64 << bit).
mask = leaf.inner.qsmask.
Release leaf.lock.
If mask == 0:
rcu_report_qs_up(leaf). // Propagate to parent.
7. Else:
Release leaf.lock.
8. Drain rcu_percpu[cpu].cb_segments — migrate pending callbacks to the
current CPU's segments (the offlined CPU will never process them).
// Migration is done under the local CPU's preemption-disabled section.
// Callbacks are appended to the current CPU's `next` segment.
3.4.8 NUMA-Aware Tree Construction¶
The tree is constructed with NUMA affinity: leaf nodes covering CPUs on the same NUMA domain are grouped under the same interior node when possible. This ensures that the most frequent lock acquisitions (leaf-level QS reporting) hit NUMA-local locks.
rcu_build_tree NUMA heuristic:
1. Sort CPUs by NUMA proximity domain (from ACPI SRAT or device tree).
2. Assign leaf nodes in NUMA-domain order:
- CPUs 0-63 on NUMA node 0 → leaf 0.
- CPUs 64-127 on NUMA node 0 → leaf 1.
- CPUs 128-191 on NUMA node 1 → leaf 2.
- Etc.
3. Group leaf nodes by NUMA domain under interior nodes:
- If NUMA node 0 has 2 leaves and NUMA node 1 has 2 leaves,
the root has 2 interior children (one per NUMA domain), each
with 2 leaf children.
- This groups NUMA-local leaves together, minimizing cross-node
lock traffic during QS propagation.
4. If the topology doesn't divide evenly, remaining CPUs fill partial
leaf nodes. The last leaf node may have fewer than leaf_fan_out CPUs.
Per-architecture notes:
| Architecture | NUMA discovery | Notes |
|---|---|---|
| x86-64 | ACPI SRAT + MADT | Standard path. Intel and AMD systems with 2-8 NUMA nodes. |
| AArch64 | ACPI SRAT or DT numa-node-id |
Server-class ARM (Ampere Altra, Graviton) uses ACPI. Embedded uses DT. |
| ARMv7 | Single NUMA node (UMA) | All CPUs in one leaf group. Tree depth = 1. |
| RISC-V 64 | DT numa-node-id |
NUMA support varies by platform. Single-node typical today. |
| PPC32 | Single NUMA node (UMA) | All CPUs in one leaf group. |
| PPC64LE | ACPI SRAT (PowerVM) or DT (KVM) | POWER9/10 NUMA with up to 16 nodes. |
| s390x | Single NUMA node (UMA) | z/VM LPARs present as single-node. |
| LoongArch64 | ACPI SRAT | Loongson 3C5000 has 4 NUMA nodes. |
rcu_read_lock() / rcu_read_unlock() algorithms (non-preemptible RCU):
UmkaOS uses non-preemptible RCU: rcu_read_lock() disables preemption via
preempt_count, so any context switch is a quiescent state. This is simpler
and lower overhead than preemptible RCU (no deferred quiescent state tracking
needed on the preempt path), and appropriate for a kernel targeting <5% overhead.
RCU readers cannot sleep or be preempted. Grace periods are detected by
quiescent state tracking (context switch, idle, user return) with bottom-up tree
propagation — no synchronization between readers and the GP kthread.
rcu_read_lock():
1. Increment CpuLocal.preempt_count (disables preemption).
2. Increment CpuLocal.rcu_nesting.
Cost: ~2 instructions (two CpuLocal register writes, no memory barriers,
no cache-line bouncing). Preemption remains disabled while rcu_nesting > 0.
(A descheduled task has by definition passed through a quiescent state,
since context switches only occur when preempt_count == 0.)
rcu_read_unlock():
1. Decrement CpuLocal.rcu_nesting.
2. If rcu_nesting == 0:
If CpuLocal.rcu_passed_quiesce == false:
Set CpuLocal.rcu_passed_quiesce = true.
// Do NOT acquire the leaf node's lock here.
// The actual tree propagation is deferred to the next scheduler
// tick or context switch via rcu_check_callbacks(). This avoids
// a spinlock acquisition on every outermost RCU drop (critical
// on high-IOPS paths such as NVMe/conntrack/routing).
3. Decrement CpuLocal.preempt_count (re-enables preemption if count == 0).
4. If preempt_count == 0 and resched_pending: call schedule().
UmkaOS Tree-RCU design choices vs. Linux:
- Hierarchical tree with runtime-discovered geometry: Tree depth and fan-out are
computed at boot from actual CPU count and NUMA topology. No CONFIG_RCU_FANOUT
compile-time constant — the tree adapts to hardware.
- 4-segment callback pipeline: done/wait/next_ready/next segments advance
as pointer swaps (O(1)) at GP completion. Compared to Linux's segmented callback
list (which uses intrusive linked-list splicing), UmkaOS uses pre-allocated ring
buffers — no container_of pointer arithmetic, no per-callback heap allocation.
- Single GP kthread: The hierarchical tree makes per-NUMA-node GP threads
unnecessary — CPUs propagate QS reports bottom-up through the tree, and the GP
kthread only monitors the root. This reduces kthread count from O(NUMA_nodes) to 1.
- FQS tree walk: The force-quiescent-state scan descends only into subtrees with
holdout CPUs (guided by the root's qsmask), not all CPUs sequentially.
- rcu_call overflow: Context-aware: task context falls back to
rcu_synchronize() + direct execution (never drops callbacks); atomic context
uses a pre-allocated 64-slot overflow_buf drained at the next timer tick via
rcu_tick_drain_overflow(). Both paths log warnings; only a full overflow buffer
in atomic context (catastrophic RCU stall) drops a callback and logs at error level.
/// Defer a callback to run after an RCU grace period (non-blocking).
///
/// This is the preferred way to free memory from atomic contexts. The callback
/// will be invoked in the RCU worker thread context after all pre-existing
/// readers have completed.
///
/// # Safety
/// - The callback must not access any data that was freed before the callback runs.
/// - The callback runs in a kernel thread context (not interrupt context), so
/// it may block, but should complete quickly to avoid delaying other callbacks.
///
/// # Example
/// ```rust
/// // Defer Box::drop after grace period
/// let ptr = Box::into_raw(old_value);
/// unsafe { rcu_defer_free(ptr) }; // callback will reconstruct Box and drop it
/// ```
pub unsafe fn rcu_defer_free<T>(ptr: *mut T);
/// Defer Box::drop after an RCU grace period (non-blocking).
///
/// This is a convenience wrapper around `rcu_defer_free()` for the common case
/// of freeing a Box<T> after an RCU grace period.
///
/// # Safety
/// Same as `rcu_defer_free()` — ptr must have been obtained from `Box::into_raw()`.
pub unsafe fn rcu_call_box_drop<T>(ptr: *mut T) {
// Reconstruct the Box and let it drop after the grace period.
rcu_defer_free(ptr);
}
RCU Slice Lifetime Safety Example (KRL):
The KeyRevocationList in Section 9.3 demonstrates correct RcuSlice usage:
// Boot-time KRL allocation (bump-allocated, 'static lifetime):
let boot_krl = KeyRevocationList {
revoked_keys: unsafe { RcuSlice::from_raw(boot_keys_ptr) }, // lives forever
revoked_count: boot_count,
// ... other fields
};
// Runtime KRL allocation (slab-allocated, RCU-managed):
let rt_keys = Box::try_new_in_slice(count, slab_allocator)?;
let rt_krl = Box::try_new(KeyRevocationList {
revoked_keys: unsafe { RcuSlice::from_raw(Box::as_slice_ptr(rt_keys)) },
revoked_count: count,
// ... other fields
})?;
// rt_krl is published via RcuCell::update(). Old KRL (if any) is freed
// by the RCU callback after the grace period, including its revoked_keys array.
/// Compile-time lock ordering: deadlock prevention via the type system.
/// A Lock<T, 30> can only be acquired while holding a Lock<_, N> where N < 30.
/// Lock levels are spaced by 10 (0, 10, 20, ...) to allow future insertions
/// between adjacent levels without full renumbering.
/// Attempting to acquire locks out of order is a compile-time error.
///
/// **Compiler feature note**: The `Assert<{ HELD < LEVEL }>: IsTrue` trait-bound
/// pattern shown below requires `#![feature(generic_const_exprs)]` (nightly,
/// tracking issue #76560, `incomplete` status as of 2025 — not on a
/// stabilization path). The **stable alternative** is `const { assert!(HELD < LEVEL) }`
/// (inline const, stable since Rust 1.79). Implementation SHOULD use the stable
/// form; the spec uses the trait-bound form for clarity of intent. Both produce
/// identical compile-time failures on ordering violations with zero runtime cost.
pub struct Lock<T, const LEVEL: u32> {
inner: SpinLock<T>, // or MutexLock<T> for sleeping locks
}
impl<T, const LEVEL: u32> Lock<T, LEVEL> {
/// Construct a lock wrapping `value`. `const fn` so a `Lock<T, LEVEL>`
/// can appear in `static` initializers (e.g. `handles: Lock::new(ArrayVec::new())`).
pub const fn new(value: T) -> Self {
Self { inner: SpinLock::new(value) }
}
pub fn lock<const HELD: u32>(&self, _proof: &LockGuard<HELD>) -> LockGuard<LEVEL>
where
Assert<{ HELD < LEVEL }>: IsTrue,
// Stable alternative (Rust 1.79+):
// const { assert!(HELD < LEVEL, "lock ordering violation") }
{
LockGuard::new(self.inner.lock())
}
/// Acquiring the first lock in a chain requires no proof.
/// Any level can be the starting lock — the constraint is that no other
/// lock is currently held. The returned `LockGuard<LEVEL>` then constrains
/// all subsequent lock acquisitions to levels strictly greater than LEVEL.
///
/// **Enforcement**: The type system alone cannot prevent a caller from
/// invoking `lock_first()` while already holding a `LockGuard` from a
/// different scope. Therefore, `lock_first()` performs a **runtime check**
/// in DEBUG builds: it reads the per-CPU `max_held_level` field and panics
/// if any lock is currently held. The check is a cheap per-CPU atomic load
/// + branch (~2-3 cycles) but is `#[cfg(debug_assertions)]`-gated
/// (ESC-0431: `max_held_level` is a debug-only `CpuLocalBlock` field, see
/// [Section 3.2](#cpulocal-register-based-per-cpu-fast-path)) — it compiles out of
/// release builds. The `lock()` method's compile-time ordering guarantee
/// (via `HELD < LEVEL`) remains the primary enforcement in EVERY build; the
/// debug-only runtime check in `lock_first()` closes the residual
/// unrelated-scope loophole during development.
///
/// **Cross-session ABBA prevention**: Lock ordering is enforced by a global
/// total order defined at compile time via the `LEVEL` const type parameter.
/// Two sessions acquiring lock A (level 2) then lock B (level 5) always
/// acquire in the same order because the type system enforces `HELD < LEVEL`
/// at every step.
pub fn lock_first(&self) -> LockGuard<LEVEL> {
assert_no_locks_held(); // per-CPU max_held_level == LEVEL_NONE
LockGuard::new(self.inner.lock())
}
}
/// **Per-CPU `max_held_level`** — a field of `CpuLocalBlock`, see
/// [Section 3.2](#cpulocal-register-based-per-cpu-fast-path):
///
/// max_held_level: AtomicU32 — level of the DEEPEST `Lock<T, LEVEL>` currently
/// held on this CPU, or `LEVEL_NONE` (0) when none is held.
/// Real lock levels start at 1, so 0 is an unambiguous
/// sentinel. Touched on the local CPU only, but a lock
/// acquired in hardirq context can update it while a
/// task-context path is mid-access, so it is `AtomicU32`
/// (`Relaxed`) — the same async-same-CPU reason
/// `need_resched` is atomic; access is field-scoped
/// (`CpuLocal::max_held_level()`), never a whole-block
/// reference (ESC-0431). `LockGuard::new` records the
/// previous value and stores the acquired `LEVEL` (always
/// the new maximum, since the ordering invariant forces
/// `LEVEL` above every held level); `LockGuard::drop`
/// restores the previous value. `#[cfg(debug_assertions)]`
/// — the field and its `LockGuard` bookkeeping exist ONLY
/// in debug builds (ESC-0431); release builds carry
/// neither the field nor the store/restore, and rely on
/// the compile-time `HELD < LEVEL` bound alone.
const LEVEL_NONE: u32 = 0;
/// Panic if this CPU already holds any `Lock<T, LEVEL>`.
///
/// Reads the per-CPU `max_held_level` and panics unless it is `LEVEL_NONE`.
/// Called by `Lock::lock_first()` to close the one loophole the compile-time
/// `HELD < LEVEL` bound cannot cover: beginning a fresh lock chain while a guard
/// from an unrelated scope is still alive. DEBUG builds only: `max_held_level`
/// is a `#[cfg(debug_assertions)]` field on `CpuLocalBlock`
/// ([Section 3.2](#cpulocal-register-based-per-cpu-fast-path), ESC-0431), so this
/// runtime cross-scope check compiles out of release builds (the compile-time
/// `HELD < LEVEL` bound stays in force in every build; only the residual
/// unrelated-scope loophole is unguarded in release). Reads via the field-scoped
/// `CpuLocal::max_held_level()` projection — one per-CPU atomic load plus a
/// branch (~2-3 cycles) in debug. Distinct from the richer debug-only
/// `held_locks` stack ([Section 3.5](#locking-strategy--lock-contention-tracking)).
#[cfg(debug_assertions)]
#[inline]
fn assert_no_locks_held() {
let held = CpuLocal::max_held_level().load(Relaxed);
assert!(
held == LEVEL_NONE,
"lock_first() called while this CPU already holds a lock (level {held})"
);
}
/// Release-build no-op: the runtime unrelated-scope check is debug-only (the
/// `max_held_level` field it reads exists only under `debug_assertions`).
#[cfg(not(debug_assertions))]
#[inline(always)]
fn assert_no_locks_held() {}
/// **Lock ordering: ZERO exceptions.**
///
/// The `Lock<T, LEVEL>` compile-time ordering system enforces a TOTAL order
/// on all lock acquisitions: a thread holding a lock at level N may only
/// acquire locks at levels > N. There are NO escape hatches, no
/// `lock_read_unchecked()`, no compile-time call-site caps, no runtime
/// fallback validators.
///
/// **History**: The spec previously required a `lock_read_unchecked()` method
/// for the page fault path, which acquired the former level-90 invalidation rwsem for read under
/// `VMA_LOCK(105, read)` -- a descending-level violation. This exception was
/// eliminated by replacing that rwsem with `InvalidateSeq`
/// (a lockless seqcount). The fault path now performs two atomic loads instead
/// of acquiring a lock:
///
/// **Fault path lock chain (strictly ascending):**
/// `VMA_LOCK(105, read)` -> `PAGE_LOCK(180)` -> `PTL(185)`
///
/// No lock ordering violation. No exception needed.
///
/// This means `Lock<T, LEVEL>` is UNIVERSAL: every lock acquisition in the
/// entire kernel goes through the compile-time ordering check. Debug builds
/// additionally validate at runtime via the per-CPU `held_locks` stack
/// ([Section 3.5](#locking-strategy--lock-contention-tracking)).
Lock categories: Subsystems define named lock categories to group related lock levels and prevent cross-subsystem lock violations:
/// Named lock category for subsystem-level lock grouping.
/// Each category maps to a range of lock levels. The runtime debug lock-order checker
/// validates that locks from different categories are never held simultaneously
/// unless explicitly permitted in the cross-category ordering table.
#[repr(u32)]
pub enum LockCategory {
/// Core kernel locks (scheduler run queues, memory allocator).
Core = 0,
/// Filesystem and block layer locks.
Fs = 1,
/// Network stack locks.
Net = 2,
/// Windows Emulation Architecture (NT object manager, WEA syscalls).
WEA = 3,
/// Driver subsystem locks (device registry, KABI VTable).
Driver = 4,
}
3.4.8.1.1 Lock Ordering¶
Lock level assignment table (authoritative; used by all Lock<T, LEVEL> instantiations):
Level 0 is reserved for locks that must be acquirable while the lock-ordering
checker is active and before any subsystem lock is held. Locks at level 0 are
never acquired while holding another level-0 lock; they are independent entry
points into the lock graph. FUTEX_BUCKET is the primary example: futex_wake()
acquires a bucket lock and then calls scheduler::enqueue(), which acquires
RQ_LOCK (level 50). The bucket lock must therefore be below TASK_LOCK (level 20)
to keep the acquisition order valid.
PI futex and RT_MUTEX: Priority-inheritance (PI) futex operations acquire
FUTEX_BUCKET (level 0) to locate the waiter, then walk the priority inheritance
chain via RT_MUTEX (level 10) to propagate priority boosting. The RT_MUTEX lock
protects the PI waiter tree and the priority inheritance chain; it must sit between
FUTEX_BUCKET and TASK_LOCK because PI chain walking reads (but does not modify)
the task struct, and may call rt_mutex_adjust_prio() which acquires TASK_LOCK.
The ordering is therefore: FUTEX_BUCKET(0) -> RT_MUTEX(10) -> TASK_LOCK(20).
Levels 30-40 cover per-task sub-locks
(SIGHAND_LOCK < FDTABLE_LOCK), taken after the task lock but before the
scheduler's run queue lock. Levels 70-90 cover capability and memory management
locks (CAP_TABLE_LOCK < VM_LOCK < ADDR_SPACE_LOCK), reflecting the invariant that
capability validation precedes address space mutation, and VMA-level decisions precede
page table modifications. Filesystem locks (FS_SB_LOCK < INODE_LOCK < DENTRY_LOCK)
follow the same outer-to-inner principle: superblock state is acquired before per-inode
operations, which in turn precede dentry cache manipulation.
Lock levels are spaced by 10 (0, 10, 20, ..., 260) with intermediate values (e.g., 105, 125) used for locks that must nest between adjacent primary levels. This provides 9 insertion points between any two adjacent levels for future additions.
| Level | Lock Name | Subsystem | Category |
|---|---|---|---|
| 0 | FUTEX_BUCKET |
FutexBucket spinlock | Core |
| 0 | IRQ_DESC_LOCK |
Interrupt descriptor | Core |
Level-0 non-co-holding proof:
FUTEX_BUCKETandIRQ_DESC_LOCKboth use level 0. The compile-time ordering system rejectsLock<T, 0>→Lock<U, 0>acquisition (0 < 0is false). These two locks are never co-held in any code path:FUTEX_BUCKETis taken by futex syscalls (process context only);IRQ_DESC_LOCKis taken by interrupt setup/teardown and interrupt-descriptor access. No futex code path touches IRQ descriptors, and no IRQ setup code path touches futex hash buckets. | 10 |RT_MUTEX|SpinLock— per-RtMutexwaiter tree lock — protects the priority inheritance waiter tree and PI chain. Acquired by theFUTEX_LOCK_PIpath afterFUTEX_BUCKETto walk and adjust the PI chain. Released before acquiringTASK_LOCKfor priority adjustment. | Core | | 12 |CRASH_LOCK|SpinLock— per-domainDomainDescriptor.crash_lock— protects domain revocation, ring state transition, and the NMI park+ack rendezvous (Step 2a broadcasts the crash NMI; every CPU parks and acks — ejection is NOT broadcast but deferred to at-resume: a window ejects when its unpark installs the deny-all image and faults) during crash recovery Steps 1-2a (exception/NMI context) and during the FMA reload preamble's gate blocks P1/P4-P6 (process context). Always acquired with NO ordered lock held: the exception handler takes it on fresh exception entry, and the preamble takes it holding onlyrecovery_mutex(a sleeping Mutex — separate table; sleeping-mutex → SpinLock nesting is legal). The ONLY ordered lock acquired under it isXA_LOCK(178): the exception-context Step 2′ recovery-descriptor walk (walk_recovery_descriptors) holdscrash_lockacross its iteration of the per-domainrecovery_descriptorsXArray, taking that XArray's internalxa_lockfor each bounded batch — soCRASH_LOCK(12) → XA_LOCK(178)is a declared increasing-order chain (seeregister_recovery_descriptor, Section 11.9); theisolate_fnhooks themselves run OUTSIDE any XArray lock and are non-sleeping by contract. Otherwise nothing ordered is acquired under it (only atomics, the arch domain image table, and the NMI IPI machinery). The interrupted task's held locks are NOT part of the exception handler's acquisition chain (distinct context). Declared atkabi/kabi-domain-runtime.md(DomainDescriptor). | Driver | | 15 |NSPROXY_LOCK|RwLock— the level shared by the per-sub-namespace object locks (MNT_NS_LOCK,UTS_NS_LOCK,IPC_NS_LOCK,NET_NS_LOCK,TIME_NS_LOCK,USER_NS_LOCK,IMA_NS_LOCK), taken during the independent per-namespace copy in fork /unshare()— they are never co-held (each namespace is copied independently in sequence). Namespace-SET replacement is NOT serialized at this level: swappingTask::namespace_set— and the paired credential commit on the CLONE_NEWUSER arms — runs underTASK_LOCK(20), the authoritative replacement serializer (see that row). BelowPROCESS_TREE_WRITE_LOCK(18)because namespace propagation precedes parent-list mutation. Declared atcontainers/namespace-architecture.md(NsProxy struct). | Container | | 18 |PROCESS_TREE_WRITE_LOCK|RwLock<()>— global RwLock for cold-path multi-task atomic mutations:exit_task()reparent (Step 12),ptrace_attach()reparent, process-group / session leadership changes. Renamed from Linux'stasklist_lockto discourage hot-path use — hot paths use RCU + per-PID atomic fields. Read-held duringfor_each_task()walks; write-held during reparent. Acquired beforePROCESS_LOCK(25)in reparent path. Declared atprocess/process-and-task-management.md(globalPROCESS_TREE_WRITE_LOCK: Lock<RwLockInner, 18>). Used atprocess/process-lifecycle-teardown.md:1169. | Core | | 20 |TASK_LOCK|Lock<(), 20>—Task::task_lock— per-task structure lock. Serializes non-atomic commits of the task's swappable resource state: thenamespace_setArcSwapswap/store (setns(2)/unshare(2)/ clone commit), the pairedinstall_credentials()+namespace_set.store()single-critical-section commit (CLONE_NEWUSER arms), and the fs root/pwd reset window. Authoritative serializer for namespace-set replacement (NOTNSPROXY_LOCK(15)— the CLONE_NEWUSER commit needs credentials andnamespace_setin ONE critical section, which a namespace-set-scoped level-15 lock cannot cover). O(1) spin critical-section budget: no sleeping, no allocation, noArc<NamespaceSet>drop inside the section (swap-then-drop-outside idiom);TRACKED_REGISTRY_LOCK(135)work is pre-allocated before the section. Chain-start acquisition through thetask_lock()accessor; callers holding a lower level usetask_lock.lock(&guard). Declared atprocess/process-and-task-management.md(Task.task_lock). | Core | | 22 |PID_TABLE_LOCK|SpinLock— write lock for the global task tablePID_TABLE: XArray<TaskId, Arc<Task>>(Section 8.1). Serializes WRITERS only (fork step 16 insert,reap_task()'spid_table_detach(),collapse_thread_group()'s ProcessId-alias re-point). ALL reads — including the lookup-and-pin primitivefind_task_by_tid()— are RCU-only and may observe the collapse_thread_group multi-structure PID swap mid-flight; that is safe (each map read is individually atomic, TaskIds are never reused, and a stale pin resolves to the ZOMBIE old leader sharing the sameArc<Process>/SignalHandlers), but implementations MUST NOT assume the intermediate state is unobservable to readers. Acquired in fork for child insert, in exec'scollapse_thread_group()for the PID swap, and inexit_task()Step 12 for child reparent. Ordering:PROCESS_TREE_WRITE_LOCK(18) → PID_TABLE_LOCK(22) → PROCESS_LOCK(25). Distinct from per-namespacePID_MAP_LOCK(47)(namespace-local pid_t alloc/free). Declared atprocess/process-and-task-management.md. | Core | | 25 |PROCESS_LOCK|SpinLock<()>—Process::lock— per-Processlock serializing thread-group membership mutations (thread_group.tasksmutators ALSO holdSIGLOCK(40)— dual-lock contract, so signal paths may iterate under 40 alone), children XArray insert/remove,task_listpointer mutation, thepid_group_nodelink/unlink/move plus the pairedProcess.pgrpload/store (job-control process-group membership — the node's LOCATION is owned by this lock;PGRP_MEMBERS_LOCK(27)nests under it for list structure only, Section 8.7), and the delayed-leader claim-eligibility evaluation (a ZOMBIE thread-group leader is claimable only when it is the SOLE task linked onthread_group.tasks—tasks.len() == 1— read under this lock inreap_task()'s merged unlink/handoff scope and in thewait4()/waitid()reap scan;thread_group.count == 0is only a lock-free pre-filter, Section 8.1). Acquired in fork for parent insert-child and inexit_task()reparent for parent's children-list mutation. Held briefly; never taken with IRQs disabled. Declared inProcessstruct inprocess/process-and-task-management.mdaspub lock: Lock<(), 25>. | Core | | 27 |PGRP_MEMBERS_LOCK|Lock<IntrPidList, 27>— per-ProcessGroupmembersintrusive member-list lock (Section 8.7). Serializes list STRUCTURE only: walkers (send_signal_to_pgrp,group_has_stopped_member,kill(-pgid)fan-out, orphan walks) hold it ALONE. The LOCATION of apid_group_node— which group it is linked into — is owned by the enclosingPROCESS_LOCK(25), NOT by this lock: every mover (setpgid, exec-collapsec2, fork step-17 link, fork rollback row 20,reap_task()step 5c) holds25THEN27(25 → 27 ascending) with the pairedProcess.pgrpload/store inside the 25 section, and a move between two groups is two SEQUENTIAL 27 scopes (unlink-from-old strictly BEFORE link-into-new — one embedded node cannot be on two lists), NEVER two 27 holds at once. Level derivation (must sit strictly between 25 and 30): movers give25 < members;send_signal_to_pgrpholdsmembersacross per-member delivery, which nestsSIGHAND_LOCK(30) → SIGLOCK(40)(the row-30 dual-lock rule), forcingmembers < 30; the 26-29 band is free, 27 chosen. Held briefly; a walker may nest 30/40 under it during delivery (27 → 30 → 40 ascending). Declared inProcessGroupinprocess/process-groups-and-sessions.mdaspub members: Lock<IntrPidList, 27>. | Core | | 30 |SIGHAND_LOCK|SpinLock—SignalHandlers::lock— writer-serialization lock for the signal disposition table. Dual-lock writer rule: every mutation of a publishedaction[]table (sigaction(),force_sig(), SA_RESETHAND, exec's signal reset) holds this lock AND nestsSIGLOCK(40)inside it around the store (30→40 ascending); delivery-path multi-fieldSigActionreads holdSIGLOCK(40)only, copy-path reads (deep_copy_dispositions(), exec's pre-swap copy) hold this lock only — either lock excludes writers. Held briefly; never taken with IRQs disabled. Declared atprocess/process-and-task-management.md(SignalHandlers). | Core | | 32 |SIGNAL_STRUCT_LOCK|Lock<Option<i32>, 32>—ThreadGroup.exit_code— protects the thread-group-level exit code (the group-exit initiation word). NOTgroup_stop_count(anAtomicU32onThreadGroupwhose mutations are ALL serialized bySIGLOCK(40)— see the field's contract) and NOToom_score_adj(lock-freeAtomicI16onProcess). Because 40 may not nest a 32 acquisition (descending),do_signal_stop()'s initiation gate substitutes a fatal-signal-pending check for readingexit_codeunder SIGLOCK. Distinct fromSIGHAND_LOCK(30)(signal handler table) andSIGLOCK(40)(per-task pending queue). Acquired byexit_task()Step 3 (no other lock held) and exec'scollapse_thread_group()Phase 1 (under PROCESS_LOCK(25); guard dropped BEFORE SIGLOCK(40) — 25→32 then 25→40, both ascending; holding it into a 40 acquisition would be a compile-rejected descending chain). Declared atprocess/process-and-task-management.md(ThreadGroup). | Core | | 40 |SIGLOCK|SpinLock— per-sighand signal queue lock — serializes signal delivery, pending mask updates, group stop state (Task.jobctlstop bits andThreadGroup.group_stop_count— all mutations under this lock), and delivery-path multi-fieldSigActionreads (writers hold 30 AND 40 — seeSIGHAND_LOCK(30)row). Also the second half of theThreadGroup.tasksdual-lock mutation contract (list mutations holdPROCESS_LOCK(25)AND this lock; readers hold either). Acquired underSIGHAND_LOCK(30)during signal delivery; chains toRQ_LOCK(50)viatry_to_wake_up()when waking the target task. IRQ-safe. | Core | | 40 |FDTABLE_LOCK|SpinLock—FdTable::inner— serializes fd alloc/close/dup within a shared file descriptor table. Held briefly for O(1) fd operations. Mutually exclusive withSIGLOCK— both at level 40 means the compiler rejects holding both simultaneously (40 < 40is false). Never co-held in any Linux or UmkaOS code path. | Core | | 42 |FS_STRUCT_LOCK|RwLock—FsStruct::lock— per-task filesystem context (cwd, root, umask) lock. Read-held during fork'scopy_fs()and during path lookup root resolution; write-held duringchdir(),chroot(),fchdir(). BelowPI_LOCK(45)because path lookup may sleep awaiting a dentry while holding read; PI propagation must not nest inside path lookup. Declared atprocess/process-and-task-management.md(FsStruct). | Fs | | 43 |PT_REG_LOCK|LockIrqSafe<(), 43>— per-task ptrace run-gate (Task.pt_reg_lock) — serializes a tracer's inspection window (register-state access, cross-domain memory copy underwith_domain_access(), Section 11.3) against ANY transition of the tracee out of a ptrace stop. Three acquisition chains: (1) resume/detach —PROCESS_TREE_WRITE_LOCK(18)(detach / tracer-exit auto-detach) orSIGLOCK(40)(PTRACE_CONT/SYSCALL/SINGLESTEP/LISTEN resume) →PT_REG_LOCK(43)→try_to_wake_up()'sPI_LOCK(45)/RQ_LOCK(50), held across the TASK_TRACED-exit state store and wake; (2) fatal wake —SIGLOCK(40)→PT_REG_LOCK(43)→PI_LOCK(45)/RQ_LOCK(50):signal_wake_up(task, fatal=true)'s TRACED-masked wake (SIGKILL kills a ptrace-stopped tracee), whose senders include hard-IRQ contexts (scheduler-tick RLIMIT_CPU/RLIMIT_RTTIME hard-limit SIGKILL, posix-timer SIGKILL from HrTimer expiry) — hence IRQ-safe: every acquisition islock_irqsave()(a tick SIGKILL on the holder's own CPU would otherwise self-deadlock); (3) tracer access — acquired with no ordered lock held, then held across the preemption-off + IRQs-off composed-access window, whose closure acquires NOTHING (leaf on this path; the debugger-sideput_usertransfer happens strictly OUTSIDE the lock — it can sleep). Holder must verify the tracee is stopped AND fully off-CPU after acquiring (-ESRCH otherwise). All holds are short and bounded (word/register-block copies, state store + wake), keeping the IRQs-off windows bounded. Level derivation: > 40 (resume and fatal-wake paths hold SIGLOCK), < 45 (the held-across wake chain). Declared atprocess/process-and-task-management.md(Task struct); contract at Section 20.4. | Observability | | 45 |PI_LOCK| Priority inheritance chain — protectsHeldMutexeslist for PI chain walking. BelowRQ_LOCK(50)so thattry_to_wake_up()can acquirePI_LOCKthenRQ_LOCKwithout inversion. | Core | | 46 |OOM_NOTIFY_LOCK|SpinLock— the/dev/oomnotification ring (OOM_NOTIFY_RING: Lock<OomNotifyRing, 46>, Section 4.5). LEAF:oom_notify()mutates the ring under it and releases BEFORE wakingOOM_NOTIFY_WAITERS(two-phase — the wake chain takes the waitqueue lock andRQ_LOCK(50), never nested under this lock);/dev/oomread()/poll()handlers acquire nothing while holding it. Declared atmemory/oom-killer.md. | Core | | 47 |PID_MAP_LOCK|SpinLock— per-PidNamespacePID Idr lock (PidNamespace::pid_map.lock). Distinct fromPID_TABLE_LOCK(22)(the global TaskId-keyed task table). PID_MAP_LOCK is per-PID-namespace and protects namespace-local pid_t alloc/free within a single namespace; it ALSO serializes mutations of the same namespace'spgids/sidsgroup/session registries and the pid-number pin/unpin/sentinel-free compound ops (registry reads stay RCU/lock-free) — Section 17.1. Acquired during fork's per-namespace PID allocation traversal. BelowRQ_LOCK(50)because PID alloc precedes scheduler enqueue. Declared atcontainers/namespace-architecture.md(PidNamespace). | Container | | 50 |RQ_LOCK| Scheduler run queue | Core | | 52 |RCU_NODE_LOCK|SpinLock<RcuNodeInner, 52>— per-RcuNodequiescent-state lock (RcuNode.lock, this file). Acquired during QS propagation (rcu_report_qs_leaf()/rcu_report_qs_up()) and GP-kthread tree init/FQS. Never nested with another RcuNode lock (leaf and each ancestor acquired sequentially, each released before the next) and never acquired with RQ_LOCK(50) held —rcu_report_qs_leaf()'s documented precondition; its call sites (timer_tick_handler()step 3,finish_task_switch()step 1a after the rq-lock release, GP kthread) all hold no runqueue lock. The root-clear wake (scheduler::unblock(gp_kthread)→ RQ_LOCK(50)) happens strictly AFTER all RcuNode locks are released (two-phase discipline), so the apparent 52→50 descent never occurs as a nested acquisition. | Core | | 53 |HRTIMER_BASE|SpinLock— per-CPU hrtimer expiry-tree lock (timekeeping-and-clock-management.md, Timer Infrastructure). Protects the per-CPU hrtimer RB tree and the next-expiry hardware programming. ABOVERQ_LOCK(50)because timers are armed while holding a runqueue lock (start_dl_replenishment_timer()under RQ_LOCK indl_task_tick(); CBSreplenish_timerre-arm;EevdfRunQueue.bandwidth_timer) — a legal 50→53 ascent. Expiry callbacks run with this lock RELEASED (the expiry handler unlinks the timer under the lock, drops it, then invokes the callback — Linux__run_hrtimerdiscipline), so a callback acquiring RQ_LOCK(50) (dl_replenish_timer_fn, CBS replenish) is a fresh acquisition, not a 53→50 descent. | Core | | 55 |BINFMT_MISC_LOCK|RwLock— global binfmt_misc registration table lock. Read-held during exec'sload_binary()(binfmt format probing); write-held during/proc/sys/fs/binfmt_misc/registeradmin ops. BelowPERF_EVENTS_LOCK(65)because exec format probing happens before perf event setup. Declared atprocess/process-and-task-management.md(binfmt_misc registration). | Core | | 65 |PERF_EVENTS_LOCK|SpinLock— per-Task::perf_eventslock — serializes per-task perf event list mutation. Acquired during exec for inheritable-event copy and in the scheduler's perf-event switch-in/out hooks. BelowIO_URING_TCTX_LOCK(68)because perf events are set up before io_uring TCTX during exec. Declared atobservability/perf-events.md. | Observability | | 68 |IO_URING_TCTX_LOCK|SpinLock— per-Task::io_uring_tctxlock — serializes io_uring task context mutation across exec (inherited/reset rules) and ring close. BelowCAP_TABLE_LOCK(70)because io_uring TCTX is reinit'd before capability-table apply during exec. Declared atsysapi/io-uring-subsystem.md. | Storage | | 70 |CAP_TABLE_LOCK| Capability table write lock — serializes capability slot insert/remove/revoke across a task's capability space. Revocation invariant: duringcap_revoke(), the per-CapEntrychildrenspinlock is held for at most O(256) iterations per workqueue item. No recursive spinlock acquisition — each delegation tree level is processed by a separate workqueue item (Section 9.1). | Core | | 80 |I_RWSEM|RwLock— per-inode read-write semaphore — serializes file read/write/truncate/fallocate. This is the most contended filesystem lock. Ordering:I_RWSEM<VM_LOCKin the truncate path (truncatetakesI_RWSEM(write)thenVM_LOCK(write)to unmap pages). The page fault path does NOT acquireI_RWSEM— truncation-fault coordination is provided byInvalidateSeq(Section 4.8), a lockless seqcount. The fault path's lock chain is strictly ascending:VMA_LOCK(105, read)->PAGE_LOCK(180)->PTL(185). No lock ordering exception is needed. Truncate holdsI_RWSEM(write)and incrementsInvalidateSeqbefore mutating the page cache; faults detect the mutation via two atomic seq loads and retry. | Fs | | 100 |VM_LOCK| VMA tree write lock (mmap_lock equivalent) — protects the per-process virtual memory area tree during mmap/munmap/fault handling | Core | | 105 |VMA_LOCK|RwLock<()>— per-VMA lock (Vma::vm_lock).mmap_locknests outsidevm_lock. Page fault fast path acquires onlyvm_lock.read(). VMA modification acquiresmmap_lock.write()thenvm_lock.write(). A thread holdingvm_lock.read()must never acquiremmap_lockin any mode. See Section 4.8. | Core | | 106 |I_MMAP_LOCK|RwLock<IntervalTree<VmaRef>>— per-AddressSpacefile-VMA interval tree (AddressSpace::i_mmap), reverse mapping for file pages. Level derivation (must sit above 105 and below 108): chain 1 (mmap):VM_LOCK(100, w)→I_MMAP_LOCK(w)— establish_mapping Step 8 registration andvma_merge()remove/reinsert (single critical section). Chain 2 (munmap):VM_LOCK(100, w)→VMA_LOCK(105, w)→I_MMAP_LOCK(w)— destroy_mapping removes the entry with the per-VMA write lock held, forcing the level ABOVE 105 (a level in the 100-105 gap would invert this chain). Chain 3 (truncate):I_RWSEM(80, w)→I_MMAP_LOCK(r)→PTL(185)— the truncation reverse-map walk visits mapping VMAs and zaps their PTEs. Chain 4 (rmap walk, reclaim/migration):I_MMAP_LOCK(r)→PAGE_LOCK(180)/PTL(185). BelowMAPLE_TREE_WRITE_LOCK(108)sovma_merge()MAY update the maple tree inside the i_mmap critical section if needed (106 → 108 ascending). Interval-tree nodes are intrusive in the VMA entries — no allocation under the lock. Declared atvfs/virtual-filesystem-layer.md(AddressSpace::i_mmap). | Fs | | 108 |MAPLE_TREE_WRITE_LOCK|SpinLock— internal maple tree write lock for the per-process VMA tree (mm->vma_tree). Acquired during VMA insert/remove for tree-structure mutation. Held briefly: a single allocation-aware tree store. Outer requirement:VM_LOCK(100, write)must be held — only the writer of mmap_lock may modify the tree.VMA_LOCK(105)may also be held at the per-VMA scope. Leaf — never acquires further locks. Declared atmemory/virtual-memory-manager.md(MapleTree internal). | Core | | 110 |ADDR_SPACE_LOCK| Address space page table lock — serializes page table modifications (map/unmap/protect) within a single address space; acquired underVM_LOCK| Core | | 120 |BUDDY_LOCK| Per-NUMA buddy allocator | Core | | 125 |SLAB_DEPOT_LOCK|SpinLock— per-SlabCachemagazine depot lock — serializes depot full/empty magazine exchange on the magazine-miss slow path. Ordering:BUDDY_LOCK(120) < SLAB_DEPOT_LOCK(125) < SLAB_LOCK(130)— a conservative reservation with NO actual nesting in current code: the free slow path copies the spare magazine's objects into a stack-local drain buffer UNDER the depot lock, DROPS the depot lock, and only then callsdrain_objects_to_slabs()(which takesSLAB_LOCK). The reserved ordering exists so future code that does nest acquires in the level-legal direction. See Section 4.3 for the complete slab lock ordering chain. | Core | | 130 |SLAB_LOCK|SpinLock— per-NUMA slab partial list (node_partial) — serializes slab insertion/removal from the per-node partial list. Never held together withSLAB_DEPOT_LOCKin current code: the drain path acquires it only AFTER the depot lock is released (drain-buffer design — see the 125 row; the 125 < 130 ordering is a conservative reservation, not observed nesting). The alloc slow path drops this lock before callingslab_grow()to avoid self-deadlock (slab_grow()re-acquires it internally). | Core | | 135 |TRACKED_REGISTRY_LOCK|SpinLock— per-TypeRegistrywriter lock guardingliveinstance-pointer set andfreedepot. Hot allocation path uses per-CPU magazines and does NOT acquire this lock. Slow path (magazine miss, free depot drain, registration, deregistration) acquires this. Migration walk (live evolution Phase B) acquires write mode for the entire walk under stop-the-world. Above all caller-held locks during allocation —VM_LOCK(100),VMA_LOCK(105),MAPLE_TREE_WRITE_LOCK(108),ADDR_SPACE_LOCK(110),BUDDY_LOCK(120),SLAB_LOCK(130). Leaf —alloc_tracked()does not call into any further ordered lock (per-type sub-regions are carved at registration time; no runtime growth). Declared atdevice-classes/live-kernel-evolution.md(TypeRegistrystruct, see Section 13.18). | Core | | 136 |KABI_ENTRIES_LOCK|Lock<XArray<Arc<RegistryEntry>>, 136>—GlobalServiceRegistry.entriesWRITE lock (publish/unpublish/set_migrating; reads are RCU viawith_provider()— never under this lock). LEAF: all publish/wake flows are two-phase (collect → drop → act), so nothing ordered is acquired under it, andpublish()allocates its entry and counter BETWEEN its two lock passes (no allocation under this level).GlobalServiceRegistry.deferredis NOT on this axis: it is a SLEEPING leaf mutex (its per-service wakeupVecs grow on the heap — theKABI_DEPENDENT_TABLEprecedent), taken by callers holding no SpinLock, never co-held withentries. Declared atkabi/kabi-domain-runtime.md(GlobalServiceRegistry). Replaces an earlier file-local hierarchy that asserted levels 40/50/60/70 — colliding withSIGLOCK/FDTABLE_LOCK(40),RQ_LOCK(50), andCAP_TABLE_LOCK(70). | Driver | | 137 |KABI_MODULES_LOCK|Lock<ArrayVec<ModuleDescriptor, MAX_MODULES_PER_DOMAIN>, 137>— per-DomainServicemodule table (DomainService.modules). Warm module-lifecycle lock (register, resolve_all phases 1/3, announce_ready phase 1, rebind phases 1/3, migration snapshot). May nestKABI_HANDLES_LOCK(138)under it; NOTHING else ordered is acquired under it — registry publication, deferred-module wakes (→try_to_wake_up()→RQ_LOCK(50)), and sleeping ring setup all run two-phase AFTER this lock is dropped (holding it across a wake would be a 137→50 descent, and holding it acrosswake_deferred_modules()re-enteringresolve_all()was a self-deadlock). Declared atkabi/kabi-domain-runtime.md(DomainService). | Driver | | 138 |KABI_HANDLES_LOCK|Lock<ArrayVec<Option<KabiHandleOpaque>, MAX_DEPS_PER_MODULE>, 138>— per-ModuleDescriptorhandle-slot array (ModuleDescriptor.handles, the per-domain binding record). Acquired underKABI_MODULES_LOCK(137); leaf below (nothing ordered under it, no sleeping under it — rebind pre-creates rings lock-free in its phase 2 and swaps in phase 3). ALL slot access (init retrieval, rebind, grant invalidation) is under this lock: there are NO lock-free readers (per-call dispatch uses owned handle copies protected by the provider generation check). Declared atkabi/kabi-domain-runtime.md(ModuleDescriptor). | Driver | | 139 |KABI_IRQ_RINGS_LOCK|LockIrqSafe<ArrayVec<IrqRingDescriptor, MAX_IRQ_RINGS_PER_DOMAIN>, 139>— per-DomainServiceIRQ-ring table (DomainService.irq_rings). IRQ-safe LEAF: acquired in process context at IRQ-ring registration/retire (bounded descriptor append/extract), handle issuance (bounded lookup + grant record), crash disconnect (notify_crash()state stores), and the crash trampoline'sconsumer_exitedwalk; IRQ-safety retained for the exception-adjacent crash paths. The timer-expiry hot path does NOT acquire it (module-keyedTIMER_ROUTE_TABLE, RCU — an earlier revision took this lock IRQ-off from the timer softirq on EVERY cross-domain expiry). Bounded hold (descriptor append/extract/lookup / state store), nothing ordered acquired under it — dispatch-table writes run in a separate phase underIRQ_DISPATCH_LOCK(174). Declared atkabi/kabi-domain-runtime.md(DomainService). | Driver | | 140 |WORKQUEUE_LOCK| Per-workqueue drain / flush serialization | Core | | 150 |FS_SB_LOCK| Per-superblock lock — protects superblock-level state (mount flags, fs-wide counters, journal commit); acquired before per-inode locks during mount/unmount and fs-wide operations | Fs | | 160 |INODE_LOCK| Per-inode metadata SpinLock (VFS) — protects inode attributes (size, timestamps, link count). Distinct fromI_RWSEMwhich serializes I/O operations.INODE_LOCKis a SpinLock for quick metadata updates;I_RWSEMis an RwLock for I/O serialization. | Fs | | 170 |WRITEBACK_LOCK| RETIRED / reserved — no SpinLock user. Formerly a per-inode writeback-stateSpinLockthat "serialized writeback initiation". Writeback serialization is now owned by the per-inodeI_WRITEBACKstate bit — a CAS onInode.i_stateinwriteback_single_inode()is the SOLE per-inode writeback-owner signal (Section 14.1; Section 4.6). The VFSAddressSpace::writeback_lockis a sleepingMutex<WritebackState>cursor container taken only by theI_WRITEBACKCAS winner (uncontended by invariant) — it is on the sleeping axis, NOT this SpinLock ordering table, and it excludes nothing. The level number is retained (not reassigned) to avoid renumbering the rows below. | Fs | | 171 |BLK_QUIESCE_LOCK|SpinLockIrqSafe— per-BlockDevicequiesce_queuelock — serializes the bounded staging deque holding bios during driver quiescence (live evolution QUEUE_FLAG_QUIESCING), tier re-binding, and crash recovery. Acquired bybio_submit_raw()step 0, the Hold-classified dispatch error arms (any context — hence IRQ-safelock_irqsave), andblk_resume_quiesced()(process context). Leaf — never nested withBLK_REQUEUE_LOCK(172)orBLK_INFLIGHT_LOCK(173)(dispatch helpers release each before touching another). Declared inBlockDeviceinstorage/block-io-and-volume-management.mdasquiesce_queue: Lock<BoundedDeque<RequeueEntry, MAX_QUIESCE_QUEUE_DEPTH>, 171>. | Storage | | 172 |BLK_REQUEUE_LOCK|SpinLockIrqSafe— per-BlockDevicerequeue_listlock — serializes requeue operations on a block device's bounded requeue deque (4096 entries). Acquired bybio_submit_raw()on the direct-dispatch AGAIN path and byblk_kick_requeue()from completion IRQ context — seestorage/block-io-and-volume-management.mdbio_submit_raw()andblk_kick_requeue(). Always uses IRQ-safe acquisition (lock_irqsave) — completion IRQ may fire while submitter holds the lock, so the lock disables local IRQs while held.blk_kick_requeue()drains entries into a localArrayVec<RequeueEntry, 64>scratch buffer, releases the lock, processes each entry (re-acquiring only if AGAIN re-pushes). BelowPAGE_LOCK(180)because page locks are released before bio submission. Declared inBlockDevicestruct instorage/block-io-and-volume-management.mdaspub requeue_list: Lock<BoundedDeque<RequeueEntry, 4096>, 172>. | Storage | | 173 |BLK_INFLIGHT_LOCK|SpinLockIrqSafe— per-shard lock of the per-BlockDevicein-flight bio table (BlkInflightTable.shards[i], XArray keyed bybio.generation; shard count = min(nr_online_cpus, 64) pow2, discovered at registration). Acquired byblk_dispatch_bio()enrollment (process/softirq/completion-IRQ context) and bybio_complete()/bio_release_quarantined()unenrollment (IRQ context — hence IRQ-safe). Crash recovery'scapture_all()walks all shards (cold). Leaf — no lock acquired under it; never held together with 171/172. Hot-path cost (~30-50 cy/I/O, sharded-uncontended) is compensated by the deletedIoRequest.sglper-request ArrayVec copy — seestorage/block-io-and-volume-management.md§Per-Device In-Flight Bio Table. Declared there. | Storage | | 174 |IRQ_DISPATCH_LOCK|Lock<(), 174>— global serializer for ALL mutation of the IRQ dispatch table (IRQ_DISPATCH_TABLE) and theIrqOwner.next_sharedchains:register_irq_ring()vector wiring (head-replace inserts) andunregister_irq_ring()unlink/mask surgery. Process context ONLY — the IRQ hot path (generic_irq_handler) reads the table and walks the chains via RCU, lockless. May acquire under it:XA_LOCK(178)(the dispatch-table XArray writes it covers) — strictly ascending; nothing else ordered. NEVER nested withKABI_IRQ_RINGS_LOCK(139): registration/retire take 139 in a SEPARATE phase (139's leaf/IRQ-off row forbids ordered acquisition under it — this lock exists precisely so dispatch-table writes happen outside 139). Declared atkabi/kabi-domain-runtime.md(dispatch-table statics). | Driver | | 175 |DSM_FETCH_COMPLETION|WaitQueue— DSM remote page fetch completion. A DSM driver fault worker (requester side) blocks here while waiting for RDMA data arrival + InvAck collection. The waiter holds NO ordered lock while parked: Core releasedVMA_LOCK(105, read)before blocking the faulting task on the FaultRequest wait entry (Section 4.15 step 5a), and the DSM fault worker acquires no Core lock before waiting. Core's post-wake chain isVMA_LOCK(105, read)→PAGE_LOCK(180)→PTL(185)(re-acquired fresh with revalidation). TASK_KILLABLE wait (only SIGKILL interrupts). See Section 6.12. | DSM | | 176 |DSM_DIR_ENTRY|Lock<(), 176>—DsmDirEntry::lock, home-node directory entry lock. Serializes multi-field directory updates (state + owner + sharers); the seqlock writer store onDsmDirectoryEntry::sequence(a seqcount, not an ordered lock; thesequencebump is a plain load+store, NOT an atomic RMW/CAS) requires this lock held. Acquired by home-side DSM message handlers that enter the DSM path holding no lower ordered lock. Never held across a network round-trip or any blocking call (Section 6.6 invariant 1). May acquire under it:DSM_ENTRY_WAITQ(177)(wakeups) and, on the home==owner self-invalidation path,PAGE_LOCK(180)/PTL(185)— strictly ascending. Declared atdsm/dsm-coherence-protocol-moesi.md(DsmDirEntry). | DSM | | 177 |DSM_ENTRY_WAITQ|Lock<(), 177>—DsmDirectoryEntry::entry_lock, per-entry wait-queue lock for blocking on the transientInvalidating→Modified(owner=requester)transition (a local, bounded home-side operation under the requester-counted invalidation model, Section 6.5). Protects waiter enrollment and broadcast wakeup on the entry's wait queue (seqlocks cannot be held acrossschedule()). Acquired afterDSM_DIR_ENTRY(176)on the wakeup path (seqlock writer THEN entry_lock — encoded by 176 < 177), or with no ordered lock held on the waiter path. Leaf on the blocking path. Declared atdsm/dsm-page-ownership-model.md(DsmDirectoryEntry). | DSM | | 178 |XA_LOCK|SpinLock— generic XArray-internal writer lock (xa_lock): the shared ordering level for the internalxa_lockof every integer-keyedXArrayinstance whose structural insert/erase/replace is NOT the page-cachepage_cache.pagestree — the VFSinode_cache, the(sb_dev, ino)-keyedDIRTY_INTENT_INDEX, theIRQ_DISPATCH_TABLE, and the swap cache. Read access usesrcu_read_lock()instead — RCU readers don't acquire this lock. Acquired with IRQs disabled (xa_lock_irq) when called from completion/IRQ contexts. Ascending chains that fix this level:INODE_LOCK(160)→XA_LOCK(178)→ICACHE_LRU_LOCK(179)(inode_cache),IRQ_DISPATCH_LOCK(174)→XA_LOCK(178)(IRQ_DISPATCH_TABLE), andXA_LOCK(178)→DIRTY_INTENT_LOCK(182)(DIRTY_INTENT_INDEX insert — the list is published underxa_lock, which is dropped BEFOREintent_lockis taken; sequential, not nested). The page-cachepage_cache.pageswriter lock is a separate level —XA_LOCK(181)— becausepage_cache.pagesmust nest UNDERPAGE_LOCK(180), a constraint this generic level cannot carry (inode_cache requires178 < ICACHE_LRU_LOCK(179); one shared level cannot be both> 180and< 179). Declared per-instance at the citing sections (vfs/virtual-filesystem-layer.mdinode_cache +DIRTY_INTENT_INDEX,kabi/kabi-domain-runtime.mdIRQ_DISPATCH_TABLE,memory/swap-subsystem.mdswap cache). | Fs | | 179 |ICACHE_LRU_LOCK|SpinLock<IntrusiveList<Inode>, 179>— global inode-cache LRU list lock. The two-phase inode shrinker (inode_cache_evict()) isolates a batch of up toINODE_EVICT_BATCH(32) candidates from the LRU head under THIS lock ALONE (phase 1, bounded hold), drops it, then revalidates and disposes each candidate underi_lockwith NOlru_lockheld (phase 2). Inbound ascending chains:INODE_LOCK(160)→179(revival unlink viainode_lru_remove_if_linked(), and the dirty/I_WRITEBACKrotate-to-tail viainode_lru_add_tail()) andXA_LOCK(178)→179. LEAF — nothing ordered is acquired under it; the shrinker NEVER takesi_lockwhile holdinglru_lock, so the compile-time strictly-ascending levels replace thelru_lock → i_lockinversion Linux resolves withspin_trylock(&inode->i_lock)ininode_lru_isolate(). Declared atvfs/virtual-filesystem-layer.md(inode LRU struct +ICACHE_LRU_LOCKconst). | Fs | | 180 |PAGE_LOCK| Per-page lock (page cache) — acquired afterINODE_LOCKduring writeback submission and per-inode dirty page iteration. The sync path nesting is:FS_SB_LOCK(150)→INODE_LOCK(160)→PAGE_LOCK(180)(writeback ownership is theI_WRITEBACKstate bit, not an ordered lock — retiredWRITEBACK_LOCK(170)no longer sits in this chain). In the page fault path:VMA_LOCK(105, read)→PAGE_LOCK(180)→PTL(185)— strictly ascending, zero exceptions. In the DSM fault path the network wait happens with NO ordered lock held (VMA_LOCKreleased before blocking, Section 4.15 step 5a); the post-wake install chain is the sameVMA_LOCK(105, read)→PAGE_LOCK(180)→PTL(185). | Fs | | 181 |XA_LOCK|SpinLock— per-AddressSpacepage_cache.pagesXArray writer lock (xa_lock), the page-cache page tree. Acquired by Core page-cache services for atomic insert/erase/replace on the page cache XArray. Read access usesrcu_read_lock()instead — RCU readers don't acquire this lock. Always acquired with IRQs disabled (xa_lock_irq) when called from completion contexts. VFS drivers do NOT hold this lock directly — page cache metadata is not mapped into VFS Tier-1 domains; all mutations go through Core viakabi_call!. This invariant closes the FLOW-04-08 deadlock class (crashed VFS driver cannot leavexa_lockheld). Level derivation: nests UNDERPAGE_LOCK(180)— the page is LOCKED first, THEN the structural insert/erase takesxa_lockunder it (memory/page-cache.md— page held since insertion guarantees the slot is present to erase);180 → 181ascending, belowDIRTY_INTENT_LOCK(182)andPTL(185). SPLIT from the genericXA_LOCK(178)level (which stays for inode_cache /DIRTY_INTENT_INDEX/IRQ_DISPATCH_TABLE/ swap cache):page_cache.pagesnests ABOVEPAGE_LOCK(180)while those generic instances must stay BELOWICACHE_LRU_LOCK(179), so a single shared level is impossible. See Section 14.2 for the rationale and Section 4.4 for the protocol. Declared atmemory/page-cache.md(AddressSpace struct). | Fs | | 182 |DIRTY_INTENT_LOCK|SpinLock<(), 182>— per-DirtyIntentListleaf spinlock guarding one inode's dirty-extent intent list (the crash-durability reservation records that let Core flush committed dirty data directly to the block device after a cross-domain filesystem crash; Section 14.1). Acquired by the Core dirty-extent write path (reserve/reserve_and_commit/flush_extent_complete) and by the crash-recovery bypass flush iteration. Leaf — nothing ordered is acquired under it (list nodes are intrusive; no allocation under the lock). Level derivation: aboveXA_LOCK(178)because inserting a new per-inode list into the(sb_dev, ino)-keyedDIRTY_INTENT_INDEXpublishes the list underXA_LOCK, DROPSXA_LOCK, and only THEN takes the freshly-inserted list'sintent_lock(178 → 182 sequential, not nested — matching the row-178 gloss and theSbDirtyIntents::by_inoinsert site invfs/virtual-filesystem-layer.md); abovePAGE_LOCK(180)and belowPTL(185). Replaces the earlier (incorrect) design that mandatedi_rwsemfor intent-list integrity — the overflow drain-and-retry now runs on the Core write path viawriteback_single_inode(), never under the caller's exclusivei_rwsem(writeback takes noi_rwsem; it uses anI_WRITEBACKCAS). Declared atvfs/virtual-filesystem-layer.md(DirtyIntentListstruct, with theDIRTY_INTENT_LOCKconst). | Fs | | 185 |PTL|SpinLock— per-page-table-page lock (Page Table Lock) — serializes translation-entry modifications within a single page-table page, at every table level: leaf PTE pages viapte_lockptr(), huge-leaf entry pages viahuge_pte_lockptr()(Section 4.8). Acquired by the fault handler (file fault, COW fault) afterPAGE_LOCKand before writing the PTE. Also acquired by truncation's reverse-map PTE-zap walk, and bytry_to_unmap()during page reclaim. Same-level multi-acquisition is sanctioned in exactly two shapes: (a) address-ordered pairs at the SAME table level vialock_pair_ordered()(mremap-class transfers, Section 4.15); (b) top-down parent-table-page-before-child-table-page for the same VA (any future split/collapse path). Neither shape can form a cycle: (a) is a total order by address within one level, (b) is a strict parent-before-child hierarchy, and a thread holding a child-level lock never requests its parent. Leaf-level lock: no further locks acquired under PTL. The fault path nesting is:VMA_LOCK(105, read)→PAGE_LOCK(180)→PTL(185). | Core | | 190 |DENTRY_LOCK| Dentry cache per-entry lock — protects dentry reference counts, parent/child linkage, and name hash chain membership | Fs | | 200 |MOUNT_LOCK|SpinLock-level global mount-table lock — short, non-sleeping mount-table bookkeeping. DISTINCT from the per-namespace SLEEPINGmount_lock(aMutex, NOT aLock<T, LEVEL>) thatcopy_tree/mount_filesystem/pivot_roothold across mount-tree mutation and RCU publication (Section 14.6; sleeping-axis, not in this SpinLock table). The two are different locks on different axes and must not be conflated — a path holding the sleeping per-namespacemount_lockmay still take this SpinLock for a bounded table update, never the reverse. | Fs | | 210 |HIERARCHY_LOCK|RwLock<()>— cgroup hierarchy lock — serializes cgroup creation and deletion (write-lock) across the cgroup tree. Task migration holds a read-lock (concurrent migrations allowed). Acquired before per-cgroup subsystem locks. Never held across filesystem operations (noGfpFlags::KERNEL_NOFSconcern — cgroup ops do not allocate from filesystem-backed paths). Sleeping lock — allocation is legal while held: this is aRwLock(sleeping), so non-FS slab allocations (controller state viachild_controller_alloc, theCgroupstruct, the cgroupfs inode) run legally under a held write; the spin-classCgroup.children_lock(a leafSpinLock, not on this numeric table) nests under it ONLY for the allocation-free O(1) RCU publish, and a SpinLock is never held across an allocation —cgroup_mkdir/cgroup_rmdir/write_subtree_controlall prepare (allocate under the sleeping write) then commit (publish under the leaf SpinLock). Cgroup task migration protocol: The two-phase release is MANDATORY — failure to releaseHIERARCHY_LOCKbefore acquiringRQ_LOCK(50)is a deadlock (level 210 > 50, compile-time rejected by theLock<T, LEVEL>mechanism). Protocol: (1) record migration intent and update cgroup membership underHIERARCHY_LOCK(read), (2) releaseHIERARCHY_LOCK, (3) acquireRQ_LOCKto perform the actual runqueue dequeue/enqueue andGroupEntitytransfer. The task'scgroup_migration_state: AtomicU8(CgroupMigrationStateenum: None/Migrating/Complete) prevents concurrent migrations of the same task between phases. See Section 17.2 for the full migration protocol specification. | Container | | 215 |CGROUP_TASKS_LOCK|RwLock<XArray<()>>— per-cgroup task-set lock (Cgroup.tasks, Section 17.2). Write-held by migration step 10 (membership move),commit_fork()(child insert), anddetach_exiting_task()(member remove); read-held bycgroup.procs/cgroup.threadsreaders and the freezer walk. Same-level tiebreaker: when two instances are held simultaneously (migration step 10's source+target pair), acquire in ascendingCgroupIdorder (parallel toRQ_LOCK's CPU-ID convention); release in reverse. Placed betweenHIERARCHY_LOCK(210)(migration holds 210-read when acquiring this) andEVM_LOCK(220). Fork/exit acquisitions are legal: fork holds only the sleeping 3a rwsem;detach_exiting_taskholds no spin-class lock at Step 8a. | Container | | 220 |EVM_LOCK|RwLock— per-superblock EVM (Extended Verification Module) lock — serializes xattr integrity verification and HMAC recomputation. Read-held during file open (IMA check); write-held during xattr update. Must be afterINODE_LOCK(xattr ops hold inode lock). | Security | | 230 |SOCK_LOCK| Per-socket protocol lock.TcpCb.lockIS this lock (one lock, one name — the per-socket lock protectingTcpMutableState; networking files citeSOCK_LOCK(230)). TX enqueue path chainsSOCK_LOCK(230) → QDISC_LOCK(235). Released before cross-domain callbacks (sk_data_readyetc. — see Section 16.2). | Net | | 232 |ZCOPY_TABLE_LOCK|SpinLockIrqSafe— write-side lock of the per-net-namespace MSG_ZEROCOPY in-flight table (ZCOPY_INFLIGHT;ZcopyTxRecordinsert/remove — lookups are RCU reads, never under this lock). Acquired bysendmsg()step 2c (Tier 0 syscall context, pin time), by the Tier 0 TX completion consumer's record retirement (completion context — hence IRQ-safelock_irqsave), and by the NIC-crash reclamation sweep (cold). Leaf — no lock acquired under it; per the two-phase discipline the completion sequence does its RCU-read work (unpin, error-queue post, WritableSpace wake) FIRST and takes this lock only for the final record removal. Never co-held withSOCK_LOCK(230)/QDISC_LOCK(235)— those live inside umka-net's domain and cannot even be attempted from the Tier 0 contexts that take this lock; the 230 < 232 < 235 placement records Net-group membership, not an observed nesting. Declared atnetworking/socket-operation-dispatch.md§MSG_ZEROCOPY In-Flight Table. | Net | | 235 |QDISC_LOCK|SpinLock— per-qdisc queue lock — serializes enqueue/dequeue/drain on one queueing discipline instance. Acquired afterSOCK_LOCK(230)on the TX enqueue path; theqdisc_run()drain loop acquires it with no socket lock held. Never held across driverkabi_call!— drain dequeues under the lock, releases, then submits. Declared atnetworking/traffic-control-and-queue-disciplines.md(Qdisc). | Net | | 240 |CONNTRACK_BUCKET| Conntrack bucket | Net | | 250 |DEV_REG_LOCK| Device registry | Driver | | 260 |VTABLE_LOCK| KABI vtable swap | Driver | | 270 |EFI_RUNTIME_LOCK| UEFI runtime call serialization — leaf, IRQs disabled. Cold-path only (variable reads, time set, reboot). Never nested. | Boot |
Sleeping Mutex Ordering (separate from SpinLock levels — sleeping Mutexes
use Mutex<T>, not Lock<T, LEVEL>, and are acquired only in process context):
| Order | Mutex Name | Subsystem | Constraint |
|---|---|---|---|
| 1 (outermost) | EVOLUTION_MUTEX |
Live evolution | Must be outermost — no lock may be held when acquiring. Held for the entire Phase A/A'/B/C evolution sequence. |
| 1a | RECOVERY_MUTEX |
Crash recovery | Per-domain DomainDescriptor.recovery_mutex (kabi/kabi-domain-runtime.md). Held by the crash recovery worker (or a one-shot recovery kthread) for the entire process-context recovery: the FMA reload preamble plus Steps 3-9 of Section 11.9. Acquired with NO other sleeping mutex held (the worker acquires it directly after dequeuing a request). Ordered between EVOLUTION_MUTEX and CPU_HOTPLUG_LOCK because the recovery path, while holding it, may legitimately acquire everything ordered after: CPU_HOTPLUG_LOCK (read — stable CPU set for the ejection sweep and consumer-thread affinity at Step 8), KABI_REGISTRY_MUTEX (Step 8 registry re-publication during reload), and OOM_LOCK (transitively — Step 8 GFP_KERNEL allocation when the recovery pool is exhausted). It must NEVER acquire EVOLUTION_MUTEX (crash-recovery reload is a direct MBS reload, not a Phase A/A'/B/C evolution). Conversely, an evolution abort that falls back to crash recovery does so by PUSHING a request and (optionally) waiting on a ReloadCompletion — never by acquiring this mutex while holding EVOLUTION_MUTEX directly; the 1 < 1a ordering nevertheless makes even a direct acquisition legal. Step 9 recovery callbacks run under this mutex — see the callback constraint list in Section 11.9. Distinct per-domain instances: recoveries of DIFFERENT domains may hold their respective mutexes concurrently (parallel recovery); same-order multi-hold is permitted here because no code path acquires two domains' recovery mutexes simultaneously (each recovery execution owns exactly one domain). |
| 2 | CPU_HOTPLUG_LOCK |
CPU hotplug | RwSem — global CPU hotplug coordination lock. Read-held by code paths that must observe a stable CPU set (per-CPU iteration with strict snapshot semantics, scheduler invariant checks). Write-held by CPU online/offline operations, by the standalone stop_the_world() wrapper (non-evolution callers), and by evolution orchestration from Phase A step 5b — acquired BEFORE KABI_REGISTRY_MUTEX(3), keeping the order 1 → 2 → 3 clean; Phase B's stop_the_world_locked() BORROWS the pre-held guard instead of acquiring (acquiring at that point, with order 3 already held, would invert 3 → 2). After Phase B the orchestration downgrades write → read (CpuHotplugWriteGuard::downgrade(), atomic — readers unblocked, hotplug writers still excluded) and releases only after watchdog_disarm() (invariant LIV-2a — makes the sleeping-lock-free stop_the_world_nmi() watchdog revert sound). The cpu_hotplug_lock_write() / cpu_hotplug_lock_read() API and the CpuHotplugWriteGuard / CpuHotplugReadGuard types are defined at Section 13.18. |
| 3 | KABI_REGISTRY_MUTEX |
KABI registry | Acquired AFTER EVOLUTION_MUTEX (and AFTER CPU_HOTPLUG_LOCK if both are needed) for write-side updates to the global KABI service registry during component swap. |
| 3a | THREADGROUP_RWSEM |
Process / cgroup | Per-Process SLEEPING RwLock<()> (Process::threadgroup_rwsem, process/process-and-task-management.md). Read-held by create_task() / create_task() step 1 through step 18b (released after commit_fork() — the child is fully linked into the group); write-held by the cgroup.procs write handler Step 1a through migration Step 14 (before the per-thread RQ phase; released strictly AFTER HIERARCHY_LOCK(210) — step 13 — because the rwsem release's waiter wake chain acquires waiters_lock and RQ_LOCK(50), which must not run under a held level-210 lock); read-held by single-thread cgroup.threads moves AND by detach_exiting_task() (exit_task Step 8a) for its whole charge-release sequence — the exit-vs-migration serialization. Moved OFF the SpinLock-leveled table (formerly level 205): the fork-side hold window sleeps (tracked/slab allocation, copy_page_tables, put_user) and nests ordered SpinLocks under it (PID_TABLE_LOCK(22), PROCESS_LOCK(25), SIGHAND_LOCK(30), PID_MAP_LOCK(47), TRACKED_REGISTRY_LOCK(135)) — legal for a sleeping lock, but a compile-rejected descending acquisition and an illegal sleep for any Lock<T, 205>. Ordered before OOM_LOCK(4): allocations inside the hold window may enter the OOM path; never acquired while holding any SpinLock or a later-ordered sleeping mutex. Distinct per-Process instances never co-held (fork/migration each operate on exactly one process' group). Better than Linux: Linux's cgroup_threadgroup_rwsem is global; UmkaOS scopes per-Process, eliminating the global scalability wall. See Section 17.2. |
| 3b | EXEC_CRED_BARRIER |
Process / ptrace | Per-Process SLEEPING RwLock<()> (Process::exec_cred_barrier, process/process-and-task-management.md). Serializes cross-process credential-derived observers against the target's own execve() credential/dumpability straddle. exec holds WRITE for one span from just before credential preparation (exec step 1c) to just after the dumpable downgrade (exec step 5e); observers hold READ (interruptible acquisition — a blocked observer returns -EINTR/-ERESTARTNOINTR) for the whole sample->decide->target-touching act of each credential-gated cross-process operation (ptrace_access_permitted() and its consumers: PTRACE_ATTACH/SEIZE admission, /proc/PID/{mem,stat,status,maps,smaps,environ,io} gated reads, process_vm_readv/writev, pidfd_getfd, prlimit64 pid≠0, perf_event_open pid>0). Concurrent readers of the same target run in parallel (RwSem). Constraints: interruptible READ acquisition; READ holds are bounded — never held while waiting for a tracee stop transition, never across an observer-side user copy beyond the target-touching act; ordered after THREADGROUP_RWSEM(3a), before OOM_LOCK(4) (allocations and user copies inside both hold windows may enter the OOM path); never acquired with any SpinLock held. No single task nests 3a and 3b: fork takes 3a only (auto-attach is barrier-exempt); exec takes 3b only (the collapse nests SpinLocks SIGNAL_STRUCT_LOCK(32)/PROCESS_LOCK(25)/SIGLOCK(40) under it — legal under a sleeping lock). Distinct per-Process instances never co-held. Deadlock-free across the exec WRITE-over-collapse hold because fatal signals permeate all ptrace stops (Section 20.4). This is NOT a credential-commit lock — the credential pointer is published by RCU copy-on-write with no lock (there is no cred_lock; Section 9.9). |
| 4 | OOM_LOCK |
OOM killer | Global OOM serialization Mutex (static OOM_LOCK: Mutex<()>). Ensures only one OOM kill sequence runs at a time — both global and per-cgroup OOM paths serialize on it INTERNALLY to invoke_oom_killer() (callers, including the memcg try_charge() path, never acquire it — invoke_oom_killer() is non-reentrant). Acquired after mmap_lock (read or write) — the allocation path may hold mmap_lock when triggering OOM. Neither OOM Mutex conflicts with EVOLUTION_MUTEX — they operate in non-overlapping code paths. See Section 4.5. |
| 5 | HIBERNATE_CANDIDATES |
OOM / hibernation | Global RwLock<Vec<Arc<MemCgroup>>> (Section 4.5) — registry of hibernation-eligible memcgs. Read-held by OOM resolution Step 0's candidate scan (under OOM_LOCK(4), hence ordered after it) and by nothing else; write-held by the cgroupfs memory.hibernate_priority handler and cgroup rmdir (both with no other sleeping mutex held — a legal order-5 acquisition). Never held across hibernate_cgroup() (dropped before the state-claim CAS); no lock of either axis is acquired under it. |
Sleeping Mutexes and SpinLocks do not directly nest (a sleeping Mutex
must not be acquired with a SpinLock held, because Mutex::lock() may sleep).
The Lock<T, LEVEL> compile-time system only covers SpinLocks. The sleeping
Mutex ordering is documented here for implementer reference.
SpinLock<T> vs Lock<T, LEVEL> policy: Use Lock<T, LEVEL> (ordered) for
ALL locks that participate in cross-subsystem locking paths — this is the default.
Bare SpinLock<T> (unordered) is permitted ONLY for:
1. Per-CPU locks held exclusively with IRQs disabled (e.g., RCU cb_segments),
where cross-CPU acquisition is structurally impossible.
2. Subsystem-internal leaf locks (see below) that are never held across subsystem
boundaries and never nest with any ordered lock.
Every bare SpinLock<T> must have a comment justifying why it does not use
Lock<T, LEVEL>. If in doubt, use Lock<T, LEVEL> — the compile-time check is
free at runtime.
This table will be extended as subsystems are implemented. The ordering invariant is: a thread holding a lock at level N may only acquire locks at levels > N. Cross-category acquisitions (e.g., Core lock then Fs lock) follow the same rule — the numeric level is the sole ordering criterion.
Subsystem-internal locks not listed above (e.g., per-NIC TX queue lock, per-pipe buffer lock, per-timer wheel bucket lock) are considered subsystem-internal: they are only acquired within a single subsystem and never held across subsystem boundaries. These do not need global level assignments — their ordering is enforced within the subsystem's own code. Only locks that participate in cross-subsystem acquisition chains require a level in this table.
Exempt locks:
RcuPerCpu::cb_segmentsspinlocks are explicitly exempt from this table. They are per-CPU, held with IRQs disabled, and callable fromrcu_call()under arbitrary lock contexts (includingDropimplementations). See thecb_segmentsdoc comment for the full rationale.Known cross-subsystem locks not yet assigned levels (to be added as subsystems are implemented):
TcpCb.lock(networking),run_lock/memslots_update_lock/io_bus_lock(KVM),wb.list_lock(writeback), IPC message queue locks, audit ring lock,f_pos_lock(file position), capability space locks. These locks may participate in cross-subsystem chains and will be assigned levels as their nesting relationships are fully specified during implementation.
Known limitation — total ordering only: The const-generic approach enforces a
total order on lock levels (level 0 < level 1 < level 2 < ...). This prevents
deadlocks caused by circular lock chains, but it cannot express a partial order
where two locks at the same conceptual level are safe to acquire together (because
they protect independent subsystems). In practice, this means some subsystems that
Linux allows to lock "in parallel" must be assigned adjacent but distinct levels in
UmkaOS, potentially over-constraining the lock graph. If this becomes a scalability
issue, the fallback is to introduce a lock_independent() API that takes two locks
at the same level with a static proof that their domains are disjoint (e.g., per-CPU
locks on different CPUs). For now, the total-order approach covers all known subsystem
interactions, and runtime debug lock-order checking (debug-mode only) validates that no partial-order
case is missed.
3.5 Locking Strategy¶
UmkaOS eliminates all "big kernel lock" patterns that plague Linux scalability:
| Linux Problem | UmkaOS Solution |
|---|---|
| RTNL global mutex | Per-table RCU + per-route fine-grained locks |
Linux dcache_lock contention |
Per-directory RCU + per-inode locks |
zone->lock on NUMA |
Per-CPU page lists + per-NUMA-node pools |
tasklist_lock |
RCU-protected process table + per-PID locks |
Linux files_lock (file table) |
Per-fdtable RCU + per-fd locks |
inode_hash_lock |
Per-bucket RCU-protected hash chains |
3.5.1 Locking Primitive Types¶
UmkaOS defines four concrete locking types used throughout all subsystems. These are the
actual implementations underlying Lock<T, LEVEL> (Section 3.4) and all per-subsystem
locks in the lock level table above.
3.5.1.1.1 RawSpinLock¶
A bare spinlock that disables preemption but does not save or restore IRQ state.
Modeled after Linux raw_spinlock_t. Suitable for: scheduler internals (runqueue locks),
interrupt-handler paths where IRQs are already disabled, and short critical sections
where the caller manages IRQ state explicitly. No RAII guard — the lock is manually
acquired and released, because a guard cannot enforce the caller's IRQ context.
/// Bare spinlock. Disables preemption on acquire; does NOT save/restore IRQ state.
///
/// # Safety
/// The caller is responsible for IRQ state. This type is correct only when:
/// - Called from an interrupt handler (IRQs already disabled), OR
/// - The caller has explicitly disabled IRQs (holds an `IrqDisabledGuard`), OR
/// - The critical section provably contains no IRQ-sensitive operations.
///
/// For general use, prefer `SpinLock<T>` which manages IRQ state automatically.
///
/// # Algorithm model: two generic bodies, boot-time selection
///
/// RawSpinLock has exactly TWO algorithm bodies — a queued spinlock
/// (qspinlock, MCS-derived) and a ticket lock — both written ONCE in
/// portable Rust atomics in `umka-nucleus/src/sync/spinlock.rs` over the shared
/// `state: AtomicU32` word. There is NO per-architecture lock implementation
/// and NO `arch::current::spinlock` seam: per-architecture involvement
/// shrinks to the existing atomic primitives (`core::sync::atomic` abstracts
/// LL/SC vs CAS) plus the `cpu_relax()` spin-wait hint (declared below).
/// Both algorithms provide starvation-free (FIFO) acquisition.
///
/// WHICH body an architecture runs — and which slow-path variant (native vs
/// paravirt) — is decided ONCE at boot by instruction patching, never by
/// runtime dispatch. See [Section 3.5](#locking-strategy--rawspinlock-algorithm-selection)
/// for the candidate table, the state-word encodings, and the normative
/// algorithm pseudocode.
///
/// # Context contract
///
/// - **Core image only.** Lock and unlock execute with the Core isolation
/// image live (`active_domain == 0`). In-domain Phase-3 driver code cannot
/// take ANY spinlock: `preempt_count` lives on the Core-keyed
/// `CpuLocalBlock` ([Section 3.2](#cpulocal-register-based-per-cpu-fast-path)), and
/// the in-domain access discipline reaches Core services — locking
/// included — only through the Core brackets
/// ([Section 12.8](12-kabi.md#kabi-domain-runtime)). Driver-private serialization inside an
/// open window comes from the ring consumer model, not from spinlocks.
/// - **NMI-forbidden.** `lock()`/`try_lock()` debug-assert `!in_nmi()`.
/// NMI-context serialization uses `NmiSpinlock` (below). This is what
/// bounds the queue-node nesting depth to three contexts (task, softirq,
/// hardirq).
/// - **Preemption placement.** `preempt_count` is incremented BEFORE the
/// first acquisition attempt (so a hold can never begin preemptible) and
/// decremented — with the `need_resched` check — AFTER the release store.
/// `try_lock()` failure decrements it again before returning.
///
/// # Hold Time Budget
///
/// RawSpinLock critical sections **must complete within 10 microseconds**.
/// This constraint feeds the 50us worst-case interrupt latency guarantee
/// ([Section 3.4](#cumulative-performance-budget)):
///
/// | Component | Budget |
/// |-----------|--------|
/// | Spinlock hold (worst case) | 10 us |
/// | Interrupt dispatch (vector lookup + context save) | 5 us |
/// | IRQ handler first-level (acknowledge + enqueue) | 10 us |
/// | Margin (cache misses, cross-NUMA, contention) | 25 us |
/// | **Total worst-case interrupt latency** | **50 us** |
///
/// Subsystems requiring longer critical sections must use `Mutex<T>` (which
/// allows preemption and sleeping). Debug builds assert hold time via
/// a per-lock hold-time timestamp comparison (warn if > 10us, panic if > 50us in
/// debug lock-checking mode).
pub struct RawSpinLock {
/// Lock state word, shared by both algorithm bodies; the ENCODING is
/// algorithm-specific (see "State-Word Encodings" below). The all-zero
/// word is the initial state and a valid UNLOCKED state under BOTH
/// encodings — the property the boot-time algorithm re-selection relies
/// on ("Early-Boot Default and the Patch Window" below). Note the ticket
/// encoding also has non-zero unlocked states (`owner == next != 0`
/// after use), so "non-zero ⇒ locked" holds only under the qspinlock
/// encoding.
state: AtomicU32,
}
impl RawSpinLock {
/// Construct an unlocked lock (usable in `static`/`const` init). The
/// all-zero word is unlocked under both algorithm encodings.
pub const fn new() -> Self {
Self { state: AtomicU32::new(0) }
}
/// Acquire the lock (spin-wait). Disables preemption. Does NOT touch IRQs.
///
/// The uncontended shape is inline and fixed per algorithm; contention
/// leaves through a boot-patched direct call (see "Fast Path and Slow-Path
/// Dispatch" below for the normative body).
///
/// # Safety
/// See struct-level safety note. The caller must ensure IRQ state is correct.
pub unsafe fn lock(&self) {
cpu_local::preempt_count_inc();
debug_assert!(!in_nmi(), "RawSpinLock is NMI-forbidden — use NmiSpinlock");
if lock_algo_is_queued() {
// Uncontended: one CAS on the whole word (0 → locked).
if self.state
.compare_exchange_weak(0, Q_LOCKED, Ordering::Acquire, Ordering::Relaxed)
.is_ok()
{
return;
}
// Contended: boot-selected slow path — a PATCHED DIRECT CALL
// ([Section 2.16](02-boot-hardware.md#extended-state-and-cpu-features--patched-direct-calls)).
// Default target qspin_lock_slow; re-pointed to qspin_lock_slow_pv
// when a hypervisor was detected at boot.
alternative_call!(
default: qspin_lock_slow,
alt: (arch_raw::HYPERVISOR, qspin_lock_slow_pv),
args: (&self.state)
);
} else {
ticket_lock(&self.state);
}
}
/// Release the lock. Re-enables preemption (checking `need_resched`).
///
/// # Safety
/// Must be called by the same CPU that called `lock()`.
pub unsafe fn unlock(&self) {
if lock_algo_is_queued() {
// qspin_unlock() is itself a boot-patched site: a single inline
// store-release of the locked byte on the native path (no call);
// byte-patched to `call qspin_unlock_pv` under a hypervisor —
// the pv unlock must additionally wake a halted queue head. See
// "Paravirt Slow-Path Variant" below.
qspin_unlock(&self.state);
} else {
ticket_unlock(&self.state);
}
cpu_local::preempt_count_dec_and_test_resched();
}
/// Try to acquire the lock once (non-blocking). Returns `true` if acquired.
///
/// **Semantics under queueing**: `try_lock()` NEVER enqueues and NEVER
/// barges past waiters. Under the qspinlock encoding it succeeds only
/// when the whole word is 0 (free, no pending claimant, empty queue);
/// under the ticket encoding only when `owner == next` (no tickets
/// outstanding). A `false` return therefore means "held OR contended",
/// which is the algorithm-independent failure predicate callers may rely
/// on.
///
/// # Safety
/// See `lock()`.
pub unsafe fn try_lock(&self) -> bool {
cpu_local::preempt_count_inc();
debug_assert!(!in_nmi(), "RawSpinLock is NMI-forbidden — use NmiSpinlock");
let acquired = if lock_algo_is_queued() {
self.state
.compare_exchange(0, Q_LOCKED, Ordering::Acquire, Ordering::Relaxed)
.is_ok()
} else {
ticket_trylock(&self.state)
};
if !acquired {
cpu_local::preempt_count_dec_and_test_resched();
}
acquired
}
}
3.5.1.1.1.1 RawSpinLock Algorithm Selection¶
Pseudocode convention: Code in this section uses Rust syntax and follows Rust ownership, borrowing, and type rules.
&selfmethods use interior mutability for mutation. Atomic fields use.store()/.load(). All#[repr(C)]structs haveconst_assert!size verification. See CLAUDE.md §Spec Pseudocode Quality Gates.
Why boot-time patching and not AlgoDispatch. AlgoDispatch
(Section 3.10) is disqualified for lock paths on
two independent grounds. First, its dispatch is an indirect call — unaffordable
on the single hottest code path in the kernel and a Spectre-v2 (branch target
injection) surface at exactly the site executed most often. Second, its init
contract is wrong for locks: AlgoDispatch::get() before phase-9 init() is
undefined behavior, but RawSpinLocks are taken from boot phase 2 onward. The
division of labor is therefore: AlgoDispatch serves phase-9+ consumers that
tolerate one predicted indirect call (crypto, checksums, compression, memcpy);
instruction patching (code_alternative! /
Section 2.16)
serves lock paths, pre-phase-9 consumers, and Spectre-sensitive call sites —
zero indirect branches, and a baked-in default that is valid from the first
instruction.
The selection POLICY reuses AlgoDispatch semantics — a priority-ordered
candidate list evaluated against CpuFeatureTable.universal
(Section 2.16) plus
platform data (hypervisor presence) — but the MECHANISM is patching: the
winning candidate is burned into the instruction stream once, in the
alt_patch_all() window.
Candidate table (normative; first matching row wins):
| Architecture | Candidates (priority order) | Selection basis | Baked-in default (pre-patch) |
|---|---|---|---|
| x86-64 | pv-qspinlock → qspinlock | CPUID.1:ECX[31] hypervisor bit (arch_raw::HYPERVISOR) |
qspinlock |
| AArch64 | qspinlock | single candidate — compile-time const, no patch | qspinlock |
| ARMv7 | ticket | single candidate — compile-time const, no patch | ticket |
| RISC-V 64 | qspinlock → ticket | qspinlock iff (Zabha ∧ Zacas) ∨ Ziccrse in the universal ISA intersection | qspinlock (demoted at patch time if the extensions are absent — see the patch-window section) |
| PPC32 | ticket | single candidate — compile-time const, no patch | ticket |
| PPC64LE | pv-qspinlock → qspinlock | device tree: shared-processor LPAR / KVM guest node | qspinlock |
| s390x | pv-qspinlock → qspinlock | STSI machine type: LPAR shared / z/VM / KVM (SIE guest is the norm on s390x) | qspinlock |
| LoongArch64 | pv-qspinlock → qspinlock | CPUCFG KVM signature | qspinlock |
Per-architecture rationale (Linux master verified where cited):
- x86-64: qspinlock matches Linux (
kernel/locking/qspinlock.c); the pv variant halts a spinning vCPU instead of burning its timeslice (kernel/locking/qspinlock_paravirt.h, pv_wait/pv_kick hypercalls). - AArch64: qspinlock; LDAXR/STLXR pairs serve the word CAS, and the
event-stream/WFE strategy lives inside
cpu_relax(). Single candidate —lock_algo_is_queued()is a compile-timetrue. - ARMv7: ticket (LDREX/STREX on the 32-bit word) — matches Linux
arch/arm/include/asm/spinlock.h("ARMv6 ticket-based spin-locking", verified against master). The reservation model does not efficiently support MCS queue-node spinning; ticket provides FIFO fairness at lower overhead on the core counts ARMv7 systems ship. - RISC-V 64: qspinlock requires the forward-progress guarantee of
cmpxchg()/xchg(): CAS via Zabha ∧ Zacas, or LR/SC with Ziccrse, provide it — verified against Linux masterarch/riscv/Kconfig(RISCV_QUEUED_SPINLOCKShelp text) andarch/riscv/kernel/setup.c(riscv_spinlock_init(): Zabha+Zacas first, else Ziccrse, else the combo static key falls back to ticket). UmkaOS mirrors the combo selection at patch time; the two-term predicate(Zabha ∧ Zacas) ∨ Ziccrseis expressible as one sum-of-products condition descriptor (Section 2.16) —Zabhaiscaps::RV_ZABHA,Zacas/Ziccrseare RISC-Varch_rawbits, giving the qspinlock candidate condition(caps::RV_ZABHA & arch_raw::RV_ZACAS) | arch_raw::RV_ZICCRSE. (An earlier revision of this section said "UmkaOS uses qspinlock unconditionally for fairness" — that was WRONG: without the forward-progress extensions the qspinlock LR/SC loops may livelock under contention.) - PPC32: ticket. Intentional UmkaOS improvement: Linux PPC32 uses a
simple test-and-set spinlock (
arch/powerpc/include/asm/simple_spinlock.h) which is unfair under contention; UmkaOS uses a ticket lock for starvation prevention. - PPC64LE: qspinlock; POWER cores benefit from MCS-style local spinning
on NUMA-distant workloads. Under a hypervisor (shared-processor LPAR or
KVM), Linux master uses a bespoke pv-aware qspinlock
(
arch/powerpc/lib/qspinlock.c: yield-to-preempted-owner via the hypervisor yield count, steal windows, LinuxMUST_Q). UmkaOS keeps the generic body and maps the pv seam toH_CONFER/H_PROD(see the pv section); the PPC-specific steal/yield tunables are documented future refinements, not v1. - s390x (no LL/SC — CS/CDS compare-and-swap serves the atomic seam;
core::sync::atomicCAS compiles to CS): qspinlock. Linux master implements its own queued spinlock for s390 (arch/s390/lib/spinlock.c: owner-CPU-in-word + per-CPU 4-nodespin_waitqueue, NIAI 4/8 cache-intent hints, directed yield viasmp_yield_cputo a preempted holder, runtime queued-vs-classic choice onCIF_DEDICATED_CPU) — architecturally the same MCS family. UmkaOS uses the one generic body; the s390x pv seam implementspv_wait()as bounded spin + diag 0x9c directed yield toward the holder/predecessor andpv_kick()as a no-op (correct: kicks are latency hints, never correctness — every wait re-checks its predicate). The NIAI access-intent hint is a documentedcpu_relax()-level refinement inside the s390x arch module, not a separate algorithm. - LoongArch64 (LL/SC + AMO): qspinlock — Linux master uses the generic
qspinlock (
arch/loongarch/include/asm/qspinlock.hincludesasm-generic/qspinlock.h), with pv support and Linuxvcpu_is_preemptedkeyed on the KVM CPUCFG signature. UmkaOS adds the pv row on the same detection. (Linux'svirt_spin_lock()test-and-set fallback for hypervisors WITHOUT pv support is unnecessary here: the degenerate yield-basedpv_wait()covers that case without a third algorithm body.)
Future candidate row (documented, NOT v1): a NUMA-aware qspinlock variant (CNA-style — grouping waiters by NUMA node to reduce cross-socket cache-line migration) is an admissible additional slow-path candidate. It changes only the slow-path function target and the queue-node layout, not the state word, so it slots into the same patched-call machinery. It is recorded here so the candidate table is understood as extensible; no v1 code path references it.
3.5.1.1.1.2 State-Word Encodings¶
Both encodings share state: AtomicU32; 0 is unlocked in both.
Queued (qspinlock) — Linux-exact layout, verified against master
include/asm-generic/qspinlock_types.h (the Linux NR_CPUS < 16K variant):
/// Queued-spinlock state-word encoding (fewer-than-16K-CPU layout):
/// bits 0-7 locked byte (Q_LOCKED = 1 held; Q_SLOW = 3 pv-halted head)
/// bits 8-15 pending byte (only bit 8 is ever set; a full byte so the
/// pending→locked hand-over is one halfword store)
/// bits 16-17 tail index (queue-node context nesting level of the tail)
/// bits 18-31 tail CPU + 1 (0 = queue empty)
pub const Q_LOCKED: u32 = 1;
/// Pv-halted marker in the locked byte (see the paravirt section).
pub const Q_SLOW: u32 = 3;
pub const Q_PENDING: u32 = 1 << 8;
pub const Q_LOCKED_MASK: u32 = 0xFF;
pub const Q_LOCKED_PENDING_MASK: u32 = 0xFFFF;
pub const Q_TAIL_IDX_SHIFT: u32 = 16;
pub const Q_TAIL_CPU_SHIFT: u32 = 18;
pub const Q_TAIL_MASK: u32 = !0u32 << Q_TAIL_IDX_SHIFT;
/// Encode a queue tail: (cpu + 1) so an all-zero tail means "empty".
pub fn q_encode_tail(cpu: u32, idx: u32) -> u32 {
((cpu + 1) << Q_TAIL_CPU_SHIFT) | (idx << Q_TAIL_IDX_SHIFT)
}
/// Decode a tail into the owning CPU's queue node.
pub fn q_decode_tail(tail: u32) -> &'static QSpinNode {
let cpu = (tail >> Q_TAIL_CPU_SHIFT) - 1;
let idx = (tail >> Q_TAIL_IDX_SHIFT) & 0x3;
&qspin_nodes(cpu).nodes[idx as usize]
}
Capacity/longevity analysis (14-bit tail-CPU field): the field stores
cpu + 1 in 14 bits with 0 reserved for "queue empty", so cpu + 1 ≤ 16383
⇒ CPU ids 0..=16382 ⇒ at most 16,383 CPUs (2¹⁴ − 1: one code point
lost to the empty encoding — the same bound as Linux's NR_CPUS < 16K
layout). MAX_CPUS (4096, a link-time array-capacity hint, not a runtime
limit) sits 4× below that. Because CPU count is bounded by hardware
topology — not a monotonic counter — the u64 counter policy does not apply;
the bound is validated at boot:
assert!(num_possible_cpus() <= 16383) in spinlock_boot_init(), panicking
with a message naming this encoding. If a future platform exceeds it, the
encoding switches to the Linux NR_CPUS ≥ 16K variant (pending shrinks to
one bit, tail widens) — a spec change, never a silent wrap.
Ticket:
/// Ticket state-word encoding:
/// bits 0-15 owner (ticket now being served)
/// bits 16-31 next (next ticket to hand out)
/// Unlocked ⟺ owner == next. Grabbing a ticket is fetch_add(1 << 16) — the
/// carry out of bit 31 when `next` wraps 0xFFFF→0 falls off the word, so
/// both halves wrap independently (mod 2^16); ordering comparisons use
/// wrapping u16 equality, valid while waiters ≤ 65,535 (bounded by the
/// 16,383-CPU cap above).
pub const T_NEXT_SHIFT: u32 = 16;
pub const T_OWNER_MASK: u32 = 0xFFFF;
3.5.1.1.1.3 Sub-Word Atomic Projections¶
Three operations touch a sub-word of the state (the qspinlock unlock byte
store, the pending→locked halfword hand-over, and the ticket owner bump).
These use typed projections rather than full-word RMWs — the reason the
qspinlock unlock is ONE plain store-release instead of a ~20-cycle
fetch_and:
/// Atomic view of the qspinlock locked byte (bits 0-7 of the state word).
/// Byte offset 0 on little-endian targets; 3 on big-endian (s390x — PPC32
/// is big-endian too but runs the ticket algorithm).
pub fn q_locked_byte(state: &AtomicU32) -> &AtomicU8 {
// SAFETY: same-object mixed-size atomic access, confined to these three
// projections. Every access to the state word is atomic (no plain
// accesses to race with), and all eight supported architectures provide
// coherent mixed-size atomics on one naturally aligned word — the same
// guarantee Linux's `struct qspinlock` byte/halfword union relies on.
unsafe { &*(state as *const AtomicU32 as *const AtomicU8).add(Q_BYTE_OFFSET) }
}
/// Atomic view of the locked+pending halfword (bits 0-15).
/// Halfword offset 0 on little-endian; 2 on big-endian.
pub fn q_locked_pending_half(state: &AtomicU32) -> &AtomicU16;
/// Atomic view of the ticket `owner` halfword (bits 0-15).
/// Halfword offset 0 on little-endian; 2 on big-endian.
pub fn t_owner_half(state: &AtomicU32) -> &AtomicU16;
/// Endian-dependent byte offset of the least-significant byte.
#[cfg(target_endian = "little")]
pub const Q_BYTE_OFFSET: usize = 0;
#[cfg(target_endian = "big")]
pub const Q_BYTE_OFFSET: usize = 3;
The projections only ever LOAD and STORE (plain atomic accesses — available
on every target). The design's only sub-word RMWs are two pv byte
compare-exchanges — qspin_unlock_pv's fast-case CAS and the head-wait
loop's 1 → Q_SLOW flip (W2) — both confined to pv candidates; they
compile to a masked word-CAS loop on targets without native byte CAS
(RISC-V has no pv row; s390x CS masks the word) — semantically identical,
since a tail-bit change simply retries.
3.5.1.1.1.4 Queue Nodes¶
MCS queue nodes live OUTSIDE the lock word, in a per-CPU node set. They are a
separate boot-time per-CPU allocation — deliberately NOT CpuLocalBlock
fields: nodes are touched only on contention (violating the block's
every-syscall field criterion), and the MCS protocol requires cross-CPU
writes into another CPU's node — an enqueuing successor links itself by
writing its predecessor's next, and a releasing predecessor hands off by
writing its successor's locked — either of which would break the
"cross-CPU writes to another CPU's CpuLocalBlock are never permitted"
invariant (Section 3.2).
/// One MCS queue node. `next`/`locked` are written cross-CPU by the MCS
/// protocol (the successor writes `next` to link in; the predecessor writes
/// `locked` to hand off); `cpu` and `pv_state` are the paravirt extension,
/// ignored by the native slow path.
// kernel-internal, not KABI
pub struct QSpinNode {
/// Successor node, linked by the CPU that enqueues behind this one.
next: AtomicPtr<QSpinNode>,
/// MCS hand-off flag: 0 = wait; 1 = you are now the queue head.
locked: AtomicU32,
/// Owning CPU id (constant per node set entry; used by pv_kick).
cpu: u32,
/// Paravirt vCPU state: PV_RUNNING / PV_HALTED / PV_HASHED (pv section).
pv_state: AtomicU8,
}
/// Nesting contexts that may hold a RawSpinLock concurrently on one CPU:
/// task, softirq, hardirq. NMI is FORBIDDEN (NmiSpinlock), so three suffice.
pub const QSPIN_MAX_CONTEXTS: usize = 3;
/// One CPU's node set, cache-line aligned against false sharing.
// kernel-internal, not KABI
#[repr(align(64))]
pub struct QSpinNodeSet {
nodes: [QSpinNode; QSPIN_MAX_CONTEXTS],
/// Nesting cursor: index of the next free node on this CPU. AtomicU32
/// (Relaxed) because hardirq/softirq entry can interleave with a task-
/// context increment on the same CPU — same rationale as
/// `softirq_pending` ([Section 3.2](#cpulocal-register-based-per-cpu-fast-path)).
count: AtomicU32,
}
/// Per-CPU node sets: boot-allocated slice of length `num_possible_cpus()`
/// (runtime-discovered — no compile-time CPU cap), allocated in
/// `spinlock_boot_init()` (boot phase 1, before the first contended lock is
/// possible: pre-SMP execution is single-CPU, and single-CPU contention
/// never reaches the queue path). Indexed by CPU id.
pub fn qspin_nodes(cpu: u32) -> &'static QSpinNodeSet;
3.5.1.1.1.5 Fast Path and Slow-Path Dispatch¶
/// Boot-patched algorithm predicate.
///
/// - Architectures with exactly ONE candidate algorithm (AArch64, ARMv7,
/// PPC32): a compile-time `const` — the dead branch is removed at build
/// time, and no patch record is emitted.
/// - Runtime-choice architectures (riscv64): a static-key-style patched
/// branch (`code_alternative!` two-byte branch flip): the baked-in default
/// arm is QUEUED; `alt_patch_all()` flips it to the ticket arm when the
/// forward-progress extensions are absent from the universal intersection.
/// Never a memory load, never an indirect branch.
pub fn lock_algo_is_queued() -> bool;
/// Spin-wait hint, declared in `arch::current::cpu` (the ONLY per-arch
/// surface the lock algorithms touch beyond `core::sync::atomic`):
///
/// | Architecture | Expansion |
/// |---|---|
/// | x86-64 | `pause` |
/// | AArch64 | `yield` (WFE/event-stream strategies live behind this hint) |
/// | ARMv7 | `yield` |
/// | RISC-V 64 | `pause` (Zihintpause; encoded as a fence hint — a NOP on cores without it) |
/// | PPC32 / PPC64LE | HMT low→medium priority nop pair (`or 1,1,1` / `or 2,2,2`) |
/// | s390x | compiler barrier (no cheap unprivileged hint; NIAI-prefixed loads in the arch module refine hot spins) |
/// | LoongArch64 | compiler barrier (`dbar 0` hint where profitable) |
pub fn cpu_relax();
The slow path is a patched direct call: the call instruction's target is
re-pointed once at boot via alternative_call!
(Section 2.16
— the general facility; the lock slow paths are its first consumer). Where an
architecture's candidate table yields a single slow-path implementation, the
"patched call" degenerates to an ordinary direct call and no patch record is
emitted (ticket on ARMv7/PPC32).
3.5.1.1.1.6 Native Queued Slow Path (normative)¶
Follows Linux master kernel/locking/qspinlock.c (queued_spin_lock_slowpath)
structurally: a pending-bit optimistic phase for the first waiter, then MCS
queueing.
/// Bounded wait for a transient (pending=1, locked=0) hand-over word.
const Q_PENDING_LOOPS: u32 = 1;
/// Contended acquisition. Entered from `lock()` after the fast-path CAS
/// failed. Preemption is already disabled; IRQs are caller-managed.
pub fn qspin_lock_slow(state: &AtomicU32) {
let mut val = state.load(Ordering::Relaxed);
// ── Phase P: pending-bit optimistic spin (first waiter, no queue) ──
// A (0,pending,0) word is a hand-over in progress; give it one bounded
// re-read so we don't queue behind a lock that is about to be ours.
if val == Q_PENDING {
let mut cnt = Q_PENDING_LOOPS;
while val == Q_PENDING && cnt > 0 {
cpu_relax();
cnt -= 1;
val = state.load(Ordering::Relaxed);
}
}
if val & !Q_LOCKED_MASK == 0 {
// No pending claimant, no queue: claim the pending byte.
val = state.fetch_or(Q_PENDING, Ordering::Acquire);
if val & !Q_LOCKED_MASK == 0 {
// Pending is ours. Wait for the holder, then take the lock.
if val & Q_LOCKED_MASK != 0 {
while state.load(Ordering::Acquire) & Q_LOCKED_MASK != 0 {
cpu_relax();
}
}
// pending → locked hand-over: we own both low bytes; the tail
// halfword may change concurrently, so store only the low half.
// Ordering: acquisition ordering came from the Acquire loads
// above (matches Linux clear_pending_set_locked).
q_locked_pending_half(state).store(Q_LOCKED as u16, Ordering::Relaxed);
return;
}
// Raced with a concurrent pending/tail claimant. If the pending bit
// was not already set, WE set it — undo before queueing.
if val & Q_PENDING == 0 {
state.fetch_and(!Q_PENDING, Ordering::Relaxed);
}
}
// ── Phase Q: MCS queue ──
let cpu = cpu_local::cpu_id();
let set = qspin_nodes(cpu);
let idx = set.count.fetch_add(1, Ordering::Relaxed) as usize;
if idx >= QSPIN_MAX_CONTEXTS {
// Unreachable under the three-context rule (NMI is forbidden); kept
// as a robustness fallback mirroring Linux: spin on trylock without
// a node.
while state
.compare_exchange_weak(0, Q_LOCKED, Ordering::Acquire, Ordering::Relaxed)
.is_err()
{
cpu_relax();
}
set.count.fetch_sub(1, Ordering::Relaxed);
return;
}
let node = &set.nodes[idx];
node.locked.store(0, Ordering::Relaxed);
node.next.store(core::ptr::null_mut(), Ordering::Relaxed);
// The queueing cache line was cold; retry once — the lock may have been
// released while we set up.
if state
.compare_exchange(0, Q_LOCKED, Ordering::Acquire, Ordering::Relaxed)
.is_ok()
{
set.count.fetch_sub(1, Ordering::Relaxed);
return;
}
// Publish this node as the new tail. Release orders the node init above
// before the tail becomes visible to the successor.
let tail = q_encode_tail(cpu, idx as u32);
let mut old = state.load(Ordering::Relaxed);
loop {
let new = (old & !Q_TAIL_MASK) | tail;
match state.compare_exchange_weak(old, new, Ordering::Release, Ordering::Relaxed) {
Ok(_) => break,
Err(v) => old = v,
}
}
if old & Q_TAIL_MASK != 0 {
// Queue was non-empty: link behind the previous tail, wait for the
// MCS hand-off on OUR OWN node (CPU-local spinning — the point of
// MCS: no shared-word cache-line bouncing while queued).
let prev = q_decode_tail(old & Q_TAIL_MASK);
prev.next.store(node as *const QSpinNode as *mut QSpinNode, Ordering::Release);
while node.locked.load(Ordering::Acquire) == 0 {
cpu_relax();
}
}
// Queue head: wait for the owner AND any pending claimant to drain.
// (Pending waiters entered before we queued; they win by design.)
let mut val = state.load(Ordering::Acquire);
while val & Q_LOCKED_PENDING_MASK != 0 {
cpu_relax();
val = state.load(Ordering::Acquire);
}
// Claim. If we are the ONLY queued waiter (tail still names our node),
// take the lock and empty the queue in one CAS.
if val & Q_TAIL_MASK == tail
&& state
.compare_exchange(val, Q_LOCKED, Ordering::Relaxed, Ordering::Relaxed)
.is_ok()
{
// Acquisition ordering came from the Acquire wait loop above.
set.count.fetch_sub(1, Ordering::Relaxed);
return;
}
// Someone queued behind us (or is mid-publish): take the lock, then
// hand MCS headship to the successor.
q_locked_byte(state).store(Q_LOCKED as u8, Ordering::Relaxed);
let mut next = node.next.load(Ordering::Acquire);
while next.is_null() {
cpu_relax();
next = node.next.load(Ordering::Acquire);
}
// SAFETY: a non-null `next` points into the successor CPU's static
// QSpinNodeSet, published by its Release store in the enqueue path.
unsafe { (*next).locked.store(1, Ordering::Release) };
set.count.fetch_sub(1, Ordering::Relaxed);
}
/// Native queued unlock: one store-release of the locked byte. Inline at
/// every `unlock()` site; byte-patched to `call qspin_unlock_pv` when the
/// pv candidate won (the ONLY difference pv makes to the unlock site).
#[inline(always)]
pub fn qspin_unlock(state: &AtomicU32) {
q_locked_byte(state).store(0, Ordering::Release);
}
3.5.1.1.1.7 Ticket Algorithm (normative)¶
/// Ticket acquisition: grab a ticket with one fetch_add; if the lock was
/// free the same instruction acquired it (fast path — one RMW, no CAS
/// loop); otherwise spin in the out-of-line wait body.
pub fn ticket_lock(state: &AtomicU32) {
let t = state.fetch_add(1 << T_NEXT_SHIFT, Ordering::Acquire);
let my = (t >> T_NEXT_SHIFT) as u16;
if (t & T_OWNER_MASK) as u16 == my {
return; // owner == our ticket: acquired
}
// Single implementation — the "patched call" degenerates to a plain
// direct call on ticket architectures.
ticket_lock_slow(state, my);
}
/// Out-of-line ticket wait: spin until `owner` reaches our ticket.
/// FIFO by construction (tickets are served in issue order).
pub fn ticket_lock_slow(state: &AtomicU32, my: u16) {
loop {
let owner = (state.load(Ordering::Acquire) & T_OWNER_MASK) as u16;
if owner == my {
return;
}
// Proportional backoff: each waiter ahead of us must complete a
// full critical section — relax once per ticket of distance.
let ahead = my.wrapping_sub(owner);
for _ in 0..ahead {
cpu_relax();
}
}
}
/// Ticket release: bump `owner`. Only the holder writes the owner half
/// while held, so load(Relaxed) + store(Release) suffices — no RMW.
pub fn ticket_unlock(state: &AtomicU32) {
let owner = t_owner_half(state).load(Ordering::Relaxed);
t_owner_half(state).store(owner.wrapping_add(1), Ordering::Release);
}
/// Ticket trylock: succeed only when no tickets are outstanding
/// (owner == next); never issues a ticket on failure.
pub fn ticket_trylock(state: &AtomicU32) -> bool {
let val = state.load(Ordering::Relaxed);
if (val & T_OWNER_MASK) != (val >> T_NEXT_SHIFT) {
return false;
}
state
.compare_exchange(
val,
val.wrapping_add(1 << T_NEXT_SHIFT),
Ordering::Acquire,
Ordering::Relaxed,
)
.is_ok()
}
3.5.1.1.1.8 Paravirt Slow-Path Variant¶
On a hypervisor, a vCPU spinning on a lock whose holder (or queue
predecessor) is PREEMPTED burns its whole timeslice. The pv candidate keeps
the same state word, fast path, and queue discipline as the native body and
substitutes halt-instead-of-spin at the two unbounded wait loops. Protocol
per Linux master kernel/locking/qspinlock_paravirt.h (pv_wait/pv_kick,
_Q_SLOW_VAL, lock hash), simplified: UmkaOS v1 omits Linux's hybrid
lock-stealing and wait-early heuristics (documented tunables below).
Arch seam (declared in arch::current::cpu; the generic body never
names a hypervisor interface):
/// Block this vCPU while `word` still reads `expected` (bounded, may wake
/// spuriously — callers ALWAYS re-check in a loop). Implementations:
/// x86-64 KVM halt + kick hypercalls; PPC64LE H_CONFER (yield to a named
/// vCPU); s390x bounded spin + diag 0x9c directed yield (degenerate form —
/// yields rather than halts; pv_kick is then a no-op); LoongArch64 KVM
/// pv-IPI wait/kick. Correctness NEVER depends on a kick arriving: a kick
/// only ends a wait early.
pub fn pv_wait(word: &AtomicU32, expected: u32);
/// Wake the pv_wait on the named CPU, if any (latency hint, not a
/// correctness edge).
pub fn pv_kick(cpu: u32);
Generic pv structures (in umka-nucleus/src/sync/spinlock.rs, compiled only
into pv-candidate builds):
/// QSpinNode.pv_state values.
pub const PV_RUNNING: u8 = 0;
/// Halted in the node-wait loop (W1); woken by the predecessor's W3 kick.
pub const PV_HALTED: u8 = 1;
/// Queue head, registered in PV_LOCK_HASH and halted on the lock word (W2);
/// woken by qspin_unlock_pv.
pub const PV_HASHED: u8 = 2;
/// Register `(state, node)` in PV_LOCK_HASH before the queue head halts.
/// Open addressing; a free entry always exists (capacity ≥ 4 × possible
/// CPUs, one blocked lock per vCPU ⇒ load factor ≤ 25%).
pub fn pv_hash(state: &AtomicU32, node: &'static QSpinNode);
/// Remove and return the node registered for `state`. Called by the
/// unlocker after observing Q_SLOW in the locked byte — the registration is
/// guaranteed present (W2 hashes BEFORE flipping the byte to Q_SLOW).
pub fn pv_unhash(state: &AtomicU32) -> &'static QSpinNode;
pv node state (QSpinNode.pv_state): PV_RUNNING = 0, PV_HALTED = 1
(halted in the node-wait loop), PV_HASHED = 2 (queue head, registered in
the lock hash). Lock hash: PV_LOCK_HASH — a boot-sized open-addressing
table of (lock_word_addr, node_ptr) pairs, capacity
4 * num_possible_cpus() entries rounded up to a power of two (a vCPU
blocks on at most one lock, so load factor ≤ 25%); pv_hash(state, node)
inserts, pv_unhash(state) removes and returns the node. Cache-line-aligned
entries, boot-allocated in spinlock_boot_init() on pv-candidate platforms
only.
/// Spin budget before halting (Linux SPIN_THRESHOLD analog). Registered as
/// a bounded ML-tunable parameter ([Section 23.1](23-ml-policy.md#aiml-policy-framework-closed-loop-kernel-intelligence));
/// the bounds [1<<10, 1<<20] are kernel-enforced.
pub static PV_SPIN_THRESHOLD: AtomicU32 = AtomicU32::new(1 << 15);
/// Paravirt contended acquisition: qspin_lock_slow with three substitutions.
pub fn qspin_lock_slow_pv(state: &AtomicU32) {
// Identical to qspin_lock_slow EXCEPT:
//
// (W1) Node-wait loop ("wait for MCS hand-off"): spin up to
// PV_SPIN_THRESHOLD; still 0 → pv_state.store(PV_HALTED, Relaxed)
// then fence(SeqCst), then re-check node.locked; if still 0,
// pv_wait(&node.locked, 0). On return: pv_state.store(PV_RUNNING,
// Relaxed) and loop (spurious wakes re-enter the spin).
//
// The fence is load-bearing. W1 and W3 form a Dekker (store-
// buffering) pattern on the pair {pv_state, node.locked}: each
// side STORES one location then LOADS the other, and a store
// followed by a load of a DIFFERENT location is unordered under
// every release/acquire combination — StoreLoad reordering is
// legal even on x86 TSO. Unfenced, both loads can read stale
// values: W1 halts on a stale locked == 0 while W3 reads a stale
// PV_RUNNING and skips the kick. With a full fence on BOTH sides
// (here, and in W3 below), at least one side observes the
// other's store: either W1's re-check sees locked == 1 and
// returns without halting, or W3 sees PV_HALTED and kicks. Linux
// master fences the same two points and draws this exact
// diagram (Linux `kernel/locking/qspinlock_paravirt.h::pv_wait_node`):
// Linux "[S] pn->state = VCPU_HALTED / MB / [L] pn->locked" vs
// "[S] next->locked = 1 / MB / [RmW] pn->state", implemented as
// Linux `smp_store_mb(pn->state, VCPU_HALTED);` before
// Linux `if (!READ_ONCE(node->locked))`). Store + fence(SeqCst) is
// chosen over a swap RMW as the exact Linux `smp_store_mb` analog:
// pv_state has a single writer at this point, so an RMW buys no
// additional ordering over the fenced plain store. The bounded
// pv_wait contract (arch seam above) remains the LIVENESS
// backstop against lost hypervisor kicks — it is not the
// correctness story for this window; without the fence every
// missed wake would cost a full halt timeout.
//
// (W2) Head-wait loop ("wait for owner+pending to drain"): spin up to
// PV_SPIN_THRESHOLD; then pv_hash(state, node), and flip the
// locked byte 1 → Q_SLOW via a byte compare-exchange:
// - CAS fails because the byte is 0: the holder released —
// pv_unhash(state) and claim as in the native body.
// - CAS succeeds: pv_state.store(PV_HASHED, Relaxed);
// pv_wait(state-low-word, Q_SLOW-in-locked-byte). The unlocker
// sees Q_SLOW, unhashes, stores 0, kicks us (see
// qspin_unlock_pv). On wake, re-enter the head-wait loop.
//
// (W3) After handing MCS headship to the successor
// (`next.locked.store(1, Release)`): fence(SeqCst), then if
// next.pv_state == PV_HALTED (Relaxed load — the fence supplies
// the ordering), pv_kick(next.cpu) — the successor halted in W1.
// This is the kicker side of W1's Dekker pattern: without a full
// fence between the hand-off store and the pv_state load, this
// side's StoreLoad can reorder just as W1's can. Linux places
// its kicker-side barrier at the same point
// (Linux `kernel/locking/qspinlock_paravirt.h::pv_kick_node`):
// Linux `smp_mb__before_atomic()` precedes the pn->state access, with
// the comment "The write to next->locked … must be ordered
// before the read of pn->state").
let _ = state; // body = qspin_lock_slow with W1-W3 substituted
}
/// Paravirt unlock (patched in place of the inline native byte store):
pub fn qspin_unlock_pv(state: &AtomicU32) {
// Fast case: locked byte is Q_LOCKED — plain release, nobody halted.
if q_locked_byte(state)
.compare_exchange(Q_LOCKED as u8, 0, Ordering::Release, Ordering::Relaxed)
.is_ok()
{
return;
}
// Byte was Q_SLOW: the queue head hashed itself and halted (W2).
let node = pv_unhash(state);
q_locked_byte(state).store(0, Ordering::Release);
pv_kick(node.cpu);
}
Deferred pv refinements (recorded, not v1): Linux's hybrid
queued/unfair stealing window, wait-early on preempted predecessors
(Linux vcpu_is_preempted), and the PPC64LE steal/MUST_Q tunables. All are
slow-path-internal, so they slot in behind the same patched call without
touching the fast path or the ABI of the state word.
3.5.1.1.1.9 Early-Boot Default and the Patch Window¶
- Baked-in default valid from the first instruction. Every lock site's
default — the compile-time branch arm of
lock_algo_is_queued()and the link-time slow-path call target from the candidate table's default column — is functional with NO initialization: the first RawSpinLock acquisition happens in boot phase 2, seven phases beforealt_patch_all(). - Re-pointed exactly once.
alt_patch_all()(boot phase 9, insidecpu_features_freeze()step 5 — Section 2.16) runs on the BSP with IRQs disabled and APs not yet started, and applies the first matching candidate row. No further lock-path patching ever occurs at runtime (live evolution patches only newly loaded module images). - Cross-algorithm soundness invariant. Re-selection is sound only if
every lock word is interpretable under the NEW algorithm at the instant of
the switch. The only encoding-compatible state is the all-zero word, so
the invariant is: at the patch window, every RawSpinLock word is 0.
This holds because (a) the window is single-CPU with IRQs off and the BSP
holds no RawSpinLock across
alt_patch_all()(asserted: the debug lock-tracking stack must be empty), so every lock is unlocked; and (b) the pre-patch default on every runtime-choice architecture is QSPINLOCK, whose unlocked word is exactly 0 (a ticket default would be unsound: a used-and-released ticket lock rests atowner == next != 0, which decodes as locked-plus-queue garbage under qspinlock). This is why the RISC-V default is qspinlock even though the machine may lack the forward-progress extensions: pre-patch execution is BSP-only (APs start after phase 9), and the LR/SC livelock risk that motivates the ticket demotion requires multi-hart contention that cannot yet exist. - Locks created after patching start at word 0 (
new()), a valid unlocked state under whichever algorithm won.
3.5.1.1.1.10 Slow-Path Live Evolution (design note, not v1 scope)¶
The slow-path implementations are replaceable through the evolution
text-patching machinery (Section 13.18) under a quiescence
precondition: no waiters queued on any lock and no CPU executing inside a
slow-path function (established by the stop-the-world window — parked CPUs
are at IPI rendezvous, not spinning in a slow path; any lock they held has
word-state but no in-flight slow-path frames once the rendezvous drains).
Re-pointing the alternative_call! sites to a new implementation is then the
same operation as the boot patch. The fast-path inline shapes and the state
word are NOT evolvable (they are compiled into every caller); an encoding
change is a rebuild, by design.
3.5.1.1.2 SpinLock<T>¶
RAII spinlock that saves and restores IRQ state on acquire/release. This is the
correct default for any data shared between normal kernel context and interrupt handlers.
Equivalent to Linux spinlock_t (acquired with spin_lock_irqsave). The protected
data T is only accessible through the returned guard, ensuring the lock is always
held when the data is accessed.
/// IRQ-saving spinlock. Disables preemption AND saves/restores IRQ state.
///
/// This is the correct default for data shared with interrupt handlers.
/// Acquiring this lock is safe from any context (normal kernel, softirq, hardirq).
///
/// The protected value `T` is accessible only through `SpinLockGuard<'_, T>`,
/// which re-enables IRQs and preemption on drop.
pub struct SpinLock<T> {
inner: RawSpinLock,
data: UnsafeCell<T>,
}
/// Opaque, architecture-defined saved interrupt-state token.
///
/// Produced when a lock disables interrupts on acquire and consumed to restore
/// the exact prior state on release. Its representation is per-architecture
/// (x86-64 `RFLAGS`, AArch64 `DAIF`, RISC-V `sstatus.SIE`, PPC `MSR[EE]`,
/// s390x PSW system mask, LoongArch64 `CRMD.IE`); generic code never inspects
/// it — it is only saved and later restored verbatim through `arch::current`.
/// The underlying `IrqFlags` type and the raw `save_and_disable()`/`restore()`
/// primitives are defined in [Section 3.8](#interrupt-handling).
pub type ArchIrqFlags = arch::current::interrupts::IrqFlags;
/// Guard returned by `SpinLock::lock()`. Releases lock and restores IRQs on drop.
pub struct SpinLockGuard<'a, T> {
lock: &'a SpinLock<T>,
/// Saved IRQ flags (RFLAGS on x86, DAIF on AArch64, SSTATUS.SIE on RISC-V, etc.).
saved_flags: ArchIrqFlags,
}
impl<T> SpinLock<T> {
/// Acquire the lock: save IRQ state, disable IRQs and preemption, spin-wait.
/// Returns a guard that provides exclusive access to `T`.
pub fn lock(&self) -> SpinLockGuard<'_, T>;
/// Acquire without saving IRQ state. Use this when the caller already holds
/// an `IrqDisabledGuard` ([Section 3.8](#interrupt-handling--local-interrupt-saverestore-archcurrentinterrupts)),
/// avoiding a redundant save/restore pair (~1-3 cycles saved on the fast path).
///
/// # Safety
/// The caller must hold an `IrqDisabledGuard` for the entire duration of the
/// returned guard's lifetime. Dropping the `IrqDisabledGuard` while the
/// `SpinLockGuard` is still alive would re-enable IRQs while the lock is held.
pub unsafe fn lock_nosave<'a>(
&'a self,
_irq: &'a IrqDisabledGuard,
) -> SpinLockGuard<'a, T>;
}
impl<T> Deref for SpinLockGuard<'_, T> {
type Target = T;
}
impl<T> DerefMut for SpinLockGuard<'_, T> {}
impl<T> Drop for SpinLockGuard<'_, T> {
fn drop(&mut self) {
// Release the RawSpinLock, then restore IRQ state from saved_flags.
// Order matters: release spin first, then re-enable IRQs. If IRQs
// were re-enabled while the lock was still held, an interrupt handler
// on the same CPU could try to acquire the same lock, causing a
// single-CPU deadlock. Releasing the lock first prevents this.
}
}
/// Acquire two DISTINCT same-level `SpinLock`s in a deadlock-free,
/// address-ordered sequence and return both guards. Any two callers acquiring
/// the same pair — in either argument order — always lock lower-address-first,
/// so no ABBA deadlock is possible even though both locks sit at the same
/// lock-hierarchy level (the compile-time `Lock<T, LEVEL>` ordering does not
/// distinguish two peers of one level). Used for the page-table-lock pair in
/// mremap PTE moves ([Section 4.15](04-memory.md#extended-memory-operations)) and anywhere two peers
/// of the same level must be held simultaneously.
///
/// For distinct locks the `Distinct` guards are in ARGUMENT order
/// (`(guard_a, guard_b)`), independent of which lock was physically acquired
/// first — callers name guards by their arguments; acquisition order is an
/// internal detail.
///
/// **Aliasing-safe**: `a` and `b` may be the same lock. When a caller cannot
/// rule out aliasing (e.g. two mremap virtual ranges that fall in the same PTE
/// page-table page), passing the same lock twice is fine — the helper detects
/// it via `ptr::eq` and acquires ONCE (`PairGuards::Same`), avoiding the
/// self-deadlock a second acquire of the same lock would cause. Distinct locks
/// are acquired in address order (deadlock-free) and returned as
/// `PairGuards::Distinct`.
pub fn lock_pair_ordered<'a, T>(
a: &'a SpinLock<T>,
b: &'a SpinLock<T>,
) -> PairGuards<'a, T> {
if core::ptr::eq(a, b) {
// Same lock aliased through both arguments — acquire once.
return PairGuards::Same(a.lock());
}
if (a as *const SpinLock<T> as usize) < (b as *const SpinLock<T> as usize) {
let ga = a.lock();
let gb = b.lock();
PairGuards::Distinct(ga, gb)
} else {
let gb = b.lock();
let ga = a.lock();
PairGuards::Distinct(ga, gb)
}
}
/// Guards returned by `lock_pair_ordered`. Holding this value keeps the
/// underlying lock(s) acquired; dropping it releases them.
pub enum PairGuards<'a, T> {
/// `a` and `b` were the same lock; acquired once. One guard covers both
/// argument names.
Same(SpinLockGuard<'a, T>),
/// `a` and `b` were distinct; both held. In ARGUMENT order
/// (`(guard_a, guard_b)`), independent of physical acquisition order.
Distinct(SpinLockGuard<'a, T>, SpinLockGuard<'a, T>),
}
lock_nosave and IrqDisabledGuard interaction: SpinLock::lock() internally
performs irq_save() + RawSpinLock::lock(). If the caller already holds an
IrqDisabledGuard (from PerCpu::get_mut_nosave(), see
Section 3.8), use
lock_nosave() to skip the redundant save/restore. This mirrors the
get_mut_nosave() pattern and saves ~1-3 cycles on the fast path.
3.5.1.1.3 MMIO Barrier Batching (io_sync flag — PPC64LE)¶
On PPC64LE, every MMIO write requires a preceding sync instruction (~100+ cycles on
POWER9) to order the store with respect to cacheable memory. When a driver performs
multiple MMIO writes within a single critical section (common for NIC ring doorbell
sequences), each MMIO write would pay the full sync cost independently.
UmkaOS batches these barriers using a per-CPU io_sync flag in the CpuLocal block:
/// Per-CPU MMIO barrier deferred-sync flag.
/// PPC64LE only; zero-sized on all other architectures.
pub struct IoSyncFlag {
/// Set to `true` by `mmio_write_*()`. Checked and cleared by
/// `SpinLock::unlock()`. When set, unlock issues `sync` before
/// the store-release that releases the lock.
#[cfg(target_arch = "powerpc64")]
pending: Cell<bool>,
}
impl IoSyncFlag {
/// Construct a cleared flag (usable in `static`/`const` init). Constructs
/// the all-zero-bits value; zero-sized on non-PPC64LE legs. The
/// `#[cfg]`-mirrored field init serves all 8 legs with one body.
pub const fn new() -> Self {
Self {
#[cfg(target_arch = "powerpc64")]
pending: Cell::new(false),
}
}
}
Protocol:
mmio_write_32(addr, val)on PPC64LE: emitsync; stw val, 0(addr), then setCpuLocal::io_sync.pending = true.SpinLock::unlock()on PPC64LE: ifio_sync.pendingistrue, emitsyncbefore the lock release store and clear the flag. This ensures all MMIO writes within the critical section are ordered before the lock release.mmiowb()(explicit MMIO write barrier): emitsyncand cleario_sync.pending. Used at the end of a critical section where MMIO writes must be visible to the device before the lock is released to another CPU.
Cost saving: If a critical section performs N MMIO writes, the naive approach issues
N sync instructions. With io_sync, only one sync is issued (in the unlock path),
saving (N-1) × ~100 cycles. This is the same optimization Linux implements via
paca->io_sync on PowerPC.
Other architectures: On x86-64, MMIO writes are naturally ordered by TSO. On AArch64,
dsb st provides the MMIO write barrier at lower cost (~20-40 cycles). The io_sync
flag is a zero-sized type on non-PPC architectures and the unlock check compiles away.
3.5.1.1.4 NmiSpinlock¶
An NMI-safe spinlock that guards a hardware serialization point (no Rust
payload) rather than a T. It exists for resources that are entered from both
normal context and NMI/panic context and are not re-entrant — notably the
platform EFI runtime services, serialized by the pstore EFI backend
(Section 20.7), whose SetVariable calls may originate
from boot-time registration, systemd-driven erase, and MCE-originated NMI
panic dumps.
/// NMI-safe spinlock guarding an external, non-re-entrant hardware resource
/// (no protected `T`, unlike `SpinLock<T>`).
///
/// Safe to acquire from NMI/panic context: `lock()` masks NMIs and IRQs on the
/// local CPU for the critical section, so a nested NMI on the same CPU cannot
/// re-enter and self-deadlock.
///
/// # Architecture NMI masking
/// NMI masking is arch-specific. On architectures without maskable NMIs
/// (x86-64), `lock()` uses a per-CPU owner marker so a nested NMI detects the
/// reentry and never spins on a lock its own CPU already holds — it falls
/// through rather than corrupting the protected resource.
///
/// # Hold Time Budget
/// Critical sections are bounded like `RawSpinLock` (≤ 10 us, no sleeping, no
/// heap) so the 50 us worst-case interrupt latency guarantee holds.
pub struct NmiSpinlock {
/// Lock state word. 0 = unlocked; non-zero = locked.
state: AtomicU32,
/// CPU id currently holding the lock (`u32::MAX` = unlocked). Used to
/// detect same-CPU NMI reentry on architectures without maskable NMIs.
owner_cpu: AtomicU32,
}
/// Guard returned by `NmiSpinlock::lock()`. Releases the lock and restores
/// NMI/IRQ state on drop.
pub struct NmiSpinlockGuard<'a> {
lock: &'a NmiSpinlock,
}
impl NmiSpinlock {
/// Construct an unlocked NMI-safe spinlock (usable in `static` init).
pub const fn new() -> Self {
Self { state: AtomicU32::new(0), owner_cpu: AtomicU32::new(u32::MAX) }
}
/// Acquire the lock: mask NMIs + IRQs on the local CPU, then spin-wait.
/// Returns a guard that releases the lock and restores NMI/IRQ state on
/// drop. Safe from NMI/panic context.
pub fn lock(&self) -> NmiSpinlockGuard<'_>;
}
3.5.1.1.5 IntrusiveList<T> — Intrusive Doubly-Linked List¶
An intrusive doubly-linked list used throughout UmkaOS for lock-free and low-overhead
queues. Elements embed a link field (IntrusiveLink<T>) rather than allocating
separate list nodes. This avoids heap allocation and improves cache locality.
UmkaOS vs Linux list_head: Linux embeds raw prev/next pointers in each
element with no ownership or membership tracking. An element can silently be on
multiple lists simultaneously, or removed from the wrong list — a common source of
kernel bugs. UmkaOS's design uses a sealed HasIntrusiveLink<T> trait with type-system-guided
single-link-per-type enforcement: each element type embeds at most one
IntrusiveLink<T>, so it can participate in at most one list at a time via that
link. Runtime double-insertion detection is provided by a debug-only in_list: bool
flag in IntrusiveLink (checked on push, panics on double-insert in debug builds).
Pinning (Pin<&mut T>) prevents moving an element while it is linked, which would
corrupt the list.
/// An intrusive doubly-linked list. Elements must embed an `IntrusiveLink<T>`
/// field to participate in the list.
///
/// **UmkaOS vs Linux `list_head`**: Linux embeds raw `prev`/`next` pointers in each
/// element with no ownership tracking. This allows elements to be on multiple lists
/// simultaneously (a common source of bugs: removing from the wrong list). UmkaOS's
/// design enforces single-link-per-type membership via the owned `IntrusiveLink`.
/// A debug-only `in_list` flag in `IntrusiveLink` provides runtime double-insertion
/// detection (panics in debug builds).
///
/// **Pinning**: Elements must be `Pin`ned before they can be inserted. Moving an element
/// while it is linked would corrupt the list. The `Pin<&mut T>` API at insertion ensures
/// the element address is stable for the element's lifetime in the list.
///
/// **Performance**: Equivalent to `list_head` — O(1) insert/remove, O(n) iteration.
/// No heap allocation; all pointers live in the embedded `IntrusiveLink` fields.
pub struct IntrusiveList<T: HasIntrusiveLink<T>> {
/// Sentinel node. In a NON-EMPTY list, `head.next` points to the first
/// element, `head.prev` points to the last element, and the boundary
/// elements point back at the sentinel's address. An EMPTY list has
/// `head.next == head.prev == null` — the deferred-sentinel state
/// established by `new()` and restored by whichever removal empties the
/// list. Invariant: `len == 0 ⇔ head.next == head.prev == null`.
head: IntrusiveLink<T>,
/// Number of elements currently in the list. O(1) cached count.
len: usize,
}
/// The link field that must be embedded in each element type `T` that participates
/// in an `IntrusiveList<T>`. Each element may embed at most ONE `IntrusiveLink<T>`
/// (embedding two would require the element to be in two lists simultaneously, which
/// is allowed by the data structure but should be avoided — use separate wrapper types).
pub struct IntrusiveLink<T> {
/// Next element in the list (or &sentinel if this is the tail).
next: *mut IntrusiveLink<T>,
/// Previous element in the list (or &sentinel if this is the head).
prev: *mut IntrusiveLink<T>,
/// Zero-size marker; `T` is the owning type that contains this link.
_marker: PhantomData<T>,
/// Debug-only double-insertion guard. Set to `true` when inserted into
/// a list, cleared on removal. In debug builds, `insert_*()` methods
/// assert `!in_list` before linking. In release builds, this field is
/// compiled out (`#[cfg(debug_assertions)]`) — zero overhead.
///
/// **Release-build consequence**: Without this guard, double-insertion
/// in release builds creates two lists sharing the same link node.
/// Removing from either list corrupts the other (next/prev pointers
/// updated for the wrong list). This is the intentional trade-off:
/// debug builds catch the programming error; release builds omit
/// the per-node overhead (1 byte + alignment padding per link).
/// The debug assertion is the primary defense; correct callers never
/// trigger the condition in production.
#[cfg(debug_assertions)]
in_list: bool,
}
impl<T> IntrusiveLink<T> {
/// An unlinked hook (`next == prev == null`). `const` so a container may
/// initialize its embedded link in a struct literal without allocation.
pub const fn new() -> Self {
IntrusiveLink {
next: core::ptr::null_mut(),
prev: core::ptr::null_mut(),
_marker: PhantomData,
#[cfg(debug_assertions)]
in_list: false,
}
}
/// True if this hook is currently linked into a list. A linked hook always
/// has a non-null neighbour on at least one side (the sole element of a
/// list points its ends at the sentinel, never null); a fresh or removed
/// hook has both pointers null. Callers must hold whatever lock serializes
/// the owning list. Used to make list removal idempotent (remove only if
/// still linked — e.g. ptrace detach after a drain-loop pop).
pub fn is_linked(&self) -> bool {
!self.next.is_null() || !self.prev.is_null()
}
}
/// Marker trait: `T` has an `IntrusiveLink<T>` field accessible via `link()`.
/// # Safety
/// The returned `IntrusiveLink` must be embedded in `self` at a stable address
/// (i.e., `self` must be pinned). Moving `self` while linked is undefined behavior.
pub unsafe trait HasIntrusiveLink<T> {
/// Return a pointer to the embedded link field.
fn link(this: *mut T) -> *mut IntrusiveLink<T>;
/// Recover the owning `T*` from a `*mut IntrusiveLink<T>` (via `offsetof`).
/// # Safety: `link` must point to the link field of a valid `T`.
unsafe fn from_link(link: *mut IntrusiveLink<T>) -> *mut T;
}
impl<T: HasIntrusiveLink<T>> IntrusiveList<T> {
/// Create an empty list. `const` so lists can appear in `static`
/// initializers (`EXT4_MOUNTS`, `TFM_REGISTRY`) and inside `const fn`
/// constructors (`Mutex::new`, `RwLock::new`, `RwLockInner::new`,
/// `WaitQueueHead::new`).
///
/// **Deferred-sentinel contract.** The sentinel starts NULL
/// (`head.next == head.prev == null`), NOT self-referential: a by-value
/// constructor cannot know the value's final address, so no constructor
/// can establish `head.next == &head`. The sentinel linkage is instead
/// established by the first insertion (`push_back`/`push_front`, or
/// `splice_back` into an empty `self`), which runs with the list at a
/// stable address (`&mut self`) and links the boundary elements to the
/// sentinel's CURRENT address. The removal that empties the list
/// (`pop_front`/`pop_back`/`remove` of the last element; `splice_back`
/// draining `other`) resets both sentinel pointers to null, maintaining
/// the invariant `len == 0 ⇔ head.next == head.prev == null`.
///
/// Consequences:
/// - `len()`/`is_empty()` are correct in the null-sentinel state (`len == 0`).
/// - `pop_front()`/`pop_back()` on an empty list return `None` without
/// dereferencing the null sentinel (the `len == 0` check short-circuits).
/// - `iter()` over an empty list yields nothing: `head.next.is_null()` ⇒
/// the iterator is constructed already-exhausted (no null deref).
/// - An EMPTY list is always movable (null pointers carry no address); a
/// NON-EMPTY list must not be moved (its boundary elements hold the
/// sentinel's address). This is the "must not be moved while non-empty"
/// precondition relied on by tracked-allocation bridges such as
/// `sighand_alloc`
/// ([Section 8.1](08-process.md#process-and-task-management--tracked-allocation-fsstruct-signalhandlers-fdtable)).
pub const fn new() -> Self {
IntrusiveList { head: IntrusiveLink::new(), len: 0 }
}
/// Number of elements in the list.
pub fn len(&self) -> usize { self.len }
/// True if the list has no elements.
pub fn is_empty(&self) -> bool { self.len == 0 }
/// Insert `elem` at the back of the list. `elem` must be pinned.
/// # Safety: `elem` must not already be in any IntrusiveList.
pub unsafe fn push_back(&mut self, elem: *mut T) { ... }
/// Insert `elem` at the front of the list.
/// # Safety: same as `push_back`.
pub unsafe fn push_front(&mut self, elem: *mut T) { ... }
/// Remove and return the front element, or `None` if empty.
pub fn pop_front(&mut self) -> Option<*mut T> { ... }
/// Remove and return the back element, or `None` if empty.
pub fn pop_back(&mut self) -> Option<*mut T> { ... }
/// Remove `elem` from the list (it must currently be in this list).
/// # Safety: `elem` must be in this list.
pub unsafe fn remove(&mut self, elem: *mut T) { ... }
/// Iterate over elements in order (front to back).
pub fn iter(&self) -> IntrusiveListIter<'_, T> { ... }
/// Move all elements from `other` to the back of `self`. O(1).
/// Re-links the moved boundary elements to `self`'s sentinel and resets
/// `other` to the empty null-sentinel state (leaving `other` movable).
/// No-op if `other` is empty.
pub fn splice_back(&mut self, other: &mut Self) { ... }
}
/// Iterator over an `IntrusiveList`. Elements are yielded as raw pointers.
pub struct IntrusiveListIter<'a, T: HasIntrusiveLink<T>> {
current: *mut IntrusiveLink<T>,
sentinel: *const IntrusiveLink<T>,
_lifetime: PhantomData<&'a T>,
}
/// Type alias: a bare, type-erased link node embedded in element structs. The
/// `IntrusiveLink<()>` layout is identical for every element type (the type
/// parameter is a `PhantomData` marker only), so a struct embeds one
/// `IntrusiveListNode` and its `HasIntrusiveLink<Self>` impl projects the node as
/// `IntrusiveLink<Self>`. Used corpus-wide (`WaitQueueEntry.link`,
/// `MutexWaiter.node`, `RwWaiter.node`, `Dentry.d_sibling`, `Page.lru`, …).
///
/// There is deliberately NO `IntrusiveList<()>` head alias: `IntrusiveList<T>`
/// requires `T: HasIntrusiveLink<T>`, which `()` cannot satisfy, so the form is
/// ill-formed. A list is always typed by its real element
/// (`IntrusiveList<MutexWaiter>`, `IntrusiveList<WaitQueueEntry>`,
/// `IntrusiveList<Dentry>`), even though the embedded node stays type-erased.
pub type IntrusiveListNode = IntrusiveLink<()>;
3.5.1.1.6 Mutex<T>¶
A sleeping lock. The caller blocks (is descheduled) if the lock is contended.
Must not be acquired from interrupt context or while holding a SpinLock or
RawSpinLock (sleeping under a spinlock causes a deadlock — the spinner cannot
be scheduled out, but the sleeping thread cannot progress). Equivalent to Linux
struct mutex.
/// Sleeping mutual exclusion lock. Blocks (deschedules) on contention.
///
/// # Contexts
/// - Safe to acquire from normal kernel task context.
/// - MUST NOT be acquired from interrupt context (hardirq, softirq, NMI).
/// - MUST NOT be acquired while holding a `SpinLock` or `RawSpinLock`.
/// Doing so would put the spinlock holder to sleep — the deadlock detector
/// ([Section 3.4](#cumulative-performance-budget), lock level ordering) treats Mutex as higher-level than
/// SpinLock in the lock hierarchy.
pub struct Mutex<T> {
/// Lock state: 0 = unlocked, 1 = locked (no waiters), 2 = locked (waiters present).
state: AtomicU32,
/// List of tasks blocked waiting for this mutex. Protected by `waiters_lock`.
/// Typed `IntrusiveList<MutexWaiter>` — the same element-typed pattern as
/// `RwLock.waiters: IntrusiveList<RwWaiter>` and `WaitQueueHead`. Each
/// `MutexWaiter` embeds a type-erased `node: IntrusiveListNode`
/// (= `IntrusiveLink<()>`, layout-identical for every element type); the
/// `MutexWaiter: HasIntrusiveLink<MutexWaiter>` impl projects that node, so
/// `pop_front()` yields `*mut MutexWaiter` directly (the list performs the
/// `container_of` recovery internally). No `IntrusiveList<()>`: `()` cannot
/// satisfy the `HasIntrusiveLink<()>` element bound, so that form is ill-formed.
waiters: IntrusiveList<MutexWaiter>,
/// Spinlock protecting the `waiters` list. Held only briefly during enqueue/dequeue.
waiters_lock: RawSpinLock,
data: UnsafeCell<T>,
}
/// Task node embedded in `Mutex::waiters` when a task is blocked.
pub struct MutexWaiter {
task: *mut Task,
node: IntrusiveListNode,
}
/// SAFETY: `MutexWaiter` embeds exactly one link hook — the type-erased
/// `node: IntrusiveListNode` (= `IntrusiveLink<()>`, layout-identical for every
/// element type since the type parameter is a `PhantomData` marker only). The
/// impl projects that node as `IntrusiveLink<MutexWaiter>` so the list can
/// yield `*mut MutexWaiter` directly. Each `MutexWaiter` lives in the blocked
/// task's stack frame and is pinned there for the whole time it is linked into
/// `Mutex::waiters` (it is unlinked before the frame unwinds), so the link
/// address is stable for its linked lifetime (the `HasIntrusiveLink`
/// move-safety contract). `from_link` is the exact inverse of `link`.
unsafe impl HasIntrusiveLink<MutexWaiter> for MutexWaiter {
fn link(this: *mut MutexWaiter) -> *mut IntrusiveLink<MutexWaiter> {
// SAFETY: `this` is a valid `MutexWaiter`; project to its type-erased
// `node` field and reinterpret it as `IntrusiveLink<MutexWaiter>`
// (`IntrusiveLink<()>` and `IntrusiveLink<MutexWaiter>` are
// layout-identical — see `IntrusiveListNode`).
unsafe { &raw mut (*this).node as *mut IntrusiveLink<MutexWaiter> }
}
unsafe fn from_link(link: *mut IntrusiveLink<MutexWaiter>) -> *mut MutexWaiter {
// SAFETY: `link` is the `node` field of a live `MutexWaiter`; subtract
// the field offset to recover the container (container_of).
let offset = core::mem::offset_of!(MutexWaiter, node);
unsafe { (link as *mut u8).sub(offset) as *mut MutexWaiter }
}
}
/// Guard returned by `Mutex::lock()`. Releases lock on drop.
pub struct MutexGuard<'a, T> {
mutex: &'a Mutex<T>,
}
impl<T> Mutex<T> {
/// Construct an unlocked mutex owning `data`. `const` — `Mutex` appears
/// in `static` initializers (e.g. `EXT4_MOUNTS`, `BTRFS_MOUNTS`,
/// `TFM_REGISTRY.lock`); the wait list starts in the null-sentinel empty
/// state (see `IntrusiveList::new()`).
pub const fn new(data: T) -> Self {
Mutex {
state: AtomicU32::new(0),
waiters: IntrusiveList::new(),
waiters_lock: RawSpinLock::new(),
data: UnsafeCell::new(data),
}
}
/// Acquire the lock. Blocks if contended. Returns when the lock is held exclusively.
pub fn lock(&self) -> MutexGuard<'_, T>;
/// Non-blocking acquire. Returns `Some(guard)` if acquired, `None` if contended.
pub fn try_lock(&self) -> Option<MutexGuard<'_, T>>;
}
impl<T> Deref for MutexGuard<'_, T> {
type Target = T;
}
impl<T> DerefMut for MutexGuard<'_, T> {}
impl<T> Drop for MutexGuard<'_, T> {
fn drop(&mut self) {
// Fast path: CAS state 1 → 0 (no waiters).
if self.mutex.state
.compare_exchange(1, 0, Ordering::Release, Ordering::Relaxed)
.is_ok()
{
return;
}
// Slow path: state == 2 (waiters present).
// 1. Acquire waiters_lock (lock level 25, below RQ_LOCK(50)).
// SAFETY: IRQ state managed by the caller; waiters_lock is held briefly.
unsafe { self.mutex.waiters_lock.lock() };
// 2. Dequeue the first waiter from the list.
// `pop_front()` returns `Option<*mut MutexWaiter>` — the typed list
// recovers the containing `MutexWaiter` from its embedded node via
// `HasIntrusiveLink::from_link` (container_of) internally.
let waiter_opt = self.mutex.waiters.pop_front();
// 3. Update state while still holding waiters_lock:
// - If list is now empty: state = 0 (no waiters, unlocked).
// - If list still has waiters: leave state at 2 (waiters present,
// but lock is now logically available for the woken task).
if self.mutex.waiters.is_empty() {
self.mutex.state.store(0, Ordering::Release);
}
// State remains 2 if waiters exist — the woken task will CAS 2→1
// (or 2→2 if more waiters arrive concurrently) on wakeup.
// 4. Release waiters_lock BEFORE calling scheduler::unblock().
// This avoids holding waiters_lock across scheduler code (which
// acquires RQ_LOCK at level 50 — higher than waiters_lock at 25).
unsafe { self.mutex.waiters_lock.unlock() };
// 5. Wake the dequeued waiter OUTSIDE waiters_lock.
// The woken task's lock() path handles the race where the lock
// has been re-acquired by another task between steps 4 and 5
// (CAS loop on state in the lock path).
if let Some(waiter) = waiter_opt {
// SAFETY: `waiter` is the `*mut MutexWaiter` the typed list recovered
// (container_of on the embedded node) while `waiters_lock` was held.
// The MutexWaiter is stack-allocated by the waiting task and remains
// valid until that task returns from `Mutex::lock()`; the task
// pointer is valid until the task is freed.
unsafe { scheduler::unblock((*waiter).task) };
}
}
}
3.5.1.1.7 RwLock<T>¶
A sleeping reader-writer lock. Multiple concurrent readers OR one exclusive writer. New readers are blocked when a writer is waiting (writer preference — prevents writer starvation). Must not be acquired from interrupt context.
/// Sleeping reader-writer lock. Many readers OR one writer. Writer-preferring.
///
/// # Contexts
/// Same restrictions as `Mutex<T>`: task context only, not under a SpinLock.
///
/// # Writer preference
/// When a writer is waiting, new `read_lock()` calls block. This prevents writer
/// starvation at the cost of reduced read throughput under write pressure.
pub struct RwLock<T> {
/// Packed state word:
/// bit 31 = WRITER_HELD (a writer currently holds the lock)
/// bit 30 = WRITER_WAITING (at least one writer is enqueued in `waiters`)
/// bits [29:0] = active reader count
///
/// Readers check `state & (WRITER_HELD | WRITER_WAITING)` atomically before
/// incrementing the reader count. If either bit is set, the reader takes the
/// slow path (enqueues on the waiters list and sleeps). This provides writer
/// preference without requiring readers to acquire `waiters_lock` on the fast
/// path — the WRITER_WAITING bit is set/cleared by writers under `waiters_lock`
/// and is visible to readers via the atomic state word.
///
/// Reader count limit: 2^30 - 1 (1,073,741,823). Safe because reader count
/// is bounded by the number of schedulable tasks (max ~64K per NUMA node in
/// practice). Exceeding this wraps into the writer-waiting bit, causing
/// incorrect lock behavior. `mem::forget` on `RwLockReadGuard` leaks the
/// reader count — this is a programming error, not an operational risk.
/// Debug builds: assert!(reader_count < (1 << 30)) in read() acquisition.
///
/// **Longevity analysis**: 2^30 overflow requires ~1.07 billion leaked read guards.
/// At one leaked guard per second (sustained `mem::forget` bug), overflow occurs
/// after ~34 years. This is a programming error scenario, not an operational risk --
/// the 50-year uptime target applies to correct programs. The debug assertion
/// catches the leak during development. A saturating check in release builds
/// (one compare-and-branch in `read()`) is a viable future hardening option but
/// is not included in the initial design due to cost on a sleeping-lock fast path.
/// Exempt from u64 widening: this is a bounded refcount (not a monotonic identifier).
state: AtomicU32,
/// Queued readers and writers. Each entry carries a `WaiterKind` tag.
waiters: IntrusiveList<RwWaiter>,
/// Protects the `waiters` list.
waiters_lock: RawSpinLock,
data: UnsafeCell<T>,
}
/// RwLock state word bit layout constants.
const RWLOCK_WRITER_HELD: u32 = 1 << 31;
const RWLOCK_WRITER_WAITING: u32 = 1 << 30;
const RWLOCK_READER_MASK: u32 = (1 << 30) - 1;
pub enum WaiterKind { Reader, Writer }
pub struct RwWaiter {
kind: WaiterKind,
task: *mut Task,
node: IntrusiveListNode,
}
/// Guard for shared read access. Multiple `RwLockReadGuard`s can coexist.
pub struct RwLockReadGuard<'a, T> {
lock: &'a RwLock<T>,
}
/// Guard for exclusive write access. Uniquely held.
pub struct RwLockWriteGuard<'a, T> {
lock: &'a RwLock<T>,
}
impl<T> RwLock<T> {
/// Construct an unlocked lock owning `data`. `const` — legal in `static`
/// initializers; the wait list starts in the null-sentinel empty state
/// (see `IntrusiveList::new()`).
pub const fn new(data: T) -> Self {
RwLock {
state: AtomicU32::new(0),
waiters: IntrusiveList::new(),
waiters_lock: RawSpinLock::new(),
data: UnsafeCell::new(data),
}
}
/// Acquire shared read access. Blocks if a writer holds or is waiting for the lock.
pub fn read(&self) -> RwLockReadGuard<'_, T>;
/// Acquire exclusive write access. Blocks until all readers and the current
/// writer (if any) release the lock.
pub fn write(&self) -> RwLockWriteGuard<'_, T>;
/// Non-blocking read acquire.
pub fn try_read(&self) -> Option<RwLockReadGuard<'_, T>>;
/// Non-blocking write acquire.
pub fn try_write(&self) -> Option<RwLockWriteGuard<'_, T>>;
}
3.5.1.1.8 RwLockInner¶
Reader-writer bookkeeping state carrying no data payload — the reader
count, writer flag, and waiter list of an RwLock, factored out for use as
the payload T of a leveled Lock<RwLockInner, LEVEL>
(Section 3.4). This is the pattern used
where read access is served by RCU and only writers require mutual exclusion
plus compile-time level ordering — e.g. PROCESS_TREE_WRITE_LOCK
(Section 8.1), whose readers walk RCU-published state and
whose writers take the leveled lock over this inner state. The protected data
lives elsewhere (RCU-published); only the acquisition bookkeeping lives here.
/// Reader-writer acquisition state with no data payload. Layout mirrors the
/// state word / waiter list of `RwLock<T>` above.
pub struct RwLockInner {
/// Packed state word: bit 31 = writer held, bit 30 = writer waiting,
/// bits [29:0] = active reader count (same layout as `RwLock<T>`).
state: AtomicU32,
/// Queued readers and writers awaiting acquisition.
waiters: IntrusiveList<RwWaiter>,
/// Protects `waiters`.
waiters_lock: RawSpinLock,
}
impl RwLockInner {
/// Construct an unlocked inner state (zero readers, no writer, empty wait
/// list). `const` so it can initialize a `static Lock<RwLockInner, LEVEL>`.
/// The wait list starts in the null-sentinel empty state — constness rests
/// on the deferred-sentinel contract of `IntrusiveList::new()`.
pub const fn new() -> Self {
RwLockInner {
state: AtomicU32::new(0),
waiters: IntrusiveList::new(),
waiters_lock: RawSpinLock::new(),
}
}
}
3.5.1.1.9 Condvar¶
A condition variable for Mutex-guarded condition waiting. Wraps a WaitQueueHead
and provides atomic mutex-release-and-sleep semantics that prevent lost-wakeup races.
Equivalent to Linux's wait_event pattern but paired with Mutex<T> rather than
a spinlock-protected wait queue.
/// Condition variable paired with `Mutex<T>`. Allows a thread to atomically
/// release a mutex and sleep until a condition becomes true.
///
/// Built on `WaitQueueHead` ([Section 3.6](#lock-free-data-structures--waitqueuehead-blocking-wait-queue)):
/// the condition variable IS a WaitQueueHead with Mutex-aware wait/notify
/// methods. The underlying WaitQueueHead's spinlock-protected waiter list
/// provides the atomicity guarantee between mutex release and sleep.
///
/// # Contexts
/// - `wait()` and `wait_interruptible()` MUST NOT be called from interrupt
/// context (they sleep). Same restrictions as `Mutex<T>`.
/// - `notify_one()` and `notify_all()` may be called from any context
/// (including interrupt context), same as `WaitQueueHead::wake_up()`.
pub struct Condvar {
wq: WaitQueueHead,
}
impl Condvar {
/// Create a new condition variable.
pub const fn new() -> Self {
Self { wq: WaitQueueHead::new() }
}
/// Release the mutex, sleep until notified, then re-acquire the mutex.
///
/// Protocol:
/// 1. Enqueue current task on `self.wq` as an exclusive waiter.
/// 2. Drop `guard` (releases the mutex — Mutex::state transitions).
/// 3. `schedule()` — task sleeps.
/// 4. On wakeup: re-acquire the mutex via `mutex.lock()`.
/// 5. Return the new `MutexGuard`.
///
/// The enqueue (step 1) happens while the mutex is still held, and the
/// WaitQueueHead's internal spinlock is held during the enqueue+state-change
/// sequence. This prevents the lost-wakeup race: if `notify_one()` is called
/// between steps 1 and 2, the waiter is already enqueued and will be woken.
pub fn wait<'a, T>(&self, guard: MutexGuard<'a, T>) -> MutexGuard<'a, T> {
let mutex = guard.mutex;
// Protocol:
// 1. Lock self.wq.lock (WQ internal spinlock).
// 2. Enqueue current task as exclusive waiter on self.wq.
// 3. Set task state to `TaskState::UNINTERRUPTIBLE`.
// 4. Unlock self.wq.lock.
// 5. Drop guard → releases Mutex (MutexGuard::drop transitions state).
// The WQ enqueue (step 2) happens BEFORE mutex release (step 5),
// so a concurrent notify_one() between steps 4 and 5 will find
// our waiter on the queue and set us `TaskState::RUNNING` before we sleep.
// 6. schedule() → suspend until notify_one/notify_all wakes us.
// 7. On wakeup: re-acquire mutex via mutex.lock().
//
// NOTE: Unlike wait_event(), Condvar::wait() does NOT take a condition
// closure. The condition check is the CALLER's responsibility after
// re-acquiring the mutex. Standard usage pattern:
// let mut guard = mutex.lock();
// while !condition(&*guard) {
// guard = condvar.wait(guard);
// }
// // condition is true and mutex is held
//
// Implementation:
self.wq.prepare_to_wait_exclusive(); // steps 1-4
core::mem::drop(guard); // step 5: release mutex
schedule(); // step 6: sleep
finish_wait(&self.wq); // remove from WQ if not already
mutex.lock() // step 7: re-acquire mutex
}
/// Like `wait()`, but returns `Err(KernelError::Interrupted)` if the
/// sleeping task receives a signal. The mutex is always re-acquired
/// before returning (even on signal interruption).
pub fn wait_interruptible<'a, T>(
&self,
guard: MutexGuard<'a, T>,
) -> Result<MutexGuard<'a, T>, KernelError> {
let mutex = guard.mutex;
// Same protocol as wait() but with TASK_INTERRUPTIBLE.
self.wq.prepare_to_wait_exclusive_interruptible();
core::mem::drop(guard);
schedule();
let interrupted = signal_pending(current_task());
finish_wait(&self.wq);
let new_guard = mutex.lock();
if interrupted {
Err(KernelError::Interrupted)
} else {
Ok(new_guard)
}
}
/// Wake one waiter (the highest-priority exclusive waiter).
pub fn notify_one(&self) {
self.wq.wake_up_one();
}
/// Wake all waiters.
pub fn notify_all(&self) {
self.wq.wake_up_all();
}
}
3.5.1.1.10 Lock Hierarchy Summary¶
The four types form a strict containment hierarchy. A thread may hold a higher-level lock while acquiring a lower-level one, but not the reverse:
| Type | Sleeps? | IRQ-safe? | Can hold while acquiring... |
|---|---|---|---|
RawSpinLock |
No | Caller-managed | (nothing lower) |
SpinLock<T> |
No | Yes (saves/restores) | RawSpinLock |
Mutex<T> |
Yes | No (task context only) | RawSpinLock, SpinLock<T> |
RwLock<T> |
Yes | No (task context only) | RawSpinLock, SpinLock<T> |
Critical constraint: holding a Mutex or RwLock guard and then acquiring a
SpinLock is permitted (Mutex is at a higher conceptual level and does not spin).
The reverse — acquiring a SpinLock and then calling Mutex::lock() — would put
the spinlock holder to sleep, violating the spinlock contract. The const-generic
lock level system (Section 3.4) enforces this at compile time by assigning
SpinLock-backed locks to lower numeric levels than Mutex-backed locks.
Lock hierarchy migration protocol: When a subsystem needs to change which lock
protects a data structure (e.g., replacing a SpinLock with a Mutex or splitting
a coarse lock into per-object fine-grained locks), the following protocol applies:
1. Both old and new locks must co-exist during the transition. The new lock is
introduced alongside the old one; all readers acquire both (old then new).
2. Writers are migrated one call site at a time. Each call site is updated to
acquire the new lock instead of the old; the old lock acquisition is removed.
3. Once all call sites use the new lock, the old lock is removed in a separate
commit. This ensures bisectability.
4. During the co-existence period, the lock ordering rule is: old lock must always
be acquired before the new lock (never reversed), to prevent deadlocks.
5. The const-generic lock level for the new lock must be assigned a level that
does not conflict with existing hierarchies. The lock_level_check! macro
validates this at compile time.
3.5.2 Preemption and Interrupt Context Model¶
UmkaOS uses separate per-CPU fields for preemption depth and interrupt context,
stored in CpuLocalBlock (see Section 3.2).
This differs from Linux's packed preempt_count u32 (which encodes preemption
depth, softirq count, hardirq count, and NMI state in a single word with bit-field
packing). UmkaOS uses separate fields for clarity and type safety:
/// In CpuLocalBlock (see cpulocal section for full struct):
///
/// preempt_count: u32 — preemption-disable nesting depth only.
/// Incremented by preempt_disable() / SpinLock::lock().
/// Decremented by preempt_enable() / SpinLock::unlock().
///
/// irq_count: u32 — hardirq nesting count.
/// Incremented by irq_enter().
/// Decremented by irq_exit().
///
/// softirq_count: u32 — softirq (bottom-half) nesting count.
/// Incremented by local_bh_disable().
/// Decremented by local_bh_enable().
///
/// need_resched: AtomicBool — set by IPI/scheduler tick when a reschedule is pending.
/// Checked on preempt_enable() and return-from-interrupt.
/// AtomicBool because IPIs write from remote CPUs (see §3.1.2).
The preemptibility check is preempt_count == 0 && irq_count == 0 && softirq_count == 0
(three loads, all from the same cache line in CpuLocalBlock). The in_interrupt() check
is irq_count > 0 || softirq_count > 0. The in_softirq() check is softirq_count > 0.
UmkaOS uses separate typed fields instead of Linux's packed preempt_count bitfield — see
Section 3.2.
Why not Linux's packed format? Linux packs everything into one u32 so that the
return-from-interrupt fast path can test preempt_count == 0 as a single branch.
UmkaOS trades that single-compare trick for separate typed fields that are easier
to reason about, debug, and extend. The two-field check is still a single cache line
access and adds at most one extra compare instruction — negligible on modern
out-of-order CPUs.
BPF compatibility: BPF programs and tracing tools that read Linux's packed
preempt_count are handled by the BPF helper layer, which synthesizes the expected
packed format from the separate fields when accessed via bpf_get_preempt_count().
NMI tracking: NMI state is tracked via a separate in_nmi: bool field in
CpuLocalBlock (not bit-packed). At most one NMI can be active per CPU
(non-nestable on all supported architectures).
/// Query helpers — FUSED field-scoped reads of per-CPU `CpuLocalBlock` fields.
/// No whole-block reference is formed (`get()` is deleted, ESC-0431): the plain
/// counters are read via the offset-primitive family (sound as-is — an asm
/// single-instruction access, not a Rust reference spanning the field, per the
/// ESC-0431 "counters stay plain" ruling), and the atomic `in_nmi` flag via its
/// single-field `&'static AtomicBool` projection.
/// True if executing in any interrupt context (hardirq or softirq).
/// Used by sleeping-lock debug assertions (sleeping in interrupt = bug).
/// Matches Linux's `in_interrupt()` which checks both HARDIRQ_MASK and
/// SOFTIRQ_MASK bits in the packed preempt_count.
#[inline(always)]
pub fn in_interrupt() -> bool {
// Fused field-scoped reads of the plain u32 counters — no whole-block ref.
unsafe {
arch::current::cpu::cpu_local_read_u32::<{ core::mem::offset_of!(CpuLocalBlock, irq_count) }>() > 0
|| arch::current::cpu::cpu_local_read_u32::<{ core::mem::offset_of!(CpuLocalBlock, softirq_count) }>() > 0
}
}
/// True if preemption is currently enabled (preempt depth == 0 and
/// not in any interrupt context). Does NOT check need_resched.
/// All three fields must be zero: preempt nesting, hardirq depth, and
/// softirq depth. UmkaOS checks these as separate fields; Linux checks
/// `preempt_count() == 0` where the packed word includes all three.
#[inline(always)]
pub fn preemptible() -> bool {
// Fused field-scoped reads of the plain u32 counters — no whole-block ref.
unsafe {
arch::current::cpu::cpu_local_read_u32::<{ core::mem::offset_of!(CpuLocalBlock, preempt_count) }>() == 0
&& arch::current::cpu::cpu_local_read_u32::<{ core::mem::offset_of!(CpuLocalBlock, irq_count) }>() == 0
&& arch::current::cpu::cpu_local_read_u32::<{ core::mem::offset_of!(CpuLocalBlock, softirq_count) }>() == 0
}
}
/// True if executing in NMI context.
#[inline(always)]
pub fn in_nmi() -> bool {
// Single-field `&'static AtomicBool` projection — sound in any context
// (the field is NMI-written; the whole-block reference `get()` used here
// before is deleted, ESC-0431).
CpuLocal::in_nmi().load(Relaxed)
}
Interaction with locking primitives:
| Primitive | Effect on CpuLocalBlock fields |
|---|---|
RawSpinLock::lock() |
preempt_count += 1 |
RawSpinLock::unlock() |
preempt_count -= 1 + check need_resched |
SpinLock::lock() |
Saves IRQ flags, disables IRQs, preempt_count += 1 |
SpinLock::unlock() |
preempt_count -= 1, restores IRQ flags, check need_resched |
irq_enter() |
irq_count += 1 |
irq_exit() |
irq_count -= 1 + process pending softirqs if irq_count == 0 && softirq_count == 0 |
local_bh_disable() |
softirq_count += 1 |
local_bh_enable() |
softirq_count -= 1 + run pending softirqs if softirq_count == 0 && irq_count == 0 |
Mutex::lock() / RwLock::read() |
No change (these sleep, preempt must be enabled) |
The preempt_enable() decrement is the primary voluntary preemption point: after
decrementing, if preempt_count == 0 && irq_count == 0 && softirq_count == 0 and
need_resched is set, schedule() is called immediately. This is how preemptive
multitasking works in UmkaOS: every SpinLock::unlock() and preempt_enable() is an
implicit preemption check. The need_resched flag consumed here is the per-CPU
Eager MIRROR (the per-task TIF bits are the authoritative request); schedule()
itself clears both at its step 2a — full setter/consumer/clear inventory:
Section 7.1. UmkaOS uses
separate irq_count (hardirq) and softirq_count (BH) fields in CpuLocalBlock
instead of Linux's packed bitfield
— see Section 3.2.
3.5.3 Lock Contention Tracking¶
Lock contention is the primary scalability bottleneck on systems with many cores. UmkaOS provides lock contention instrumentation for debugging and performance analysis, integrated with the tracepoint subsystem (Section 20.2) and FMA (Section 20.1).
Tracepoints — two static tracepoints on every Lock<T, LEVEL>:
/// Emitted when a thread begins waiting for a contended lock.
/// `lock_addr`: address of the lock. `level`: compile-time lock level.
/// `caller`: return address of the `lock()` call site.
tracepoint!(lock_contention_begin, lock_addr: usize, level: u32, caller: usize);
/// Emitted when the thread acquires the lock (contention resolved).
/// `wait_ns`: nanoseconds spent waiting. Zero means the lock was
/// acquired on the first attempt (no contention — tracepoint still
/// fires for consistent tracing, but can be filtered by `wait_ns > 0`).
tracepoint!(lock_contention_end, lock_addr: usize, level: u32, wait_ns: u64);
These tracepoints are compiled in unconditionally but have zero cost when no tracer is attached (static branch, same as all UmkaOS tracepoints). When a tracer is attached, overhead is <1% of the contended lock path (Section 3.4). In production deployments where tracepoint overhead is unacceptable, lock contention tracing can be disabled at runtime via the tracepoint enable/disable mechanism.
3.5.3.1 64-bit Atomics on 32-bit Legs — the Semantic Family¶
The load-bearing rule: there is NO single sound 32-bit emulation of
AtomicU64. The correct fallback depends entirely on the USE SEMANTICS — a
torn-tolerant hint, a lossless counter, and a tear-free correctness value need
three DIFFERENT protocols. The spec therefore defines a THREE-MEMBER semantic
family, all cfg-split on target_has_atomic = "64". The compiler predicate IS
the architecture abstraction here: no arch/*/ module is involved, because
"does this target have a native 64-bit atomic" is exactly what the predicate
answers (true on all 64-bit legs and ARMv7-A via LDREXD/STREXD; false only on
PPC32 among supported targets).
/// Bounded retry count for the 32-bit torn-read loops below.
const ATOMIC_U64_READ_RETRIES: usize = 8;
/// **AtomicU64Cell — wait-free, torn-tolerant load/store.** On
/// `target_has_atomic = "64"` a transparent `AtomicU64`; on 32-bit legs a lo/hi
/// `AtomicU32` pair: `store` = `hi` (Relaxed) then `lo` (caller's ordering);
/// `load` = `lo` (caller's ordering) + `hi` (Relaxed). A reader in the carry
/// window may observe a value off by up to 2^32 — EVERY consumer MUST document
/// its tear tolerance at the use site. Wait-free on both legs; both variants
/// are exactly 8 bytes.
// kernel-internal, not KABI
#[cfg(target_has_atomic = "64")]
#[repr(transparent)]
pub struct AtomicU64Cell(AtomicU64);
#[cfg(not(target_has_atomic = "64"))]
#[repr(C)]
pub struct AtomicU64Cell {
/// Low 32 bits. Placed first so a consumer with an entry-stub offset
/// constraint (e.g. `CpuLocalU64`) reads the bounded low half first.
lo: AtomicU32,
hi: AtomicU32,
}
#[cfg(target_has_atomic = "64")]
impl AtomicU64Cell {
pub const fn new(v: u64) -> Self { Self(AtomicU64::new(v)) }
pub fn load(&self, order: Ordering) -> u64 { self.0.load(order) }
pub fn store(&self, v: u64, order: Ordering) { self.0.store(v, order) }
}
#[cfg(not(target_has_atomic = "64"))]
impl AtomicU64Cell {
pub const fn new(v: u64) -> Self {
Self { lo: AtomicU32::new(v as u32), hi: AtomicU32::new((v >> 32) as u32) }
}
/// Store `hi` (Relaxed) then `lo` with the caller's ordering as the release
/// point. Torn-tolerant.
pub fn store(&self, v: u64, order: Ordering) {
self.hi.store((v >> 32) as u32, Ordering::Relaxed);
self.lo.store(v as u32, order);
}
/// Load `lo` with the caller's ordering, `hi` Relaxed. Torn-tolerant.
pub fn load(&self, order: Ordering) -> u64 {
let lo = self.lo.load(order) as u64;
let hi = self.hi.load(Ordering::Relaxed) as u64;
(hi << 32) | lo
}
}
const_assert!(core::mem::size_of::<AtomicU64Cell>() == 8);
/// **AtomicU64Counter — lossless add/sub accumulator** (`add`/`sub`/`load`/
/// `reset`, plus a RACY-tolerated `fetch_max`). On `target_has_atomic = "64"` a
/// transparent `AtomicU64`; on 32-bit legs a lo/hi `AtomicU32` pair with a
/// carry-propagating `add` — no increment is EVER lost, though a reader may
/// observe a value that transiently deviates from the true total in EITHER
/// direction, bounded by 2^32, between a half update and the carry/borrow
/// propagation. `add` can read transiently LOW (low half wrapped up, high-half
/// carry not yet propagated). `sub` is the same carry-propagating protocol
/// applied to `wrapping_neg(delta)` (subtraction mod 2^64), so it can read
/// transiently HIGH by up to 2^32: the low half is written before the high half
/// decrements — e.g. `0x1_0000_0005 - 10` momentarily reads `0x1_FFFF_FFFB`
/// (hi still `1`, lo already `0xFFFF_FFFB`) before the high half drops to `0`,
/// settling at the correct `0xFFFF_FFFB`. Consumers MUST tolerate a bounded
/// transient error in BOTH directions. The stored value is a sum of signed
/// deltas that is never negative by the caller's usage invariant.
///
/// **Contract split (normative)**: only `add` is LOSSLESS. `fetch_max` on the
/// 32-bit leg raises the value with two plain stores (hi then lo) and is
/// RACY-TOLERATED — it may interleave with a concurrent `add` or another
/// `fetch_max` and momentarily lose or mix an update. Use `fetch_max` ONLY for
/// debug-hint consumers (exactly the original `DebugStatU64::update_max`
/// contract); a value that needs BOTH exact accumulation and an exact maximum
/// on a 32-bit leg must not mix the two on this type. Both variants 8 bytes.
// kernel-internal, not KABI
#[cfg(target_has_atomic = "64")]
#[repr(transparent)]
pub struct AtomicU64Counter(AtomicU64);
#[cfg(not(target_has_atomic = "64"))]
#[repr(C)]
pub struct AtomicU64Counter {
lo: AtomicU32,
hi: AtomicU32,
}
#[cfg(target_has_atomic = "64")]
impl AtomicU64Counter {
pub const fn new(v: u64) -> Self { Self(AtomicU64::new(v)) }
pub fn add(&self, delta: u64) { self.0.fetch_add(delta, Ordering::Relaxed); }
/// Subtract `delta`. Never drives the value negative by the caller's usage
/// invariant (the value is a running sum of signed deltas that stays >= 0).
pub fn sub(&self, delta: u64) { self.0.fetch_sub(delta, Ordering::Relaxed); }
pub fn fetch_max(&self, candidate: u64) { self.0.fetch_max(candidate, Ordering::Relaxed); }
pub fn load(&self) -> u64 { self.0.load(Ordering::Relaxed) }
pub fn reset(&self) -> u64 { self.0.swap(0, Ordering::Relaxed) }
}
#[cfg(not(target_has_atomic = "64"))]
impl AtomicU64Counter {
pub const fn new(v: u64) -> Self {
Self { lo: AtomicU32::new(v as u32), hi: AtomicU32::new((v >> 32) as u32) }
}
/// Carry-propagating add: the low-half `fetch_add` detects carry-out from
/// its returned previous value and propagates it with a high-half
/// `fetch_add`. No increment is ever lost.
pub fn add(&self, delta: u64) {
let old_lo = self.lo.fetch_add(delta as u32, Ordering::Relaxed);
let carry = ((old_lo as u64 + (delta & 0xFFFF_FFFF)) >> 32) as u32;
// wrapping_add: the hi half wraps mod 2^32 exactly as a native
// u64::fetch_add would, so a hi-half wrap is CORRECT, not a panic.
let hi_add = carry.wrapping_add((delta >> 32) as u32);
if hi_add != 0 {
self.hi.fetch_add(hi_add, Ordering::Relaxed);
}
}
/// Subtract `delta`: the identical carry-propagating protocol as `add`
/// applied to `wrapping_neg(delta)` (value + (2^64 - delta) mod 2^64 =
/// value - delta mod 2^64). No borrow path is needed. The stored value is
/// a sum of signed deltas that is never negative by the usage invariant.
pub fn sub(&self, delta: u64) {
self.add(delta.wrapping_neg());
}
/// Raise to `max(current, candidate)` with two plain stores (hi then lo).
/// The monotone-max value may transiently mix halves; the additive path
/// stays exact regardless.
pub fn fetch_max(&self, candidate: u64) {
if candidate <= self.load() { return; }
self.hi.store((candidate >> 32) as u32, Ordering::Relaxed);
self.lo.store(candidate as u32, Ordering::Relaxed);
}
/// hi/lo/hi re-read loop (bounded retries); a torn read in the carry/borrow
/// window deviates from the true value by at most 2^32 in EITHER direction
/// (transiently LOW after an `add` carry-out, transiently HIGH after a `sub`
/// borrow — see the type-level contract above).
pub fn load(&self) -> u64 {
for _ in 0..ATOMIC_U64_READ_RETRIES {
let hi1 = self.hi.load(Ordering::Acquire);
let lo = self.lo.load(Ordering::Acquire);
let hi2 = self.hi.load(Ordering::Acquire);
if hi1 == hi2 {
return ((hi1 as u64) << 32) | lo as u64;
}
}
((self.hi.load(Ordering::Acquire) as u64) << 32)
| self.lo.load(Ordering::Acquire) as u64
}
/// Reset to zero, returning the previous value.
pub fn reset(&self) -> u64 {
let lo = self.lo.swap(0, Ordering::Relaxed) as u64;
let hi = self.hi.swap(0, Ordering::Relaxed) as u64;
(hi << 32) | lo
}
}
const_assert!(core::mem::size_of::<AtomicU64Counter>() == 8);
/// **AtomicU64Exact — tear-free load/store/RMW for CORRECTNESS values.** On
/// `target_has_atomic = "64"` a transparent `AtomicU64`; on 32-bit legs a
/// leaf-`SpinLock`-guarded `u64` — EVERY access (`load`/`store`/`fetch_add`)
/// takes the lock, so loads and stores are mutually serialized and no torn
/// value is ever observable. COST, stated honestly (32-bit legs): readers as
/// well as writers acquire the leaf lock (IRQ-saving), so a reader blocks while
/// a writer holds it and writers are NOT wait-free. Using `AtomicU64Exact` on a
/// HOT PATH on a 32-bit leg REQUIRES an explicit justification comment at the
/// use site.
///
/// **Finite-tag stall rule — class (iii) RETIRED**
/// ([Section 3.6](#lock-free-data-structures--the-finite-tag-stall-rule)): the value is
/// authoritative truth read under the leaf lock; there is no optimistic
/// generation/seqlock protocol on any leg, so no stall-window or tag-wrap
/// argument exists to be made.
// kernel-internal, not KABI
#[cfg(target_has_atomic = "64")]
#[repr(transparent)]
pub struct AtomicU64Exact(AtomicU64);
#[cfg(not(target_has_atomic = "64"))]
#[repr(C)]
pub struct AtomicU64Exact {
/// The 64-bit value, guarded by a leaf `SpinLock` on 32-bit legs. Finite-tag
/// stall rule class (iii) RETIRED: every access takes the lock; there is no
/// `gen`/seqlock and no optimistic reader retry. Kernel-internal, not KABI —
/// NOT offset- or ABI-critical.
val: SpinLock<u64>,
}
#[cfg(target_has_atomic = "64")]
impl AtomicU64Exact {
pub const fn new(v: u64) -> Self { Self(AtomicU64::new(v)) }
pub fn load(&self, order: Ordering) -> u64 { self.0.load(order) }
pub fn store(&self, v: u64, order: Ordering) { self.0.store(v, order) }
pub fn fetch_add(&self, delta: u64, order: Ordering) -> u64 {
self.0.fetch_add(delta, order)
}
}
#[cfg(not(target_has_atomic = "64"))]
impl AtomicU64Exact {
pub const fn new(v: u64) -> Self {
Self { val: SpinLock::new(v) }
}
/// Class (iii) read: take the leaf lock and read the authoritative value —
/// no optimistic retry. A SeqCst caller additionally gets an explicit
/// `fence(SeqCst)` BEFORE the read (the lock's Acquire alone does not place
/// the read in the global SeqCst order).
pub fn load(&self, order: Ordering) -> u64 {
if order == Ordering::SeqCst {
core::sync::atomic::fence(Ordering::SeqCst);
}
let guard = self.val.lock();
*guard
}
/// Class (iii) write under the leaf lock. A SeqCst caller additionally gets a
/// `fence(SeqCst)` AFTER the store.
pub fn store(&self, v: u64, order: Ordering) {
let mut guard = self.val.lock();
*guard = v;
if order == Ordering::SeqCst {
core::sync::atomic::fence(Ordering::SeqCst);
}
}
/// Tear-free RMW under the leaf lock; returns the previous value. A SeqCst
/// caller additionally gets a `fence(SeqCst)` AFTER the update (see `store`).
pub fn fetch_add(&self, delta: u64, order: Ordering) -> u64 {
let mut guard = self.val.lock();
let cur = *guard;
*guard = cur.wrapping_add(delta);
if order == Ordering::SeqCst {
core::sync::atomic::fence(Ordering::SeqCst);
}
cur
}
}
// The 64-bit variant is exactly 8 bytes (transparent AtomicU64). The 32-bit
// variant is a single leaf `SpinLock<u64>` (size arch-dependent) and is NOT
// offset- or ABI-critical — kernel-internal, never crosses a KABI/wire boundary.
#[cfg(target_has_atomic = "64")]
const_assert!(core::mem::size_of::<AtomicU64Exact>() == 8);
Classification rule (normative): every logically-64-bit atomic value in this
spec MUST be one of the three family members (or a documented canonical instance
below). Tear tolerance is a correctness judgment about the value's CONSUMERS —
it is made SPEC-SIDE, never builder-side. An unclassified bare AtomicU64
encountered by the builder on a non-target_has_atomic = "64" target is an
ESCALATION, not a self-fix: the builder owns only the MECHANISM legs and may
ship mechanism errata (the SL-stream pattern), but the tear-tolerance judgment
belongs to the spec.
Canonical instances: DebugStatU64 (below) IS the family's AtomicU64Counter
member with debugfs extras (reset-on-read, bounded read retries);
CpuLocalU64 (Section 3.2) IS the
AtomicU64Cell member with entry-stub offset constraints (lo placed first);
EarlyLogByteCounter (Section 2.3) IS the AtomicU64Counter
member's boot-time instance; LeAtomicU64
(Section 6.1) IS the endian-wrapped,
RDMA/MMIO-visible AtomicU64Cell variant — same target_has_atomic = "64"
cfg-split, load/store on all legs, compare_exchange/fetch_add on 64-bit legs
only and governed by the NIC-RMW-ownership rule. These existing types are NOT
renamed or re-protocoled — each simply carries a family-membership note pointing here.
This legislates away the defect class ESC-0320 / ESC-0406 (PPC32
max-atomic-width = Some(32), no core AtomicU64) — arch-named content is
acceptable here because it is per-arch content (ESC-0120 exemption).
Per-lock contention counters (debug builds only, #[cfg(debug_assertions)]):
/// Bounded retry count for the 32-bit torn-read loop in `DebugStatU64::load()`.
const DEBUG_STAT_READ_RETRIES: usize = 8;
/// 64-bit debug-statistics cell — the storage type of every
/// `LockContentionStats` field. It IS the `AtomicU64Counter` member of the
/// 64-bit-atomic semantic family
/// ([Section 3.5](#locking-strategy--64-bit-atomics-on-32-bit-legs-the-semantic-family))
/// with debugfs extras (reset-on-read, bounded read retries).
///
/// On architectures with native 64-bit atomics (`target_has_atomic = "64"`:
/// all 64-bit legs, plus ARMv7-A via LDREXD/STREXD) this is a transparent
/// `AtomicU64`. On 32-bit legs without native 64-bit atomics (PPC32 is the
/// only supported one) it is a lo/hi `AtomicU32` pair with the wait-free,
/// torn-tolerant protocol documented on each method.
///
/// **Why not the spinlock fallback used by the accelerator CBS fields**
/// ([Section 22.2](22-accelerators.md#accelerator-scheduler), FIX-030)? These cells are updated on the
/// contended-acquisition path of every `Lock<T, LEVEL>`; nesting a spinlock
/// inside lock instrumentation would need its own level assignment and adds
/// debug-build overhead to every contended acquire. CBS budgets are
/// correctness values and need the lock; contention statistics are
/// debug-only diagnostics with no correctness consumers, so a wait-free
/// cell that tolerates rare torn observations is the better fit.
#[cfg(target_has_atomic = "64")]
pub struct DebugStatU64(AtomicU64);
/// 32-bit-leg variant: lo/hi `AtomicU32` pair. See the 64-bit variant's doc.
#[cfg(not(target_has_atomic = "64"))]
pub struct DebugStatU64 {
/// Low 32 bits of the logical u64 value.
lo: AtomicU32,
/// High 32 bits of the logical u64 value.
hi: AtomicU32,
}
#[cfg(target_has_atomic = "64")]
impl DebugStatU64 {
/// Zero cell. `const` — the stats are embedded in `Lock<T, LEVEL>`
/// values, including `static` locks, in debug builds.
pub const fn new() -> Self { Self(AtomicU64::new(0)) }
/// Add `delta` to the cell.
pub fn add(&self, delta: u64) { self.0.fetch_add(delta, Ordering::Relaxed); }
/// Raise the cell to `max(current, candidate)`.
pub fn update_max(&self, candidate: u64) { self.0.fetch_max(candidate, Ordering::Relaxed); }
/// Read the cell.
pub fn load(&self) -> u64 { self.0.load(Ordering::Relaxed) }
/// Reset to zero, returning the previous value (debugfs reset-on-read).
pub fn reset(&self) -> u64 { self.0.swap(0, Ordering::Relaxed) }
}
#[cfg(not(target_has_atomic = "64"))]
impl DebugStatU64 {
/// Zero cell (see 64-bit variant).
pub const fn new() -> Self {
Self { lo: AtomicU32::new(0), hi: AtomicU32::new(0) }
}
/// Add `delta`. No increment is ever lost: the low-half `fetch_add`
/// detects carry-out from its returned previous value and propagates it
/// with a high-half `fetch_add`. A reader between the low-half wrap and
/// the carry propagation observes a transiently low value (see `load`).
pub fn add(&self, delta: u64) {
let old_lo = self.lo.fetch_add(delta as u32, Ordering::Relaxed);
// Carry into the high word: low-half overflow plus the delta's own
// high half. The sum is computed in u64 so it cannot itself overflow.
let carry = ((old_lo as u64 + (delta & 0xFFFF_FFFF)) >> 32) as u32;
// wrapping_add: the hi half wraps mod 2^32 exactly as a native
// u64::fetch_add would, so a hi-half wrap is CORRECT, not a panic.
let hi_add = carry.wrapping_add((delta >> 32) as u32);
if hi_add != 0 {
self.hi.fetch_add(hi_add, Ordering::Relaxed);
}
}
/// Raise the cell to `max(current, candidate)`. Racy by design: if the
/// candidate exceeds the current value, the halves are written with two
/// plain stores (hi then lo). Two concurrent writers may interleave,
/// leaving the cell a mix of both writers' halves — TOLERATED:
/// `max_wait_ns` is a debug hint read by humans via debugfs, and the
/// additive counters remain exact regardless.
pub fn update_max(&self, candidate: u64) {
if candidate <= self.load() {
return;
}
self.hi.store((candidate >> 32) as u32, Ordering::Relaxed);
self.lo.store(candidate as u32, Ordering::Relaxed);
}
/// Read via a hi/lo/hi re-read loop: retry while the high word moved
/// under the read. Bounded — after `DEBUG_STAT_READ_RETRIES` attempts the
/// (possibly torn) value is accepted. A torn read in the carry window is
/// off by at most 2^32 in EITHER direction (stale lo with new hi
/// over-reads; new lo with stale hi under-reads — ≈ 4.3 s of accumulated
/// wait time either way); readers are debugfs consumers, which tolerate this.
pub fn load(&self) -> u64 {
for _ in 0..DEBUG_STAT_READ_RETRIES {
let hi1 = self.hi.load(Ordering::Acquire);
let lo = self.lo.load(Ordering::Acquire);
let hi2 = self.hi.load(Ordering::Acquire);
if hi1 == hi2 {
return ((hi1 as u64) << 32) | lo as u64;
}
}
((self.hi.load(Ordering::Acquire) as u64) << 32)
| self.lo.load(Ordering::Acquire) as u64
}
/// Reset to zero, returning the previous value (debugfs reset-on-read).
/// An `add` racing the two swaps may be split between the returned value
/// and the freshly zeroed cell — never lost, at worst attributed to the
/// next sampling interval.
pub fn reset(&self) -> u64 {
let lo = self.lo.swap(0, Ordering::Relaxed) as u64;
let hi = self.hi.swap(0, Ordering::Relaxed) as u64;
(hi << 32) | lo
}
}
/// Per-lock debug statistics. Embedded in Lock<T, LEVEL> when debug
/// assertions are enabled. Not present in release builds (zero overhead).
/// The contended acquisition path updates the cells via
/// `contention_count.add(1)`, `total_wait_ns.add(wait_ns)`,
/// `max_wait_ns.update_max(wait_ns)`.
pub struct LockContentionStats {
/// Number of times this lock was acquired with contention (waited > 0 ns).
pub contention_count: DebugStatU64,
/// Cumulative wait time in nanoseconds.
pub total_wait_ns: DebugStatU64,
/// Maximum single wait time in nanoseconds.
pub max_wait_ns: DebugStatU64,
}
Exposed via /sys/kernel/debug/locks/<lock_name>/contention_count,
total_wait_ns, max_wait_ns (read with DebugStatU64::load(); reset on
read via reset()).
Torn-read tolerance (32-bit legs): the lo/hi protocol never loses an
increment (carry propagation is a second fetch_add, not a read-modify-write
of both halves), but a debugfs reader can transiently observe a value that is
low by up to 2^32 during a carry window, and max_wait_ns can mix the halves
of two concurrently observed maxima. This is acceptable because the fields are
#[cfg(debug_assertions)]-only diagnostics with no programmatic consumers —
nothing branches on them; they exist for humans reading debugfs. Correctness
values must NOT use DebugStatU64: warm-path counters use PerCpuCounter
(Section 3.1,
which carries its own 32-bit fallback note), and CBS budget fields use the
FIX-030 spinlock-protected fallback (Section 22.2).
Runtime lock ordering validation (debug builds only):
In debug builds, a per-CPU held_locks: ArrayVec<HeldLockEntry, 16> stack
tracks currently held lock addresses and levels:
/// Debug-only per-CPU lock tracking entry.
/// Stored in a per-CPU ArrayVec (max depth 16 — deepest observed nesting
/// in Linux is ~12; 16 provides headroom). IRQ context pushes onto the
/// same stack: since IRQ handlers acquire locks at levels higher than
/// any non-IRQ lock they nest inside, the ordering check remains valid.
#[cfg(debug_assertions)]
struct HeldLockEntry {
/// Address of the lock instance (for identification in diagnostics).
addr: usize,
/// Lock level (from the const generic LEVEL parameter).
level: u32,
}
On each lock acquisition, the runtime checker verifies that the new lock's
level is strictly greater than all currently held locks. Violation -> BUG()
with a diagnostic message showing the lock ordering chain. This is
equivalent to Linux's lockdep but leverages the compile-time level system --
the runtime check catches cases where the static checker cannot (e.g.,
locks acquired through dynamic dispatch).
The lock ordering system has ZERO escape hatches: no lock_read_unchecked(),
no compile-time call-site caps, no read-vs-write mode tracking. Every lock
acquisition -- read or write -- must satisfy the strictly-ascending level
invariant. This was made possible by replacing the former invalidation rwsem
that forced a descending-level exception in the page fault path, with
InvalidateSeq (a lockless seqcount that requires no lock acquisition at all).
See Section 3.4.
Release builds rely solely on the compile-time guarantee.
3.6 Lock-Free Data Structures¶
Where possible, lock-free data structures replace locked ones:
- MPSC ring buffers: For all cross-domain communication (io_uring-style)
- RCU-protected radix trees: For page cache lookups
- Per-CPU freelists: For slab allocation (no cross-CPU contention)
- Atomic reference counts: For capability tokens and shared objects
- Sequence locks (seqlock): For rarely-written, frequently-read data (e.g., system
time, mount table snapshot). See
SeqLock<T>specification below. - IDR (Integer ID Radix tree): For integer-keyed namespaces (PIDs, IPC IDs, file descriptors). See Section 3.6 below.
3.6.1 SeqLock<T> — Sequence Lock¶
A sequence lock optimized for rarely-written, frequently-read data. Readers never block and never modify shared state (no atomic RMW). Writers are mutually excluded by a spinlock. Readers detect concurrent writes via a sequence counter and retry.
Used by: timekeeping (Section 7.8), vDSO data page (Section 2.22), mount table snapshots, DSM coherence metadata (Section 6.5), IMA policy (Section 9.5).
/// Sequence lock for read-optimized, rarely-written shared data.
///
/// # Design
/// - `seq` is an even number when no write is in progress; odd during a write.
/// - Readers snapshot `seq`, copy the data, re-read `seq`. If the two `seq`
/// values differ or either is odd, the read was torn and must be retried.
/// - Writers acquire `lock`, increment `seq` to odd, write data, increment
/// `seq` to even, release `lock`.
///
/// # Constraints
/// - `T: Copy` required: readers copy the entire `T` into a local variable.
/// Non-Copy types would require partial reads that could observe torn state.
/// - Readers MUST NOT hold references into the SeqLock data — only copies.
/// - Writers must not panic while holding the write lock (seq would remain odd,
/// causing all readers to spin forever).
///
/// # Memory Ordering (per architecture)
///
/// | Operation | x86-64 (TSO) | AArch64 / ARMv7 | RISC-V | PPC32/PPC64LE | s390x (near-TSO) | LoongArch64 |
/// |-----------|-------------|-----------------|--------|---------------|------------------|-------------|
/// | Reader: seq load | Plain MOV (compiler barrier) | `LDAR` (load-acquire) | `fence r,r` + load | `lwsync` + load | Plain load (compiler barrier) | `dbar 0x14` (load-load) + load |
/// | Reader: data read | Plain MOV | Plain LDR (after acquire on seq) | Plain load (after fence) | Plain load (after lwsync) | Plain load | Plain load (after dbar) |
/// | Reader: seq re-read | `fence(Acquire)` (no-op on TSO) + plain MOV | `DMB ISHLD` + plain LDR | `fence r,r` + plain load | `lwsync` + plain load | Plain load (no-op — near-TSO) | `dbar 0x14` + plain load |
/// | Writer: seq increment (odd) | Plain MOV + compiler barrier | `STLR` (store-release) | `fence w,w` + store | `lwsync` + store | Plain store (compiler barrier) | `dbar 0x12` (store-store) + store |
/// | Writer: data write | Plain MOV | Plain STR | Plain store | Plain store | Plain store | Plain store |
/// | Writer: seq increment (even) | Plain MOV + compiler barrier | `STLR` (store-release) | `fence w,w` + store | `lwsync` + store | Plain store (compiler barrier) | `dbar 0x12` + store |
///
/// On x86-64 and s390x (near-TSO), the hardware guarantees loads are not reordered past
/// loads and stores are not reordered past stores. Compiler barriers
/// (`core::hint::black_box` or `asm!("" ::: "memory")`) suffice; no hardware fence
/// needed. s390x provides sequential-consistency for most operations; only store-load
/// reordering is possible, which does not affect seqlock correctness (the reader's
/// load-load path is naturally ordered). Full serialization when needed uses
/// `BCR 15,0`.
///
/// On LoongArch64, the memory model is weakly ordered (similar to ARM). The `dbar`
/// instruction with hint values controls barrier granularity: `0x14` = load-load
/// barrier, `0x12` = store-store barrier, `0x00` = full barrier.
///
/// **Note**: The dbar hint values `0x14` and `0x12` are Loongson
/// implementation-specific (3A5000/3C5000 series). The LoongArch ISA
/// architecture manual (Volume 1, §2.2.10.1) defines `dbar 0` as a full
/// barrier but leaves partial-barrier hint values as implementation-defined.
/// A future LoongArch CPU from a different vendor may treat unrecognized
/// hint values as `dbar 0` (full barrier). The Rust compiler (via LLVM's
/// LoongArch backend) emits correct barrier instructions for
/// `Ordering::Acquire`/`Release` regardless of these specific hint values,
/// so generated code is always correct. The hint values in the table above
/// document current Loongson hardware behavior for performance analysis,
/// not correctness requirements.
///
/// On the remaining weakly-ordered architectures (ARM, RISC-V, PPC), explicit
/// acquire/release barriers are required to prevent the CPU from reordering data
/// reads/writes past the sequence counter.
pub struct SeqLock<T: Copy> {
/// Sequence counter. Even = idle, odd = write in progress.
seq: AtomicU32,
/// Writer mutual exclusion.
lock: RawSpinLock,
/// Protected data.
data: UnsafeCell<T>,
}
// SAFETY: SeqLock is Sync when T is Send (writers need exclusive access;
// readers only copy T values).
unsafe impl<T: Copy + Send> Sync for SeqLock<T> {}
impl<T: Copy> SeqLock<T> {
/// Create a new SeqLock with the given initial value.
pub const fn new(val: T) -> Self {
Self {
seq: AtomicU32::new(0),
lock: RawSpinLock::new(),
data: UnsafeCell::new(val),
}
}
/// Simple reader: returns a copy of the protected data.
/// Automatically retries on torn reads. Suitable for single-value
/// reads where the retry loop is simple.
///
/// # Panics (debug only)
/// Logs a warning if the retry count exceeds 1000 (indicates a
/// writer is holding the lock for too long or a livelock).
pub fn read(&self) -> T {
loop {
let s1 = self.read_begin();
// SAFETY: we will check for torn read via read_retry.
let val = unsafe { *self.data.get() };
if !self.read_retry(s1) {
return val;
}
core::hint::spin_loop();
}
}
/// Begin an optimistic read. Returns the current sequence number.
/// The caller reads the protected data, then calls `read_retry(seq)`
/// to check for concurrent writes. Use this for multi-field reads
/// where the caller needs to read multiple related fields consistently.
///
/// ```rust
/// loop {
/// let seq = seqlock.read_begin();
/// let (a, b, c) = unsafe { read_fields(&*seqlock.data_ptr()) };
/// if !seqlock.read_retry(seq) {
/// break (a, b, c);
/// }
/// }
/// ```
#[inline(always)]
pub fn read_begin(&self) -> u32 {
loop {
let s = self.seq.load(Ordering::Acquire);
if s & 1 == 0 {
return s;
}
// Odd = write in progress; spin until even.
core::hint::spin_loop();
}
}
/// Check if a concurrent write occurred since `read_begin()` returned
/// `start_seq`. Returns `true` if the read must be retried (torn).
///
/// **Memory ordering**: The `fence(Acquire)` before the `Relaxed` load
/// matches Linux's `smp_rmb()` semantics exactly. The fence ensures all
/// preceding data reads (between `read_begin()` and this call) are
/// ordered before the sequence counter re-read. Without this fence, a
/// weakly-ordered CPU could reorder the seq re-read before data reads,
/// observing a clean seq while the data was actually torn.
///
/// On x86-64 and s390x (TSO): `fence(Acquire)` is a no-op (hardware
/// provides load-load ordering natively). Zero cost.
/// On AArch64/ARMv7: emits `dmb ishld` (load-load + load-store barrier).
/// On RISC-V: emits `fence r,r` (load-load barrier).
/// On PPC32/PPC64LE: emits `lwsync` (load-load + load-store barrier).
///
/// Why not `load(Acquire)`? An Acquire load on the seq counter provides
/// ordering for loads *after* the load, not *before* it. We need the
/// reverse: ensure all data loads *before* this point complete before the
/// seq re-read. `fence(Acquire)` provides this "loads before the fence
/// complete before loads after the fence" guarantee.
///
/// Why not `SeqCst`? Wastes ~20 cycles on x86-64 (`MFENCE`) for ordering
/// that is not needed. The seqlock read path only requires load-load
/// ordering, not a full store-load fence.
///
/// Why not `compiler_fence`? Unsound on 6/8 architectures. A compiler
/// fence prevents the compiler from reordering, but the CPU can still
/// reorder loads across it on ARM, RISC-V, PPC, and LoongArch.
#[inline(always)]
pub fn read_retry(&self, start_seq: u32) -> bool {
core::sync::atomic::fence(Ordering::Acquire);
let s = self.seq.load(Ordering::Relaxed);
s != start_seq
}
/// Acquire write access. Mutually exclusive with other writers.
/// Preemption is disabled for the duration (RawSpinLock).
///
/// # Safety
/// Caller must ensure IRQ state is appropriate (see RawSpinLock safety).
pub unsafe fn write_lock(&self) {
self.lock.lock();
// Increment seq to odd (write in progress).
// Single-writer (RawSpinLock ensures exclusion), so plain load+store
// suffices — saves ~8-18 cycles vs atomic RMW (LOCK XADD on x86).
let s = self.seq.load(Ordering::Relaxed);
self.seq.store(s + 1, Ordering::Release);
}
/// Release write access. Increments seq to even (write complete).
///
/// # Safety
/// Must be called by the same CPU that called `write_lock()`.
pub unsafe fn write_unlock(&self) {
// Release ordering ensures all data writes are visible before
// the seq increment that signals "write complete" to readers.
// Single-writer: plain load+store, not atomic RMW.
let s = self.seq.load(Ordering::Relaxed);
self.seq.store(s + 1, Ordering::Release);
self.lock.unlock();
}
/// Write a new value. Convenience wrapper around write_lock/write_unlock.
///
/// # Safety
/// See `write_lock()`.
pub unsafe fn write(&self, val: T) {
self.write_lock();
*self.data.get() = val;
self.write_unlock();
}
/// Raw pointer to the data (for multi-field reads in read_begin/read_retry
/// patterns). The pointer is valid for the lifetime of the SeqLock.
/// MUST only be dereferenced between read_begin() and read_retry() calls.
pub fn data_ptr(&self) -> *const T {
self.data.get()
}
}
Longevity analysis for seq: AtomicU32: The sequence counter increments by 2 per
write (once to odd, once to even). At 1 billion writes per second (extreme), wrap
occurs in ~4.3 seconds — but wrap is safe. The read_retry check uses equality
(s != start_seq), which correctly detects wrap-around writes. A reader preempted
between read_begin() and read_retry() for exactly 2^31 write cycles would
observe a false-clean read. At typical write rates (timekeeping at 1 KHz), this
requires ~25 days of continuous preemption — impractical, and the data would be
functionally harmless (the reader gets the last-written value, which is the most
recent available regardless of wrap). SeqLock readers are fully preemptible (no
PreemptGuard requirement); the safety argument rests on the impracticality of
2^31 writes during a single preemption window, not on preemption being disabled.
The u32 seq is retained for Linux vDSO ABI compatibility (the vDSO exposes u32 seq
to userspace). Kernel-internal SeqLock paths with unbounded preemption latency
(e.g., SIGSTOP) are acceptable because a stale-but-consistent read is no worse
than the stale time value the reader would get anyway.
3.6.2 Compare-and-Swap Semantics Differ by Architecture¶
All UmkaOS lock-free algorithms that update shared state use compare-and-swap (CAS) operations,
exposed in Rust as AtomicT::compare_exchange and AtomicT::compare_exchange_weak. The
semantics of these operations differ materially across ISAs, and algorithms must be written
to be correct on the weakest variant.
x86 CMPXCHG — guaranteed single-attempt success:
On x86, CMPXCHG is an atomic read-modify-write instruction that succeeds if and only if
the current memory value matches the expected value. There are no spurious failures: if no
other CPU has modified the location since the load, the CMPXCHG will succeed on the first
attempt, always. This makes CAS on x86 appear to be a simple conditional store, and
algorithms that assume single-try success work correctly there.
ARM LDXR/STXR — load-exclusive/store-exclusive with spurious failure:
AArch64 and ARMv7 implement CAS using load-exclusive (LDXR) and store-exclusive (STXR)
instruction pairs. The store-exclusive can fail even when no competing CPU has modified the
target address. The CPU's exclusive monitor — a hardware reservation mechanism — can be
cleared by an interrupt, a context switch, a hypervisor VM exit, a cache line eviction, or
other microarchitectural events that have no relation to conflicting memory accesses. When
the exclusive monitor is cleared, STXR returns a failure status and the CAS must be
retried. This spurious failure is architecturally specified and guaranteed to occur
occasionally in practice.
RISC-V LR/SC — load-reserved/store-conditional with spurious failure:
RISC-V uses load-reserved (LR.W/LR.D) and store-conditional (SC.W/SC.D) pairs with
identical semantics. The RISC-V specification explicitly permits SC to spuriously fail even
in the absence of competing accesses. The reservation may be invalidated by any exception,
interrupt, or other event during the LR/SC sequence.
PowerPC ldarx/stdcx. — load-and-reserve/store-conditional:
PowerPC uses lwarx/stwcx. (32-bit) and ldarx/stdcx. (64-bit) with the same
spurious-failure semantics. An interrupt, context switch, or cache invalidation between the
load-reserve and store-conditional will cause the store-conditional to indicate failure,
requiring retry.
Design implication for UmkaOS lock-free algorithms: Every lock-free algorithm in UmkaOS that modifies shared state through a CAS loop MUST be written as an unconditional retry loop that tolerates spurious failure. On x86, the retry branch is never taken in practice; on ARM, RISC-V, and PowerPC, it will be taken occasionally and the algorithm must converge correctly regardless of how many spurious failures occur before a genuine success.
A CAS loop that assumes at most one retry after a genuine conflict is incorrect on ARM/RISC-V/PPC. A CAS loop that checks for spurious failure explicitly but does not re-load the current value before retrying is also incorrect — the retry must reload the current memory value and re-evaluate the expected value before each subsequent attempt.
3.6.3 LL/SC Restrictions on Non-Cacheable (MMIO) Memory¶
On architectures that implement atomics via LL/SC pairs, atomic operations cannot be used on MMIO registers or other non-cacheable memory regions. This is a hard architectural constraint, not a performance guideline.
| Architecture | Restriction | Consequence |
|---|---|---|
| PowerPC | lwarx/stwcx. require Memory Coherence Required (M=1) storage. Cache-inhibited or write-through storage triggers a data storage or alignment exception. |
Atomic operations on device registers must use lock-based access or device-specific compare-and-swap commands (e.g., PCIe AtomicOps). The generic AtomicT API must never be used on MMIO addresses. |
| ARM (AArch64/ARMv7) | LDXR/STXR on Device memory type is implementation-defined. Some cores (Cortex-A57, erratum 832075) deadlock on exclusive + device load. |
Exclusive load/store sequences targeting Device-nGnRnE or Device-nGnRE memory may silently fail or hang. MMIO registers must be accessed via plain loads/stores with appropriate barriers (ldr/str + dsb), never via exclusive pairs. |
| RISC-V | LR/SC on I/O memory (PMA = non-cacheable) is implementation-defined. The spec permits but does not require support. |
Assume LR/SC on MMIO fails. Use lock-based MMIO access or PCI AtomicOps where the device supports them. |
| x86-64 | LOCK CMPXCHG on uncacheable (UC) memory works correctly but is not guaranteed to be atomic with respect to device-side state. |
x86 atomics on MMIO are architecturally permitted but should still be avoided — the device's view of atomicity differs from the CPU's. |
Design rule: The AtomicT Rust types (AtomicU32, AtomicU64, etc.) must never be
used to access memory-mapped I/O registers. All MMIO access goes through the dedicated
mmio_read_* / mmio_write_* functions (which emit plain loads/stores with architecture-
appropriate barriers), or through the volatile read_volatile / write_volatile wrappers.
Compile-time enforcement: MMIO regions are typed as MmioRegion<T>, not as raw pointers
to atomic types.
Rust compare_exchange vs compare_exchange_weak:
- compare_exchange on platforms that use LR/SC or LDXR/STXR internally wraps
the operation in a loop that retries on spurious failure, presenting the caller with
x86-equivalent "fail only on genuine conflict" semantics. This loop is invisible to the
caller and adds retries inside the atomic operation itself.
- compare_exchange_weak exposes the underlying hardware semantics directly, permitting
spurious failure to propagate to the caller. The caller's retry loop must handle it.
In UmkaOS lock-free algorithms that already contain an explicit retry loop (the standard
pattern for CAS-based updates), compare_exchange_weak is preferred. Using
compare_exchange inside an outer retry loop results in double-looping on ARM/RISC-V:
the inner hidden loop retries spurious failures, and the outer loop retries genuine
conflicts. The compare_exchange_weak variant eliminates the inner loop, letting the
outer loop handle both cases, reducing instruction count and branch pressure on the
architectures where LR/STXR instructions are used.
3.6.4 Idr<T> — Integer ID Allocator¶
Idr<T> is a radix-tree-based integer ID allocator, equivalent to Linux's struct idr
(reimplemented on XArray since Linux 4.20). It maps small non-negative integer keys to
values with O(1) average-case lookup and integrated next-ID allocation.
/// Radix-tree-based integer ID allocator.
///
/// Internally uses a 64-ary radix tree (6 bits per level). For the common
/// case of <64K entries, the tree is 3 levels deep (6+6+6 = 18 bits).
/// Full u32 range: 6 levels (6×6 = 36 > 32 bits). Full u64 range: 11
/// levels (6×11 = 66 > 64 bits). Sparse ID spaces incur only the cost of
/// allocated nodes — unpopulated subtrees are null pointers with no memory.
/// Each node is cache-line-aligned (64 bytes metadata + 512 bytes pointers).
///
/// **Concurrency model** (matches Linux's IDR/XArray):
/// - **Reads**: RCU-protected. `lookup()` takes an `&RcuReadGuard` and
/// performs a lock-free radix tree walk; the returned borrow is tied to
/// the guard's lifetime. Multiple readers on different CPUs never contend.
/// - **Writes**: Serialized by an internal `SpinLock`. `idr_alloc*()` and
/// `idr_remove()` acquire the lock, update the tree, then publish changes
/// via RCU (store-release on the node pointer). Freed nodes are reclaimed
/// after an RCU grace period.
/// - **No external locking required** for basic operations. Callers that
/// need atomic read-modify-write sequences (e.g., "find and update if
/// exists") must hold their own lock.
pub struct Idr<T> {
/// Root of the radix tree.
root: *mut IdrNode<T>,
/// Next ID hint (speeds up sequential allocation).
next_id: AtomicU32,
/// Write-side lock (protects tree mutations).
lock: SpinLock<()>,
}
/// Interior or leaf node of the `Idr<T>` 64-ary radix tree.
///
/// Each level consumes 6 key bits, selecting one of 64 child `slots` as
/// `(id >> shift) & 0x3F`. Interior nodes (`shift > 0`) hold child
/// `*mut IdrNode<T>` pointers; leaf nodes (`shift == 0`) hold `*mut T` value
/// pointers. Slots are type-erased to `AtomicPtr<()>` so one node layout serves
/// both levels and readers walk lock-free under RCU. `#[repr(align(64))]`
/// cache-line-aligns the node; the metadata occupies the first cache line and
/// the 64 × 8-byte slot array (512 B) follows.
#[repr(align(64))]
struct IdrNode<T> {
/// Bit `i` set ⇔ slot `i` is occupied. Enables an O(1) lowest-free-slot
/// scan (`(!bitmap).trailing_zeros()`) for `idr_alloc*()` without touching
/// the pointer array.
bitmap: u64,
/// Number of occupied slots (`bitmap.count_ones()`). A node reaching
/// `count == 0` after a remove is unlinked and freed after a grace period.
count: u16,
/// Key bits already consumed above this node: `0` at the leaf, `6` one
/// level up, `12` two up, …
shift: u8,
/// Explicit padding: `parent` needs 8-byte alignment; `bitmap`(8) +
/// `count`(2) + `shift`(1) = 11 bytes, so 5 bytes of padding follow.
_pad: [u8; 5],
/// Parent node, or null at the root — used to prune emptied nodes on remove.
parent: *mut IdrNode<T>,
/// 64 child slots. Published store-release by the write side; read
/// lock-free under `RcuReadGuard`. Null = empty. A removed node is queued to
/// the per-CPU `RcuCallbackRing` and freed one grace period later, so a
/// concurrent `lookup()` walker never dereferences freed memory.
slots: [AtomicPtr<()>; 64],
}
Operations (canonical API): this list is normative. Idr<T> is defined ONCE,
here; every call site across the specification uses exactly these spellings —
a call site using any other method name on an Idr is a bug.
const fn new() -> Idr<T>: Empty allocator.constsoIdrvalues can bestaticinitializers (e.g., the vsock CID allocator, Section 16.24).idr_alloc(value: T) -> Result<u32, IdrError>: Allocate the LOWEST free ID in[1, u32::MAX)and storevalue. Shorthand foridr_alloc_range(1, u32::MAX, value).idr_alloc_range(min: u32, max: u32, value: T) -> Result<u32, IdrError>: Allocate the LOWEST free ID in the half-open range[min, max). Half-open is the RustRangeidiom: bounds compose without ±1 arithmetic (a..bthenb..ctile exactly), andu32::MAXworks as an exclusive end without overflow. (Coincides with Linuxidr_alloc()'s convention,lib/idr.c— convenient for porting call sites, but the Rust idiom is the reason.)Err(IdrError::Exhausted)when every ID in the range is allocated.lookup<'g>(&self, id: u32, guard: &'g RcuReadGuard) -> Option<&'g T>: Lock-free RCU radix walk. The explicit guard parameter ties the returned borrow to the RCU read-side critical section at compile time — the borrow cannot outlive the guard. Callers extractCopyvalues with.copied().idr_remove(id: u32) -> Option<T>: Remove the entry and free the ID — it is immediately reusable by a subsequentidr_alloc*(). RCU-deferred node reclamation.idr_replace(id, value) -> Result<T, IdrError>: Overwrite the value stored at an ALLOCATED id and return the old value, WITHOUT freeing the id — the id never passes through a free state, so a concurrentidr_allocunder the same internal SpinLock can never claim it (a remove+alloc emulation would open exactly that window).Err(IdrError::NotFound)if the id is unallocated. The value swap is a store-release on the leaf slot; concurrent RCU readers observe either the old or the new value, never a torn state. Linux precedent:lib/idr.c idr_replace()(same semantics, verified against torvalds/linux master). Load-bearing user: exec'scollapse_thread_group()PID remap (Section 8.1).iter<'g>(&self, guard: &'g RcuReadGuard) -> impl Iterator<Item = (u32, &'g T)>: Iterate all live entries in ascending ID order under the RCU guard. Per-node RCU snapshot semantics: entries inserted or removed concurrently with the walk may or may not be observed.with_lock<R>(&self, f: impl FnOnce(&mut IdrLocked<'_, T>) -> R) -> R: COMPOUND-OPERATION view. Acquires the Idr's internal writer SpinLock (forPidNamespace.pid_mapthat lock IS the namespace'sPID_MAP_LOCK(47)— master-table row 47), runs the closure against a locked view, releases on return.IdrLockedexposes the non-relocking accessorslookup(id) -> Option<&T>,remove(id) -> Option<T>,replace(id, value) -> Result<T, IdrError>, andalloc_range(min, max, value) -> Result<u32, IdrError>— same semantics as the free methods above minus the per-call lock. Use when a check-then-act sequence must be atomic against other allocator writers (the PID pin/unpin/sentinel-free compound ops, Section 17.1, are the load-bearing users). The closure runs under a SpinLock: no sleeping, no allocation, no nested ordered-lock acquisition at or below the Idr lock's level. Single-op call sites keep using the plain methods —with_lockis for multi-op atomicity only.
Error type and Errno mapping:
/// Idr operation failure. Converted to a Linux-compatible errno at the
/// syscall boundary via the `From` impl below.
pub enum IdrError {
/// Every ID in the requested range is allocated.
Exhausted,
/// `idr_replace()` targeted an unallocated ID.
NotFound,
}
impl From<IdrError> for Errno {
fn from(e: IdrError) -> Errno {
match e {
// Linux `kernel/pid.c` `alloc_pid()` parity: fork() reports EAGAIN when
// the namespace's pid_max space is exhausted.
IdrError::Exhausted => Errno::EAGAIN,
IdrError::NotFound => Errno::ENOENT,
}
}
}
Allocation discipline: Idr<T> is a RECYCLING (lowest-free) allocator —
Linux IDR semantics. PidNamespace.pid_map REQUIRES this discipline:
userspace pid_t numbers are recycled per-namespace exactly as on Linux
(Section 8.1). Kernel-internal
identifiers that must NEVER be reused (TaskId, ProcessId, namespace IDs) are
NOT allocated from an Idr — they are plain monotonic AtomicU64 counters
(e.g., NEXT_TASK_ID). Use an Idr only for integer keyspaces where reuse
is required (ABI-mandated recycling) or harmless (bounded handle tables with
generation-tagged validation).
Usage: PID namespaces (Section 17.1), SysV IPC ID allocation (Section 17.1), file descriptor tables (Section 8.1), and any kernel subsystem mapping small integers to objects.
3.6.5 WaitQueueHead — Blocking Wait Queue¶
A WaitQueueHead is a spinlock-protected intrusive list of waiters. Kernel code
that needs to block until a condition is true (e.g., pipe readable, socket writable,
child exited) inserts itself into a WaitQueueHead and calls schedule(). When the
condition changes, the producer calls wake_up() to unblock one or all waiters.
/// Flags controlling waiter behavior in a `WaitQueueHead`.
bitflags! {
pub struct WaitFlags: u32 {
/// Exclusive waiter: only ONE exclusive waiter is woken per wake_up() call.
/// Non-exclusive waiters are always woken. This prevents thundering herd
/// on resources where only one waiter can make progress (e.g., accept()).
const EXCLUSIVE = 1 << 0;
/// Interruptible by signals. If unset, uses `TaskState::UNINTERRUPTIBLE`.
const INTERRUPTIBLE = 1 << 1;
/// Bookmark entry used for safe list iteration during wake_up_all().
/// Not a real waiter — never invokes wakeup function.
const BOOKMARK = 1 << 2;
}
}
/// Intrusive hash-bucket list node (Linux `hlist_node` shape). Embedded in an
/// object that lives in an RCU-protected chained hash table (e.g. dentry
/// `d_hash`) so the element can be removed in O(1) without a separate map entry
/// pointing at it. Forward-linked with a back-pointer to the previous element's
/// `next` slot (`pprev`) — hash-bucket chaining, the sanctioned exception to the
/// "no IntrusiveList as a general container" rule ([Section 3.13](#collection-usage-policy)),
/// NOT a general-purpose list.
pub struct HashListNode {
/// Next element in the bucket chain; null at the tail.
pub next: *mut HashListNode,
/// Address of the previous element's `next` field (the bucket head's slot
/// for the first element), enabling O(1) self-removal.
pub pprev: *mut *mut HashListNode,
}
/// A single waiter registered on a WaitQueueHead.
/// Embedded in the waiter's stack frame or task struct.
pub struct WaitQueueEntry {
/// Wakeup function called when the waiter is unblocked.
/// For normal task sleep: sets task state to `TaskState::RUNNING` and
/// calls `try_to_wake_up()` (enqueues on runqueue).
/// Custom wakeup functions are used by epoll (ep_poll_callback),
/// io_uring poll wakeups, and autoremove waiters.
pub wakeup: fn(*mut WaitQueueEntry) -> bool,
/// Link in the wait queue list (intrusive).
pub link: IntrusiveListNode,
/// Pointer to the waiting task.
/// **Design tradeoff**: The full Arc cost per wait/wake cycle is ~10-20 cycles:
/// `Arc::clone` on insertion costs ~5-10 cycles (one `LOCK XADD` for refcount
/// increment, NOT a heap allocation), and the paired `Arc::drop` on removal
/// costs another ~5-10 cycles (one `LOCK XADD` decrement + deallocation branch).
/// This prevents use-after-free when a task is killed (SIGKILL) while on a
/// wait queue — a historically common bug class in Linux where raw
/// `task_struct*` with manual ordering guarantees has caused subtle UAF issues.
/// UmkaOS pays ~10-20 cycles per wait/wake cycle to eliminate this entire bug
/// class. The wait path already takes the WaitQueueHead spinlock (~10-30
/// cycles), so the Arc pair adds ~33-67% to the spinlock cost, or ~25-40%
/// to the total wait/wake cost (~30-50 cycles overall). The design decision
/// is sound: the total cost remains small, and the UAF prevention is worth it.
pub task: Arc<Task>,
/// Waiter flags: EXCLUSIVE, INTERRUPTIBLE, BOOKMARK.
pub flags: WaitFlags,
/// Private data for the wakeup function (e.g., poll key for epoll).
pub private: usize,
}
/// SAFETY: `WaitQueueEntry` embeds exactly one link hook — the type-erased
/// `link: IntrusiveListNode` (= `IntrusiveLink<()>`, layout-identical for every
/// element type since the type parameter is a `PhantomData` marker only). The
/// impl projects that node as `IntrusiveLink<WaitQueueEntry>` so
/// `IntrusiveList<WaitQueueEntry>` can yield `*mut WaitQueueEntry` directly. A
/// `WaitQueueEntry` lives in the waiter's stack frame or task struct and is
/// pinned there for the whole time it is linked into a `WaitQueueHead` (it is
/// removed from the queue before the frame unwinds), so the link address is
/// stable for its linked lifetime (the `HasIntrusiveLink` move-safety
/// contract). `from_link` is the exact inverse of `link`.
unsafe impl HasIntrusiveLink<WaitQueueEntry> for WaitQueueEntry {
fn link(this: *mut WaitQueueEntry) -> *mut IntrusiveLink<WaitQueueEntry> {
// SAFETY: `this` is a valid `WaitQueueEntry`; project to its type-erased
// `link` field and reinterpret it as `IntrusiveLink<WaitQueueEntry>`
// (`IntrusiveLink<()>` and `IntrusiveLink<WaitQueueEntry>` are
// layout-identical — see `IntrusiveListNode`).
unsafe { &raw mut (*this).link as *mut IntrusiveLink<WaitQueueEntry> }
}
unsafe fn from_link(link: *mut IntrusiveLink<WaitQueueEntry>) -> *mut WaitQueueEntry {
// SAFETY: `link` is the `link` field of a live `WaitQueueEntry`;
// subtract the field offset to recover the container (container_of).
let offset = core::mem::offset_of!(WaitQueueEntry, link);
unsafe { (link as *mut u8).sub(offset) as *mut WaitQueueEntry }
}
}
/// Opaque handle to a registered waiter, returned when a task enqueues itself
/// on a `WaitQueueHead` and consumed to wake or dequeue that specific waiter
/// later. Follows the opaque-handle ABA-safety convention: the value packs a
/// waiter identifier with a generation, so a token that names an
/// already-dequeued (and possibly reused) waiter is rejected rather than waking
/// the wrong task — never a raw pointer to the entry, which would reintroduce
/// the use-after-free hazard `WaitQueueEntry`'s `Arc<Task>` exists to prevent.
/// Held by subsystems that must remember *which* waiter to resume when an
/// out-of-band event resolves (e.g. the userfaultfd pending-fault table's
/// `WaitEntry`, [Section 4.15](04-memory.md#extended-memory-operations)).
#[repr(transparent)]
#[derive(Clone, Copy, PartialEq, Eq)]
pub struct WaitQueueToken(pub u64);
/// Head of a wait queue. Holds a spinlock and the list of waiters.
/// Embedding one in a struct means "tasks can block waiting for this object".
/// Used by: PipeBuffer (read_wait/write_wait), sockets, epoll fds, futexes,
/// page fault waits, inode event waits.
pub struct WaitQueueHead {
/// Protects the waiter list and coordinates with the waker.
pub lock: SpinLock<()>,
/// List of `WaitQueueEntry` nodes. Non-exclusive waiters are at the front
/// (FIFO order); exclusive waiters are appended at the tail. This ordering
/// ensures `wake_up()` wakes ALL non-exclusive waiters plus exactly ONE
/// exclusive waiter, preventing thundering herd while still broadcasting
/// to poll/epoll listeners.
///
/// **Priority ordering within exclusive waiters**: exclusive entries are
/// sorted by task scheduling class (DL > RT > CFS) so that higher-priority
/// tasks are woken first. This prevents priority starvation among waiters
/// (a low-priority exclusive waiter cannot indefinitely precede a
/// high-priority one in the wake order). Note: this is *not* priority
/// inversion prevention — true PI requires a PI-mutex mechanism
/// ([Section 8.5](08-process.md#real-time-guarantees--priority-inheritance-protocol)) that boosts
/// the lock holder's priority, not just the wake order of waiters.
///
/// Typed `IntrusiveList<WaitQueueEntry>` (each entry embeds a type-erased
/// `link: IntrusiveListNode`, projected by `WaitQueueEntry:
/// HasIntrusiveLink<WaitQueueEntry>`). Not `IntrusiveList<()>`, which is
/// ill-formed — `()` cannot satisfy the element bound
/// ([Section 3.5](#locking-strategy--intrusivelistt-intrusive-doubly-linked-list)).
pub head: IntrusiveList<WaitQueueEntry>,
}
/// Outcome of an interruptible, timed wait (`wait_event_interruptible_timeout`):
/// the wait can end because the condition became true, the deadline expired, or
/// a signal is pending. It carries the legs that neither `wait_event_timeout`
/// (a `bool`, no signal leg) nor `wait_event` (a `Result`, no timeout leg)
/// expresses on its own.
pub enum InterruptibleWaitResult {
/// `condition()` was observed true (on entry or after a wakeup).
Ready,
/// The timeout elapsed before the condition became true.
TimedOut,
/// A signal is pending; the caller should unwind with `EINTR`.
Interrupted,
}
impl WaitQueueHead {
/// Construct an empty wait queue (unlocked spinlock, empty waiter list).
/// `const` — `WaitQueueHead` appears in `static` initializers (e.g. the
/// fsync page-wait table, `OOM_NOTIFY_WAITERS`) and inside `const fn`
/// constructors (`Completion::new`, `Condvar::new`). Constness rests on
/// the null-sentinel deferred-init contract of `IntrusiveList::new()`
/// ([Section 3.5](#locking-strategy--intrusivelistt-intrusive-doubly-linked-list)).
pub const fn new() -> Self;
/// Block until `condition()` returns true. Interruptible by signals.
/// Returns `Ok(())` when condition is true, `Err(EINTR)` if interrupted.
///
/// **Algorithm** (standard wait-loop; avoids lost-wakeup race):
/// ```
/// 1. Allocate WaitQueueEntry on caller's stack with wakeup=default_wakeup.
/// 2. Lock self.lock.
/// 3. Insert entry into self.head:
/// - Non-exclusive (flags & EXCLUSIVE == 0): prepend to head (front).
/// - Exclusive: append to tail, sorted by scheduling class priority
/// (DL > RT > CFS) among other exclusive entries.
/// 4. Set current task state to TASK_INTERRUPTIBLE.
/// 5. Unlock self.lock.
/// 6. Check condition(). If true:
/// Set task state to `TaskState::RUNNING`.
/// Remove entry from queue (lock → splice out → unlock).
/// Return Ok(()).
/// 7. Call schedule(). Suspend until wake_up_one/wake_up_all is called.
/// 8. On resume: set task state to `TaskState::RUNNING`.
/// Check for pending signal: if signal_pending(), go to step 9.
/// Check condition(). If true: remove entry, return Ok(()).
/// Otherwise: set state to TASK_INTERRUPTIBLE, go to step 7 (retry loop).
/// 9. Signal path: remove entry from queue, return Err(EINTR).
/// ```
/// Step 4 before step 6 is essential: a waker calling `wake_up_one()` between
/// steps 5 and 6 sets the task `TaskState::RUNNING` before `schedule()` is called, so
/// `schedule()` will return immediately — the wakeup is never lost.
pub fn wait_event<F: Fn() -> bool>(&self, condition: F) -> Result<(), Errno>;
/// Block until `condition()` returns true. NOT interruptible by signals.
/// Returns only when the condition is true (no EINTR).
/// Same algorithm as `wait_event` but uses `TaskState::UNINTERRUPTIBLE` at step 4
/// and skips the signal check at step 8.
pub fn wait_event_uninterruptible<F: Fn() -> bool>(&self, condition: F);
/// Block until `condition()` returns true, interruptible ONLY by fatal
/// signals (SIGKILL, or group-exit in progress). Returns `Err(EINTR)`
/// only for those. Same algorithm as `wait_event` but uses
/// TASK_KILLABLE at step 4; the step-8 signal check tests
/// `fatal_signal_pending()` instead of `signal_pending()`.
/// Linux equivalent: `wait_event_killable()`. Used where an
/// uninterruptible wait would make a hung resource unkillable (e.g.,
/// exec's `collapse_thread_group()` waiting for sibling threads to exit,
/// [Section 8.1](08-process.md#process-and-task-management)).
pub fn wait_event_killable<F: Fn() -> bool>(&self, condition: F) -> Result<(), Errno>;
/// Block until `condition()` returns true OR `timeout_ns` nanoseconds
/// elapse, whichever comes first. Returns `true` if the condition was
/// met before the timeout, `false` if the timeout expired.
///
/// **Algorithm** (matches `wait_event` with timer-based wakeup):
/// ```
/// 1. Check condition() — if true, return true immediately.
/// 2. Allocate WaitQueueEntry on stack (exclusive = false).
/// 3. Add entry to self.head (under self.lock).
/// 4. Set current task state to `TaskState::UNINTERRUPTIBLE`.
/// 5. Arm a high-resolution timer (hrtimer) with `timeout_ns` duration.
/// The timer callback wakes the waiting task via
/// `scheduler::unblock(entry.task)` — which runs the wake predicate
/// (CAS from the observed sleep state to `TaskState::RUNNING`, rejecting
/// ZOMBIE/DEAD) and enqueues it on its CPU's runqueue — identical to a
/// wake_up() from the condition producer.
/// 6. Re-check condition() — if true, cancel timer, remove entry, return true.
/// 7. Call schedule() to yield the CPU.
/// 8. On wakeup: check if timer expired (timer callback sets a local flag).
/// If expired: remove entry from queue, return false (timeout).
/// 9. Re-check condition() — if true, cancel timer, remove entry, return true.
/// Otherwise: go to step 4 (retry loop — spurious wakeup).
/// ```
///
/// The timer ensures bounded wait even if the condition producer never
/// calls `wake_up()`. The timer is cancelled on the success path to
/// avoid a spurious wakeup after return.
///
/// MUST NOT be called from atomic context (sleeps).
pub fn wait_event_timeout<F: Fn() -> bool>(
&self,
timeout_ns: u64,
condition: F,
) -> bool;
/// Block until `condition()` is true, `timeout_ns` nanoseconds elapse, or a
/// signal arrives — whichever comes first. Interruptible
/// (TASK_INTERRUPTIBLE). The signal-and-timeout counterpart of
/// `wait_event_timeout` (which is uninterruptible) and of `wait_event`
/// (which has no timeout). Linux equivalent:
/// `wait_event_interruptible_timeout()`.
///
/// Returns:
/// - `InterruptibleWaitResult::Ready` — `condition()` observed true;
/// - `InterruptibleWaitResult::TimedOut` — the deadline was reached first;
/// - `InterruptibleWaitResult::Interrupted` — a signal is pending (caller
/// returns `EINTR`).
///
/// **Algorithm**: identical to `wait_event_timeout` (check `condition()`
/// first; stack `WaitQueueEntry`; arm an hrtimer for `timeout_ns` whose
/// callback wakes via `scheduler::unblock` exactly like `wake_up()`;
/// enqueue-before-recheck to close the lost-wakeup race; retry loop on
/// spurious wakeups) with two changes: the entry is enqueued
/// TASK_INTERRUPTIBLE, and after each `schedule()` the task tests
/// `signal_pending()`. A pending signal returns `Interrupted`, an expired
/// timer returns `TimedOut`, and `condition()` observed true (on entry or
/// after any wake) returns `Ready`. The timer is cancelled on every
/// non-`TimedOut` return.
pub fn wait_event_interruptible_timeout<F: Fn() -> bool>(
&self,
timeout_ns: u64,
condition: F,
) -> InterruptibleWaitResult;
/// Wake waiters: all non-exclusive waiters PLUS exactly one exclusive waiter.
///
/// This is the standard Linux `wake_up()` semantic. It prevents thundering
/// herd: only one exclusive waiter is woken (the highest-priority one at the
/// tail), while all non-exclusive waiters (epoll, poll) are always notified.
///
/// **Algorithm**:
/// ```
/// 1. Lock self.lock.
/// 2. If self.head is empty: unlock, return.
/// 3. Walk the list from head:
/// For each non-exclusive entry: call entry.wakeup(&entry).
/// For the first exclusive entry encountered: call entry.wakeup(&entry),
/// then STOP (do not wake further exclusive waiters).
/// 4. Remove woken entries from the list.
/// 5. Unlock self.lock.
/// ```
/// Wakeup callbacks run under the lock to prevent the woken task from
/// re-inserting before the walk completes. The lock hold time is bounded
/// by the number of non-exclusive waiters (typically ≤ epoll fd count).
pub fn wake_up(&self);
/// Wake up exactly one waiter (the first in FIFO order, ignoring flags).
/// Used when the caller knows exactly one waiter should proceed.
///
/// **Algorithm**:
/// ```
/// 1. Lock self.lock.
/// 2. If self.head is empty: unlock, return.
/// 3. Pop the first WaitQueueEntry from self.head.
/// 4. Unlock self.lock.
/// 5. Call entry.wakeup(&entry):
/// default_wakeup: calls scheduler::unblock(entry.task), which runs
/// the wake predicate (CAS from the observed sleep state to
/// `TaskState::RUNNING`, rejecting ZOMBIE/DEAD) and enqueues the task on its
/// CPU's runqueue. Routing through the predicate — rather than a bare
/// `entry.task.state = TaskState::RUNNING` store followed by an enqueue —
/// is what prevents re-waking a task that has since become
/// ZOMBIE/DEAD.
/// ```
/// Unlocking before calling `scheduler::unblock` avoids holding the wait queue
/// spinlock during scheduler operations (which may acquire runqueue locks).
pub fn wake_up_one(&self);
/// Wake up all waiters simultaneously (broadcast).
///
/// **Algorithm**:
/// ```
/// 1. Lock self.lock.
/// 2. Drain self.head into a local list (swap head pointer to empty list).
/// 3. Unlock self.lock.
/// 4. For each entry in local list: call entry.wakeup(&entry).
/// ```
/// Draining to a local list (step 2) means wakeups run outside the lock,
/// avoiding contention between woken tasks and new waiters inserting.
pub fn wake_up_all(&self);
/// Low-level wait entry: enqueue the current task as an **exclusive** waiter
/// (appended to the tail, ordered by scheduling class as in `wait_event`)
/// and set its state to `TaskState::UNINTERRUPTIBLE`, then return WITHOUT sleeping.
/// The caller drops any held locks, calls `schedule()`, then `finish_wait()`.
/// Used by primitives such as `Condvar` whose condition is re-checked in
/// caller code after re-acquiring a lock, so the closure-based `wait_event`
/// does not fit. Enqueue-before-sleep ordering closes the lost-wakeup race
/// exactly as documented for `wait_event` (step 4 before step 6).
pub fn prepare_to_wait_exclusive(&self);
/// Like `prepare_to_wait_exclusive`, but sets `TASK_INTERRUPTIBLE` so a
/// pending signal wakes the task. The caller inspects `signal_pending()`
/// after `schedule()` and before `finish_wait()`.
pub fn prepare_to_wait_exclusive_interruptible(&self);
}
/// Low-level wait teardown: dequeue the current task's waiter entry from `wq`
/// if it is still linked (a wakeup may already have removed it) and set the
/// task state back to `TaskState::RUNNING`. The teardown counterpart to
/// `WaitQueueHead::prepare_to_wait_exclusive`; idempotent — safe to call whether
/// or not the task was actually woken through `wq`. Linux equivalent:
/// `finish_wait()`.
pub fn finish_wait(wq: &WaitQueueHead);
Usage: PipeBuffer (§17.3.2), socket wait queues, epoll event delivery,
page fault wait-on-writeback, waitpid() child-exit notification.
3.6.6 IntrusiveNode<T> — Keyed Intrusive List Hook¶
A doubly-linked list hook embedded directly in a container struct and
carrying an inline key of type T. Embedding the hook avoids a per-node heap
allocation on insertion (the node storage is part of the enclosing struct),
and the inline key lets the owning list order or identify nodes without a
separate key lookup — e.g. Task.sibling_node: IntrusiveNode<ProcessId>
(Section 8.1) threads a task into its parent's child
list keyed by ProcessId. A hook is either linked into exactly one list or unlinked;
is_linked() distinguishes the two states. Distinct from the untyped
IntrusiveListNode (which carries links only, no key).
/// Keyed intrusive list hook. `T` is the inline key.
pub struct IntrusiveNode<T> {
/// Next hook in the owning list; null when this is the tail or unlinked.
next: AtomicPtr<IntrusiveNode<T>>,
/// Previous hook in the owning list; null when this is the head or unlinked.
prev: AtomicPtr<IntrusiveNode<T>>,
/// Inline key identifying/ordering this node within its list.
key: T,
}
impl<T> IntrusiveNode<T> {
/// Create an unlinked hook holding `key`.
pub const fn new(key: T) -> Self {
IntrusiveNode {
next: AtomicPtr::new(core::ptr::null_mut()),
prev: AtomicPtr::new(core::ptr::null_mut()),
key,
}
}
/// The node's key.
pub fn key(&self) -> &T {
&self.key
}
/// True if this hook is currently linked into a list. A linked hook has a
/// non-null neighbour on at least one side (both null ⇒ unlinked; the
/// sole element of a list points its ends at the list head, never null).
pub fn is_linked(&self) -> bool {
!self.next.load(Ordering::Acquire).is_null()
|| !self.prev.load(Ordering::Acquire).is_null()
}
}
3.6.7 Completion — One-Shot (or Multi-Shot) Signaling Primitive¶
A counter-based synchronization primitive for "producer does work, consumer waits for
completion" patterns. Built on WaitQueueHead + AtomicU32. Equivalent to Linux's
struct completion (include/linux/completion.h).
Used by: device probe completion, firmware load wait, module init synchronization, workqueue flush, AHCI command completion (Section 15.4), DMA fence completion, driver unload synchronization.
/// Counter-based completion primitive.
///
/// `done` tracks the number of completions:
/// - 0 = not yet completed; `wait()` will block.
/// - 1..u32::MAX-1 = completed N times; `wait()` decrements and returns.
/// - u32::MAX = "complete all" sentinel; `wait()` returns without decrement.
///
/// Supports three patterns:
/// 1. **One-shot**: `complete()` once, `wait()` once. Standard case.
/// 2. **Pre-completion**: `complete()` before `wait()`. The waiter returns
/// immediately (done > 0).
/// 3. **Barrier**: `complete_all()` sets done to u32::MAX, waking ALL
/// current and future waiters until `reinit()` is called.
pub struct Completion {
/// Completion counter. 0 = pending, >0 = done.
done: AtomicU32,
/// Wait queue for blocked waiters.
wq: WaitQueueHead,
}
impl Completion {
/// Create a new, uncompleted Completion.
pub const fn new() -> Self {
Self {
done: AtomicU32::new(0),
wq: WaitQueueHead::new(),
}
}
/// Block until the completion is signaled (done > 0).
///
/// If `done` is already > 0 (pre-completion or multi-completion),
/// atomically decrements `done` and returns immediately (no sleep).
/// If `done` == u32::MAX (complete_all), returns without decrement.
///
/// **Atomicity**: Uses a CAS loop to atomically check-and-decrement `done`.
/// The previous load+fetch_sub pattern had a TOCTOU race: two concurrent
/// waiters could both observe `done > 0`, both call `fetch_sub(1)`, and
/// underflow `done` below the intended value. The CAS loop ensures that
/// exactly one waiter consumes each completion increment.
///
/// **Wait-flavor set** (normative, closed until a consumer motivates
/// more): `wait()` = uninterruptible; `wait_timeout(timeout_ns) -> bool`
/// = uninterruptible with a deadline; `wait_killable() -> Result<(), Errno>`
/// = fatal-signal-interruptible. There is deliberately NO
/// plain-interruptible flavor — no consumer needs one, and the only
/// signal-aware waiter in the corpus (`vfork()`) has a contract that is
/// exactly killable ([Section 8.1](08-process.md#process-and-task-management)).
///
/// MUST NOT be called from atomic context.
pub fn wait(&self) {
// Uninterruptible flavor: `wait()` returns `()` and documents
// unconditional blocking, so it MUST NOT be built on the
// interruptible `wait_event` (which returns `Result<(), Errno>`) —
// an EINTR return there would leave the caller past the wait with
// no completion consumed and no way to observe it.
self.wq.wait_event_uninterruptible(|| {
loop {
let d = self.done.load(Ordering::Acquire);
if d == u32::MAX {
return true; // complete_all sentinel: always ready
}
if d == 0 {
return false; // not complete — sleep
}
// Try to atomically consume one completion: d → d-1.
if self.done.compare_exchange_weak(
d, d - 1,
Ordering::AcqRel, Ordering::Acquire,
).is_ok() {
return true;
}
// CAS failed (concurrent complete() or another waiter) — retry.
}
});
}
/// Block until the completion is signaled or `timeout_ns` nanoseconds
/// elapse. Returns `true` if completed, `false` on timeout.
///
/// Uses the same CAS loop as `wait()` for atomicity. See `wait()` for
/// the TOCTOU race rationale.
///
/// MUST NOT be called from atomic context.
pub fn wait_timeout(&self, timeout_ns: u64) -> bool {
self.wq.wait_event_timeout(timeout_ns, || {
loop {
let d = self.done.load(Ordering::Acquire);
if d == u32::MAX {
return true; // complete_all sentinel
}
if d == 0 {
return false; // not complete — sleep or timeout
}
if self.done.compare_exchange_weak(
d, d - 1,
Ordering::AcqRel, Ordering::Acquire,
).is_ok() {
return true;
}
}
})
}
/// Block until the completion is signaled, interruptible ONLY by fatal
/// signals (SIGKILL, or a group exit already in progress).
///
/// - `Ok(())` — exactly one completion was consumed, via the same CAS
/// check-and-decrement closure `wait()` uses (the `complete_all`
/// sentinel returns without decrement).
/// - `Err(EINTR)` — a fatal signal was delivered. The completion count is
/// **NOT** consumed: a concurrent or later `complete()` remains
/// observable to other waiters.
///
/// Built on `WaitQueueHead::wait_event_killable` (TASK_KILLABLE,
/// `fatal_signal_pending()` test — see that method above). The
/// not-consumed-on-EINTR rule is part of the contract, not an
/// implementation detail: callers that unwind on `Err(EINTR)` must not
/// assume the producer's signal was absorbed.
///
/// MUST NOT be called from atomic context.
pub fn wait_killable(&self) -> Result<(), Errno> {
self.wq.wait_event_killable(|| {
loop {
let d = self.done.load(Ordering::Acquire);
if d == u32::MAX {
return true; // complete_all sentinel: always ready
}
if d == 0 {
return false; // not complete — sleep (killable)
}
if self.done.compare_exchange_weak(
d, d - 1,
Ordering::AcqRel, Ordering::Acquire,
).is_ok() {
return true;
}
// CAS failed (concurrent complete() or another waiter) — retry.
}
})
}
/// Non-blocking check. Returns `true` if the completion has been
/// signaled (done > 0). Does NOT consume the completion.
pub fn try_wait(&self) -> bool {
self.done.load(Ordering::Acquire) > 0
}
/// Signal one waiter. Increments `done` and wakes the
/// highest-priority waiter on the wait queue.
///
/// If `done == u32::MAX` (the `complete_all` sentinel), this is a no-op:
/// `complete_all()` has already been called, and incrementing would wrap
/// `done` to 0, silently destroying the sentinel and causing all future
/// `wait()` calls to block forever. The guard is unconditional (not
/// debug-only) because sentinel destruction is a latent correctness bug
/// that would manifest silently in production, not just in tests.
///
/// **Atomicity**: Uses a CAS loop to atomically check-and-increment `done`,
/// matching the CAS loop in `wait()`. The previous load-then-fetch_add
/// pattern had a TOCTOU race: between the `load(Relaxed)` guard check and
/// the `fetch_add(1, Release)`, a concurrent `complete_all()` could store
/// `u32::MAX`. The `fetch_add` would then wrap `u32::MAX` to 0, destroying
/// the sentinel. The CAS loop prevents this by re-checking the value before
/// committing the increment.
///
/// **Cost**: ~2-5 cycles more than the previous non-atomic path in the
/// uncontended case (one CAS instead of one load + one fetch_add). Negligible
/// for a completion signaling path.
///
/// May be called from any context (including hardirq).
pub fn complete(&self) {
loop {
let prev = self.done.load(Ordering::Relaxed);
if prev == u32::MAX {
// complete_all() sentinel already set — wake is harmless.
break;
}
debug_assert!(prev < u32::MAX - 1, "Completion counter overflow");
if self.done.compare_exchange_weak(
prev, prev + 1, Ordering::Release, Ordering::Relaxed,
).is_ok() {
break;
}
// CAS failed (concurrent complete() or complete_all()) — retry.
// spin_loop() maps to YIELD on ARM, PAUSE on x86, nop on RISC-V.
// This mitigates LL/SC contention on weakly-ordered architectures
// where spurious CAS failures thrash the cache line reservation.
// The loop is bounded by N concurrent callers (at most N*K
// iterations where K is the LL/SC retry factor, typically 2-4).
core::hint::spin_loop();
}
self.wq.wake_up_one();
}
/// Signal ALL current and future waiters. Sets `done` to u32::MAX
/// (sentinel) and wakes all waiters on the wait queue.
///
/// After `complete_all()`, any call to `wait()` returns immediately
/// until `reinit()` is called. Used for barrier patterns where all
/// waiters should proceed (e.g., module init gate).
///
/// May be called from any context (including hardirq).
pub fn complete_all(&self) {
self.done.store(u32::MAX, Ordering::Release);
self.wq.wake_up_all();
}
/// Reset the completion for reuse. Sets `done` to 0.
///
/// MUST only be called when no waiters are blocked (typically after
/// `complete_all()` + all waiters have returned from `wait()`).
/// Calling `reinit()` with blocked waiters is a programming error.
pub fn reinit(&self) {
debug_assert!(
self.wq.is_empty(),
"Completion::reinit() called with blocked waiters"
);
// Release ordering is defensive: by contract, no concurrent readers
// exist when reinit() is called. Relaxed would suffice for correctness.
// Release is used for consistency with complete()/complete_all() and
// costs nothing on TSO architectures (x86-64, s390x) and ~1-2 cycles
// on weakly-ordered architectures (ARM, RISC-V, PPC).
self.done.store(0, Ordering::Release);
}
}
Longevity analysis for done: AtomicU32: Exempt from u64 requirement as a
bounded refcount (not a monotonic identifier). The counter is bounded by
complete_all() (sets to u32::MAX sentinel) or reinit() (resets to 0).
Normal complete() increments by 1; wait() decrements by 1. Wrap past
u32::MAX-1 is detected by a debug assertion. In practice, done rarely
exceeds single digits (one-shot pattern). The only risk is a bug calling
complete() without matching wait() — the debug assertion catches this.
3.6.8 oneshot — One-Shot Value-Carrying Channel¶
Completion (above) signals a wait/wake edge but carries no payload. A
oneshot channel transfers exactly one value from a single producer to a
single consumer and then closes. It is the "submit a request to a worker, block
for the typed result" primitive.
Used by: the module loader (DriverLoadRequest::result_tx carries
Result<DriverHandle, KernelError>, Section 11.4),
KABI demand-loading (Section 12.7),
and any warm/cold-path "request → typed reply" hand-off. The shared state is one
heap allocation (Arc<Inner<T>>), so oneshot::channel is a warm/cold-path
primitive — never used on a hot path.
pub mod oneshot {
/// Shared state behind a one-shot channel. Kernel-internal, not KABI.
struct Inner<T> {
/// The single value in flight. `Some` after `send`, taken by `recv`.
slot: SpinLock<Option<T>>,
/// Wait/wake edge: completed by `send` (value ready) or by the
/// sender's `Drop` (channel closed with no value).
ready: Completion,
/// Set by `send` so the sender's `Drop` knows a value was delivered
/// and must not also signal channel-closed.
sent: AtomicBool,
}
/// Producing half. Consumed by `send` — a value can be sent at most once.
/// Not `Clone` (single producer).
pub struct Sender<T> {
inner: Arc<Inner<T>>,
}
/// Consuming half. Consumed by `recv`. Not `Clone` (single consumer).
pub struct Receiver<T> {
inner: Arc<Inner<T>>,
}
/// Returned by `recv` when the `Sender` was dropped without sending —
/// the request was abandoned (worker cancelled/panicked before replying).
pub struct RecvError;
/// Create a connected `(Sender, Receiver)` pair. One heap allocation.
pub fn channel<T>() -> (Sender<T>, Receiver<T>) {
let inner = Arc::new(Inner {
slot: SpinLock::new(None),
ready: Completion::new(),
sent: AtomicBool::new(false),
});
(Sender { inner: Arc::clone(&inner) }, Receiver { inner })
}
impl<T> Sender<T> {
/// Deliver the value and wake any blocked `recv`. Consumes `self`, so
/// exactly one value can ever be sent. The `sent` flag is set before
/// `complete()` so the subsequent `Drop` of `self` is a no-op.
pub fn send(self, value: T) {
*self.inner.slot.lock() = Some(value);
self.inner.sent.store(true, Ordering::Release);
self.inner.ready.complete();
}
}
impl<T> Drop for Sender<T> {
fn drop(&mut self) {
// If dropped without a successful `send`, wake the receiver so
// `recv` returns `Err(RecvError)` instead of blocking forever.
if !self.inner.sent.load(Ordering::Acquire) {
self.inner.ready.complete();
}
}
}
impl<T> Receiver<T> {
/// Block until the value arrives (or the sender closes the channel).
/// Consumes `self`. MUST NOT be called from atomic context (`wait`
/// may sleep). Returns `Err(RecvError)` if the sender dropped first.
pub fn recv(self) -> Result<T, RecvError> {
self.inner.ready.wait();
self.inner.slot.lock().take().ok_or(RecvError)
}
/// Non-blocking poll. `Some(value)` if a value has arrived (and takes
/// it), `None` if still pending or already taken.
pub fn try_recv(&self) -> Option<T> {
self.inner.slot.lock().take()
}
}
}
3.6.9 CacheLinePadded<T> — Fixed 64-Byte Cache-Line Padding¶
CacheLinePadded<T> places a value alone on a fixed 64-byte cache line on
every architecture. It is the deterministic-size counterpart to
Section 17.3's CacheAligned<T>, whose alignment tracks the target's per-arch
CACHE_LINE_SIZE (32 on PPC32, 64 default, 128 on PPC64LE, 256 on s390x). Use
CacheAligned<T> for pure false-sharing avoidance (it matches the real line size
on each target); use CacheLinePadded<T> when a #[repr(C)] struct's byte
accounting is fixed by a const_assert! and the wrapped field must be exactly 64
bytes regardless of target (e.g. per-CPU doorbell/ring headers whose 64-byte
members are load-bearing for the enclosing struct's asserted size — the VFS and
socket per-CPU ring sets).
/// A value placed alone on a fixed 64-byte cache line. `align(64)` on ALL
/// targets (contrast `CacheAligned<T>` in [Section 17.3](17-containers.md#posix-ipc), which uses the
/// per-arch `CACHE_LINE_SIZE`). `size_of::<CacheLinePadded<T>>()` is `size_of::<T>()`
/// rounded up to a multiple of 64. Kernel-internal — never crosses a wire/KABI
/// boundary, so the fixed alignment carries no cross-arch ABI hazard.
#[repr(C, align(64))]
pub struct CacheLinePadded<T>(pub T);
const_assert!(align_of::<CacheLinePadded<u8>>() == 64);
const_assert!(size_of::<CacheLinePadded<AtomicU64>>() == 64);
3.6.10 SpscRing<T, N> — Lock-Free Single-Producer Single-Consumer Ring Buffer¶
A fixed-capacity, NMI-safe, lock-free ring buffer for unidirectional data flow between exactly one producer and one consumer. This is a core concurrency primitive used across multiple subsystems wherever a single writer must communicate with a single reader without any locking.
/// Lock-free single-producer single-consumer ring buffer.
///
/// # Safety Guarantees
/// - **NMI-safe**: The producer may run in NMI context (no locks, no allocation,
/// no blocking). The consumer runs in task or softirq context.
/// - **Wait-free producer**: `try_push()` completes in O(1) with no CAS loops.
/// It either succeeds or returns `Err(Full)` — never spins.
/// - **Lock-free consumer**: `try_pop()` completes in O(1).
/// - **Memory ordering**: Producer uses `Release` store on `head`; consumer uses
/// `Acquire` load on `head`. Symmetric for `tail`. This is sufficient for all
/// architectures including ARMv7 and PPC32 (no fences needed beyond the
/// atomic ordering).
///
/// # Capacity
/// `N` must be a power of two (enforced at compile time). Effective capacity
/// is `N - 1` to distinguish full from empty.
///
/// # Type constraint
/// `T: Copy` — elements are memcpy'd into/out of the ring. No destructors
/// run inside the ring (NMI safety requires no heap interaction).
pub struct SpscRing<T: Copy, const N: usize> {
/// Ring buffer storage. Cache-line padded to avoid false sharing
/// between producer (writes to `buf[head]`) and consumer (reads from
/// `buf[tail]`).
buf: [MaybeUninit<T>; N],
/// Next write position (producer-owned). Indexed modulo N.
/// Cache-line aligned to avoid false sharing with `tail`.
/// **Longevity**: AtomicU32 wrapping is correct by design. The distance
/// `head.wrapping_sub(tail)` is always in [0, N) because the producer
/// cannot advance head beyond tail + N (full check). Power-of-two
/// bitmask indexing (`idx & (N - 1)`) is wrap-safe. No operational
/// risk from counter wrapping — indices are relative, not absolute.
head: CacheAligned<AtomicU32>,
/// Next read position (consumer-owned). Indexed modulo N.
tail: CacheAligned<AtomicU32>,
}
impl<T: Copy, const N: usize> SpscRing<T, N> {
/// Compile-time bounds: N must fit in u32 (head/tail are AtomicU32) and
/// must be a power of two for efficient modulo via bitmask.
///
/// Placed as an associated const inside the `impl` block so the assertions
/// have access to the generic `N` parameter. A bare `const _: () = { ... }`
/// at module level does not have `N` in scope — it requires the assertions
/// to appear inside the type's own `impl` block where `N` is bound.
const _ASSERT: () = {
assert!(N <= u32::MAX as usize, "SpscRing N exceeds u32::MAX");
assert!(N.is_power_of_two(), "SpscRing N must be a power of two");
};
/// Push an element. Returns `Err(Full)` if the ring is full.
/// NMI-safe: no allocation, no lock, no blocking.
pub fn try_push(&self, val: T) -> Result<(), RingError>;
/// Pop an element. Returns `Err(Empty)` if the ring is empty.
pub fn try_pop(&self) -> Result<T, RingError>;
/// Number of elements currently in the ring.
pub fn len(&self) -> u32;
/// Whether the ring is empty.
pub fn is_empty(&self) -> bool;
}
pub enum RingError {
Full,
Empty,
}
Subsystem usage:
| Subsystem | Type Parameter | Capacity | Producer Context | Consumer Context |
|---|---|---|---|---|
| perf PMU | RawSample |
per-event (configurable) | NMI/PMI handler | perf reader kthread |
| seccomp notify | SeccompNotif |
256 | syscall entry | supervisor process |
| EDAC | u64 (timestamp) |
UE_BURST_WINDOW |
MCE handler (NMI) | EDAC poller kthread |
| IMA | ImaMeasurement |
4096 | file open path | IMA worker kthread |
| audit | AuditRecord |
8192 | syscall exit | audit daemon kthread |
| userfaultfd | UffdMsg |
2048 | page fault handler | uffd monitor process |
| MCE log | MceLogEntry |
32 | NMI handler | mcelog reader |
| pstore | PstoreRecord |
64 | panic/NMI | pstore flush kthread |
All uses share the same SpscRing<T, N> implementation — no per-subsystem
reimplementation. The ring is instantiated inline (no heap allocation for the
ring itself; the containing struct allocates it as a field).
3.6.11 BroadcastRing<T, N> — Single-Producer Multi-Reader Broadcast Ring¶
A fixed-capacity ring with exactly one producer and any number of
independent readers, each tracking its own read cursor outside the ring.
Unlike SpscRing, the producer never waits for a reader: the write cursor
advances unconditionally, and a reader that falls more than N events behind
loses the oldest unread events (overrun), detected by comparing cursor distance
to capacity. This is the broadcast / fan-out primitive — one hardware event
stream delivered to many consumers (e.g. an evdev device ring read by every open
fd, Section 21.3).
/// Single-producer, multi-reader broadcast ring.
///
/// # Model
/// - **One producer**: advances `write_idx` with a `Release` store after writing
/// `buf[write_idx & (N-1)]`. Never blocks and never consults any reader cursor
/// — a slow reader cannot stall the producer.
/// - **Many readers**: each holds its own external cursor (not stored in the
/// ring). A reader loads `write_idx` (`Acquire`), computes
/// `pending = write_idx.wrapping_sub(reader_idx)`, and reads
/// `buf[reader_idx & (N-1)]`.
///
/// # Overrun detection
/// Because the producer overwrites the oldest slot on wrap, a reader detects
/// loss when `pending > N` (its cursor is older than the oldest live slot). The
/// number of dropped events is `pending - N`; the reader resynchronizes by
/// setting `reader_idx = write_idx - N` and (for evdev) emitting `SYN_DROPPED`.
///
/// # Capacity / type
/// `N` is a power of two (bitmask indexing). `T: Copy` — elements are memcpy'd;
/// no destructors run inside the ring (single-writer, NMI/IRQ-safe publish).
pub struct BroadcastRing<T: Copy, const N: usize> {
/// Ring storage; slot `i` is `buf[i & (N-1)]`.
buf: [MaybeUninit<T>; N],
/// Producer write cursor (monotonic, modular). Readers compare their own
/// cursor against this. **Longevity**: `AtomicU32` wrapping is correct by
/// design — `write_idx.wrapping_sub(reader_idx)` is a *relative* distance,
/// never an absolute count, so wrap does not corrupt overrun detection.
write_idx: AtomicU32,
}
impl<T: Copy, const N: usize> BroadcastRing<T, N> {
const _ASSERT: () = {
assert!(N <= u32::MAX as usize, "BroadcastRing N exceeds u32::MAX");
assert!(N.is_power_of_two(), "BroadcastRing N must be a power of two");
};
/// Create an empty ring: `write_idx == 0`, all slots uninitialized.
/// `const fn` so the ring can be constructed inline — as a struct-field
/// initializer or in a `static` — with no heap allocation, matching the
/// section convention that rings are instantiated inline by their
/// containing struct (e.g. `InputDevice.event_ring`,
/// [Section 21.3](21-user-io.md#input-subsystem--input-device-registration)). An empty ring is
/// immediately usable: a reader whose cursor equals the write cursor
/// observes `Err(BroadcastRead::Empty)`, and no slot is ever read
/// before the producer's `Release`-published write to it.
pub const fn new() -> Self;
/// Publish `val` at the current write cursor and advance it. Wait-free, O(1),
/// IRQ/NMI-safe. Overwrites the oldest slot when the ring is full — this is
/// intentional (readers detect the drop via overrun).
pub fn publish(&self, val: T);
/// Read the element at `reader_idx`. Returns `Ok(val)` on success,
/// `Err(BroadcastRead::Empty)` if `reader_idx == write_idx` (nothing new), or
/// `Err(BroadcastRead::Overrun { dropped })` if the reader fell more than `N`
/// behind (caller must resynchronize its cursor).
pub fn read_at(&self, reader_idx: u32) -> Result<T, BroadcastRead>;
/// Current producer cursor (`Acquire`), for readers computing `pending`.
pub fn write_cursor(&self) -> u32;
}
pub enum BroadcastRead {
/// No new events (reader cursor caught up to the producer).
Empty,
/// Reader overran; `dropped` events were lost. Caller must resync its cursor.
Overrun { dropped: u32 },
}
Producer contract: exactly one writer. Multiple writers require external
serialization (a SpinLock around publish), which reintroduces blocking and
is out of scope for this primitive — use SpscRing per reader instead.
3.6.12 RingBuf<T, N> — Bounded FIFO Ring Buffer¶
A fixed-capacity FIFO queue with inline storage and no internal
synchronization. Where SpscRing targets the lock-free single-producer /
single-consumer case with T: Copy, RingBuf is the general bounded queue for
callers that already hold a lock and need move (non-Copy) semantics — e.g. a
per-client MIDI event FIFO held as Mutex<RingBuf<SeqEvent, N>>
(Section 21.4). Elements are moved in and out;
Drop runs on any elements still queued when the ring is dropped.
Callers requiring concurrent access wrap it in a lock (Mutex/SpinLock); it
is deliberately not atomic so the single-threaded path pays nothing.
/// Fixed-capacity FIFO ring buffer. Inline storage, no heap allocation, no
/// internal locking (caller-synchronized). Capacity is exactly `N` (unlike
/// `SpscRing`, whose lock-free full/empty disambiguation costs one slot).
///
/// # Type constraint
/// None — `T` may be non-`Copy`; elements are moved. `Drop` releases any
/// elements still in the ring.
pub struct RingBuf<T, const N: usize> {
/// Ring storage; `head`/`tail`/`len` track occupancy.
buf: [MaybeUninit<T>; N],
/// Index of the next slot to write (modulo N).
head: usize,
/// Index of the next slot to read (modulo N).
tail: usize,
/// Number of occupied slots, in `0..=N`. Distinguishes full from empty
/// (so all `N` slots are usable).
len: usize,
}
impl<T, const N: usize> RingBuf<T, N> {
/// Create an empty ring.
pub const fn new() -> Self;
/// Push an element at the tail. Returns `Err(value)` (giving the element
/// back) if the ring is full — the MIDI FIFO drops it and increments a
/// `lost` counter.
pub fn push(&mut self, value: T) -> Result<(), T>;
/// Pop the oldest element from the head, or `None` if empty.
pub fn pop(&mut self) -> Option<T>;
/// Peek the oldest element (head) without removing it, or `None` if empty.
/// Used by the N_TTY canonical read to inspect the head line record's
/// terminator and length before deciding how many bytes to consume, popping
/// the record only once fully delivered ([Section 21.1](21-user-io.md#tty-and-pty-subsystem)).
pub fn front(&self) -> Option<&T>;
/// Number of queued elements.
pub fn len(&self) -> usize;
/// Whether the ring holds no elements.
pub fn is_empty(&self) -> bool;
/// Whether the ring is at capacity (`len == N`).
pub fn is_full(&self) -> bool;
/// Discard all queued elements, resetting the ring to empty (drops each `T`).
/// Used by the N_TTY input flush when signal-generating input arrives without
/// `NOFLSH` ([Section 21.1](21-user-io.md#tty-and-pty-subsystem)); this clears committed input on ^C/^\/^Z.
pub fn clear(&mut self);
/// Total capacity `N`.
pub const fn capacity(&self) -> usize { N }
}
3.6.13 Static Keys — Zero-Overhead Static Branches¶
A static key (jump label) is a boolean condition whose runtime read cost is
zero. The check site is a single machine instruction — a NOP when the key is
disabled, an unconditional branch when enabled — so a disabled key consumes no
branch-predictor slot, no data cache line, and no load. Writers pay a one-time
instruction-patching cost to flip the site; readers pay nothing. This is the
mechanism behind LSM hook bypass (Section 9.8),
tracepoint dispatch (Section 20.2), and
ML observation gating (Section 23.1),
all of which must be free when inactive.
Merit (not Linux parity). The value is the ~0-cycle disabled read. A plain
AtomicBool flag costs a load plus a data-dependent branch on every hot-path
check — LSM has_cap(), every tracepoint site, every observe_kernel! — and at
the hundreds of sites hit per syscall that is measurable. Patching the code
itself removes both the load and the branch. UmkaOS uses the portable jump-label
form on all eight architectures; the ISA-specific NOP/branch encodings and
I-cache serialization live behind arch::current::text_patch and are tabulated
per-arch in Section 20.2 — never in this
generic type.
/// A static key: a runtime-flippable boolean whose read site is a patched
/// instruction (NOP when disabled, branch when enabled). Read cost ~0; write
/// cost is a one-time code patch across all registered sites.
///
/// Kernel-internal, not KABI: a `StaticKey` never crosses a domain, wire, or
/// userspace boundary. Its authoritative state lives in non-Evolvable storage
/// (a Tier-0 `static` or a Nucleus `PersistentSlot`) — see "Static Keys and
/// Live Evolution" below for why.
pub struct StaticKey {
/// Enable refcount. `0` = disabled (every site is `NOP`); `> 0` = enabled
/// (every site branches). A refcount, not a bool, so independent enablers
/// share one key: the code is patched only on the `0 → 1` and `1 → 0` edges.
/// `AtomicI32` (not `u64`): bounded by the number of concurrent enablers,
/// and the signed type makes a disable-without-matching-enable underflow
/// detectable — the same rationale that exempts refcounts from the u64
/// counter policy.
enabled: AtomicI32,
}
impl StaticKey {
/// Const constructor for a key that starts **disabled** (sites are `NOP`).
/// Usable in a `static` initializer.
pub const fn new_disabled() -> Self {
Self { enabled: AtomicI32::new(0) }
}
/// Const constructor for a key that starts **enabled** (sites branch).
pub const fn new_enabled() -> Self {
Self { enabled: AtomicI32::new(1) }
}
/// Read the key. At a hot-path site prefer the `static_key_enabled!` macro,
/// which the compiler places inline as the patched instruction. This method
/// is the fallback data-path read (a single relaxed load) used where the key
/// is reached dynamically and no patch site was emitted. `&self`: a reader
/// never mutates the key.
#[inline(always)]
pub fn is_enabled(&self) -> bool {
self.enabled.load(Relaxed) > 0
}
/// Enable the key (writer side; warm/cold path only). Increments the
/// refcount; on the `0 → 1` edge it patches every registered call site from
/// `NOP` to a branch via `arch::current::text_patch`, which serializes
/// against concurrent execution on all CPUs. `&'static self`: patch sites
/// reference the key by a stable address, so a `StaticKey` must outlive
/// every site — enforced by requiring a `'static` receiver.
pub fn enable(&'static self) {
if self.enabled.fetch_add(1, AcqRel) == 0 {
arch::current::text_patch::patch_static_key(self, true /* to_branch */);
}
}
/// Disable the key (writer side). Decrements; on the `1 → 0` edge it patches
/// every site back to `NOP`. Underflow (more disables than enables) is a bug
/// caught by the debug assertion on the pre-decrement value.
pub fn disable(&'static self) {
let prev = self.enabled.fetch_sub(1, AcqRel);
debug_assert!(prev > 0, "StaticKey disabled more times than enabled");
if prev == 1 {
arch::current::text_patch::patch_static_key(self, false /* to_branch */);
}
}
}
/// Reader used where a `StaticKey` is reached through a field path rather than a
/// bare `static` (e.g. `static_key_enabled(&LSM_REGISTRY.file_hooks_active)`).
/// Equivalent to `key.is_enabled()`.
#[inline(always)]
pub fn static_key_enabled(key: &StaticKey) -> bool {
key.is_enabled()
}
/// Enable a static key by reference — the by-name API the ML-policy and
/// tracepoint subsystems call.
pub fn static_key_enable(key: &'static StaticKey) {
key.enable();
}
/// Disable a static key by reference.
pub fn static_key_disable(key: &'static StaticKey) {
key.disable();
}
The hot-path read is emitted by a macro so the compiler inlines the patch site directly into the caller:
/// Compile-time-inlined static-branch check. Expands to the patched instruction
/// at the call site (a `NOP` when the key is disabled, a branch when enabled).
/// Prefer this at hot-path sites; use `static_key_enabled()` when the key is
/// reached dynamically. `$key` must resolve to a `&'static StaticKey`.
macro_rules! static_key_enabled {
($key:expr) => {{
// Emits the jump-label site: a NOP patched to a branch when $key is
// enabled. `static_branch_site` records this site's address in the
// owning module's `.umka_static_keys` section, keyed by $key's address,
// so the site can be re-patched if the module is reloaded.
$crate::arch::current::text_patch::static_branch_site(&$key)
}};
}
3.6.13.1 Call-Site Registration¶
Each expansion of static_key_enabled! places a record — (key_address,
site_address) — into a per-module .umka_static_keys linker section (the
UmkaOS analogue of Linux's struct jump_entry in __jump_table). When
enable()/disable() flips a key, arch::current::text_patch walks the set of
sites registered for that key address and rewrites each one. A key with no
registered sites (e.g., enabled before its module loaded) simply records the
refcount; the sites are patched to the correct state at registration time (next
subsection).
3.6.13.2 Static Keys and Live Evolution¶
Live evolution (Section 13.18) replaces an Evolvable module's
.text wholesale: the new binary arrives with its jump-label sites in their
compiled default state (NOP), and — because Evolvable modules are required to
be stateless (no .data/.bss; the load-time validator rejects mutable statics)
— a StaticKey can never live inside an Evolvable module. Its authoritative
enabled refcount lives in non-Evolvable storage: a Tier-0 static (e.g.,
LSM_REGISTRY) or a Nucleus PersistentSlot. The patch sites, by contrast,
are embedded in whatever code reads the key — which may be Evolvable and thus
ephemeral.
This is the UmkaOS-specific design point. The safe resolution is re-apply on load, from persistent state:
- When a module is loaded (first load or evolution reload), the loader walks the
module's
.umka_static_keyssection. For each(key_address, site_address)record it reads the referenced key's currentenabledrefcount and patches the fresh site to the matching state (NOPif0, branch if> 0). - This happens while the new module's code is quiesced and not yet spliced into
the live call graph (Phase A/B of the evolution protocol, or the initial load
before the module's entry points are published), so no CPU can observe a
half-patched site. The patch uses the same
arch::current::text_patchserialization as a normalenable()/disable(), so the ordering rules are identical. - Correspondingly, unloading a module removes its sites from the key's site set
before the
.textis freed, so a later flip never writes into reclaimed memory.
The refcount surviving in persistent storage is what makes this correct: a key
enabled before a reader module existed, or across a reload of that module, is
re-materialized in the fresh code at load time rather than silently reverting to
its compiled NOP default. This mirrors Linux's jump_label_add_module() / jump_label_apply_nops()
sequence, but is stated here as a first-class
consequence of the Nucleus/Evolvable split rather than an afterthought.
Alternative considered (rejected): storing the key inside the Evolvable module and re-deriving state from a checkpoint on reload. Rejected because it duplicates the authoritative enable state into swappable memory, reintroducing exactly the mutable-static hazard the stateless-module rule exists to prevent, and it races the reload window (the checkpoint must be captured and replayed atomically with the swap). Keeping the state non-Evolvable and re-applying to sites is simpler and has a single source of truth.
3.6.14 Notifier Chains — Ordered RCU Callback Lists¶
crate::notify is the kernel's generic notifier chain: an ordered set of
callbacks that subsystems register to be told about an event (a power-supply
state change, an orderly reboot, a device hotplug). It is the one framework all
subsystems use instead of each hand-rolling a callback list, and it is defined
here (not in any device class) because it is a general concurrency primitive.
Readers (notify()) traverse the chain lock-free under RCU; writers
(register/unregister) serialize on an internal SpinLock. This matches the
read-mostly access pattern — chains are notified far more often than they are
mutated.
// umka-nucleus/src/notify/mod.rs — generic ordered callback chain.
use core::ffi::c_void;
use core::sync::atomic::{AtomicPtr, Ordering};
/// Outcome of one notifier callback, controlling chain traversal.
#[repr(u32)]
pub enum NotifierResult {
/// Continue to the next block in the chain.
Proceed = 0,
/// Stop the chain: later (lower-priority) blocks are not called. Used to
/// veto or short-circuit an action (e.g. a reboot notifier that aborts a
/// pending shutdown).
Stop = 1,
}
/// Type-erased dispatch trampoline stored in a `NotifierBlock`. Given the block
/// and the chain event, it recovers the typed owner and invokes the owner's
/// handler, returning whether traversal should continue. Monomorphized per
/// owner type by `NotifierBlock::new` / `register_reboot_notifier`.
pub type NotifierTrampoline =
fn(nb: &NotifierBlock, event: u64, data: *mut c_void) -> NotifierResult;
/// A node in a `NotifierChain`. Embedded (intrusively) in the owner struct so a
/// callback can recover its owner via `container_of`.
///
/// **Kernel-internal, not KABI/wire.** The sizes of the linkage pointer and the
/// trampoline pointer are implementation-defined; a `#[repr(C)]` owner that
/// embeds a `NotifierBlock` therefore cannot assert a fixed total struct size
/// (e.g. `WatchdogRebootHook`, the subsystem-level reboot hook,
/// [Section 13.19](13-device-classes.md#hardware-watchdog-framework)).
pub struct NotifierBlock {
/// Next block in the priority-ordered list (RCU-published). Null at tail.
next: AtomicPtr<NotifierBlock>,
/// Trampoline invoking the owner's typed handler. Set at construction.
call: NotifierTrampoline,
/// Higher priority runs earlier. Equal priorities run in registration order.
priority: i32,
}
impl NotifierBlock {
/// Construct a block whose `call` trampoline invokes `handler(&mut owner)`.
/// The owner is recovered from this block by `container_of` at notify time, so
/// `Self` MUST be embedded in the owner `T` and constructed in place; the
/// recovery offset is fixed by the per-`T` monomorphization of this
/// constructor. `priority`: higher runs earlier.
pub const fn new<T>(handler: fn(&mut T), priority: i32) -> Self;
}
/// An ordered callback chain. One instance per event source (a per-device
/// `NotifierChain` field, or a global chain such as `REBOOT_NOTIFIER_CHAIN`).
pub struct NotifierChain {
/// Head of the priority-ordered block list (RCU-published).
head: AtomicPtr<NotifierBlock>,
/// Serializes register/unregister; readers never acquire it.
write_lock: SpinLock<()>,
}
impl NotifierChain {
/// Construct an empty chain. `const` so it can initialize a struct field or
/// a `static` global.
pub const fn new() -> Self;
/// Link `nb` into the chain in descending `priority` order. Warm path
/// (registration is rare). Returns `KernelError::EEXIST` if `nb` is already
/// linked into a chain. `nb` must outlive its registration (typically it is
/// embedded in a long-lived owner struct or a `static`).
pub fn register(&self, nb: &'static NotifierBlock) -> Result<(), KernelError>;
/// Unlink `nb`. SYNCHRONOUSLY waits out any in-flight `notify()` (RCU grace
/// period) before returning, so no callback runs after this returns.
pub fn unregister(&self, nb: &'static NotifierBlock);
/// Notify all registered blocks in priority order with `event`/`data`. Stops
/// early if a block returns `NotifierResult::Stop`, returning that result;
/// otherwise returns `NotifierResult::Proceed`. Runs under `rcu_read_lock()`
/// — callbacks must not sleep and must not register/unregister on this chain.
pub fn notify(&self, event: u64, data: *mut c_void) -> NotifierResult;
}
Longevity: event is u64, so no chain event code wraps within the
operational lifetime. Registration/unregistration is idempotent and leak-free —
unregister() fully unlinks a block, and re-registration of the same block is
rejected until it is unlinked.
3.6.15 RcuList<T> — RCU-Protected Singly-Linked List¶
RcuList<T> is the kernel's generic RCU-protected singly-linked list for
append-mostly, iterate-often collections whose readers must run lock-free under
an &RcuReadGuard. Readers traverse without acquiring any lock; writers
serialize among themselves via an external lock (the container's), publish
new nodes with a Release store, and reclaim unlinked nodes only after a grace
period (rcu_call) — so a reader that already observed a node keeps a valid
&T for the duration of its read section. Consumers:
Section 20.2 (RcuList<BpfProg>, the per-callsite attached-probe
list) and Section 17.2 (RcuList<NetlinkPortId>, a per-CPU taskstats
listener list).
// umka-nucleus/src/rcu/list.rs — generic RCU-protected singly-linked list.
use core::marker::PhantomData;
use core::sync::atomic::{AtomicPtr, Ordering};
/// A node owns its `value` by heap allocation. Reclaimed via `rcu_call` after a
/// grace period once unlinked.
struct RcuListNode<T> {
/// Next node (RCU-published). Null at tail.
next: AtomicPtr<RcuListNode<T>>,
/// Payload.
value: T,
}
/// RCU-protected singly-linked list. `T` is stored by value inside heap nodes.
pub struct RcuList<T> {
/// Head of the list (RCU-published). Null = empty.
head: AtomicPtr<RcuListNode<T>>,
}
impl<T> RcuList<T> {
/// Construct an empty list. `const` so it can initialize a struct field.
pub const fn new() -> Self {
Self { head: AtomicPtr::new(core::ptr::null_mut()) }
}
/// Prepend `value` (O(1)). The caller MUST hold the container's write lock —
/// concurrent writers are NOT internally serialized. Readers concurrent with
/// this call see either the old or the new head, never a torn node (the node
/// is fully initialized before the `Release` publish).
pub fn push_front(&self, value: T) {
let node = Box::into_raw(Box::new(RcuListNode {
next: AtomicPtr::new(self.head.load(Ordering::Relaxed)),
value,
}));
// Release publishes the fully-initialized node to lock-free readers.
self.head.store(node, Ordering::Release);
}
/// Iterate the list under RCU. Each yielded `&T` lives only as long as
/// `guard`. Lock-free; safe concurrently with `push_front`/`remove`.
pub fn iter<'g>(&self, _guard: &'g RcuReadGuard) -> RcuListIter<'g, T> {
RcuListIter {
// Acquire pairs with `push_front`'s Release store.
cur: self.head.load(Ordering::Acquire),
_guard: PhantomData,
}
}
/// Unlink the first node whose value satisfies `pred` and schedule its
/// storage for RCU-deferred free (`rcu_call`). The caller MUST hold the
/// container's write lock. Returns `true` if a node was removed. `T: 'static`
/// so the deferred free closure outlives the grace period.
pub fn remove<F: Fn(&T) -> bool>(&self, pred: F) -> bool
where
T: 'static,
{
// Single-writer walk (readers may run concurrently). `prev` is the atomic
// link pointing at the current node.
let mut prev = &self.head;
loop {
let cur = prev.load(Ordering::Acquire);
if cur.is_null() {
return false;
}
// SAFETY: `cur` is a live node kept valid by the write-lock (no other
// writer) and by RCU (no reclaim before grace period).
let node = unsafe { &*cur };
if pred(&node.value) {
// Unlink: point `prev` past `cur` (Release for readers).
prev.store(node.next.load(Ordering::Acquire), Ordering::Release);
// Reclaim after a grace period; a concurrent reader may still
// hold `&node.value`.
rcu_call(move || {
// SAFETY: reconstructs the Box that `push_front` leaked;
// reached only after the grace period, so no reader observes it.
drop(unsafe { Box::from_raw(cur) });
});
return true;
}
prev = &node.next;
}
}
}
/// Lock-free forward iterator over an `RcuList<T>`, valid for the lifetime of the
/// `RcuReadGuard` it borrows.
pub struct RcuListIter<'g, T> {
cur: *const RcuListNode<T>,
_guard: PhantomData<&'g RcuReadGuard>,
}
impl<'g, T> Iterator for RcuListIter<'g, T> {
type Item = &'g T;
fn next(&mut self) -> Option<&'g T> {
if self.cur.is_null() {
return None;
}
// SAFETY: the node is kept alive for the grace period (RCU); `guard`
// outlives the returned `&'g T`; Acquire pairs with the publisher's Release.
let node = unsafe { &*self.cur };
self.cur = node.next.load(Ordering::Acquire);
Some(&node.value)
}
}
Longevity: nodes are freed via rcu_call on removal — no leak. The list has
no fixed capacity; a per-node heap allocation is acceptable because inserts are
warm-path (probe attach, listener register), never per-packet.
3.6.16 TokenBucket — Lock-Free Token-Bucket Rate Limiter¶
TokenBucket is the kernel's canonical rate-limiting primitive: an atomic,
&self token bucket that subsystems embed and drive from a SHARED reference
(a &NetconsoleTarget, a &PtySlaveState) with no external lock. Subsystems
requiring rate limiting (netconsole transmit throttling, TTY signal injection,
and any other advisory event/byte-rate cap) use this type rather than
reimplementing token-bucket logic. It is defined here (not in a device class)
because it is a general concurrency primitive.
It is advisory, not a security boundary: under contention a concurrent refill
may grant or lose O(1) tokens (the refill and consume paths use Relaxed
CAS/swap), which is acceptable for rate limiting. Distinct same-named-concept
variants exist for genuinely different needs and are NOT interchangeable:
AuditTokenBucket (Section 20.2, &mut self behind a SpinLock,
u128-intermediate byte rates for the audit path). PolicyServiceRateLimiter
(Section 23.1) is the ML-policy
#[repr(C)] single-token variant. The DSM anti-entropy sender does NOT use a
distinct bucket type — its byte-rate limiter is sender-private plain state owned
by the single anti-entropy sender through &mut self
(Section 6.13), not a shared-&self atomic bucket.
// umka-nucleus/src/rcu/../sync/token_bucket.rs — generic lock-free rate limiter.
use core::sync::atomic::{AtomicU64, Ordering};
/// Lock-free token-bucket rate limiter. Kernel-internal (embedded in owner
/// structs, never crosses a wire/KABI boundary). `align(64)` keeps the hot
/// atomics on their own cache line (no false sharing with neighbouring fields).
///
/// **32-bit legs (cfg-split coupled state)**: the three mutable fields
/// (`tokens`, `last_refill_ns`, `remainder_ns`) are ONE logical state — the
/// lock-free protocol CASes `tokens` and swaps `last_refill_ns` as a coupled
/// pair. On `target_has_atomic = "64"` they are independent `AtomicU64`s driven
/// by the lock-free protocol below. On 32-bit legs without a native 64-bit
/// atomic (PPC32) there is no sound way to drive coupled 64-bit state lock-free,
/// so they move under a single per-bucket `SpinLock` as plain `u64`s — the
/// 64-bit-atomic family's explicit-lock branch
/// ([Section 3.5](#locking-strategy--64-bit-atomics-on-32-bit-legs-the-semantic-family)).
/// The public API is identical on both legs; the immutable `capacity` /
/// `ns_per_token` stay plain `u64` outside the lock on both legs.
///
/// **Hot-path justification (32-bit legs)**: `try_consume` takes the spinlock
/// once per consume/refill. Acceptable because (i) rate limiting is advisory and
/// off any per-packet fast path — callers are netconsole TX throttling and TTY
/// signal injection (bounded rate); and (ii) PPC32 (the only affected leg) runs
/// low core counts, so per-bucket lock contention is negligible.
#[cfg(target_has_atomic = "64")]
#[repr(align(64))]
pub struct TokenBucket {
/// Current whole-token count (0..=capacity).
tokens: AtomicU64,
/// Nanosecond timestamp of the last refill.
last_refill_ns: AtomicU64,
/// Fractional-nanosecond remainder carried across refills so sub-token
/// intervals are never lost (exact long-term rate, no floating point).
remainder_ns: AtomicU64,
/// Burst capacity (maximum token count).
capacity: u64,
/// Nanoseconds required to produce one token (= 1e9 / refill_rate_per_sec),
/// stored as the divisor to avoid per-refill division. `u64::MAX` = never
/// refills (rate 0).
ns_per_token: u64,
}
/// 32-bit-leg variant: the coupled mutable state lives under one per-bucket
/// spinlock (see the type doc above). Same public API as the 64-bit variant.
#[cfg(not(target_has_atomic = "64"))]
#[repr(align(64))]
pub struct TokenBucket {
/// Coupled mutable state (token count + refill timestamps) under one lock.
state: SpinLock<TokenBucketState>,
/// Burst capacity (maximum token count). Immutable after construction.
capacity: u64,
/// Nanoseconds required to produce one token (= 1e9 / refill_rate_per_sec).
/// Immutable after construction. `u64::MAX` = never refills (rate 0).
ns_per_token: u64,
}
/// The coupled `TokenBucket` mutable state, held under one `SpinLock` on 32-bit
/// legs (plain `u64`s — the lock, not the atomic width, provides tear-freedom).
#[cfg(not(target_has_atomic = "64"))]
struct TokenBucketState {
/// Current whole-token count (0..=capacity).
tokens: u64,
/// Nanosecond timestamp of the last refill.
last_refill_ns: u64,
/// Fractional-nanosecond remainder carried across refills so sub-token
/// intervals are never lost (exact long-term rate, no floating point).
remainder_ns: u64,
}
#[cfg(target_has_atomic = "64")]
impl TokenBucket {
/// Create a bucket with `capacity` burst tokens, refilling at
/// `refill_rate_per_sec` tokens/second, starting full.
pub fn new(capacity: u64, refill_rate_per_sec: u64) -> Self {
Self {
tokens: AtomicU64::new(capacity),
last_refill_ns: AtomicU64::new(0),
remainder_ns: AtomicU64::new(0),
capacity,
ns_per_token: if refill_rate_per_sec == 0 {
u64::MAX
} else {
1_000_000_000 / refill_rate_per_sec
},
}
}
/// Try to consume `n` tokens, refilling first based on elapsed time. Returns
/// `true` if `n` tokens were available (event allowed), `false` if the bucket
/// is short (rate-limited). Lock-free and `&self`, so a shared reference
/// suffices. `Relaxed` throughout — this is advisory rate limiting, not
/// synchronization.
pub fn try_consume(&self, n: u64) -> bool {
let now_ns = crate::arch::current::cpu::read_monotonic_ns();
// Refill. `swap` means concurrent callers race on the elapsed window (one
// wins the interval, others see ~0) — advisory, acceptable.
let last = self.last_refill_ns.swap(now_ns, Ordering::Relaxed);
if self.ns_per_token != u64::MAX && self.ns_per_token != 0 {
let elapsed = now_ns.saturating_sub(last);
let carry = self.remainder_ns.load(Ordering::Relaxed);
let total = elapsed.saturating_add(carry);
let new_tokens = total / self.ns_per_token;
self.remainder_ns.store(total % self.ns_per_token, Ordering::Relaxed);
if new_tokens > 0 {
let mut cur = self.tokens.load(Ordering::Relaxed);
loop {
let refilled = cur.saturating_add(new_tokens).min(self.capacity);
match self.tokens.compare_exchange_weak(
cur, refilled, Ordering::Relaxed, Ordering::Relaxed) {
Ok(_) => break,
Err(observed) => cur = observed,
}
}
}
}
// Consume `n` if available (CAS to avoid wrapping underflow).
let mut cur = self.tokens.load(Ordering::Relaxed);
loop {
if cur < n {
return false;
}
match self.tokens.compare_exchange_weak(
cur, cur - n, Ordering::Relaxed, Ordering::Relaxed) {
Ok(_) => return true,
Err(observed) => cur = observed,
}
}
}
}
/// 32-bit-leg variant: identical public API and arithmetic, but the coupled
/// state is mutated under one per-bucket spinlock (plain `u64`s) because PPC32
/// has no native 64-bit atomic to drive the lock-free protocol.
#[cfg(not(target_has_atomic = "64"))]
impl TokenBucket {
/// Create a bucket with `capacity` burst tokens, refilling at
/// `refill_rate_per_sec` tokens/second, starting full.
pub fn new(capacity: u64, refill_rate_per_sec: u64) -> Self {
Self {
state: SpinLock::new(TokenBucketState {
tokens: capacity,
last_refill_ns: 0,
remainder_ns: 0,
}),
capacity,
ns_per_token: if refill_rate_per_sec == 0 {
u64::MAX
} else {
1_000_000_000 / refill_rate_per_sec
},
}
}
/// Try to consume `n` tokens, refilling first based on elapsed time. Returns
/// `true` if `n` tokens were available (event allowed), `false` if the bucket
/// is short (rate-limited). `&self`, so a shared reference suffices; the
/// coupled state is guarded by one per-bucket spinlock (see the type doc's
/// hot-path justification). Same observable behaviour as the lock-free
/// 64-bit path.
pub fn try_consume(&self, n: u64) -> bool {
let now_ns = crate::arch::current::cpu::read_monotonic_ns();
let mut st = self.state.lock();
// Refill based on elapsed time since the last consume/refill.
let last = st.last_refill_ns;
st.last_refill_ns = now_ns;
if self.ns_per_token != u64::MAX && self.ns_per_token != 0 {
let elapsed = now_ns.saturating_sub(last);
let total = elapsed.saturating_add(st.remainder_ns);
let new_tokens = total / self.ns_per_token;
st.remainder_ns = total % self.ns_per_token;
if new_tokens > 0 {
st.tokens = st.tokens.saturating_add(new_tokens).min(self.capacity);
}
}
// Consume `n` if available.
if st.tokens < n {
return false;
}
st.tokens -= n;
true
}
}
Longevity: token counts and timestamps are u64 on both legs (independent
AtomicU64s where a native 64-bit atomic exists; plain u64s under a per-bucket
spinlock on 32-bit legs) and do not wrap within the operational lifetime. No
allocation; the bucket is embedded in its owner.
3.6.17 The Finite-Tag Stall Rule¶
Normative, corpus-wide. No correctness argument for a lock-free or optimistic protocol may assume an upper bound on how long a thread or vCPU can stall between two of its own memory operations. Hypervisor descheduling, VM pause / resume / migration, SMI, and debugger stops are UNBOUNDED, and UmkaOS explicitly runs virtualized and targets 50-year uptime, so any argument of the form "tag wrap requires 2^N events, which at a realistic rate is minutes-to-hours of continuous stall and therefore cannot happen" is INADMISSIBLE — every such passage is a defect, regardless of how large the derived stall duration is.
Every finite-generation or finite-sequence optimistic protocol — an ABA-guarded
Treiber tag, a seqlock generation, a packed [gen | idx] handle — MUST carry a
one-line site classification as exactly one of:
- (i) Stall-immune by construction — a tag wrap-to-collision requires participation of the stalled thread itself (e.g. the observed slot cannot be recycled without the observer's own release), so stalling cannot create the hazard. Correctness rests on the participation structure, never on a duration.
- (ii) Advisory — the optimistically-read value is a routing / hint snapshot only; every consumer re-validates against an authoritative source (lock-held state, or a protocol NACK / forward / retry at the recipient) before ANY irreversible effect, so a wrap-corrupted acceptance is observationally equivalent to an ordinary stale snapshot and is absorbed by the same recovery. Seqlock-snapshot sites of this class additionally obey the SEQLOCK SNAPSHOT CONTRACT (Section 6.3): snapshots route, locks decide.
- (iii) Retired — where neither (i) nor (ii) can be established, the optimistic
protocol is replaced by an explicit lock or an ownership transfer. The canonical
TokenBucket32-bit leg (a per-bucketSpinLock<TokenBucketState>) and the DSM free-list pools (Section 6.6, Section 6.5) are class (iii): the head runs under a leafSpinLockon ALL legs.
A site whose classification cannot be established from its own section is filed as
an individual ledger finding — never guessed, and never papered over with a stall
bound. Classification of the remaining corpus finite-tag / seqlock sites —
including guarded_claim (Section 3.11), expected to be class (i)
only if the claimed slot's tag cannot recycle without the claimant's own release —
is carried to a dedicated round.
3.7 Scalability Analysis: Hot-Path Metadata on 256+ Cores¶
Moving from a monolithic kernel to a hybrid one can introduce new contention bottlenecks in the "core" layer — specifically in metadata structures that every I/O operation must touch. This section analyzes UmkaOS's three highest-contention metadata subsystems on many-core machines (256-512 cores, 4-8 NUMA nodes).
1. Capability Table (Section 9.1)
Every syscall and driver invocation performs a capability check. On a 512-core machine running 10,000 concurrent processes, the capability table sees millions of lookups per second.
Design for contention: - Per-process capability tables: each process has its own capability table (a small array indexed by capability handle, typically <256 entries). Lookups are process-local with no cross-process contention. Two processes on different CPUs never touch the same capability table. - Capability creation/delegation (write path): uses per-process lock. Only contends when the same process creates capabilities from multiple threads simultaneously (rare — capability creation is a cold path). - Capability revocation: RCU-deferred. Revoking a capability marks it as invalid (atomic store), then defers memory reclamation to an RCU grace period. No lock on the read path.
Contention profile: none on read path (per-process, indexed by local handle). Comparable to Linux's file descriptor table — per-process, never a global bottleneck.
2. Unified Object Namespace / umkafs (Section 20.5)
The object namespace provides /sys/kernel/umka/ introspection. On a busy system,
monitoring tools (umka-top, prometheus-exporter) may read thousands of objects
per second.
Design for contention:
- Registry reads: the device registry (Section 11.4) is RCU-protected. Reads are
lock-free. Enumeration snapshots use a seqlock to detect concurrent modifications.
- Attribute reads (sysfs-style): most attributes are backed by atomic counters
or per-CPU counters that are aggregated on read. Example: a NIC's rx_packets
counter is per-CPU; reading it sums all per-CPU values. No lock on the write path
(per-CPU increment), brief aggregation on the read path.
- Object creation/destruction (hot-plug, process exit): per-subsystem lock, not
global. Creating a network device takes the network subsystem lock; creating a block
device takes the block subsystem lock. No global namespace lock.
- Pathological case: a monitoring tool reading every object attribute on every CPU
at 1Hz on a 512-core, 10,000-device system. The aggregation overhead is proportional
to num_cpus × num_objects — at 512 × 10,000, this is ~5M atomic reads per scan.
At ~5ns each, one scan takes ~25ms. This is acceptable for 1Hz monitoring; for
higher-frequency monitoring, tools should read only the objects they need.
Note: This 25ms scan runs in process context (a monitoring thread), not in
interrupt context. It does not affect the 50μs interrupt latency guarantee
from Section 8.5, which governs ISR entry latency — a completely orthogonal concern.
Contention profile: lock-free reads (RCU + atomics), per-subsystem writes. Bottleneck only if monitoring tools read aggressively (configurable rate limiting).
3. Page Cache and Memory Management (Section 4.1)
The page cache is the single highest-contention structure in any kernel. Every file read, every mmap fault, every page writeback touches it.
Design for contention:
- Page cache lookups: RCU-protected radix tree, per-inode (same as Linux's xarray).
Two threads faulting pages from different files never contend. Two threads faulting
different pages from the same file contend only at the xarray level (fine-grained,
per-node locking within the radix tree).
- LRU lists: per-NUMA-node LRU lists with per-CPU page batching (same design as
Linux). Pages are added to a per-CPU buffer; the buffer is drained to the LRU list
in batches of 15 pages. This amortizes the per-NUMA LRU lock to 1 acquisition per
15 page faults.
- Buddy allocator: per-NUMA-node lock, with per-CPU free page pools absorbing >95%
of allocations (Section 4.3). On a 512-core, 8-NUMA-node system, each buddy
allocator sees ~1/8 of the allocation traffic, and per-CPU pools absorb most of it.
- Per-VMA locks (see Section 4.8) eliminate mmap_lock
as a scalability bottleneck for the page fault path. On 256 cores, page faults acquire
only the per-VMA vm_lock.read(), which is per-VMA (not per-process). VMA structural
modifications (mmap/munmap) still acquire mmap_lock.write(), but these are
infrequent relative to page faults (typically <0.1% of memory operations). The remaining
contention point is the page table lock (PTL), which is
per-page-table-page and thus naturally distributed.
- Known scaling limit: extreme mmap/munmap churn (thousands of VMAs per second
per process) contends on the per-process mmap_lock.write(). UmkaOS uses the same maple
tree (lockless reads, locked writes) as Linux 6.1+. For workloads that create/destroy
VMAs at extreme rates (JVMs, some databases), this is a known bottleneck in all
current kernels. Per-VMA locks do not help here — the contention is on mmap_lock.write()
for structural modifications, not on the fault path. UmkaOS does not claim to solve
it — the contention is fundamental to the VMA data structure, not to the hybrid
architecture.
Contention profile: per-inode, per-NUMA, per-CPU on all hot paths. No new bottlenecks introduced by the hybrid architecture — UmkaOS's page cache follows the same design as Linux (which already runs on 512+ core machines).
Summary — the hybrid architecture's overhead (isolation domain switches) is orthogonal to metadata contention. The "core" layer's metadata structures use the same per-CPU, per- NUMA, RCU-based techniques that make Linux scale to 256+ cores. No new global locks are introduced. The capability system is per-process (no shared structure), the object namespace is RCU-read / per-subsystem-write, and the page cache follows Linux's proven xarray + per-CPU page batching design.
3.8 Interrupt Handling¶
- All device interrupts are routed to threaded interrupt handlers (same as Linux
IRQF_THREADED). - Top-half handlers are kept minimal: acknowledge interrupt, wake thread.
- This ensures all interrupt processing is preemptible and schedulable.
- MSI/MSI-X is preferred for all PCI devices (per-queue interrupts, no sharing).
- High-rate interrupt paths (100GbE at line-rate, NVMe at millions of IOPS): the threaded-handler model introduces scheduling latency (~1-5μs per interrupt) that could limit throughput if each packet triggered a separate interrupt. UmkaOS addresses this the same way Linux does — NAPI-style polling: the first interrupt wakes the thread, the thread then polls the device ring in a busy loop until the ring is drained, then re-enables interrupts. For 100GbE, this means the thread is woken once per batch of 64-256 packets, not once per packet. Combined with MSI-X per-queue affinity (one interrupt thread per CPU, one queue per CPU), the scheduling overhead is amortized to <100ns per packet, which is within the performance budget (Section 1.3).
3.8.1 Inter-Processor Cross-Calls (Generic)¶
Cross-CPU work is dispatched with smp_send_ipi(mask: CpuMask, kind: IpiKind).
The transport underneath is the per-architecture IPI
(arch::current::interrupts::send_ipi()); IpiKind is the generic, arch-neutral
description of what the target CPU should do. Data-free variants correspond to
the arch "action bitmask" (reschedule / TLB flush / function call / stop);
data-carrying variants pass a borrowed payload that the sender keeps live for the
duration of the synchronous cross-call (it blocks until every target
acknowledges).
/// Work requested of one or more remote CPUs via `smp_send_ipi()`.
pub enum IpiKind<'a> {
/// Wake the target CPU's scheduler to re-evaluate its runqueue.
Reschedule,
/// Full TLB flush on the target CPU (whole address space / ASID).
TlbFlushAll,
/// Invalidate a specific set of virtual address ranges on the target
/// (batched multi-range TLB shootdown). The slice is borrowed for the
/// duration of the synchronous cross-call; the sender must not drop the
/// backing storage until `smp_send_ipi()` returns. `VaRange` is defined
/// below in this section.
TlbFlushMulti(&'a [VaRange]),
/// Run `func(data)` on the target CPU (generic function-call IPI).
CallFunction { func: fn(*mut ()), data: *mut () },
/// Halt the target CPU (panic / shutdown / kexec-equivalent path).
Stop,
}
The TlbFlushMulti variant above borrows a slice of VaRange, the batched
multi-range TLB-shootdown payload:
/// A half-open virtual address range `[start, start + len)` queued for batched
/// TLB invalidation. `Copy` so it lives inline in `TlbFlushSet::ranges`.
#[derive(Clone, Copy)]
pub struct VaRange {
/// First address of the range.
pub start: VirtAddr,
/// Length of the range in bytes.
pub len: usize,
}
3.8.1.1 Synchronous Cross-Call Functions¶
Three convenience entry points wrap smp_send_ipi() +
IpiKind::CallFunction for the common "run this closure over there and
wait until done" pattern. They are the corpus-wide primitives behind PCP
drains (Section 4.2), slab magazine drains
(Section 4.3), memcg stock flushes (Section 17.2), the
NOLOCK-qdisc evolution quiesce
(Section 16.21), PMU event teardown
(Section 20.8), and the ArcSwap reclamation rendezvous
(Section 3.1).
/// Run `f` on EVERY online CPU — INCLUDING the calling CPU — and return
/// only after all of them have executed it. The target set is the online
/// mask snapshot at call time.
pub fn smp_call_function_all<F: Fn() + Sync>(f: F);
/// Run `f(cpu)` on every CPU in `mask` (the executing CPU's id is passed
/// so one closure can service a heterogeneous target set). The calling
/// CPU, if in `mask`, is included.
pub fn smp_call_function_many<F: Fn(u32) + Sync>(mask: &CpuMask, f: F);
/// Run `f` on the single CPU `cpu` (which may be the calling CPU).
pub fn smp_call_function_single<F: Fn() + Sync>(cpu: u32, f: F);
/// Like `smp_call_function_many`, but targets a LIVE `AtomicCpuMask`
/// ([Section 9.1](09-security.md#capability-based-foundation)) read directly at send time (per-word
/// `Acquire` loads) instead of a materialized `CpuMask` snapshot. The target set
/// therefore reflects online state at IPI-ISSUE time (strictly more current than
/// a prior snapshot), and NO `CpuMask`/`CpuMaskBuf` is allocated — so it is legal
/// on paths that must not allocate a mask (e.g. slab GC magazine drain,
/// [Section 4.3](04-memory.md#slab-allocator)). Same synchronous semantics and caller-context contract
/// as the three variants above.
pub fn smp_call_function_mask_atomic<F: Fn(u32) + Sync>(mask: &AtomicCpuMask, f: F);
Synchronous semantics. All three variants BLOCK until every target CPU
has executed f and acknowledged completion, via per-CPU completion flags
spin-polled by the caller — they spin-wait, they never sleep. The closure
(and everything it borrows) is only borrowed for the duration of the call;
the sender keeps the payload live until return, per the IpiKind
data-carrying rule above.
Caller context contract.
- Task context only — never from hardirq, softirq, or NMI context.
- Local IRQs MUST be enabled: two CPUs cross-calling each other with IRQs masked spin on each other's acks forever. (Preemption may be disabled or enabled; the implementation pins the CPU internally while snapshotting the target set and sending.)
- The caller MUST NOT hold any spinlock that
fcan also take: the caller spins for acks while a target executingfmay be spinning on that lock.
Target (closure) context contract.
- On remote CPUs,
fexecutes in hard-IRQ context with local IRQs disabled. When the calling CPU is itself a target,fis invoked directly on the caller under a local IRQ-disable bracket — sofruns with IRQs disabled on EVERY executing CPU, uniformly, and never in a nested-interrupt context on the caller. fmust be short and non-blocking: no sleeping, no unbounded waiting; only IRQ-safe locks per the lock-ordering table (Section 3.5).
Ordering contract (what callers may rely on):
- Execution, not delivery. The closure EXECUTES on each target CPU; the rendezvous never completes by merely posting an interrupt.
- Dispatch acquire. The caller publishes the cross-call record with
Releaseordering after everything program-ordered before the call; closure entry on each target performs the matchingAcquire. Every write the caller made before invoking the cross-call is therefore visible tofon every target. - Ack release. Each target acknowledges completion with a
Releasestore that the caller's spin-waitAcquire-loads. On return, the caller observes everythingfdid on every target. - No blanket entry barrier. The primitive does NOT promise that
interrupt entry/exit is a full memory barrier on the target — that is
not an architectural property of all eight targets (trap/exception
entry is context-synchronizing on several of them, which is not a
memory fence). An algorithm that needs a full StoreLoad fence to
execute on every CPU (membarrier-style protocols; the ArcSwap
reclamation rendezvous,
Section 3.1)
must execute
fence(SeqCst)INSIDE the closure; contracts 2 and 3 then extend that fence's ordering to the caller.
CPU hotplug. The target set is a snapshot; an IPI to a CPU that went offline between snapshot and delivery is silently dropped (the offline CPU is quiesced and runs no work the closure could need to observe or fence). A caller whose protocol must cover late-onlining CPUs synchronizes with the hotplug subsystem separately; guards/state pinned to a CPU (e.g. preemption-pinned readers) cannot outlive that CPU's offlining, which is what makes the drop safe for every current consumer.
3.8.2 Local Interrupt Save/Restore (arch::current::interrupts)¶
This section is the home of the arch::current::interrupts local-interrupt
save/restore seam: the per-architecture primitives that disable and restore
local interrupts on the current CPU, used by SpinLock<T>
(Section 3.5) and the per-CPU fast paths
(Section 3.3).
The two primitive operations are implemented per-arch in umka_nucleus::arch::current::interrupts:
/// Disable local interrupts and return the previous interrupt state (flags register
/// value on x86, DAIF on AArch64, `sstatus.SIE` on RISC-V, etc.).
/// Returns an opaque `usize` whose only valid use is as the argument to `local_irq_restore`.
/// Ordering: `local_irq_save()` acts as a compiler fence — no reads/writes are
/// reordered across it. Does NOT prevent NMIs (non-maskable interrupts).
pub fn local_irq_save() -> IrqDisabledGuard { /* arch-specific */ }
/// Restore interrupt state saved by a prior `local_irq_save()`.
/// Must be called on the same CPU that called `local_irq_save()`.
/// Calling this on a different CPU is undefined behavior.
pub fn local_irq_restore(guard: IrqDisabledGuard) { /* arch-specific via Drop */ }
Per-architecture IRQ save/restore:
The IrqDisabledGuard uses architecture-specific instructions to save and
restore the interrupt enable flag atomically. The pseudocode for each
supported architecture:
| Architecture | Save (disable IRQs, return old state) | Restore |
|---|---|---|
| x86-64 | PUSHF; POP rax (rax = eflags); CLI |
PUSH rax; POPF |
| AArch64 | MRS x0, DAIF; MSR DAIFSet, #0xF |
MSR DAIF, x0 |
| ARMv7 | MRS r0, CPSR; CPSID if |
MSR CPSR_c, r0 |
| RISC-V 64 | CSRRCI a0, sstatus, 0x2 (returns old sstatus, clears SIE) |
CSRW sstatus, a0 |
| PPC32 | MFMSR r0; RLWINM r1,r0,0,17,15; MTMSR r1 |
MTMSR r0 |
| PPC64LE | MFMSR r0; LI r1,MSR_RI; MTMSRD r1,1 (clears EE, sets RI) |
MTMSRD r0,1 |
| s390x | STNSM mask,0xFC (save PSW mask, clear I/O + external IRQ bits) |
SSM mask (restore saved PSW mask) |
| LoongArch64 | CSRRD r0, CRMD (save CRMD); CSRXCHG zero, IE_MASK, CRMD (clear IE bit) |
CSRWR r0, CRMD (restore saved CRMD) |
Notes:
- x86-64: PUSHF/POP captures the entire EFLAGS including IF (bit 9).
CLI clears IF. PUSH/POPF restores all flags including IF.
- AArch64: DAIF = Debug/SError/IRQ/FIQ mask bits. MSR DAIFSet, #0xF
sets all four mask bits (disables all async exceptions).
Restore writes the entire DAIF register, re-enabling only the
exceptions that were enabled before save.
- ARMv7: CPSR_c = control byte of CPSR. CPSID if = disable IRQ+FIQ.
- RISC-V: sstatus.SIE (bit 1) controls supervisor interrupt enable.
CSRRCI atomically reads old value and clears SIE in one instruction.
- PPC32: MSR.EE (bit 15) = external interrupt enable.
RLWINM with mask clears bit 15. MTMSR restores.
- PPC64LE: MTMSRD with L=1 only updates bits 48 (EE) and 62 (RI)
from the source register; all other MSR bits are unchanged. LI r1,MSR_RI
loads a value with EE=0, RI=1; MTMSRD r1,1 clears EE while preserving RI.
Note: Linux PPC64 BOOK3S uses software IRQ masking (PACA store) for the
common case; the hardware MSR approach shown here is the hard-disable path.
- s390x: STNSM saves the PSW mask byte and clears the specified bits.
Bits 6 (I/O interrupts) and 7 (external interrupts) are cleared by
masking with 0xFC. SSM restores the saved mask byte.
- LoongArch64: CRMD.IE (bit 2) controls interrupt enable. CSRXCHG performs
an atomic read-modify-write on the CSR: zero with IE_MASK clears IE.
CSRWR restores the entire CRMD register from the saved value.
The platform-independent IrqDisabledGuard::new() calls
arch::current::interrupts::local_irq_save() which expands to the
appropriate instruction sequence above. This is zero-cost abstraction:
the compiler selects the single-architecture path at compile time.
IrqDisabledGuard is the RAII wrapper returned by local_irq_save(). Its Drop
implementation calls the per-arch restore sequence, ensuring correct pairing:
/// Proof token that IRQs are disabled on the current CPU. Created by
/// `local_irq_save()` or `irq_disabled_guard()` (the latter asserts
/// that IRQs are already disabled and constructs the token without
/// a redundant CLI). The token is `!Send` and `!Sync` — it cannot
/// be transferred to another CPU where IRQ state may differ.
///
/// **Implies preemption disabled.** Disabling hardware IRQs masks the
/// scheduler tick interrupt, so the scheduler cannot preempt the running
/// task while this guard is held. Code holding `IrqDisabledGuard` therefore
/// has the same preemption-safety guarantee as code holding `PreemptGuard`.
/// The converse is NOT true: `PreemptGuard` alone (via `preempt_disable()`)
/// does not disable IRQs — interrupt handlers can still fire and must not
/// call `get_mut_nosave()` unless they hold their own `IrqDisabledGuard`.
pub struct IrqDisabledGuard {
/// Saved interrupt flags. If this guard was created by
/// `local_irq_save()`, Drop restores them. If created by
/// `irq_disabled_guard()` (assertion-only), Drop is a no-op.
saved_flags: Option<usize>,
_not_send: PhantomData<*const ()>,
}
impl Drop for IrqDisabledGuard {
fn drop(&mut self) {
if let Some(flags) = self.saved_flags {
// SAFETY: restoring flags that we saved earlier.
unsafe { restore(flags) };
} else {
// saved_flags is None: this is the assertion-only variant created
// by `irq_disabled_guard()`. The caller is responsible for
// maintaining the IRQ-disabled state. Debug assertion verifies
// the invariant was not violated (e.g., by dropping a SpinLockGuard
// that re-enabled IRQs while a PerCpuMutRefNosave was still live).
#[cfg(debug_assertions)]
{
assert!(
!arch::current::interrupts::are_enabled(),
"IrqDisabledGuard (None variant) dropped with IRQs enabled — \
the caller violated the safety contract by re-enabling IRQs \
while this proof token was live"
);
}
}
}
}
/// Assert that IRQs are already disabled and obtain a proof token.
///
/// # Safety
///
/// The caller must guarantee that hardware interrupts remain disabled
/// for the entire lifetime of the returned guard. Specifically:
/// - IRQs must be disabled at the point of call (verified by debug assert).
/// - No other guard (e.g., `SpinLockGuard`) that might re-enable IRQs may
/// be dropped while a `PerCpuMutRefNosave` derived from this token is
/// still live. The borrow checker ties the `PerCpuMutRefNosave` lifetime
/// to this guard, but cannot track the hardware IRQ state changing via
/// an independent guard's Drop.
///
/// Violating this invariant allows interrupt handlers to race on per-CPU
/// data, producing undefined behavior.
pub unsafe fn irq_disabled_guard() -> IrqDisabledGuard {
#[cfg(debug_assertions)]
{
assert!(
!arch::current::interrupts::are_enabled(),
"irq_disabled_guard() called with IRQs enabled"
);
}
IrqDisabledGuard {
saved_flags: None,
_not_send: PhantomData,
}
}
SpinLock<T> (Section 3.5) and other callers that store the saved
token by hand — rather than in a RAII guard — use the raw form of the same
seam, plus the IrqFlags token type it produces:
/// Opaque per-architecture saved interrupt-state token. Representation is
/// architecture-defined (x86-64 `RFLAGS`, AArch64 `DAIF`, RISC-V `sstatus.SIE`,
/// PPC `MSR[EE]`, s390x PSW system mask, LoongArch64 `CRMD.IE`); generic code
/// never inspects it — it is only saved and later restored verbatim. This is the
/// value stored in `SpinLockGuard::saved_flags`
/// (aliased `ArchIrqFlags = arch::current::interrupts::IrqFlags` — see
/// [Section 3.5](#locking-strategy)).
pub type IrqFlags = /* arch-specific opaque type */ usize;
/// Disable local interrupts and return the previous interrupt state as a raw
/// `IrqFlags` token (no RAII guard). The caller is responsible for pairing it
/// with `restore()`. Used by `SpinLock<T>`, which stores the token in its guard
/// and restores it on drop. Same per-arch instruction sequences as the table above.
pub fn save_and_disable() -> IrqFlags { /* arch-specific */ }
/// Restore interrupt state from a token returned by `save_and_disable()`.
///
/// # Safety
///
/// The caller must pass an `IrqFlags` produced by `save_and_disable()` (or an
/// equivalent arch save) on the SAME CPU, exactly once. Restoring on a different
/// CPU, or with a fabricated or reused token, is undefined behavior.
pub unsafe fn restore(flags: IrqFlags) { /* arch-specific */ }
/// True if local interrupts are currently enabled on this CPU. Used by the
/// debug assertions in `IrqDisabledGuard` / `irq_disabled_guard()`.
pub fn are_enabled() -> bool { /* arch-specific */ }
3.8.2.1 Interrupt-Return Reservation Clearing¶
Invariant (arch interrupt-return requirement). The interrupt/exception return path MUST clear any load-reserved/store-conditional reservation left outstanding by an interrupt handler before it resumes the interrupted context, so the resumed context cannot complete a store-conditional against a reservation it does not own. This is the interrupt-return counterpart of the context-switch reservation clearing (Section 7.3) and is a correctness requirement, not an optimization: the single-word guarded position claim (Section 3.1) depends on it so that an interrupt taken inside a claim window cannot let a stale validation commit. Legs whose claim lowering holds no reservation across the window satisfy it vacuously (every 64-bit-atomic leg claims with a self-contained compare-exchange and carries no reservation into a handler); the load-bearing case is the one leg that lowers the claim to an ll/sc reservation held across the window.
PPC32 (per-arch content). Return from interrupt (
rfi) does NOT clear a reservation. The interrupt return sequence therefore executes a reservation-killstwcx.to the per-CPULlscDummyscratch word (CpuLocalBlock.llsc_dummy;AtomicUsize, sized to each leg's native reservation-store width — 32-bit on PPC32, sostwcx.writes the whole word) on the way out; the store value is discarded and its success or failure is irrelevant — astwcx.clears the processor's reservation either way — guaranteeing no reservation is inherited across the return.
3.8.3 s390x Interrupt Model¶
The s390x architecture uses a PSW-swap interrupt model that is fundamentally different from all other supported architectures. There is no external interrupt controller (no APIC, GIC, PLIC, or equivalent). Interrupt routing is an architectural feature of the CPU itself, mediated through the lowcore (a per-CPU memory page at fixed physical addresses).
3.8.3.1 Interrupt Classes¶
s390x defines six interrupt classes, each with a dedicated pair of PSW save/load slots in the lowcore:
| Class | Old PSW Offset | New PSW Offset | Triggers |
|---|---|---|---|
| Restart | 0x120 | 0x1A0 | SIGP restart order from another CPU |
| External | 0x130 | 0x1B0 | Signals, timers, clock comparator, SIGP external call/emergency signal |
| SVC | 0x140 | 0x1C0 | System call instruction (svc) |
| Program | 0x150 | 0x1D0 | Faults, traps, illegal instructions, addressing exceptions |
| Machine Check | 0x160 | 0x1E0 | Hardware errors, channel failures |
| I/O | 0x170 | 0x1F0 | Channel I/O completion, subchannel status pending |
Each PSW is 16 bytes (128-bit). Old PSWs span 0x120-0x17F; New PSWs span 0x1A0-0x1FF.
When an interrupt fires, the hardware atomically saves the current PSW to the "Old PSW" slot and loads the "New PSW" from the adjacent slot. The new PSW contains the entry point of the interrupt handler for that class. No software interaction is needed to claim the interrupt — the hardware dispatch is implicit in the PSW swap.
3.8.3.2 Stack Setup on Interrupt Entry¶
The PSW swap only sets the instruction pointer — it does not switch the stack pointer. Each interrupt handler's prologue must establish a valid kernel stack before any function calls. The stack setup sequence:
- Read the prefix register: The lowcore page is per-CPU, relocated via the CPU
prefix register (
SIGP SET_PREFIX). The handler reads the saved stack pointer fromLC_ASYNC_STACK(offset 0x0350 in the lowcore) for async interrupts (External, I/O, Machine Check) or uses the current kernel stack for synchronous exceptions (SVC, Program). The async stack is a per-CPU 16 KB stack separate from the process kernel stack, preventing async interrupts from overflowing a shallow kernel stack. - Save registers to lowcore:
STMG r8,r15,0x0200(__LC_SAVE_AREA, lowcore offset 0x0200,save_area: [u64; 8], 64 bytes) for callee-saved registers and the stack pointer needed for handler setup. The full 16-register save (STMG r0,r15= 128 bytes) would overwrite critical lowcore fields at 0x0240+ (includingstack_canaryand other per-CPU data). After establishing the kernel stack (step 3), the handler saves the remaining registers (r0-r7) to the stack frame. This two-phase save matches Linuxarch/s390/kernel/entry.SSTMG %r8,%r15(saving 8 registers to lowcore, then the full set to the stack). - Load kernel stack pointer:
LG r15, __LC_ASYNC_STACK(or stay on current stack for SVC/Program if already in kernel mode). For user→kernel transitions: load the per-CPU kernel stack fromLC_KERNEL_STACK(offset 0x0348). - Build a standard stack frame: 160 bytes (s390x ABI minimum). The
STPTinstruction saves the CPU timer for accounting.
s390x lowcore offset reference (verified against arch/s390/include/asm/lowcore.h
in torvalds/linux master):
| Symbol | Offset | Type | Description |
|---|---|---|---|
__LC_SAVE_AREA |
0x0200 | [u64; 8] |
Register save area for interrupt entry (r8-r15, 64 bytes; full set saved to stack) |
LC_CPU_LOCAL_BASE |
0x0340 | u64 |
Per-CPU CpuLocalBlock base pointer — read by arch::current::cpu::cpu_local_block(), written once per CPU at bring-up (Section 3.2) |
LC_KERNEL_STACK |
0x0348 | u64 |
Per-CPU kernel stack pointer |
LC_ASYNC_STACK |
0x0350 | u64 |
Per-CPU async (external/I/O) interrupt stack |
LC_MCK_STACK |
0x0368 | u64 |
Per-CPU machine check interrupt stack |
NMI (Machine Check) stack: Machine check handlers use a dedicated per-CPU stack
(LC_MCK_STACK, offset 0x0368, 8 KB) because a machine check can interrupt any
context, including other interrupt handlers. This three-stack model (kernel, async,
machine check) prevents stack overflow even under nested interrupt scenarios.
NMI/MCE stack budget per architecture:
| Architecture | NMI stack size | NMI source | Budget constraint |
|---|---|---|---|
| x86-64 | 8 KB (IST entry 2) | NMI pin, APIC NMI LVT, INT 2 |
Max call depth ~40 frames; no allocation, no sleeping, no page faults |
| AArch64 | 8 KB (dedicated SP_EL1 region) | SError (async abort), FIQ (secure NMI) | SError handler must be self-contained; FIQ reserved for secure firmware |
| ARMv7 | 4 KB (FIQ mode stack) | FIQ (used for NMI-like signaling on vexpress) | Minimal handler; logs event and returns |
| RISC-V 64 | Shared with kernel stack | No true NMI; scause MSB distinguishes |
Software convention: NMI-like IPIs limited to 2 KB stack usage |
| PPC32 | 4 KB (critical interrupt stack) | Machine check, critical input | SPR save area at fixed offsets; 1 KB for handler logic |
| PPC64LE | 8 KB (per-CPU machine-check stack) | Machine check (MCE), system reset | Per-CPU save area; handler must not touch SLB or HPT |
| s390x | 8 KB (LC_MCK_STACK) | Machine check | Lowcore save area; 6 KB usable after register save |
| LoongArch64 | 8 KB (dedicated per-CPU) | Machine error exception (Ecode=62) | Handler reads CSR.MERR*, logs, returns via ERTN |
Isolation-key treatment of these stacks splits by architecture. On x86-64 the
NMI/#DF/#MC IST stacks are the exception to the shared kernel-stack key class:
NMI is deliverable at CPL 3 under the interrupted user thread's PKRU and AC
(unprivileged WRPKRU can deny any key; POPF can clear AC), so no key grant can
make the hardware frame push safe — these stacks are mapped U=0 and keyless,
exempt from both SMAP and PKRU, and they host their handlers' entire frames (an
IST handler never pivots: the fixed-top IST reload is not nesting-safe). The
entry-ordering contract — STAC plus Core-PKRU establishment before the first
access to U=1 kernel data such as CpuLocalTransit — is normative in
Section 11.2.
On AArch64 POE / ARMv7 DACR these per-CPU stacks — like all kernel-mode
stacks — carry the shared read-write infrastructure tagging (overlay index 2 /
DACR domain 14,
Section 11.2):
exception entry writes no memory, and the entry stub's first spill stores execute
under whatever KERNEL image was live when the NMI/MCE arrived (POR_EL1/DACR are
not user-writable), so the pages must be writable under every kernel domain image.
The remaining architectures need no tagging (translation-gated kernel mappings or
no fast isolation —
Section 11.2).
3.8.3.3 I/O Interrupt Routing¶
I/O interrupts are generated by channel subsystem subchannels and float to any CPU that has the appropriate Interrupt Sub-Class (ISC) enabled. ISC bits are controlled via Control Register 6 (CR6): each of the 8 ISC classes (0-7) can be independently masked per-CPU. UmkaOS manages I/O interrupt affinity by masking ISC bits in CR6 on each CPU — this is the s390x equivalent of interrupt affinity routing on other platforms.
The UmkaOS IRQ domain for s390x maps subchannel interrupts to generic software IRQ
numbers via ISC-to-vector translation: when an I/O interrupt is received, the handler
reads the subchannel identification word (SCHID) from the lowcore I/O interruption code
area, translates the (SCHID, ISC) pair to a software IRQ number, and dispatches through
the standard IrqDescriptor path.
3.8.3.4 External Interrupts¶
External interrupts include:
- Clock comparator: fires when TOD clock reaches the programmed comparator value.
- CPU timer: fires when the per-CPU timer decrements to zero.
- SIGP external call: sent by another CPU via SIGP EXTERNAL_CALL.
- SIGP emergency signal: high-priority inter-CPU signal via SIGP EMERGENCY_SIGNAL.
- Service signal: from the service element (SE) or hypervisor.
External interrupt subclass codes are in the lowcore external interruption code field. UmkaOS routes these to dedicated handlers: timer interrupts to the timekeeping subsystem, SIGP signals to the IPI handler.
3.8.3.5 Inter-Processor Interrupts (IPI)¶
s390x uses the SIGP (Signal Processor) instruction for all inter-CPU communication:
| SIGP Order | Purpose |
|---|---|
EXTERNAL_CALL |
General-purpose IPI (schedule, TLB flush, function call) |
EMERGENCY_SIGNAL |
High-priority IPI (stop, NMI-equivalent) |
RESTART |
Boot/restart a stopped CPU |
SET_PREFIX |
Set the lowcore prefix (per-CPU page base address) |
STOP |
Halt a CPU |
SENSE |
Query CPU status |
UmkaOS maps the generic arch::current::interrupts::send_ipi() interface to
SIGP EXTERNAL_CALL for normal IPIs and SIGP EMERGENCY_SIGNAL for NMI-class events.
3.8.3.6 IrqChip Adaptation¶
The s390x IrqChip implementation differs from controller-based architectures:
- ack(): no-op (the PSW swap implicitly acknowledges the interrupt).
- mask(): disables the relevant ISC bit in CR6 for the current CPU.
- unmask(): enables the ISC bit in CR6.
- set_affinity(): adjusts ISC masks across the target CPU set.
- eoi(): no-op (no end-of-interrupt concept in the PSW-swap model).
3.8.4 LoongArch64 Interrupt Model¶
LoongArch64 uses a two-level interrupt controller architecture: the EIOINTC (Extended I/O Interrupt Controller) for general device interrupts and the LIOINTC (Legacy I/O Interrupt Controller) for UART and other legacy devices.
3.8.4.1 EIOINTC — Extended I/O Interrupt Controller¶
The EIOINTC provides 256 interrupt vectors with per-CPU routing capability. Configuration
is performed through IOCSR (I/O Control and Status Register) space, accessed via the
IOCSRRD and IOCSRWR instructions.
Key EIOINTC capabilities: - 256 vectors (0-255), each independently routable to any CPU. - Per-vector CPU affinity: routing registers specify the target CPU for each vector. UmkaOS programs these at device probe time based on IRQ affinity policy. - Per-vector priority: each interrupt source has a configurable priority level. - Enable/disable per-vector: individual interrupt lines can be masked independently.
EIOINTC initialization sequence: 1. Discover EIOINTC base via ACPI MADT or device tree. 2. Disable all 256 vectors (write zero to enable registers). 3. Set default routing: all vectors to BSP (CPU 0). 4. Configure priority levels. 5. Enable desired interrupt lines as devices are probed.
3.8.4.2 LIOINTC — Legacy I/O Interrupt Controller¶
The LIOINTC handles legacy device interrupts (UART, RTC, etc.) that do not route through
the EIOINTC. It provides a smaller set of interrupt lines (typically 32) with fixed or
limited routing. The LIOINTC cascades into the CPU's interrupt input, and UmkaOS creates
a secondary IrqDomain for LIOINTC that parents to the EIOINTC root domain.
3.8.4.3 Interrupt Enable and Masking¶
Global interrupt enable is controlled by the CSR.CRMD.IE bit (Control and Status
Register — Current Mode):
- IE = 1: interrupts enabled.
- IE = 0: interrupts disabled (masked globally).
UmkaOS sets CSR.CRMD.IE = 0 on interrupt entry (automatic by hardware) and restores it
on exception return. Per-line masking is handled through the EIOINTC vector enable
registers.
3.8.4.4 Inter-Processor Interrupts (IPI)¶
LoongArch64 IPIs use the IOCSR mailbox mechanism combined with a dedicated EIOINTC IPI
vector:
1. The sender writes the IPI message (action bitmask) to the target CPU's IOCSR mailbox
register via IOCSRWR.
2. The sender triggers the IPI by asserting the designated IPI vector in the EIOINTC
(or directly via the IOCSR IPI send register).
3. The target CPU receives the interrupt on the IPI vector, reads its IOCSR mailbox to
determine the requested action(s), clears the mailbox, and dispatches accordingly.
UmkaOS maps arch::current::interrupts::send_ipi() to this IOCSR mailbox + EIOINTC
mechanism. The IPI action bitmask encodes: reschedule, TLB flush, function call, and stop.
3.8.4.5 IrqChip Adaptation¶
The LoongArch64 EIOINTC IrqChip implementation:
- ack(): clears the pending bit for the vector in the EIOINTC status register.
- mask(): clears the enable bit for the vector via IOCSR write.
- unmask(): sets the enable bit for the vector via IOCSR write.
- set_affinity(): reprograms the per-vector routing register to target the new CPU set.
- eoi(): clears the pending bit and unmasks (standard ack-then-unmask sequence).
3.8.5 Softirq: Deferred Interrupt Processing¶
Softirqs are the bottom-half mechanism for work that cannot be done in hardirq context but must run with low latency before returning to process context. Every network packet, every timer tick, every block I/O completion, and every RCU callback batch involves softirq processing.
UmkaOS design decisions vs Linux:
- 10 softirq vectors matching Linux for /proc/softirqs ABI compatibility.
- No tasklets: tasklets are deprecated in Linux (replaced by threaded handlers and
workqueues). UmkaOS skips tasklets entirely.
- Non-preemptible by default: optimized for throughput (UmkaOS's primary server
target). Matches Linux PREEMPT_NONE behavior.
- Evolvable preemption hook: runtime-switchable threaded softirq mode for latency-
sensitive workloads (per cgroup/workload class), enabling PREEMPT_RT-style behavior
without kernel rebuild.
3.8.5.1 Softirq Vector Table¶
/// Softirq vector indices. Matches Linux `include/linux/interrupt.h` for
/// `/proc/softirqs` output compatibility.
#[repr(u32)]
pub enum SoftirqVec {
HiPriority = 0, // HI_SOFTIRQ: high-priority tasklet replacement (timer-critical)
Timer = 1, // Linux TIMER_SOFTIRQ: timer wheel expiry processing
NetTx = 2, // NET_TX_SOFTIRQ: network transmit completion
NetRx = 3, // NET_RX_SOFTIRQ: network receive (NAPI poll)
Block = 4, // BLOCK_SOFTIRQ: block I/O completion
IrqPoll = 5, // Linux IRQ_POLL_SOFTIRQ: IRQ polling (blk-iopoll)
Tasklet = 6, // Linux TASKLET_SOFTIRQ: no-op handler in UmkaOS (tasklets are
// deprecated; this slot is present solely for /proc/softirqs
// positional ABI compatibility with Linux, which exposes
// per-vector counters by position, not by name).
Sched = 7, // Linux SCHED_SOFTIRQ: scheduler load balancing
HrTimer = 8, // Linux HRTIMER_SOFTIRQ: high-resolution timer expiry
Rcu = 9, // RCU_SOFTIRQ: RCU callback processing
}
/// Total number of softirq vectors.
pub const NR_SOFTIRQS: usize = 10;
/// Per-softirq handler function type. Called with preemption disabled, IRQs
/// enabled (within the handler — softirq execution re-enables IRQs after
/// the initial hardirq context exit). The handler must not sleep.
pub type SoftirqHandler = fn();
/// Per-CPU softirq pending bitmask. Bit N is set when softirq vector N
/// has been raised and not yet processed. `AtomicU32` stored in
/// `CpuLocalBlock` ([Section 3.2](#cpulocal-register-based-per-cpu-fast-path)).
///
/// Atomicity is required because a hardirq can preempt `do_softirq()`
/// between snapshot and clear. `raise_softirq()` uses `fetch_or(bit, Relaxed)`;
/// `do_softirq()` uses `swap(0, Relaxed)` for atomic snapshot-and-clear.
/// See the `CpuLocalBlock::softirq_pending` doc comment for the full rationale.
3.8.5.2 Raising a Softirq¶
/// Mark a softirq vector as pending on the current CPU.
/// May be called from any context (hardirq, softirq, process).
/// The softirq will be processed at the next `irq_exit()` or
/// `local_bh_enable()` call on this CPU.
///
/// # Implementation
/// Atomically sets bit `vec` in the per-CPU `softirq_pending` bitmask
/// using `fetch_or(bit, Relaxed)`. No IPI is sent — softirqs are
/// CPU-local by design. `Relaxed` ordering suffices because the only
/// consumer is `do_softirq()` on the same CPU, and the pending check
/// at `irq_exit()` is ordered by the interrupt return sequence.
pub fn raise_softirq(vec: SoftirqVec) {
// Field-scoped `CpuLocal::softirq_pending()` (a single-field `&'static
// AtomicU32`), NOT a whole-block reference: `get()`/`get_mut()` are both
// deleted (ESC-0431) because NMI/IPI handlers may concurrently access other
// atomic fields (in_nmi, need_resched) — a whole-block reference's lifetime
// could overlap those async writes (clause 2, lifetime-overlap form). The
// single-field projection is sound in any context — see the CpuLocal
// reference-formation doctrine
// ([Section 3.2](#cpulocal-register-based-per-cpu-fast-path--cpulocal-reference-formation-doctrine-two-clauses)).
let _guard = local_irq_save();
CpuLocal::softirq_pending().fetch_or(1 << (vec as u32), Ordering::Relaxed);
}
/// Raise a softirq from hardirq context. Same as `raise_softirq()` but
/// inlined for use in IRQ handlers (avoids function-call overhead).
#[inline(always)]
pub fn raise_softirq_irqoff(vec: SoftirqVec) {
// IRQs already disabled in hardirq context; no save/restore needed.
// Field-scoped projection — see raise_softirq on why no whole-block ref.
CpuLocal::softirq_pending().fetch_or(1 << (vec as u32), Ordering::Relaxed);
}
3.8.5.3 Softirq Processing Algorithm¶
do_softirq() — called from irq_exit() and local_bh_enable():
Precondition: softirq_pending is non-zero AND NOT in hardirq context
(irq_count == 0) AND NOT in softirq context (softirq_count == 0).
0. Guard check (defense-in-depth): if irq_count > 0 || softirq_count > 0,
return immediately. Both callers (irq_exit, local_bh_enable) already
enforce this precondition, but the explicit check protects against future
callers that might omit it. Cost: one compare on a cold path (softirq
entry is infrequent relative to the work done inside).
1. Increment softirq_count (mark "in softirq context") to prevent re-entry.
2. Enable IRQs (softirq handlers run with IRQs enabled to reduce
interrupt latency — a key difference from hardirq context).
3. iterations = 0.
4. Loop:
a. pending = softirq_pending.swap(0, Relaxed).
// Atomic snapshot-and-clear. The swap atomically reads the current
// value and replaces it with 0 in a single operation, preventing
// the TOCTOU race where a hardirq sets a bit between the read and
// the clear. Relaxed ordering suffices: the only producer is the
// local CPU's hardirq handler, and the interrupt return sequence
// provides the necessary ordering between the hardirq write and
// the softirq read.
b. For each set bit N in pending (lowest to highest):
- Call softirq_handlers[N]().
c. iterations += 1.
d. If softirq_pending.load(Relaxed) != 0 AND iterations < MAX_SOFTIRQ_RESTART (10):
- Handlers may have raised new softirqs; loop back to (a).
e. If softirq_pending.load(Relaxed) != 0 AND iterations >= MAX_SOFTIRQ_RESTART:
- Residual softirqs remain. Wake ksoftirqd (step 5).
- Break out of loop.
5. Disable IRQs.
6. Decrement softirq_count (leave softirq context).
7. If residual softirqs remain: wake per-CPU ksoftirqd kthread.
MAX_SOFTIRQ_RESTART = 10 (matches Linux).
3.8.5.4 ksoftirqd Fallback¶
Each CPU has a dedicated ksoftirqd/N kthread (SCHED_OTHER, nice 0). It processes
softirqs that could not be fully drained in the hardirq-exit path (due to the
10-iteration limit). Without ksoftirqd, a sustained softirq storm (e.g., 100GbE
line-rate packet flood) would prevent the CPU from ever reaching process context.
ksoftirqd/N thread:
Loop:
1. If softirq_pending.load(Relaxed) == 0: schedule() (sleep until woken by do_softirq).
2. local_bh_disable().
3. While softirq_pending.load(Relaxed) != 0:
a. do_softirq() — same algorithm as above.
b. If need_resched is set: cond_resched() (yield to higher-priority tasks).
4. local_bh_enable().
3.8.5.5 Context Semantics¶
| Property | Hardirq | Softirq | Process |
|---|---|---|---|
| Preemption | Disabled | Disabled (default) | Enabled |
| IRQs | Disabled | Enabled (within handler) | Enabled |
| May sleep | No | No | Yes |
| May allocate | No (except emergency) | GfpFlags::ATOMIC_ALLOC only |
GfpFlags::KERNEL |
| Nested interrupts | Higher-priority only | Hardirqs only | All |
| SpinLock behavior | Already has IRQs off | local_bh_disable (via softirq_count) | Saves/restores IRQs |
3.8.5.6 Interaction with Locking Primitives¶
- SpinLock:
SpinLock::lock()callslocal_irq_save()and incrementspreempt_count(NOTirq_count-- see Section 3.5 for the authoritative table). Releasing callslocal_irq_restore()and decrementspreempt_count. Softirq execution is indirectly prevented while a SpinLock is held because hardware IRQs are masked -- softirq processing occurs atirq_exit()which cannot execute while IRQs are disabled. This is an indirect effect of IRQ masking, not anirq_countmechanism. local_bh_disable()/local_bh_enable(): Increments/decrements the dedicatedCpuLocalBlock::softirq_countfield. Whensoftirq_count > 0, softirqs are not processed atirq_exit().local_bh_enable()checkssoftirq_pending.load(Relaxed)and callsdo_softirq()if any softirqs are pending andsoftirq_countreaches 0. UmkaOS uses separate typed fields (irq_countfor hardirq,softirq_countfor BH) instead of Linux's packed bitfield layout — see Section 3.2.- RCU:
RCU_SOFTIRQ(vector 9) processes completed grace period callbacks. Softirq context counts as a quiescent state for RCU purposes (softirqs run between RCU read-side critical sections).
3.8.5.7 Evolvable Preemption Policy Hook¶
The default non-preemptible softirq model is optimal for throughput-oriented server workloads. For latency-sensitive workloads (real-time audio, trading), softirqs can be converted to threaded execution via an Evolvable policy hook:
/// Evolvable softirq scheduling policy.
///
/// **Nucleus** (data): `SoftirqVec` enum, `softirq_pending: AtomicU32` bitmask, handler table.
/// **Evolvable** (policy): this trait controls whether softirqs run inline
/// (non-preemptible, default) or as dedicated kthreads (preemptible).
///
/// Runtime-switchable per cgroup via the `cpu.softirq_mode` cgroup knob:
/// - `inline` (default): softirqs run in `do_softirq()` with preemption disabled.
/// - `threaded`: each softirq vector is serviced by a dedicated SCHED_FIFO kthread,
/// allowing preemption between softirq handlers.
pub trait SoftirqPolicy: Send + Sync {
/// Returns true if the given softirq vector should run as a dedicated
/// kthread rather than inline in do_softirq(). Checked per-invocation.
fn is_threaded(&self, vec: SoftirqVec) -> bool;
}
The SoftirqPolicy hook is consulted at the start of do_softirq(). When
is_threaded() returns true for a vector, the corresponding kthread is woken
instead of running the handler inline. This is the UmkaOS equivalent of Linux's
PREEMPT_RT forced-threading, but runtime-switchable rather than compile-time.
3.9 Memory Model Differences Across Architectures¶
Critical Implementation Warning: UmkaOS relies extensively on lock-free concurrency, asynchronous ring buffers, and RCU to meet its strict performance budget. All lock-free algorithms must be correct across every target architecture — not just on the architecture where they were developed and tested.
3.9.1 x86 TSO Conceals Ordering Bugs¶
x86_64 implements Total Store Ordering (TSO). Under TSO, loads are not reordered with other loads, stores are not reordered with other stores, and stores are observed in program order by all CPUs. This is a significantly stronger ordering guarantee than software actually requires for most algorithms.
The consequence for development is that lock-free code with missing memory barriers, or code that uses Ordering::Relaxed where Ordering::Release is required, will almost always execute correctly on x86 and pass all tests there. The hardware silently supplies the ordering that the programmer omitted. Bugs of this class are structurally invisible on x86 — no amount of stress-testing on x86 hardware will expose them, because the CPU never exercises the reorderings that would trigger the race.
x86 TSO pays a hidden hardware cost. The strong ordering guarantee is not free: Intel and AMD CPUs implement store buffers and memory ordering machinery that impose a hardware tax on every store, whether or not the software needs ordered visibility. Code using Ordering::Release on x86 compiles to a plain store (the hardware already provides release semantics), but the CPU still pays the store-ordering overhead internally. Software running on x86 is paying for ordering it often does not need.
The practical conclusion: x86 is an unreliable test platform for lock-free code. A lock-free algorithm that passes all tests on x86 has not been validated — it has been tested on the platform most likely to hide its bugs. Correctness on x86 is necessary but not sufficient.
3.9.2 ARM, RISC-V, and PowerPC: Explicit Memory Ordering Surfaces True Requirements¶
AArch64, ARMv7, RISC-V, and PowerPC implement relaxed memory models. The CPU is permitted to reorder independent memory reads and writes for performance — a store to address A followed by a load from address B can complete in either order if A and B are in different cache lines and there is no explicit ordering constraint between them. Stores from one CPU become visible to other CPUs at different times, and different CPUs may observe stores in different orders unless barriers enforce a consistent sequence.
This is not a deficiency in these architectures — it is the architecturally correct exposure of what ordering actually costs. These architectures give software precise control: pay for ordering when the algorithm requires it, pay nothing when it does not. A missing memory barrier on AArch64 or RISC-V produces visible, reproducible failures: sequence locks read torn data, ring buffer consumers observe the tail pointer advance before payload data is visible, and RCU readers dereference pointers to uninitialized memory. The bug that x86 conceals, ARM and RISC-V surface.
Develop and test lock-free algorithms on ARM or RISC-V. Only a platform with relaxed memory ordering can expose ordering bugs. x86 is unreliable as a correctness gate for lock-free primitives.
Implementation Mandates: To ensure lock-free algorithms are correct across all target architectures, the following rules apply:
- Explicit Rust Atomics: All shared memory synchronization must use Rust's
std::sync::atomictypes with mathematically correct memory orderings (Acquire,Release,AcqRel,SeqCst). Never rely on implicit hardware ordering; write code that is correct on the weakest memory model UmkaOS targets. - Release-Acquire Semantics: The standard pattern for lock-free publishing in UmkaOS (e.g., advancing a ring buffer head, or updating an RCU pointer) MUST pair an
Ordering::Releasestore on the producer with anOrdering::Acquireload on the consumer. - No
Relaxedin Control Flow:Ordering::Relaxedmay only be used for pure statistical counters (e.g.,rx_packets). It must never be used to synchronize visibility of other data. - Mandatory Multi-Arch CI: Lock-free primitives (MPSC queues, RCU, seqlocks) must be subjected to heavy stress-testing natively on AArch64 and RISC-V hardware (or QEMU/emulators with memory-reordering fuzzing enabled). x86_64 test passes are considered insufficient to prove the correctness of lock-free code.
3.9.3 Ordering Instruction Cost: Ring Buffer and RCU Performance by Architecture¶
The implementation mandate to use Ordering::Release/Ordering::Acquire on ring buffer
head/tail pointer updates and RCU pointer publications has measurably different instruction-
level costs across architectures. Understanding this is essential for interpreting benchmark
results and for choosing the minimum correct ordering at each synchronization point.
Ordering::Release store — instruction cost by architecture:
| Architecture | Compiled instruction | Approximate cost |
|---|---|---|
| x86-64 | Plain MOV (no additional instruction) |
~1 cycle (TSO provides release for free) |
| AArch64 | STLR (store-release) or STR + DMB ISHST |
~5-20 cycles (barrier flushes store buffer) |
| ARMv7 | STR + DMB ISHST |
~10-30 cycles |
| RISC-V | FENCE RW,W before store (or amoswap with .rl) |
~10-30 cycles |
| PPC64 | lwsync before store |
~10-20 cycles |
| PPC32 (e500) | sync before store |
~20-40 cycles (e500v1/v2 cores do NOT support lwsync — it causes an Illegal Instruction trap; sync/msync must be used instead) |
| s390x | Plain ST/STG (no additional instruction) |
~0 cycles (strongly ordered (at least TSO; sequential consistency for single-copy atomics) for single-copy atomics — release is free) |
| LoongArch64 | DBAR 0x12 (store-release ordering) + ST.D |
~15-25 cycles |
Ordering::Acquire load — instruction cost by architecture:
| Architecture | Compiled instruction | Approximate cost |
|---|---|---|
| x86-64 | Plain MOV (no additional instruction) |
~1 cycle (TSO prevents load reordering) |
| AArch64 | LDAR (load-acquire) |
~5-20 cycles |
| ARMv7 | LDR + DMB ISH |
~10-30 cycles |
| RISC-V | FENCE R,RW after load (or lr with .aq) |
~10-30 cycles |
| PPC64 | lwsync after load or isync on branch path |
~10-20 cycles |
| PPC32 (e500) | sync after load or isync on branch path |
~20-40 cycles (e500 lacks lwsync; see Release note above) |
| s390x | Plain L/LG (no additional instruction) |
~0 cycles (strongly ordered (at least TSO; sequential consistency for single-copy atomics) — acquire is free) |
| LoongArch64 | LD.D + DBAR 0x14 (load-acquire ordering) |
~15-25 cycles |
Implications for UmkaOS ring buffer throughput:
Ring buffer head/tail pointer updates require a Release store on the producer side and an Acquire load on the consumer side — this is the minimum correct ordering and cannot be reduced without introducing a race. On x86, these compile to ordinary MOV instructions; the TSO hardware silently provides the ordering. On ARM and RISC-V, each ring buffer publish or consume event pays a real instruction-level barrier cost.
The consequence is that ring buffer throughput on AArch64 and RISC-V will be measurably lower than on x86 for identical Rust source code — not because of a bug or a missing optimization, but because x86 is paying its ordering cost in hardware (through store-buffer logic and pipeline constraints that are invisible to software), while ARM and RISC-V pay it explicitly in the instruction stream. Both are paying; only the accounting differs.
Ordering::Relaxed usage in UmkaOS ring buffers:
Within the ring buffer implementation, Ordering::Relaxed is used selectively where only
atomic visibility (not ordering relative to other accesses) is required:
- Statistical counters (
rx_packets,tx_bytes, drop counters):Relaxedon both store and load. These are sampled for reporting only; no other memory access is conditioned on their value. - Per-CPU freelist size counters:
Relaxed. The size is advisory; the actual allocation uses a separate acquire-load on the freelist head pointer. - Dead-reckoning checks (e.g., "is the ring approximately empty?"):
Relaxed. If the approximate check is stale by one entry, the caller falls back to a serializing path.
Ordering::Relaxed is never used for the head/tail pointers that control whether a slot
is safe to read or write. Misuse of Relaxed on these pointers produces silent data
corruption on ARM and RISC-V; it works by accident on x86. The rule from mandate (3) is
absolute: Relaxed may not appear in any code path that controls visibility of payload
data.
3.9.4 Endianness¶
PPC32 and s390x are big-endian; all other supported architectures are little-endian.
Wire-format structs use Le types (Section 6.1). Kernel-internal
structs use native endianness. Conversion happens at RDMA/wire boundaries only.
3.10 Algorithm Dispatch and In-Kernel SIMD¶
Two closely related mechanisms let UmkaOS select the best available algorithm implementation per platform and use hardware SIMD safely from kernel code.
3.10.1 AlgoDispatch: Zero-Cost Runtime Dispatch¶
AlgoDispatch<F> holds a single function pointer chosen once at boot from a
priority-ordered list of candidates. After boot, dispatching is a plain indirect
call — no atomic, no lock, no branch on a feature flag.
Design rationale. The naïve alternative is checking a feature flag at every call site:
// Bad: branch in hot path, missed inlining, repeated flag read
if this_cpu_has_crypto(CryptoCaps::SHA2_256) { sha256_sha_ni(data, out) }
else { sha256_generic(data, out) }
AlgoDispatch moves the branch to boot time and stores the result as an immutable
function pointer. The call site becomes:
Scope (normative): AlgoDispatch serves boot-phase-9+ consumers that
tolerate one predicted indirect call — crypto, checksums, compression,
RAID parity, memory operations. Two structural limits delimit it:
calling get() before phase-9 init() is undefined behavior, and the
dispatch is an indirect call (a per-call cost and a Spectre-v2 surface).
Paths that violate either limit — lock algorithms (taken from boot
phase 2), pre-phase-9 consumers, and Spectre-sensitive call sites — use
boot-time instruction patching instead: code_alternative! for inline
shapes and alternative_call! for out-of-line implementation selection
(Section 2.16). The
RawSpinLock design is the worked example of the patching side
(Section 3.5).
Implementation:
/// A boot-initialised, immutable-after-init function pointer.
///
/// `F` is a function-pointer type, e.g. `fn(&[u8], &mut [u8; 32])`.
///
/// # Thread Safety
/// `AlgoDispatch<F>` is `Sync` when `F: Copy + Send`. After `init()`, the
/// contained function pointer is never modified, so concurrent reads from
/// multiple CPUs are safe without synchronisation.
///
/// # Heterogeneous CPU Support
/// Candidate selection uses `all_cpus_have_*()` (§2.1.2.18.3), ensuring the
/// chosen implementation is valid on every CPU in the system. A kthread using
/// the dispatched function can migrate freely without correctness issues.
/// On RISC-V systems where harts have differing ISA extensions, the scheduler
/// already constrains task placement via `isa_required` affinity (§7.1.5.9);
/// `AlgoDispatch` selects based on the universal intersection, which is the
/// correct choice for kthreads that may run on any hart.
pub struct AlgoDispatch<F: Copy + Send + 'static> {
func: UnsafeCell<MaybeUninit<F>>,
init_done: AtomicBool,
}
/// Safety: after init() sets init_done, func is immutable; read-only access
/// from multiple threads is safe.
unsafe impl<F: Copy + Send + 'static> Sync for AlgoDispatch<F> {}
impl<F: Copy + Send + 'static> AlgoDispatch<F> {
/// Construct an uninitialised dispatch slot (for use in `static` context).
pub const fn uninit() -> Self {
Self {
func: UnsafeCell::new(MaybeUninit::uninit()),
init_done: AtomicBool::new(false),
}
}
/// Boot-time initialiser. Must be called exactly once, during boot phase 9
/// (after `cpu_features_freeze()`, before any concurrent kernel code runs).
/// Panics if called twice or after concurrent access begins.
///
/// Selects the first candidate in `candidates` whose requirements are
/// satisfied by the universal CPU feature intersection. The last entry MUST
/// have empty requirements (the generic fallback); the function panics if
/// no candidate matches.
pub fn init(&self, candidates: &[AlgoCandidate<F>]) {
assert!(
!self.init_done.load(Ordering::Relaxed),
"AlgoDispatch::init called twice"
);
for candidate in candidates {
if all_cpus_have_crypto(candidate.crypto_required)
&& all_cpus_have_atomics(candidate.atomics_required)
&& min_simd_width_bytes() >= candidate.min_simd_bytes
{
// SAFETY: init_done is false, so no concurrent reads yet.
unsafe { (*self.func.get()).write(candidate.func) };
self.init_done.store(true, Ordering::Release);
log::info!(
"algo_dispatch: selected '{}' for '{}'",
candidate.name,
candidate.algo_name,
);
return;
}
}
panic!("AlgoDispatch: no candidate matched (missing generic fallback?)");
}
/// Call the dispatched implementation. Panics in debug if `init()` was
/// not called; in release, calling before init is undefined behaviour
/// (the AlgoDispatch initialisation sequence in boot phase 9 prevents this).
#[inline(always)]
pub fn get(&self) -> F {
// Acquire pairs with the Release store in init(), ensuring the func
// write is visible on weakly-ordered architectures (ARM, RISC-V).
// This Acquire load is NOT a debug check — it is a correctness
// requirement for memory ordering. On x86-64 (TSO), Acquire on
// AtomicBool compiles to a plain MOV (zero extra cost). On AArch64
// it compiles to LDAR (~1 extra cycle vs plain load). The compiler
// cannot hoist or elide this load because AtomicBool::load is opaque.
//
// **Design decision**: The ~1 cycle Acquire load per call is accepted.
// AlgoDispatch is used for crypto algorithm selection and SIMD dispatch —
// warm-path operations (per-packet or per-block-I/O, not per-instruction).
// At the highest anticipated dispatch frequency (~1M calls/sec for crypto),
// the LDAR adds ~1ms/sec — negligible. The Acquire is retained because it
// provides the happens-before with init() on weakly-ordered architectures
// and prevents the compiler from caching the init_done check across calls.
//
// **Future optimization (Phase 3+)**: A static-key / patched-NOP approach
// could eliminate the Acquire load entirely by patching the init_done check
// to a NOP at boot time. This requires self-modifying code with I-cache
// coherency across all 8 architectures — deferred in favor of the simpler,
// correct, and quantifiably cheap Acquire approach.
let ready = self.init_done.load(Ordering::Acquire);
debug_assert!(ready, "AlgoDispatch used before init()");
// SAFETY: init_done is true (Acquire) → func is fully written and immutable.
unsafe { (*self.func.get()).assume_init() }
}
}
/// One candidate in an `AlgoDispatch` selection list.
pub struct AlgoCandidate<F: Copy> {
/// Algorithm name for boot log (e.g. "sha256").
pub algo_name: &'static str,
/// Implementation variant name (e.g. "sha_ni", "sha2_ce", "zknh", "generic").
pub name: &'static str,
/// All listed cryptographic capabilities must be in the universal intersection.
pub crypto_required: CryptoCaps,
/// All listed atomic capabilities must be universal.
pub atomics_required: AtomicCaps,
/// Minimum SIMD register width required in bytes. 0 = scalar, no SIMD needed.
pub min_simd_bytes: u16,
/// The function pointer for this implementation.
pub func: F,
}
Declaration macro for ergonomic global registration:
/// Declare a module-level `AlgoDispatch` and its candidate list.
/// Candidates are listed highest-priority first; the last MUST have no
/// requirements (generic fallback).
///
/// The macro expands to a `static` `AlgoDispatch` and a `fn init_<name>()`
/// that is registered with the boot-phase-9 init table via `#[algo_init]`.
///
/// Example:
/// ```rust
/// algo_dispatch! {
/// pub static SHA256: AlgoDispatch<fn(&[u8], &mut [u8; 32])> = {
/// algo: "sha256",
/// candidates: [
/// // x86-64 SHA-NI / RISC-V Zknh / AArch64 SHA2-CE
/// { "sha_ni_or_ce", crypto: SHA2_256, simd: 0, sha256_hwaccel },
/// // AVX2 4-way parallel schedule (x86-64 without SHA-NI)
/// { "avx2_4way", crypto: {}, simd: 32, sha256_avx2_4way },
/// // NEON 4-way parallel schedule (AArch64 without SHA2-CE)
/// { "neon_4way", crypto: {}, simd: 16, sha256_neon_4way },
/// // Portable scalar — always eligible, always last
/// { "generic", crypto: {}, simd: 0, sha256_generic },
/// ],
/// };
/// }
/// ```
macro_rules! algo_dispatch { ... }
All algo_dispatch! statics in the kernel are initialised by a single call to
algo_dispatch_init_all() at boot phase 9. This function iterates the linker
section __algo_dispatch_inits (populated by #[algo_init] attribute macros on
generated init functions) and calls each init function in source-order within
each crate, crates in link order. No heap allocation occurs. Total init time is
O(candidates × algo_count), bounded in practice to < 1 ms.
3.10.2 SimdKernelGuard: Safe In-Kernel SIMD Use¶
Task FPU context (§2.1.2.17) is managed lazily: user tasks pay no cost unless they use floating-point. That mechanism handles user FPU context — the task's architectural floating-point state that must survive context switches.
A separate problem is deliberate kernel SIMD use: a kthread or kernel function that intentionally issues SIMD instructions for bulk operations (AES encryption, hash computation, SIMD memcpy, compression). Three invariants must hold:
-
Non-preemptible while SIMD is active. SIMD register state is per-CPU and not saved on preemption unless the task-FPU mechanism is engaged. Preemption mid-SIMD would corrupt the registers if the task is migrated to another CPU. Remedy: disable preemption for the SIMD region's duration.
-
Forbidden in interrupt context. An interrupt handler issuing SIMD instructions would corrupt the interrupted task's (or kthread's) SIMD state, which may be live but not yet saved (lazy save deferred). Kernel SIMD is unconditionally forbidden from IRQ handlers, NMI handlers, and softirq handlers.
-
SIMD unit must be enabled for kernel mode. Most architectures disable the FPU/SIMD unit in kernel mode by default (the mechanism that causes the #NM / EL0 FP-trap that the lazy-FPU handler catches). Kernel SIMD requires explicitly re-enabling it for the duration of the operation.
Kernel-FP policy (normative). UmkaOS kernel images are compiled with FP/SIMD
codegen disabled on EVERY architecture — a general-regs-only regime. x86-64's
pinned target is the model (-mmx,-sse,…,+soft-float, rustc_abi: Softfloat); each
of the other seven legs pins the equivalent soft-float / general-regs-only kernel
codegen (e.g. aarch64-unknown-none-softfloat; an F/D-stripped RISC-V kernel target;
FP-feature-stripped custom target JSONs for armv7/ppc32/ppc64le/s390x/loongarch64).
Consequently the FPU/SIMD unit is disabled in kernel mode by default (invariant 3
above), and general kernel code — including setjmp/longjmp, which therefore saves
GPRs only
(Section 3.2)
— never touches FP or SIMD registers. Explicit FP/SIMD instructions exist in EXACTLY
two sanctioned classes:
#[target_feature]-gated SIMD kernels invoked underSimdKernelGuard(crypto, compression, checksums): the guard saves live user FPU state, then enables the unit for the operation's duration and disables it on drop;#[target_feature]-gated scalar-FP functions of kthreads that OWN task FPU context via the lazy-trap path (the guard-free exemption stated in theSimdKernelGuarddoc below): the first FP instruction traps, the lazy-FPU handler allocates the save area and enables the unit (Section 7.3).
Any FP/SIMD instruction outside these two classes is a bug — on the FPU-disabled architectures (AArch64/ARMv7/RISC-V/s390x/LoongArch64) it traps on first issue. Per-target codegen enforcement is a builder-side pinning of each kernel target to the policy above; x86-64 already conforms.
SimdKernelGuard enforces all three invariants via RAII:
/// RAII guard for deliberate kernel-initiated SIMD/FPU use.
///
/// Acquire this guard before issuing any SIMD instruction in kernel code
/// (crypto, compression, SIMD memcpy, etc.). The guard is not required for
/// scalar floating-point in kthreads that legitimately own FPU context (e.g.
/// `PdControllerState` in the IntentOptimizer kthread, §7.3.5).
///
/// # Invariants Enforced
/// - Cannot be acquired from interrupt context (asserted at acquisition).
/// - Preemption is disabled for the guard's lifetime.
/// - The architecture's SIMD/FPU unit is enabled for kernel mode on acquisition
/// and disabled on drop.
/// - If the current task had live (unsaved) user FPU state, it is saved to the
/// task's FPU save area before kernel SIMD is enabled, so that it can be
/// restored correctly on the next return-to-user or context switch.
///
/// # Nestable (reference-counted)
/// Acquiring a `SimdKernelGuard` while one is already held on this CPU is safe:
/// the inner guard increments `simd_kernel_depth` but skips SIMD enable (already
/// active). On drop, the inner guard decrements depth but skips SIMD disable.
/// Only the outermost guard (depth transitions 0→1 on acquire and 1→0 on drop)
/// actually enables/disables the SIMD unit. Debug builds assert nesting depth
/// < 16 to catch unbounded recursion.
///
/// # Architecture-Specific Enable/Disable
///
/// | Arch | Enable in kernel mode | Disable |
/// |---------|------------------------------------|----------------------------------|
/// | x86-64 | `clts` (clear CR0.TS) | `mov cr0, cr0 | CR0_TS` |
/// | AArch64 | `CPACR_EL1.FPEN ← 0b11` | `CPACR_EL1.FPEN ← 0b00` |
/// | | `ZCR_EL1.LEN ← max` (if SVE) | (context switch restores ZCR) |
/// | ARMv7 | `FPEXC.EN ← 1` (MCR p10,7,FPEXC) | `FPEXC.EN ← 0` |
/// | RISC-V | `sstatus.FS ← 01` (Initial) | `sstatus.FS ← 00` (Off) |
/// | | `sstatus.VS ← 01` if RVV present | `sstatus.VS ← 00` |
/// | PPC32 | `MSR.VEC ← 1` (mtmsr) | `MSR.VEC ← 0` |
/// | PPC64LE | `MSR.VEC ← 1, MSR.VSX ← 1` | `MSR.VEC ← 0, MSR.VSX ← 0` |
/// | s390x | `STCTG`/`LCTG` CR0: set AFP bit | `LCTG` CR0: clear AFP bit |
/// | | (Additional Floating-Point). Enables| Disables vector register access. |
/// | | VX (vector extension) register | `arch_state` saves previous CR0. |
/// | | access for SIMD instructions. | |
/// | LoongArch64 | `CSR.EUEN.FPE ← 1` (FP enable) | `CSR.EUEN.FPE ← 0` |
/// | | `CSR.EUEN.SXE ← 1` (128-bit LSX) | `CSR.EUEN.SXE ← 0` |
/// | | `CSR.EUEN.ASXE ← 1` (256-bit LASX) | `CSR.EUEN.ASXE ← 0` |
///
/// The `arch_state` field stores whatever per-arch context is needed at drop
/// time (e.g., the previous CR0 value on x86-64, the previous CPACR_EL1 value
/// on AArch64). It is zero-sized on architectures where a single bit-set/clear
/// suffices and the previous value is known (e.g., CR0.TS is always 1 before
/// the guard; always restored to 1 on drop).
///
/// **Per-architecture `SimdKernelState` sizes:**
///
/// | Architecture | Size (bytes) | Saved state |
/// |---|---|---|
/// | x86-64 | 0 (ZST) | CR0.TS is always 1 before; restored unconditionally. |
/// | AArch64 | 8 | Previous CPACR_EL1 (u64): restore FPEN+ZEN fields on drop. |
/// | ARMv7 | 4 | Previous FPEXC (u32): restore EN bit on drop. |
/// | RISC-V 64 | 8 | Previous sstatus (u64): restore FS and VS fields on drop. |
/// | PPC32 | 4 | Previous MSR (u32): restore VEC bit on drop. |
/// | PPC64LE | 4 | Previous MSR low word (u32): restore VEC+VSX bits on drop. |
/// | s390x | 8 | Previous CR0 (u64): restore AFP bit on drop via `LCTG`. |
/// | LoongArch64 | 4 | Previous CSR.EUEN (u32): restore FPE+SXE+ASXE bits on drop. |
pub struct SimdKernelGuard {
_preempt: PreemptGuard,
arch_state: arch::current::cpu::SimdKernelState,
// Prevent Send: the guard must be dropped on the CPU that acquired it.
_no_send: PhantomData<*mut ()>,
}
impl SimdKernelGuard {
/// Acquire the guard. Panics if called from interrupt context
/// (`CpuLocal::irq_count > 0` or `CpuLocal::preempt_count` indicates IRQ depth).
#[must_use]
#[inline]
pub fn new() -> Self {
debug_assert!(
!arch::current::interrupts::in_interrupt(),
"SimdKernelGuard::new() called from interrupt context"
);
#[cfg(not(debug_assertions))]
if arch::current::interrupts::in_interrupt() {
// Panic even in release builds: returning a noop guard would
// allow the caller to execute SIMD instructions without the
// SIMD unit enabled, causing a #UD fault or silent data
// corruption. There is no safe fallback here — the caller
// expects SIMD to be available after acquiring the guard.
panic!("SimdKernelGuard::new() called from interrupt context");
}
let preempt = PreemptGuard::new();
let simd_depth = CpuLocal::simd_kernel_depth();
let depth = simd_depth.load(Ordering::Relaxed);
debug_assert!(
depth < 16,
"SimdKernelGuard nesting depth {} exceeds limit — likely unbounded recursion",
depth,
);
if depth > 0 {
// Already active on this CPU — return a noop guard that only
// increments depth. SIMD unit is already enabled by the outer guard.
simd_depth.fetch_add(1, Ordering::Relaxed);
return Self {
_preempt: preempt,
arch_state: arch::current::cpu::SimdKernelState::NOOP,
_no_send: PhantomData,
};
}
// Outermost acquisition: save task FPU state if it is live (lazy-FPU
// mechanism may not have saved it yet). After this, the save area is
// up-to-date for context switch.
arch::current::cpu::save_task_fpu_if_live();
let arch_state = arch::current::cpu::simd_kernel_enable();
// Track nesting depth in CpuLocal for is_active() check.
//
// NMI window: between save_task_fpu_if_live() and the depth increment
// below, an NMI handler would see depth == 0. This is safe because NMI
// handlers are prohibited from using SIMD (checked by the in_interrupt()
// guard above). The enable-before-increment order avoids needing to undo
// depth on enable failure.
CpuLocal::simd_kernel_depth().fetch_add(1, Ordering::Relaxed);
Self { _preempt: preempt, arch_state, _no_send: PhantomData }
}
/// Create a no-op guard that skips SIMD enable/disable. Used when:
/// - Called from interrupt context (FPU state already saved by the
/// interrupt entry trampoline on architectures that require it).
/// - The caller knows SIMD is already active (`is_active() == true`)
/// but needs a guard value for API uniformity.
///
/// The returned guard holds a `PreemptGuard` (preemption disabled)
/// but does NOT touch FPU/SIMD registers or increment `simd_kernel_depth`.
/// Drop is a no-op beyond re-enabling preemption.
pub fn noop() -> Self {
Self {
_preempt: PreemptGuard::new(),
arch_state: arch::current::cpu::SimdKernelState::NOOP,
_no_send: PhantomData,
}
}
/// True if a `SimdKernelGuard` is currently held on this CPU.
/// Use for nesting avoidance in composable functions:
/// ```rust
/// let _guard = if !SimdKernelGuard::is_active() { Some(SimdKernelGuard::new()) } else { None };
/// ```
/// Check if a SimdKernelGuard is active on the CURRENT CPU.
/// Preemption must be disabled (or an existing PreemptGuard held)
/// to ensure the CpuLocal read is on the correct CPU. If called
/// without preemption disabled, a migration between the CpuLocal
/// read and the field access could return a false positive from
/// the previous CPU, causing a #UD fault on the new CPU.
#[inline(always)]
pub fn is_active() -> bool {
let _preempt = PreemptGuard::new();
CpuLocal::simd_kernel_depth().load(Ordering::Relaxed) > 0
}
}
impl Drop for SimdKernelGuard {
fn drop(&mut self) {
// Field-scoped `CpuLocal::simd_kernel_depth()` (a single-field `&'static
// AtomicU8`) — no whole-block reference (`get()`/`get_mut()` deleted,
// ESC-0431; the former whole-block aliasing violation SF-101/SF-102 is
// now unrepresentable). fetch_sub returns the PREVIOUS value; depth
// reaches 0 when prev == 1.
let prev = CpuLocal::simd_kernel_depth().fetch_sub(1, Ordering::Relaxed);
if prev == 1 {
// Outermost guard: disable the SIMD unit.
arch::current::cpu::simd_kernel_disable(&self.arch_state);
}
// Inner guards: arch_state is NOOP, simd_kernel_disable is a no-op anyway.
// _preempt dropped here: preemption re-enabled after SIMD disabled.
}
}
A simd_kernel_depth: AtomicU8 field is added to CpuLocalBlock (§3.1.2). It tracks
the nesting depth for is_active() and for the nestable guard protocol: only the
outermost guard (depth 0→1) enables SIMD; only the last drop (depth 1→0) disables
it. The field is zero when no SimdKernelGuard is held. AtomicU8 (with Relaxed
ordering) is used instead of plain u8 to allow access via the field-scoped
CpuLocal::simd_kernel_depth() projection (a single-field &'static AtomicU8)
without ever forming a whole-block reference of either kind, which the
CpuLocal reference-formation doctrine
(Section 3.2)
prohibits — NMI handlers may concurrently access other atomic fields through the shared
reference. On x86-64,
Relaxed atomic ops compile to plain loads/stores with zero overhead.
Non-local exit (longjmp) bypasses Drop. A domain panic recovered via longjmp()
unwinds NO destructors, so a SimdKernelGuard live on the panicking domain's stack is
skipped: it neither disables the SIMD unit nor decrements simd_kernel_depth, and its
embedded PreemptGuard is likewise not dropped. The catch_domain_panic() recovery arm
repairs this out of band. The normative longjmp-out-of-a-live-guard recovery contract —
unconditional simd_kernel_disable, simd_kernel_depth = 0, and snapshot-based
preempt-count restoration — is specified in the JmpBuf region at
Section 3.2.
User FPU state needs no repair: the outermost guard saved it before enabling the unit, so
the unconditional disable makes the next user FP access trap-and-restore.
3.10.3 Combined Usage Pattern¶
A kernel subsystem using hardware-accelerated algorithms combines both mechanisms:
// ── Module level (initialised at boot phase 9) ──────────────────────────────
// Expanded AES-GCM key material: the AES round-key schedule plus the GHASH
// subkey H = AES_K(0^128), computed once when the key is set. Opaque and
// algorithm-private — kernel-internal, never crosses a KABI/wire boundary, so
// no `#[repr(C)]`/`const_assert!` is required. `[u8; 240]` holds up to 15 round
// keys × 16 B (the AES-256 schedule); shorter key sizes use a prefix.
struct AesGcmKey {
round_keys: [u8; 240],
ghash_h: [u8; 16],
}
type AesGcmFn = fn(key: &AesGcmKey, nonce: &[u8; 12],
aad: &[u8], plaintext: &[u8], ct_out: &mut [u8]);
algo_dispatch! {
pub(crate) static AES_GCM: AlgoDispatch<AesGcmFn> = {
algo: "aes-gcm",
candidates: [
// VAES + VPCLMULQDQ: 8-block parallelism (x86-64 AVX-512+)
{ "vaes_vpclmul", crypto: VAES | CLMUL, simd: 64, aes_gcm_vaes_vpclmul },
// AES-NI + PCLMULQDQ (x86-64 baseline accelerated, AArch64 PMULL)
{ "aesni_clmul", crypto: AES_BLOCK | CLMUL, simd: 16, aes_gcm_aesni_clmul },
// RISC-V Zkne + Zkg scalar
{ "zkne_zkg", crypto: AES_BLOCK | CLMUL, simd: 0, aes_gcm_zkne_zkg },
// PPC64LE vcipher + vpmsumd
{ "vcipher_ppc", crypto: AES_BLOCK | CLMUL, simd: 16, aes_gcm_vcipher_ppc },
// Portable constant-time scalar (always eligible, always last)
{ "generic", crypto: {}, simd: 0, aes_gcm_generic },
],
};
}
// ── Call site ────────────────────────────────────────────────────────────────
pub fn encrypt(key: &AesGcmKey, nonce: &[u8; 12],
aad: &[u8], plaintext: &[u8], ct_out: &mut [u8]) {
// Acquire the SIMD guard only when the selected implementation uses SIMD.
// `min_simd_bytes > 0` in the selected candidate means SIMD is needed.
// If a SimdKernelGuard is already held by a caller, reuse it (no re-entry).
let _guard = if AES_GCM_USES_SIMD && !SimdKernelGuard::is_active() {
Some(SimdKernelGuard::new())
} else {
None
};
AES_GCM.get()(key, nonce, aad, plaintext, ct_out);
}
The boolean AES_GCM_USES_SIMD is a static bool initialised alongside the
AlgoDispatch during boot phase 9; it captures whether the selected candidate has
min_simd_bytes > 0. This avoids the guard acquisition overhead on platforms
(or algorithm variants) that selected a purely scalar implementation.
3.10.4 Feature-Dependent Subsystem Catalog¶
Every kernel subsystem that benefits from CPU-specific hardware acceleration
uses AlgoDispatch for variant selection. The table below is the authoritative
catalog. New entries must be added here when a subsystem gains hardware
acceleration.
For the full kernel image structure showing how these modules are packaged and loaded, see Section 2.21.
3.10.4.1 Cryptographic Algorithms¶
| Algorithm | AlgoDispatch Static |
Priority-Ordered Variants |
|---|---|---|
| AES-GCM | AES_GCM |
VAES+VPCLMUL (x86 AVX-512) → AES-NI+CLMUL (x86) → CE+PMULL (AArch64) → Zkne+Zkg (RISC-V) → vcipher+vpmsumd (PPC64) → generic |
| AES-XTS | AES_XTS |
AES-NI (x86) → CE (AArch64) → Zkne (RISC-V) → generic |
| ChaCha20-Poly1305 | CHACHA20_POLY1305 |
AVX-512 (x86) → AVX2 (x86) → NEON (AArch64) → generic |
| SHA-256 | SHA256 |
SHA-NI (x86) → AVX2 4-way (x86) → SHA2-CE (AArch64) → NEON 4-way (AArch64) → Zknh (RISC-V) → generic |
| SHA-512 | SHA512 |
AVX2 (x86) → SHA512-CE (AArch64) → generic |
| SHA-3 / SHAKE | SHA3 |
AVX2 Keccak-4x (x86) → SHA3-CE (AArch64, FEAT_SHA3) → generic |
| SM3 | SM3 |
AVX2+AES-NI (x86) → SM3-CE (AArch64, FEAT_SM3) → Zksh (RISC-V) → generic |
| SM4 | SM4 |
AES-NI affine (x86) → SM4-CE (AArch64, FEAT_SM4) → Zksed (RISC-V) → generic |
| ML-KEM-768 | ML_KEM_768 |
AVX2 NTT (x86) → NEON NTT (AArch64) → generic |
| ML-DSA-65 | ML_DSA_65 |
AVX2 (x86) → NEON (AArch64) → generic |
| GHASH | GHASH |
PCLMULQDQ (x86) → PMULL (AArch64) → Zkg (RISC-V) → generic |
| Poly1305 | POLY1305 |
AVX2 (x86) → NEON (AArch64) → generic |
3.10.4.2 Checksum and Hash¶
| Algorithm | AlgoDispatch Static |
Priority-Ordered Variants |
|---|---|---|
| CRC32C | CRC32C |
SSE4.2 crc32 instr (x86) → CRC32 instr (AArch64) → Zbc (RISC-V) → generic |
| xxHash64 | XXHASH64 |
AVX2 (x86) → NEON (AArch64) → generic |
| Adler32 | ADLER32 |
SSSE3 (x86) → NEON (AArch64) → generic |
Incremental CRC32C API: The CRC32C dispatch slot is exposed through a
three-call incremental helper (init → update* → finalize) so a caller can
checksum several non-contiguous byte ranges without concatenating them first —
used by, e.g., the live-evolution ComponentState checksum
(Section 13.18) over its header fields plus data buffer, and by
on-disk metadata verification. This is the single canonical CRC32C helper;
subsystems MUST NOT re-implement it.
/// CRC32C (Castagnoli, polynomial 0x1EDC6F41) folding step: fold `bytes` into a
/// running CRC register and return the updated register.
type Crc32cFn = fn(crc: u32, bytes: &[u8]) -> u32;
/// Dispatch slot for the CRC32C folding step, selected once at boot phase 9
/// from the hardware variants catalogued in the table above (SSE4.2 `crc32` on
/// x86, `CRC32C{B,H,W,X}` on AArch64, Zbc on RISC-V, slice-by-8 table generic).
/// Initialised via `CRC32C.init(&[…])` alongside the other dispatch statics.
pub static CRC32C: AlgoDispatch<Crc32cFn> = AlgoDispatch::uninit();
/// Start an incremental CRC32C computation. Returns the CRC32C seed register
/// (all-ones); it is NOT a valid checksum until `crc32c_finalize` inverts it.
pub fn crc32c_init() -> u32 { 0xFFFF_FFFF }
/// Fold `bytes` into the running CRC register `crc`. May be called any number of
/// times over non-contiguous ranges. Dispatches to the hardware `CRC32C` variant
/// when the CPU provides one, else the portable table.
pub fn crc32c_update(crc: u32, bytes: &[u8]) -> u32 {
CRC32C.get()(crc, bytes)
}
/// Finish an incremental CRC32C computation: apply the final bit inversion and
/// return the checksum.
pub fn crc32c_finalize(crc: u32) -> u32 { crc ^ 0xFFFF_FFFF }
3.10.4.3 Compression¶
| Algorithm | AlgoDispatch Static |
Priority-Ordered Variants |
|---|---|---|
| zstd compress | ZSTD_COMPRESS |
AVX2 match finder (x86) → NEON (AArch64) → generic |
| zstd decompress | ZSTD_DECOMPRESS |
BMI2 bit extraction (x86) → generic |
| LZ4 | LZ4 |
AVX2 sequence match (x86) → NEON (AArch64) → generic |
| zlib/deflate | ZLIB_DEFLATE |
SSE4.2+PCLMULQDQ (x86) → CRC32+PMULL (AArch64) → generic |
3.10.4.4 Memory Operations¶
| Operation | AlgoDispatch Static |
Priority-Ordered Variants |
|---|---|---|
| memcpy (kernel) | KERNEL_MEMCPY |
ERMS+FSRM rep movsb (x86) → AVX2 (x86, no FSRM) → NEON ldp/stp (AArch64) → generic |
| memset / page zero | KERNEL_MEMSET |
ERMS rep stosb (x86) → AVX2 (x86) → DC ZVA (AArch64) → NEON stp xzr (AArch64) → generic |
| memcmp | KERNEL_MEMCMP |
SSE4.2 PCMPISTRI (x86) → NEON (AArch64) → generic |
3.10.4.5 RAID Parity¶
| Operation | AlgoDispatch Static |
Priority-Ordered Variants |
|---|---|---|
| XOR (RAID5) | RAID_XOR |
AVX-512 (x86) → AVX2 (x86) → SVE (AArch64) → NEON (AArch64) → RVV (RISC-V) → generic |
| P+Q (RAID6) | RAID_PQ |
AVX-512 (x86) → AVX2 (x86) → NEON (AArch64) → generic |
3.10.4.6 Networking¶
| Operation | AlgoDispatch Static |
Priority-Ordered Variants |
|---|---|---|
| TCP/UDP/IP checksum | NET_CSUM |
AVX2 (x86) → SSE2 (x86) → NEON (AArch64) → generic |
| RSS Toeplitz hash | NET_TOEPLITZ |
PCLMULQDQ (x86) → PMULL (AArch64) → generic |
3.10.4.7 Code Alternatives (Instruction-Level Dispatch)¶
AlgoDispatch selects which function to call. A complementary mechanism —
code_alternative! — selects which instruction to use within a function.
This covers CPU errata workarounds, new instruction adoption, and
microarchitectural tuning at the instruction level:
| Category | Examples | Mechanism |
|---|---|---|
| Spectre/Meltdown mitigations | Retpoline → eIBRS direct branch; KPTI enable/disable; VERW on context switch | code_alternative! + ErrataCaps |
| New instructions replacing old | SERIALIZE replacing CPUID; WRMSRNS replacing WRMSR; LKGS for syscall entry |
code_alternative! + arch_raw |
| Errata workarounds | LFENCE serialization; CLEARBHB (AArch64); DSB before TLBI (Cortex-A76) | code_alternative! + ErrataCaps |
| Page zeroing | DC ZVA vs STP xzr (AArch64, depends on ZVA block size) | code_alternative! |
| Lock algorithm selection | RawSpinLock qspinlock-vs-ticket branch; native-vs-paravirt slow path; pv unlock | code_alternative! (fast-path branch, unlock byte-patch) + alternative_call! (slow-path target) — see Section 3.5 |
| Power management | MWAIT hint value selection; HWP enable/disable; C-state depth | MicroarchHints + platform PM driver |
See Section 2.16
for the full code_alternative! specification, ErrataCaps bitflags,
and MicroarchHints struct.
3.10.4.8 Not Dispatched via AlgoDispatch or code_alternative!¶
The following subsystems have CPU-feature-dependent behavior but use
neither AlgoDispatch nor code_alternative!, because their dispatch
is structurally different:
| Subsystem | Mechanism | Reason |
|---|---|---|
| Isolation domain switch | arch::current::isolation (compile-time per target triple) |
Affects entire driver model topology, not a single callsite. Runtime fallback (POE → page table on AArch64) is per-driver-init, not per-call. |
| BPF JIT | Per-arch JIT backend (x86-64, AArch64, RISC-V, etc.) | Code generation, not algorithm selection. JIT emits arch-native instructions; no "variant" to dispatch. |
| Context switch | arch::current::context::context_switch() |
Per-arch assembly. Only one implementation per target triple. |
| TLB flush | arch::current::mm::flush_tlb_*() |
Hardware instruction, no variants. |
| Interrupt entry/exit | Per-arch asm (IDT/GIC/PLIC/OpenPIC vectors) | Hardware trap path, no dispatch. |
| Page table depth | MicroarchHints.page_table_levels → configured once at boot |
4-level vs 5-level is an MMU configuration, not an instruction alternative. |
3.10.4.9 Three-Level Dispatch Summary¶
Compile time Boot time (phase 9) Runtime
│ │ │
▼ ▼ ▼
arch::current:: code_alternative! AlgoDispatch
(per target triple) (instruction patching) (function pointer)
│ │ │
│ Selects arch module │ Patches instructions │ Selects algorithm
│ (x86 vs ARM vs RV) │ within arch code for │ implementation
│ │ specific CPU model │ (SHA-NI vs generic)
│ │ │
└──── compile-time ─────┴──── zero-overhead ─────┴── one indirect call ──
after patching (branch-predicted)
All three mechanisms together ensure the kernel binary adapts to the exact hardware at boot — no per-host recompilation, no runtime branches in hot paths. This is how UmkaOS avoids the "compile for your CPU" approach (Gentoo/Slackware) while still extracting maximum performance.
3.10.4.10 Module Packaging Rule¶
Default (Model A): All AlgoDispatch candidates are compiled inline into
the module that declares the algo_dispatch! static. Dead variants remain in
the image (~few KB each). This is the default for all algorithms in the catalog
above.
Exception (Model B): If a single variant implementation exceeds 64 KB of
code (e.g., a complex AVX-512 implementation with hand-tuned assembly), it
MAY be split into a separate feature-variant module loaded by AlgoDispatch
at boot phase 9. The decision is per-algorithm, documented in the variant's
algo_dispatch! declaration with a // MODEL_B: <module-name> comment.
Currently, no algorithm in the catalog requires Model B — all are under the 64 KB threshold. This section exists to define the mechanism for future use.
3.11 Workqueue / Deferred Work¶
Kernel operations that cannot complete in interrupt context — because IRQ handlers must be atomic and non-sleeping — or that must not block the calling thread need deferred execution. UmkaOS provides a structured workqueue mechanism for this.
UmkaOS improvement over Linux: Linux uses an anonymous kworker pool model where
all kernel-wide deferred work competes in a shared thread pool, causing priority
inversion, poor debuggability (ps shows meaningless kworker/0:1), and no
backpressure (the queue grows unboundedly under load). UmkaOS instead requires each
subsystem to create a named thread pool with explicit priority, bounded depth,
and CPU affinity:
psshowsumkad-net-rx-0,umkad-blk-io-3— always attributable to a subsystem- Bounded queues return
WouldBlockinstead of silently accumulating unbounded work - Priority isolation: network Rx runs SCHED_FIFO; background scan runs SCHED_IDLE
- No priority inversion: high-priority subsystems are never delayed by background tasks
3.11.1 Core Types¶
/// Opaque handle to a submitted work item. Used for cancellation.
pub struct WorkHandle(u64);
/// A unit of deferred work. `f` runs in a workqueue thread context:
/// preemptible, may sleep, may allocate with GFP_KERNEL.
///
/// # Submission context
/// `queue_work()` may be called from ANY context — process, softirq,
/// completion-consumer, or hardirq. It is a lock-free bounded
/// `BoundedMpmcRing` push (no allocation — WorkItems come from a per-CPU
/// slab, see below) followed by an IRQ-safe `WaitQueueHead::wake_up()`
/// (the same primitive completion handlers already use from IRQ context).
/// This is what makes it the designated deferral mechanism for I/O
/// completion callbacks (`blk-io` — e.g., `writeback_end_io_deferred`,
/// [Section 15.2](15-storage.md#block-io-and-volume-management)). An earlier revision forbade IRQ
/// context wholesale; the restriction applied to a pre-`BoundedMpmcRing`
/// design that could allocate on submission.
///
/// What remains process-context-only: `cancel_work_sync()`, `flush()`,
/// `drain()`, and `with_timeout()` (they sleep). For deferred FREEING of
/// RCU-protected memory, `rcu_call` ([Section 3.4](#cumulative-performance-budget))
/// remains the right primitive — it batches with grace periods; a
/// workqueue item does not.
pub struct WorkItem {
pub f: fn(*mut ()),
pub data: *mut (),
/// Deadline hint in nanoseconds from boot. `NO_DEADLINE` (`u64::MAX`) = no
/// deadline (schedule at discretion). If set and expired, the item is still
/// executed (not dropped); the deadline serves as an urgency signal
/// to the scheduler.
///
/// Uses `NO_DEADLINE` sentinel instead of `Option<u64>` to save 8 bytes:
/// `Option<u64>` is 16 bytes (no niche optimization for u64), while
/// `u64` is 8 bytes. `u64::MAX` nanoseconds is ~584 years from boot —
/// safely beyond any practical deadline. This keeps `WorkItem` at 24
/// bytes (three pointer-sized fields) instead of 32.
pub deadline_ns: u64,
}
/// Sentinel value for "no deadline". `u64::MAX` nanoseconds = ~584 years from boot.
pub const NO_DEADLINE: u64 = u64::MAX;
impl WorkItem {
/// Create a new work item.
///
/// `deadline_ns == u64::MAX` (`NO_DEADLINE`) indicates no deadline — the work
/// item is scheduled at the workqueue's discretion. All u64 values are valid;
/// `NO_DEADLINE` is not a "special" invalid value that needs validation — it is
/// simply the sentinel for "no deadline." There is no way to distinguish
/// "accidental u64::MAX" from "intentional NO_DEADLINE" — they are the same
/// value by definition.
pub fn new(f: fn(*mut ()), data: *mut (), deadline_ns: u64) -> Self {
Self { f, data, deadline_ns }
}
}
// SAFETY: WorkItem is Send; the caller must ensure `data` is valid
// for the lifetime of the work item.
unsafe impl Send for WorkItem {}
/// Delayed work: a RE-ARMABLE deferral handle, embedded in its owning
/// subsystem struct (e.g. `BdiWriteback.dwork` for periodic writeback).
/// Arming (`schedule_delayed` for `SYSTEM_WQ`, `WorkQueue::queue_delayed_work`
/// for a specific queue) starts a one-shot hrtimer (`hrtimer_start`, timer
/// framework in the scheduling chapter); on expiry the timer handler clears
/// `pending` and enqueues a FRESH `WorkItem` rebuilt from the stored `item`'s
/// fields on the target queue, so the handle is immediately re-armable —
/// periodic users re-arm from the work function itself.
pub struct DelayedWork {
/// The template the expiry handler enqueues from. `WorkItem` is NOT `Copy`
/// (it carries single-ownership move semantics for `queue_work`, which takes
/// its argument by value), but its three fields (fn pointer + context
/// pointer + deadline) are each `Copy`, so the handler rebuilds a fresh
/// `WorkItem` from them via `WorkItem::new` WITHOUT moving `item` out of the
/// shared handle — the `DelayedWork` itself is never consumed.
pub item: WorkItem,
/// Arming state: `false` = idle, `true` = timer armed, not yet fired.
/// Single-flag idempotence for `schedule_delayed` (see method).
pending: AtomicBool,
/// The one-shot timer armed by `schedule_delayed`/`queue_delayed_work`.
timer: HrTimer,
}
impl DelayedWork {
/// Create an idle (unarmed) delayed-work handle.
pub fn new(item: WorkItem) -> Self;
/// Arm this handle on the shared system work queue: after `delay`,
/// `item` is enqueued on `SYSTEM_WQ` for execution.
///
/// The `Duration` → nanoseconds conversion is explicit:
/// `delay.as_nanos() as u64`. u64 nanoseconds ≈ 584 years (see
/// `NO_DEADLINE` above) — the cast cannot truncate a real timeout.
/// `Duration::ZERO` is a valid delay meaning "enqueue at the next timer
/// opportunity"; callers whose contract treats zero as *disabled*
/// (e.g. `dirty_writeback_interval()`) must guard before calling.
///
/// **Idempotent while armed**: if a previous arming has not fired yet,
/// returns `false` and the EARLIER expiry is kept — exactly what
/// periodic re-arm callers want, matching Linux `schedule_delayed_work()`
/// on an already-pending work. Returns `true`
/// if this call armed the timer.
///
/// The expiry handler runs in timer (IRQ) context: it clears `pending`
/// (Release) BEFORE submitting a rebuilt work item via
/// `SYSTEM_WQ.get().expect("set at kernel init").queue_work(WorkItem::new(
/// self.item.f, self.item.data, self.item.deadline_ns))` (lock-free MPMC
/// ring — IRQ-safe), so a re-arm racing with expiry is never lost. The three
/// `self.item` fields are read by `Copy`, never moving the shared `item`.
/// `queue_work` returns `Result<WorkHandle, KernelError>`: the handler
/// discards the `Ok(WorkHandle)` (a periodic re-arm needs no cancellation
/// token), and on `Err(KernelError::WouldBlock)` (ring momentarily full)
/// re-arms the timer 1 ms out instead of dropping the work — bounded retry,
/// no silent loss.
pub fn schedule_delayed(&self, delay: Duration) -> bool {
if self.pending.swap(true, Ordering::AcqRel) {
return false; // Already armed — keep the earlier expiry.
}
let delay_ns = delay.as_nanos() as u64;
hrtimer_start(&self.timer, delay_ns); // one-shot; handler above
// `hrtimer_start`/`hrtimer_cancel` are the shared-ref arm/cancel
// primitives defined in [Section 7.8](07-scheduling.md#timekeeping-and-clock-management).
true
}
/// Disarm a pending (not yet fired) arming. Returns `true` if a pending
/// timer was cancelled. Uses `hrtimer_cancel()`, which SYNCHRONOUSLY
/// waits out an in-flight expiry handler (timer framework contract), so
/// after `cancel()` returns no expiry can enqueue `item` — though an
/// item enqueued by an EARLIER expiry may still be pending in the queue
/// (drain the queue or `cancel_work_sync` for full quiescence).
///
/// Owners embedding a `DelayedWork` MUST call this before teardown
/// (e.g. BDI unregister): an expiry after the owner is freed would
/// enqueue a work item whose `data` pointer dangles.
pub fn cancel(&self) -> bool {
if !self.pending.swap(false, Ordering::AcqRel) {
return false; // Idle — nothing armed.
}
hrtimer_cancel(&self.timer);
true
}
}
/// Named thread pool for deferred work.
pub struct WorkQueue {
name: &'static str,
queue: Arc<DynMpmcRing<WorkItem>>,
/// Opaque handle to a kernel thread (defined in [Section 8.8](08-process.md#resource-limits-and-accounting)).
threads: ArrayVec<KthreadHandle, WORKQUEUE_MAX_THREADS>,
/// Scheduling policy for worker threads (defined in [Section 8.1](08-process.md#process-and-task-management--kernel-thread-creation)).
sched: KthreadSchedPolicy,
cpu_mask: CpuMask,
}
/// Maximum threads in a single WorkQueue.
pub const WORKQUEUE_MAX_THREADS: usize = 64;
/// Default maximum pending items per WorkQueue.
/// Callers receive `WouldBlock` when this is reached (backpressure, not panic).
pub const WORKQUEUE_DEFAULT_DEPTH: usize = 4096;
/// Serialized (single-thread) variant: guarantees strict FIFO execution order.
/// Use for device state machines and sequenced protocol stacks.
pub struct OrderedWorkQueue(WorkQueue); // internally max_threads = 1
/// The shared default work queue (the UmkaOS analogue of Linux `system_wq`).
/// For LOW-RATE, non-latency-critical deferred work whose owner does not
/// need its own thread pool or FIFO ordering — e.g. periodic writeback
/// scheduling (`BdiWriteback.dwork`) and the AF_UNIX SCM_RIGHTS garbage
/// collector. Threads appear as `umkad-system-N`, satisfying the
/// named-workqueue rule for IRQ/completion-context deferral (the name is
/// `system`). Created during kernel init, before any subsystem that submits
/// deferred work; write-once thereafter. Held in a `BootOnceCell`, not a bare
/// `OnceCell`: the ambient `core::cell::OnceCell` is `!Sync` for every `T`, so a
/// bare `static OnceCell` does not satisfy the `Sync` bound on statics and will
/// not compile; `BootOnceCell` (§Write-Once Boot Publication Cell in
/// [Section 2.3](02-boot-hardware.md#boot-init-cross-arch)) supplies `Sync` for `T: Send + Sync`, met here by
/// `Arc<WorkQueue>`. Subsystems with ordering, latency, or throughput
/// requirements create their own named queue instead of piling onto this one.
pub static SYSTEM_WQ: BootOnceCell<Arc<WorkQueue>> = BootOnceCell::new();
/// Lock-free MPMC bounded ring buffer for work items.
///
/// Per-slot wrapper for the MPMC ring. Each slot carries a Lamport-style
/// sequence number that coordinates producers and consumers without a
/// global lock. The `seq` field is initialized to the slot's index at
/// creation time; producers advance it to `enq_head + 1` after writing
/// (publishing the slot as full), consumers advance it to `deq_tail + capacity`
/// after reading (recycling the slot for the next producer).
///
/// The sequence number protocol ensures:
/// - A producer only writes to a slot where `seq == enq_head` (slot is empty).
/// - A consumer only reads from a slot where `seq == deq_tail + 1` (slot is full).
/// - No two threads ever access the same slot's `data` concurrently.
// kernel-internal, not KABI — generic type with T-dependent size.
#[repr(C, align(64))] // cache-line aligned: prevents false sharing between adjacent slots
pub struct Slot<T> {
/// Lamport sequence number for this slot. Controls slot ownership. Initially
/// equals the slot index; advances by `N` (capacity) each producer/consumer
/// cycle, so ownership is decided RELATIVE to the claiming position, never by
/// absolute value:
/// - `seq == enq_head`: slot is empty, available for producers.
/// - `seq == deq_tail + 1`: slot is full, available for consumers.
/// - Other values: slot is being written/read by another thread.
///
/// Width-selected `AtomicClaimPos` (`AtomicU64` where the leg has 64-bit
/// atomics, native `AtomicU32` on PPC32 — keyed on `target_has_atomic = "64"`,
/// NOT pointer width;
/// [Section 3.1](#rust-ownership-for-lock-free-paths--guarded-position-claim)). The
/// protocol compares only wrapping-RELATIVE differences, so absolute
/// magnitude is never read; the full wrap-safety analysis and the reversal
/// note live on the `Slot::seq` field in the memory-ordering specification
/// below.
pub seq: AtomicClaimPos,
/// The actual data stored in this slot. Only accessed after the
/// sequence number handshake confirms exclusive ownership.
pub data: UnsafeCell<MaybeUninit<T>>,
}
/// Lock-free multi-producer multi-consumer (MPMC) bounded ring buffer.
/// Uses Lamport-style per-slot sequence numbers with a guarded position claim
/// ([Section 3.1](#rust-ownership-for-lock-free-paths--guarded-position-claim)) — see
/// [Section 3.11](#workqueue-deferred-work--boundedmpmcring-memory-ordering-specification)
/// for the full algorithm and memory ordering specification.
///
/// - Producer checks `slot.seq` (empty when `seq == enq_head`), then
/// guarded-claims `enq_head` to acquire the slot
/// - Consumer checks `slot.seq` (full when `seq == deq_tail + 1`), then
/// guarded-claims `deq_tail` to acquire the slot
///
/// **Storage is inline** (`[Slot<T>; N]`), so a `BoundedMpmcRing` needs no heap
/// allocation and its `new()` is `const` — it can initialize a `static`
/// directly (`static Q: BoundedMpmcRing<Ev, 256> = BoundedMpmcRing::new();`).
/// `N` is the capacity and MUST be a power of two `>= 2` (indexing masks with
/// `N-1`; `N == 1` breaks the seq protocol — see `new()`).
/// This is the canonical form used by every fixed-depth ring in the kernel
/// (IMA extend queue, EDAC offline queue, crash-recovery ring, seccomp notif
/// queue, FMA health ring, uffd message queue). When the capacity is only known
/// at run time (WorkQueue depth, OOM reaper victim ring), use the heap-backed
/// sibling `DynMpmcRing<T>` below.
pub struct BoundedMpmcRing<T, const N: usize> {
/// Inline slot storage. Cache-line-aligned per `Slot<T>`.
slots: [Slot<T>; N],
/// Producer claim counter, placed alone on its own cache line via
/// `CacheAligned` (per-arch `CACHE_LINE_SIZE` — 64 by default, 32 on
/// PPC32, 128 on PPC64LE, 256 on s390x; [Section 17.3](17-containers.md#posix-ipc)). This keeps the
/// producer's `enq_head` stores off the consumer's `deq_tail` line, so the
/// two hot counters never false-share — the same head/tail separation
/// `SpscRing` uses ([Section 3.6](#lock-free-data-structures)).
enq_head: CacheAligned<AtomicClaimPos>,
/// Consumer claim counter, on its own cache line (see `enq_head`).
deq_tail: CacheAligned<AtomicClaimPos>,
}
impl<T, const N: usize> BoundedMpmcRing<T, N> {
/// Construct an empty ring. `const` so it can initialize `static`s with
/// zero heap allocation. Each slot's `seq` is set to its index (the Lamport
/// "empty" invariant); `enq_head`/`deq_tail` start at 0.
pub const fn new() -> Self {
// Capacity must be a power of two so `head & (N-1)` is a valid wrap.
const { assert!(N.is_power_of_two(), "BoundedMpmcRing N must be a power of two"); }
// ...and at least 2. `N == 1` is a power of two but breaks the Lamport
// seq protocol: with one slot a producer at `head + 1` re-examines the
// SAME slot and sees `seq == head`, mistaking the still-full slot for free.
const { assert!(N >= 2, "BoundedMpmcRing N must be >= 2 (N == 1 breaks the seq protocol)"); }
// Initialize each slot with seq = index. A const `while` loop over a
// MaybeUninit array is required because `Slot<T>` is not `Copy`
// (contains `AtomicClaimPos`), so `[Slot::new(); N]` array-repeat is illegal.
let mut slots: [MaybeUninit<Slot<T>>; N] =
// SAFETY: an array of `MaybeUninit` is itself valid uninitialized.
unsafe { MaybeUninit::uninit().assume_init() };
let mut i = 0;
while i < N {
slots[i] = MaybeUninit::new(Slot {
seq: AtomicClaimPos::new(i as ClaimPos),
data: UnsafeCell::new(MaybeUninit::uninit()),
});
i += 1;
}
// SAFETY: all N elements were initialized by the loop above.
let slots = unsafe { MaybeUninit::array_assume_init(slots) };
BoundedMpmcRing {
slots,
enq_head: CacheAligned(AtomicClaimPos::new(0)),
deq_tail: CacheAligned(AtomicClaimPos::new(0)),
}
}
}
// SAFETY: BoundedMpmcRing is designed for concurrent multi-producer multi-consumer
// access; per-slot sequence numbers ensure no slot is accessed by two threads at once.
unsafe impl<T: Send, const N: usize> Sync for BoundedMpmcRing<T, N> {}
/// Runtime-sized sibling of `BoundedMpmcRing`. Identical Vyukov algorithm and
/// memory ordering, but the slot array is a single `Box<[Slot<T>]>` allocated
/// once by `with_capacity()` — for rings whose depth is a runtime parameter
/// rather than a compile-time constant. `push`/`pop` never allocate. Used by
/// `WorkQueue` (depth is a `WorkQueue::new` argument) and the OOM reaper
/// victim ring (scaled to system size, [Section 4.2](04-memory.md#physical-memory-allocator)).
pub struct DynMpmcRing<T> {
/// Heap slot storage; length is a power of two (rounded up by the ctor).
slots: Box<[Slot<T>]>,
/// `capacity - 1`; masks `head as usize` / `tail as usize` into a slot index.
/// Stays native `usize` (an index mask, not a claim counter).
mask: usize,
/// Same width-selected claim counters as `BoundedMpmcRing` (the sibling ring
/// gets the identical guarded-claim treatment;
/// [Section 3.1](#rust-ownership-for-lock-free-paths--guarded-position-claim)), and the
/// same `CacheAligned` producer/consumer separation: `enq_head` and
/// `deq_tail` each sit alone on their own cache line ([Section 17.3](17-containers.md#posix-ipc)) so
/// producer writes never invalidate the consumer's line (no false sharing).
enq_head: CacheAligned<AtomicClaimPos>,
deq_tail: CacheAligned<AtomicClaimPos>,
}
impl<T> DynMpmcRing<T> {
/// Allocate a ring with at least `capacity` slots (rounded up to a power of
/// two, with a floor of 2 — a requested `capacity` of 0 or 1 is rounded up
/// to 2, since `N == 1` breaks the seq protocol, see `BoundedMpmcRing::new`).
/// The single allocation happens here; `push`/`pop` never allocate.
/// Each slot's `seq` is initialized to its index.
pub fn with_capacity(capacity: usize) -> Self;
/// Producer side — identical to `BoundedMpmcRing::push`, including the same
/// `guarded_claim(&enq_head.0, seq_of, expected)` call shape (`enq_head`
/// is `CacheAligned<AtomicClaimPos>`, a newtype with no `Deref`, so the
/// call passes the inner `.0`); the `seq_of`
/// closure indexes with `(pos as usize) & self.mask` instead of a const
/// `N - 1`, and `expected` is `|pos| pos` as in the bounded ring.
pub fn push(&self, item: T) -> Result<(), T>;
/// Consumer side — identical to `BoundedMpmcRing::pop`, with the same
/// `guarded_claim(&deq_tail.0, seq_of, expected)` shape (`deq_tail` is
/// `CacheAligned<AtomicClaimPos>`, passed as `.0`; `seq_of` masks with
/// `self.mask`, `expected` is `|pos| pos.wrapping_add(1)`).
pub fn pop(&self) -> Option<T>;
}
// SAFETY: same per-slot sequence-number discipline as BoundedMpmcRing.
unsafe impl<T: Send> Sync for DynMpmcRing<T> {}
3.11.1.1 WorkerPool — Per-Connection Worker Threads¶
A WorkerPool is a dynamically-sized pool of worker kthreads, each dedicated to one
long-lived connection: a worker is spawned when a connection is accepted and reaped
when it closes. This differs from WorkQueue, which multiplexes short deferred
WorkItems across a fixed thread set — a WorkerPool grows and shrinks with the
number of active connections, so a blocking per-connection request loop never starves
its peers. It is the model for in-kernel servers that hold a socket open per client,
such as ksmbd (Section 15.21).
/// A dynamically-sized pool of per-connection worker kthreads.
pub struct WorkerPool {
/// Pool name; workers appear as `<name>-<id>` in process listings.
name: &'static str,
/// Live workers keyed by assigned worker id. Warm-path insert/remove
/// (connection setup/teardown), never on the per-request data path.
workers: SpinLock<XArray<KthreadHandle>>,
/// Next worker id to assign. `u64` — never wraps within the pool lifetime.
next_id: AtomicU64,
/// Current live worker count (admission control / statistics).
active: AtomicU32,
/// Upper bound on concurrent workers; connections beyond this are refused
/// (backpressure). Runtime-configured, not a compile-time constant.
max_workers: usize,
/// Scheduling policy applied to spawned workers.
sched: KthreadSchedPolicy,
}
impl WorkerPool {
/// Create an empty pool named `name`, admitting at most `max_workers`
/// concurrent workers, each scheduled per `sched`.
pub fn new(name: &'static str, max_workers: usize, sched: KthreadSchedPolicy)
-> WorkerPool;
/// Spawn a new worker running `entry`, returning its assigned worker id.
/// `Err(WouldBlock)` if `active` has reached `max_workers`.
pub fn spawn<F>(&self, entry: F) -> Result<u64>
where
F: FnOnce() + Send + 'static;
/// Reap the worker with `id` after its connection closes: joins the
/// kthread and drops its handle. Idempotent for an unknown `id`.
pub fn reap(&self, id: u64);
}
/// Handle to a single dedicated kernel worker kthread — one long-lived,
/// nameable thread owned by a specific subsystem (e.g. the EDAC poller
/// `edac_poller/0`, or a deferred offline-worker), as opposed to the shared
/// `WorkQueue` pool that multiplexes `WorkItem`s. Owns the kthread: dropping the
/// handle requests stop and joins the thread, so there is no leak across
/// subsystem teardown.
///
/// A dedicated kworker sleeps until woken — by its own timer rearm (periodic
/// pollers) or by `wake()` (event-driven workers) — runs its body once, then
/// sleeps again. Use it when a subsystem needs a private thread with a stable
/// name and predictable scheduling rather than multiplexed deferred work.
pub struct KworkerHandle {
/// Underlying kthread task id. `u64` — never recycled within uptime.
tid: AtomicU64,
/// Wake flag: set by `wake()` or the worker's timer, cleared when the
/// worker begins an iteration.
pending: AtomicBool,
/// Wait queue the worker parks on between wakeups.
waiters: WaitQueue,
}
impl KworkerHandle {
/// Spawn a dedicated kworker named `name` running `body`, scheduled per
/// `sched`. The thread runs until the handle is dropped (stop + join).
pub fn spawn<F>(name: &'static str, sched: KthreadSchedPolicy, body: F)
-> Result<KworkerHandle, KernelError>
where
F: FnMut() + Send + 'static;
/// Wake the worker to run one iteration of its body.
pub fn wake(&self);
}
3.11.2 API¶
impl WorkQueue {
/// Create a named work queue.
///
/// - `name`: threads appear as `umkad-{name}-N` in process listings
/// - `max_threads`: number of concurrent worker threads (1 ≤ N ≤ WORKQUEUE_MAX_THREADS)
/// - `queue_depth`: maximum pending items (2 ≤ N ≤ 65535). The lower bound is
/// 2, not 1: the backing `DynMpmcRing` needs ≥ 2 slots — a single-slot ring
/// breaks the Lamport seq protocol (`BoundedMpmcRing::new`).
/// - `sched`: scheduling policy for all worker threads
/// - `cpu_mask`: CPU affinity mask; `CpuMask::all()` for no restriction
///
/// Returns `Err(KernelError::InvalidArgument)` when `queue_depth` is outside
/// `2..=65535` or `max_threads` is outside `1..=WORKQUEUE_MAX_THREADS`.
pub fn new(
name: &'static str,
max_threads: usize,
queue_depth: usize,
sched: KthreadSchedPolicy,
cpu_mask: CpuMask,
) -> Result<Arc<Self>, KernelError>;
/// Submit work for asynchronous execution. Returns immediately.
///
/// Returns `WouldBlock` if the queue is at capacity. The caller must
/// handle backpressure — retry after a delay, drop the work, or use
/// a per-subsystem overflow strategy. Silent queuing of unbounded work
/// is not permitted.
pub fn queue_work(&self, item: WorkItem) -> Result<WorkHandle, KernelError>;
// NOTE: WorkItem is pre-allocated from a per-CPU slab cache (Section 4.2
// slab allocator) sized to each workqueue's configured active-work bound per CPU.
// queue_work does not allocate: it takes a `WorkItem` by value (move
// semantics) that was previously obtained from the per-CPU slab. If the
// slab is exhausted, queue_work returns ENOMEM (backpressure). No heap
// allocation occurs under any lock.
/// Arm `work` to enqueue `work.item` on THIS queue after `delay` — the
/// targeted-queue form of `DelayedWork::schedule_delayed` (which targets
/// `SYSTEM_WQ`). Same `Duration` → ns conversion and same idempotence:
/// returns `Err(KernelError::ResourceBusy)` if `work` is already armed
/// (the earlier expiry is kept — EBUSY: the handle is in use). The
/// returned handle supports `cancel_delayed_work_sync`.
pub fn queue_delayed_work(&self, work: &DelayedWork, delay: Duration)
-> Result<WorkHandle, KernelError>;
/// Cancel a pending (not yet started) work item.
/// Returns `Ok(true)` if cancelled before execution.
/// Returns `Ok(false)` if the item was already running or completed.
/// Does NOT wait for a currently-running item to finish.
pub fn cancel_work(&self, handle: WorkHandle) -> Result<bool, KernelError>;
/// Cancel a pending or running work item and block until it completes.
///
/// - If the item has not yet started: dequeued, never executed. Returns `Ok(true)`.
/// - If the item is currently running: blocks until execution finishes.
/// Returns `Ok(false)` (item ran to completion, not cancelled).
/// - If the item has already completed: returns `Ok(false)` immediately.
///
/// MUST NOT be called from atomic context (sleeps while waiting for
/// the running item to complete). This is the safe pattern for driver
/// unload — ensures no work item references freed data after return:
///
/// ```rust
/// fn driver_unload(wq: &WorkQueue, handle: WorkHandle) {
/// wq.cancel_work_sync(handle).expect("cancel failed");
/// // Safe: work item has completed or was cancelled.
/// // Data referenced by the work item can now be freed.
/// }
/// ```
pub fn cancel_work_sync(&self, handle: WorkHandle) -> Result<bool, KernelError>;
/// Same as `cancel_work_sync` for delayed work items. If the timer has
/// not yet fired, the timer is cancelled and the work item is dequeued.
/// If the timer has fired and the work item is running, blocks until
/// the running item completes.
pub fn cancel_delayed_work_sync(&self, handle: WorkHandle) -> Result<bool, KernelError>;
/// Wait for all currently-queued items to complete. Items submitted
/// concurrently with or after `flush()` is called are not waited for.
/// May sleep; must not be called from atomic context.
pub fn flush(&self);
/// Cancel all pending items and wait for any currently-running items
/// to complete. After `drain()` returns, no items from before the call
/// are in-flight.
pub fn drain(&self);
}
impl OrderedWorkQueue {
pub fn new(name: &'static str, queue_depth: usize, sched: KthreadSchedPolicy,
cpu_mask: CpuMask) -> Result<Arc<Self>, KernelError>;
// Delegates to WorkQueue with max_threads = 1.
pub fn queue_work(&self, item: WorkItem) -> Result<WorkHandle, KernelError>;
pub fn flush(&self);
pub fn drain(&self);
}
3.11.2.1 with_timeout() — Bounded Wait, Detached Continuation¶
A kthread that must make forward progress past a possibly-blocking closure
(the exit-cleanup kthread executing ExitCleanupActions,
Section 8.2) cannot
run the closure inline — a vfs_unlinkat() stuck on a dead NFS server would
wedge it forever. with_timeout() bounds the WAIT, not the work:
/// Run `f` as a work item on the caller-designated named workqueue and
/// wait (killably) for its completion, up to `deadline`.
///
/// - Completed in time: Ok(R).
/// - Timeout / caller killed: Err(KernelError::Timeout). THE WORK ITEM IS NOT
/// CANCELLED — kernel code cannot be safely interrupted at an
/// arbitrary blocking point. The closure keeps its worker thread until
/// it eventually returns (the workqueue concurrency manager spawns a
/// replacement worker up to max_threads, so the queue keeps
/// processing), its side effects still apply on late completion, and
/// its result is discarded (the completion record is refcounted; the
/// orphaned worker drops the last reference).
///
/// Requirements: `f: FnOnce() -> R + Send + 'static` (it may outlive the
/// caller's frame — no borrowed stack state; capture owned Arcs only).
/// Caller must be sleepable task context. Bounded queue-full submission
/// error is reported as Err before any wait.
pub fn with_timeout<R: Send + 'static>(
deadline: Duration,
queue: &WorkQueue,
f: impl FnOnce() -> R + Send + 'static,
) -> Result<R, KernelError>;
Call sites name their queue explicitly (anonymous submission is forbidden,
below); the exit-cleanup path uses its own exit-clean named queue (2
threads, depth 256, SCHED_OTHER — cold path).
3.11.3 Standard Named Queues¶
The following named queues are created at boot by their respective subsystems. All subsystems that need deferred work must use one of these or create their own named queue — anonymous work submission is not permitted.
| Queue Name | Threads | Depth | Policy | Subsystem | Used by |
|---|---|---|---|---|---|
net-rx |
1/NIC | 4096 | SCHED_FIFO | Network Rx path | NIC drivers (GRO aggregation, TCP/UDP demux, netfilter deferred verdicts) |
blk-io |
4 | 8192 | SCHED_FIFO | Block I/O completion | Block drivers (I/O completion callbacks, request queue drain, SCSI/NVMe status processing) |
rcu-reclaim |
1/NUMA | 16384 | SCHED_OTHER | RCU callback processing | RCU subsystem (deferred free of RCU-protected objects, slab page release, routing table entry reclaim) |
pm-async |
2 | 256 | SCHED_OTHER | Device suspend/resume (§7.2.8) | Power management (async device suspend/resume, runtime PM state transitions, wakeup source bookkeeping) |
fw-loader |
2 | 64 | SCHED_OTHER | Firmware loading (§11.5.15) | Driver framework (firmware blob fetch from filesystem, microcode upload, FPGA bitstream loading) |
dma-fence |
2 | 1024 | SCHED_FIFO | DMA fence callbacks | GPU/DMA subsystem (fence signal callbacks, buffer release, inter-engine sync completion) |
crypto |
4 | 4096 | SCHED_OTHER | Async crypto operations | Crypto API (async AES-GCM/ChaCha completions, dm-crypt block encryption, TLS record processing) |
fsync |
8 | 16384 | SCHED_OTHER | Filesystem writeback | VFS/filesystems (dirty page writeback, journal commit, inode sync, periodic flush timer callbacks) |
events |
4 | 4096 | SCHED_OTHER | General subsystem events | General deferred work (sysfs notifications, kobject cleanup, uevent dispatch, deferred probe retry) |
events-long |
2 | 1024 | SCHED_IDLE | Background maintenance tasks | Long-running background work (memory compaction, slab cache shrink, periodic health checks, debug info collection) |
hotplug |
2 | 128 | SCHED_OTHER | Device hotplug event processing (Section 11.4) | Bus subsystem (PCI/USB/platform device add/remove, driver bind/unbind, resource rebalancing) |
mod-loader |
4 | 256 | SCHED_OTHER | Driver module loading with priority ordering (Section 11.4) | Module subsystem (driver module load/init, dependency resolution, symbol relocation, signature verification) |
Security-critical invariant: Security-critical operations (IMA measurement, capability validation, LSM hooks) are synchronous in the caller's context. They are never submitted to shared workqueues and are never subject to workqueue backpressure. This prevents a saturated workqueue from delaying or dropping security checks.
3.11.3.1 Tier 1 Crash Recovery¶
When a Tier 1 driver crashes (Section 11.9), the crash recovery sequence MUST drain or cancel all pending work items from workqueues registered by the crashed driver before releasing the driver's address space. The protocol:
- Mark driver as crashed: The crash handler sets the driver's state to
CRASHEDin the device registry (Section 11.4). - Cancel pending items: For each workqueue owned by (or shared with) the crashed
driver,
drain_or_cancel_driver_work(driver_id)iterates the queue'sBoundedMpmcRingand removes anyWorkItemwhoseowner_driver_idmatches the crashed driver. Removed items are dropped without execution. - Wait for in-flight items: If a work item from the crashed driver is currently executing in a worker thread, the worker detects the crash via the driver state flag and aborts execution at the next safe point (cancellation point). The crash handler waits for in-flight items to complete or abort before proceeding.
- Release address space: Only after all work items are drained/cancelled does the crash handler unmap the driver's code and data pages. This prevents use-after-unmap faults in worker threads.
For system-global workqueues (events, events-long, etc.), step 2 filters by
owner_driver_id. The queue itself is not destroyed — only the crashed driver's
items are removed. For driver-private workqueues (created by the driver at probe
time), the entire queue is drained and destroyed.
Thread names appear in /proc/N/comm and ps output as umkad-{name}-{index}.
The 1/NIC convention means one thread per network interface card is created
by the network subsystem at NIC probe time.
Workqueue creation phase assignments (keyed to boot phases in Section 2.3):
| Phase | Queues created | Rationale |
|---|---|---|
| Phase 2.7+ (workqueue init, early services) (see canonical boot table in Section 2.3) | rcu-reclaim, events, events-long |
Created by workqueue_init_early() during Phase 2.7. Needed by core subsystems before device enumeration. Dependency: rcu_init() (Phase 2.8) creates the rcu-reclaim workqueue. The workqueue subsystem (Phase 2.7) must be initialized first. rcu_init() constructs the rcu-reclaim queue with WorkQueue::new during its init sequence. |
| Phase 4.5 (block/storage init) | blk-io, fsync, crypto |
Created during block_init() (Phase 4.5 in the canonical boot table, Section 2.3). Block I/O completion and filesystem writeback available |
| Phase 4.4a (bus enumeration) | pm-async, fw-loader, dma-fence, hotplug, mod-loader |
Device probe begins; firmware loading and PM needed |
| Phase 5.3+ (NIC probe) | net-rx (1 per NIC) |
Created dynamically as each NIC driver probes |
3.11.4 BoundedMpmcRing Memory Ordering Specification¶
The BoundedMpmcRing algorithm (Producer guarded-claim on enq_head, Consumer guarded-claim
on deq_tail, Lamport-style sequence numbers) is correct only with precisely specified memory orderings.
Without explicit orderings, the implementation is a data race on all architectures with
weak memory models (AArch64, RISC-V, PPC). TSO (x86-64) hides these bugs in testing.
Slot structure (one entry per ring position):
#[repr(C, align(64))] // cache-line aligned: prevents false sharing between adjacent slots
struct Slot<T> {
/// Sequence number. Initially equals the slot index. Advances by N (the ring
/// capacity) after each producer/consumer cycle. A producer sees `seq == head`
/// when the slot is free; a consumer sees `seq == tail + 1` when it is filled.
///
/// **Type — width-selected `AtomicClaimPos`**
/// ([Section 3.1](#rust-ownership-for-lock-free-paths--guarded-position-claim)): `AtomicU64`
/// where the leg has native 64-bit atomics, native `AtomicU32` on PPC32. The
/// split is keyed on `target_has_atomic = "64"`, NOT pointer width — the two
/// DIVERGE on ARMv7-A, where `usize` is 32-bit yet the leg has 64-bit atomics
/// (LDREXD/STREXD; Rust `armv7a-none-eabi` `max_atomic_width: Some(64)`), so a
/// plain `AtomicUsize` would wrongly give ARMv7-A a 32-bit counter. PPC32
/// (`max-atomic-width = 32`, ESC-0406) is the ONLY leg on the native-32-bit
/// branch.
///
/// **Wrapping-RELATIVE classification (all legs)**: the protocol NEVER consumes
/// an ABSOLUTE `seq` value. Every decision in `push`/`pop` is a wrapping-relative
/// distance `dif = seq.wrapping_sub(pos) as SignedClaimPos` compared against 0
/// (`dif == 0` → claim; `dif < 0` → full/empty; `dif > 0` → stale, retry). The
/// distance between any live `seq` and the head/tail examining it is bounded by
/// the capacity N, a power of two that MUST be `< 2^31` (the 32-bit sign bit is
/// the tightest ceiling, on PPC32; realistic rings are 256-4096). `dif` is cast
/// to `SignedClaimPos` (same width as the counter), never `isize`, because
/// `isize` is 32-bit on ARMv7-A and would truncate a 64-bit difference.
///
/// **Wrap-safety — two mechanisms, one per branch**:
/// - **64-bit-atomic legs** (x86-64, AArch64, ARMv7-A, RISC-V64, PPC64LE,
/// s390x, LoongArch64): a full `2^64` wrap is UNREACHABLE BY WIDTH — at an
/// extreme sustained `10^7` claims/s, `2^64 / 10^7 ≈ 5.8x10^4` years. `push`/
/// `pop` claim with a plain CAS loop (`guarded_claim`'s 64-bit lowering); a
/// stale claim cannot alias because the counter cannot return to the same
/// value. Zero residual.
/// - **PPC32** (no 64-bit atomic to widen to): the position claim is a guarded
/// `lwarx`/validate/`stwcx.` sequence (`guarded_claim`'s ll/sc lowering). The
/// argument is STRUCTURAL, not probabilistic: `lwarx` anchors a reservation on
/// the head/tail word; all validation (`slot.seq` vs the freshly-reserved
/// position, full/empty checks) runs INSIDE the window on values loaded within
/// it — ordinary loads do NOT clear a Power ISA reservation (Book II; Book E /
/// e500 agree); `stwcx.` commits the increment ONLY IF the reservation still
/// holds. A full `2^32` wrap requires `2^32` stores to that word by other CPUs,
/// and a SINGLE store to the reservation granule by another processor
/// architecturally clears the reservation — so a claimant that stalled across
/// ANY other CPU's progress has its `stwcx.` fail by construction, and the
/// stale claim cannot succeed regardless of numeric aliasing. Zero residual, no
/// lock, no preempt-disable (a context switch clears the reservation too, which
/// only forces a benign re-anchor).
///
/// **Reversal note**: earlier iterations used `AtomicU64` (unimplementable on
/// PPC32) and then `AtomicUsize` with an accepted probabilistic full-wrap ABA
/// residual; per user ruling both are superseded by the width-selected
/// `AtomicClaimPos` + `guarded_claim` design, which is zero-residual by
/// construction on every leg.
seq: AtomicClaimPos,
data: UnsafeCell<MaybeUninit<T>>,
}
Enqueue (producer side):
// impl<T, const N: usize> BoundedMpmcRing<T, N> — `N` is the compile-time
// capacity (power of two). `DynMpmcRing::push` is identical with `self.mask`
// substituted for `N - 1`.
fn push(&self, item: T) -> Result<(), T> {
// Claim a producer slot with the arch-abstracted guarded position claim
// ([Section 3.1](#rust-ownership-for-lock-free-paths--guarded-position-claim)). The
// increment commits ONLY on an unbroken claim — an unwrapped 64-bit counter
// on legs with 64-bit atomics, an intact `lwarx`/`stwcx.` reservation on
// PPC32 — so a stale `enq_head` that survived a full counter wrap CANNOT
// forge a claim. No preemption guard is needed (a lost claim only re-anchors).
//
// The primitive — not this call site — loads `slot.seq` WITHIN the claim
// window: `seq_of` hands it the slot's `seq` ADDRESS for the freshly-anchored
// position `pos`, and `expected` gives the sequence value that marks the slot
// free (a producer slot is free when `seq == head == pos`). guarded_claim
// loads `seq` (Acquire, pairing with (D) Release in the consumer) and
// classifies `seq - expected(pos)` wrapping-RELATIVE (Vyukov bounded-MPMC
// discipline): `== 0` claim, `> 0` a rival producer already took it (re-anchor),
// `< 0` ring full. Absolute magnitude is never used, so the comparison is
// width-independent; the cast (inside the primitive) is same-width
// `SignedClaimPos`, never `isize`, and is correct while capacity N < 2^31.
// Passing the address rather than a pre-read value keeps the DECIDING load
// strictly after the anchor; for that to hold, `seq_of`/`expected` must stay
// pure (no captured pre-anchor loads) — the primitive states this caller
// obligation ([Section 3.1](#rust-ownership-for-lock-free-paths--guarded-position-claim)).
let head = match guarded_claim(
&self.enq_head.0,
|pos| &self.slots[(pos as usize) & (N - 1)].seq, // producer slot's seq
|pos| pos, // free when seq == head
) {
Ok(pos) => pos,
Err(ClaimUnavailable) => return Err(item), // ring full
};
// The claim is committed (`enq_head` advanced past `head`); we own the slot.
let slot = &self.slots[(head as usize) & (N - 1)];
// (3) Write data. The Release in (4) makes this visible.
unsafe { slot.data.get().write(MaybeUninit::new(item)); }
// (4) Publish: seq = head + 1. Release pairs with the consumer's `slot.seq`
// Acquire load inside `guarded_claim` (pop).
slot.seq.store(head.wrapping_add(1), Release);
Ok(())
}
Dequeue (consumer side):
fn pop(&self) -> Option<T> {
// Claim a consumer slot via the guarded position claim (see `push`). The
// claim commits only when a producer has FILLED the slot (`seq == tail + 1`),
// so an empty ring returns `None` — `deq_tail` is never advanced onto a slot
// that may never fill. `expected` is `pos + 1` (the "filled" sequence value);
// guarded_claim loads the slot's `seq` inside the window (Acquire, pairing
// with (4) Release) and classifies `seq - (pos + 1)` wrapping-RELATIVE:
// `== 0` claim, `< 0` ring empty, `> 0` slot already consumed (re-anchor).
let tail = match guarded_claim(
&self.deq_tail.0,
|pos| &self.slots[(pos as usize) & (N - 1)].seq, // consumer slot's seq
|pos| pos.wrapping_add(1), // filled when seq == tail + 1
) {
Ok(pos) => pos,
Err(ClaimUnavailable) => return None, // ring empty
};
// The claim is committed (`deq_tail` advanced past `tail`); we own the slot.
let slot = &self.slots[(tail as usize) & (N - 1)];
// (C) Read data. Safe: the Acquire load inside `guarded_claim` established
// happens-before with the producer's write in (3).
let item = unsafe { slot.data.get().read().assume_init() };
// (D) Recycle slot: seq = tail + N. Release ensures (C) completes before the
// next producer overwrites the slot.
slot.seq.store(tail.wrapping_add(N as ClaimPos), Release);
Some(item)
}
Ordering summary:
| Operation | Ordering | Pairs with | Purpose |
|---|---|---|---|
guarded_claim(&enq_head, …) position word |
Relaxed | — | Claim a producer slot. Relaxed suffices: the guarded claim (plain CAS on 64-bit-atomic legs, lwarx/stwcx. on PPC32) provides claim mutual exclusion, while the per-slot seq field (Acquire/Release) provides ALL data-visibility guarantees. |
slot.seq.load inside guarded_claim (producer) |
Acquire | (D) Release | See recycled slot before overwriting |
slot.seq.store(head + 1) |
Release | consumer slot.seq load (Acquire) |
Make data write visible before publishing |
slot.seq.load inside guarded_claim (consumer) |
Acquire | (4) Release | See data write before reading slot |
guarded_claim(&deq_tail, …) position word |
Relaxed | — | Claim a consumer slot atomically (MPMC-safe) |
slot.seq.store(tail + N) |
Release | producer slot.seq load (Acquire) |
Recycle slot after read completes |
Per-architecture compile result:
- x86-64 (TSO): load(Acquire) = plain load; store(Release) = plain store. TSO
makes these no-ops, but the orderings must still be written correctly — they are hints
to the compiler's reordering optimizer regardless of hardware fences.
- AArch64 (weak): load(Acquire) compiles to ldar; store(Release) to stlr.
These instructions are required for correctness. Without them, the CPU can reorder
loads/stores past the sequence number checks.
- RISC-V (RVWMO): load(Acquire) → lw + fence r,rw; store(Release) → fence rw,w + sw.
- PPC64 (power model): load(Acquire) → ld + cmpw + bc + isync;
store(Release) → lwsync + std.
The position-word claim (guarded_claim,
Section 3.1) lowers separately from
the seq handshake above: on the seven legs with 64-bit atomics it is a
compare_exchange_weak retry loop on the 64-bit counter (a full wrap is unreachable by
width); on PPC32 it is a lwarx/validate/stwcx. sequence whose commit is void if any
other CPU stored to the reservation granule (structural wrap-safety, no lock). Both
branches drive the counter Relaxed — the counter is claim arbitration only; all
data-visibility ordering rides the seq Acquire/Release handshake above.
Generic rate limiter: TokenBucket (defined in
Section 3.6) is the
standard &self rate-limiting primitive: lock-free on 64-bit-atomic legs
(coupled-CAS refill) and per-bucket SpinLock on legs without a native 64-bit
atomic (PPC32) — a cfg-split whose public &self API is identical on every leg.
Subsystems requiring rate limiting from a shared reference (netconsole transmit
throttling, TTY signal injection) should use this type rather than reimplementing
token-bucket logic. Two deliberately different variants exist and are NOT interchangeable:
the audit path's SpinLock-wrapped AuditTokenBucket
(Section 20.2) and DSM's single-owner, &mut self
AntiEntropyRateLimiter (Section 6.13), which uses plain
u64 fields (no atomics) because a genuine single-sender invariant holds.
3.11.5 Cgroup Integration¶
Workqueue threads are subject to cgroup CPU bandwidth accounting
(Section 7.6). When a workqueue is created on behalf of a
specific cgroup (e.g., a cgroup-scoped memory reclaim kthread), the worker
threads are placed into that cgroup's cpu controller hierarchy. This ensures
that deferred work charged to a container consumes the container's CPU budget,
not the root cgroup's.
Accounting rules:
- Work items submitted by a task inherit that task's cgroup context. The worker
thread temporarily associates with the submitter's cgroup for the duration of
the work item execution (cgroup_work_enter(submitter_css) /
cgroup_work_exit()). CPU time consumed is charged to the submitter's
cpu.stat.
- System-global workqueues (e.g., umkad-rcu-gp, umkad-mm-compact) are
placed in the root cgroup and are not subject to per-container bandwidth limits.
- Per-cgroup workqueue thread counts are visible via
/sys/fs/cgroup/<path>/cpu.workqueue_threads.
3.12 IRQ Chip and irqdomain Hierarchy¶
Hardware interrupt controllers form a cascade hierarchy: a root controller (APIC on x86-64, GIC on AArch64/ARMv7, PLIC on RISC-V, OpenPIC/XIVE on PPC, EIOINTC on LoongArch64) or architectural interrupt mechanism (PSW-swap on s390x) may cascade into secondary controllers (GPIO expanders, PCIe MSI controllers, I2C interrupt expanders). The irqdomain abstraction maps hardware interrupt numbers to UmkaOS internal software IRQ numbers and routes them through the correct chip-level handlers.
UmkaOS improvement over Linux: Linux uses generic irq_domain_ops function tables
with void-pointer driver data, providing no type safety. UmkaOS uses typed Rust traits
(IrqChip, IrqDomain) with concrete implementations per controller type. The cascade
hierarchy is explicit at compile time, eliminating class-of-bug where a wrong ops table
is installed.
3.12.1 Core Types¶
/// Hardware interrupt number — chip-local, not globally unique.
pub type HwIrq = u32;
/// UmkaOS global software IRQ number — globally unique, allocated sequentially.
///
/// **Longevity analysis (50-year counter policy)**: SwIrq is allocated
/// monotonically via `IrqTable::next_free: AtomicU32` during device probe.
/// IRQ numbers are recycled via `IrqDomain::free()` on hot-unplug; the
/// allocator scans for free slots starting from `next_free`. At 1000
/// hot-plug cycles/second (extreme server scenario), monotonic exhaustion
/// of the u32 space would occur in ~49.7 days. In practice:
/// - Hot-plug rates are typically <10/second (years to exhaustion).
/// - IRQ numbers ARE recycled (free returns slots to the pool).
/// - With recycling, the counter wraps safely (wrapping arithmetic
/// finds free slots by scanning from the wrapped value).
/// - Total IRQ count per system is typically <10K (well within u32).
/// If a future system requires >4B lifetime IRQ allocations without
/// recycling, widen to u64 and update `IrqTable::next_free` accordingly.
pub type SwIrq = u32;
/// Software IRQs 0-31 are reserved for architecture-specific use (NMI, MCE, etc.).
/// Dynamically allocated software IRQs start at this value.
pub const IRQ_FIRST_DYNAMIC: SwIrq = 32;
/// Sentinel value: no IRQ assigned.
pub const IRQ_INVALID: SwIrq = u32::MAX;
/// IRQ trigger type, set per-IRQ at probe time from device tree or ACPI _CRS.
/// Matches Linux `include/linux/irq.h` `IRQ_TYPE_*` values for DT compatibility.
#[repr(u8)]
pub enum IrqTrigger {
/// Platform default / driver doesn't care (Linux `IRQ_TYPE_NONE`).
/// The interrupt controller uses its hardware default or device-tree
/// specified trigger type. Many DT-probed drivers specify `IRQ_TYPE_NONE`
/// and rely on the platform to configure the correct trigger.
PlatformDefault = 0,
RisingEdge = 1,
FallingEdge = 2,
BothEdges = 3,
LevelHigh = 4,
LevelLow = 5,
}
/// Return value from an interrupt handler.
#[repr(u8)]
pub enum IrqReturn {
/// IRQ was serviced; EOI will be sent.
Handled = 0,
/// IRQ was not for this handler (spurious); do not send EOI.
NotHandled = 1,
/// Primary handler requests the threaded handler be woken (irq/N kthread).
WakeThread = 2,
}
/// Chip-specific per-IRQ data stored inline in IrqDescriptor to avoid
/// pointer chasing on the hot interrupt path.
pub struct IrqChipData {
pub hw_irq: HwIrq,
/// MMIO base address of the interrupt controller (architecture-specific).
pub regs: *mut u8,
/// Bit mask for this IRQ within its controller register (e.g., 1 << pin_index).
pub mask: u32,
}
/// Per-IRQ state descriptor. One instance per allocated software IRQ number.
pub struct IrqDescriptor {
pub sw_irq: SwIrq,
pub hw_irq: HwIrq,
pub trigger: IrqTrigger,
pub chip: &'static dyn IrqChip,
pub chip_data: IrqChipData,
/// IrqDomains are registered at boot/module-load and never freed during
/// normal operation, so a static reference suffices and avoids atomic
/// refcount overhead on every interrupt.
pub domain: &'static dyn IrqDomain,
/// Primary handler (always in atomic/IRQ context, must not sleep).
/// Returns `WakeThread` to schedule the threaded handler (if registered).
/// Returns `Handled` if the interrupt was fully serviced in hardirq context.
///
/// SAFETY: `handler` and `handler_data` are registered together by `request_irq()`.
/// Type erasure via `*mut ()` is safe because the IRQ subsystem never calls a
/// handler with data from a different registration. The handler pointer is
/// immutable after registration (updates require `free_irq()` + `request_irq()`).
pub handler: Option<fn(SwIrq, *mut ()) -> IrqReturn>,
/// Threaded handler (runs in kthread context, may sleep and allocate).
/// Scheduled when the primary handler returns `WakeThread`. The IRQ line
/// remains masked at the chip until the threaded handler completes.
/// `None` for pure hardirq-only handlers. Same safety invariant as `handler`.
pub threaded_handler: Option<fn(SwIrq, *mut ()) -> IrqReturn>,
pub handler_data: *mut (),
/// Debug-only type tag for handler_data. Stores the TypeId of the concrete
/// type passed to `request_irq()`, enabling runtime detection of mismatched
/// data types during IRQ re-registration (e.g., driver reload with a
/// different handler data type). Zero-cost in release builds.
#[cfg(debug_assertions)]
pub handler_data_type_id: Option<core::any::TypeId>,
/// Action flags controlling handler behavior.
pub flags: IrqActionFlags,
/// Serializes enable/disable/set_type operations on this descriptor.
action_lock: SpinLock<()>,
/// Nesting depth counter: 0 = enabled, N > 0 = disabled N times.
depth: AtomicU32,
}
bitflags::bitflags! {
/// Flags for IRQ handler registration (mirrors Linux IRQF_* flags).
pub struct IrqActionFlags: u32 {
/// Handler runs in a dedicated kthread (threaded IRQ).
const THREADED = 1 << 0;
/// IRQ may be shared between multiple devices.
const SHARED = 1 << 1;
/// Request a one-shot handler: IRQ stays masked until threaded handler completes.
const ONESHOT = 1 << 2;
/// Do not disable this IRQ during suspend.
const NO_SUSPEND = 1 << 3;
}
}
/// Interrupt chip operations. All methods are called with preemption disabled.
/// The chip is responsible for managing the physical hardware registers.
pub trait IrqChip: Send + Sync {
/// Acknowledge (clear) a pending edge-triggered interrupt at the chip.
/// Called immediately after the interrupt is claimed.
fn ack(&self, data: &IrqChipData);
/// Mask (disable) this IRQ line at the chip. Prevents further interrupt delivery.
fn mask(&self, data: &IrqChipData);
/// Unmask (enable) this IRQ line at the chip.
fn unmask(&self, data: &IrqChipData);
/// Configure trigger type. Returns `Err(KernelError::NotSupported)` if the
/// hardware does not support the requested trigger type for this line.
fn set_type(&self, data: &IrqChipData, trigger: IrqTrigger)
-> Result<(), KernelError>;
/// Set SMP affinity: route this IRQ to the specified CPUs.
/// Returns `Ok(())` without effect if the chip does not support affinity
/// (e.g., legacy 8259A PIC with fixed routing).
fn set_affinity(&self, data: &IrqChipData, mask: &CpuMask)
-> Result<(), KernelError>;
/// Send End-Of-Interrupt to allow the next interrupt from this line.
/// Default implementation: calls `unmask`, which is correct for edge-triggered
/// controllers. Level-triggered controllers must override this to send a
/// chip-specific EOI command before unmasking.
fn eoi(&self, data: &IrqChipData) {
self.unmask(data);
}
}
/// Hardware IRQ specification as encoded in a device tree `interrupts` property
/// or ACPI `_CRS` extended interrupt descriptor. The interpretation is
/// domain-specific (each IrqDomain implementation defines the encoding).
///
/// For GIC (AArch64/ARMv7): cells[0] = IRQ type (SPI=0, PPI=1), cells[1] = INTID,
/// cells[2] = trigger flags.
/// For APIC (x86-64): cells[0] = vector number.
/// For PLIC (RISC-V): cells[0] = source number, cells[1] = trigger flags.
/// For s390x: cells[0] = ISC (0-7), cells[1] = subchannel ID (SCHID).
/// For EIOINTC (LoongArch64): cells[0] = vector number (0-255), cells[1] = trigger flags.
///
/// The `cells` array is a uniform encoding regardless of whether the source
/// is DTB (`interrupts` property) or ACPI (`_CRS` extended interrupt descriptor).
/// The IrqDomain implementation for each controller knows its source format
/// and interprets the cells accordingly — no discriminant field is needed
/// because a system uses either DTB or ACPI, never both simultaneously for
/// the same controller.
pub struct IrqSpec {
pub cells: [u32; 3],
}
/// An irqdomain: maps hardware IRQ numbers to software IRQs for one controller.
/// Domains form a tree; `parent()` returns the upstream domain for cascades.
pub trait IrqDomain: Send + Sync {
/// Translate a hardware IRQ specification (from DT `interrupts` or ACPI _CRS)
/// into a hardware IRQ number for this domain.
fn translate(&self, spec: &IrqSpec) -> Result<HwIrq, KernelError>;
/// Allocate software IRQ(s) and create descriptors for a range of hardware IRQs.
/// Called during device probe. Returns the first allocated SwIrq.
/// `count` is typically 1 for regular IRQs, N for MSI-X vectors.
fn alloc(
&self,
hw_irq: HwIrq,
count: u32,
chip_data: IrqChipData,
) -> Result<SwIrq, KernelError>;
/// Free a previously allocated IRQ range, releasing software IRQ numbers.
fn free(&self, sw_irq: SwIrq, count: u32);
/// Activate: perform final hardware-side setup (e.g., program MSI address/data
/// registers in PCIe config space). Called after alloc, before unmasking.
fn activate(&self, desc: &mut IrqDescriptor) -> Result<(), KernelError>;
/// Deactivate: reverse of activate (e.g., mask MSI in PCIe config space).
/// Called before free.
fn deactivate(&self, desc: &IrqDescriptor);
/// Parent domain in the cascade hierarchy. `None` for root domains.
/// Returns `&'static` because IrqDomains are registered at boot/module-load
/// and never freed during normal operation — consistent with the `IrqChip`
/// pattern (`&'static dyn IrqChip`). Avoids atomic refcount overhead (Arc
/// clone + drop) on the IRQ dispatch hot path during hierarchical cascading.
fn parent(&self) -> Option<&'static dyn IrqDomain>;
}
/// Global mapping from software IRQ number to IrqDescriptor. O(1) lookup by SwIrq.
///
/// The table is sized at boot based on the total IRQ count reported by all root domains.
/// No runtime resizing occurs after initialization.
pub struct IrqTable {
/// Indexed by SwIrq. `None` = not allocated.
///
/// **Collection policy exemption**: Uses a flat `Box<[Option<&'static ...>]>` array
/// instead of XArray because the key space is dense, spanning zero through the
/// boot-discovered IRQ count; it is sized once
/// at boot, and never resized. A flat array gives true O(1) indexed access
/// without XArray's radix-tree indirection overhead on the IRQ fast path.
///
/// **`&'static` rationale**: IrqDescriptors are allocated at probe time from a
/// dedicated slab and never freed during normal operation (same lifetime as
/// IrqChip and IrqDomain). The IRQ receipt flow (step 4) indexes into this
/// array on every interrupt — using `&'static` avoids atomic refcount overhead
/// (Arc clone + drop: two atomic RMW ops per interrupt). Registration and
/// deregistration paths (`request_irq()` / `free_irq()`) use the IrqTable's
/// `lock` for serialization; no Arc is needed for concurrent access safety
/// because the flow handler only borrows the descriptor for the interrupt
/// duration (preemption disabled, single-CPU access).
descriptors: Box<[Option<&'static IrqDescriptor>]>,
next_free: AtomicU32,
/// Serializes `alloc` and `free` operations only. Does NOT protect lookups.
lock: SpinLock<()>,
}
pub static IRQ_TABLE: BootOnceCell<IrqTable> = BootOnceCell::new();
3.12.2 Root Domain Implementations Per Architecture¶
Each architecture instantiates a root IrqDomain at boot. Secondary domains
(MSI, GPIO) are created by their subsystems and parent to the root domain.
| Architecture | Root Controller | HW IRQ Range | Notes |
|---|---|---|---|
| x86-64 | Local APIC + IOAPIC | vectors 0-255 | Legacy PIC (8259A) disabled at boot |
| x86-64 | PCI MSI | dynamically allocated | MsiIrqDomain, parents to APIC |
| AArch64 | GIC-v3 | SPIs 32-1019 | PPIs (16-31) per-CPU; SGIs (0-15) for IPI |
| ARMv7 | GIC-v2 | SPIs 32-1019 | Same numbering as GIC-v3 |
| RISC-V | PLIC or APLIC | sources 1-1023 | PLIC: legacy; APLIC: modern (wired + MSI modes). See APLIC note below |
| PPC32 | OpenPIC | IRQs 0-511 | External + internal sources |
| PPC64LE | XICS / XIVE | IRQs 0-65535 | XICS: pseries default + POWER8 bare-metal (via hcalls or OPAL); XIVE: POWER9+ bare-metal or pseries with CAS negotiation. Event queue per CPU on XIVE; ICP/ICS model on XICS |
| s390x | PSW-swap (architectural) | ISC 0-7 × subchannels | No external controller; lowcore PSW pairs per interrupt class; I/O floats via ISC masks in CR6 |
| LoongArch64 | EIOINTC | vectors 0-255 | Per-CPU routing via IOCSR registers; LIOINTC cascades for legacy devices |
All platforms also support:
- GpioIrqDomain: GPIO pins as IRQ sources; parents to the platform root domain.
Created by the GPIO controller driver at probe time.
- MsiIrqDomain: PCI MSI and MSI-X vectors; parents to the platform root domain.
Created by the PCIe port driver at enumeration.
3.12.3 IRQ Receipt Flow¶
The full path from hardware exception to handler completion:
1. CPU receives hardware interrupt → architecture-specific entry stub
(x86-64: IDT vector handler; AArch64: VBAR_EL1 IRQ vector;
ARMv7: IRQ vector table; RISC-V: stvec trap handler;
PPC32: external interrupt vector; PPC64LE: XICS/XIVE interrupt;
s390x: PSW swap loads new PSW from lowcore; LoongArch64: EIOINTC vector)
2. arch::current::interrupts::handle_irq() is called from the entry stub
with preemption disabled and interrupts masked.
3. Claim interrupt from controller:
- x86-64: vector is encoded in the IDT entry; no separate claim needed
- AArch64: read GIC_IAR1 to claim and get INTID
- RISC-V: read PLIC claim/complete register to claim
- PPC32: read OpenPIC IACK register
- PPC64LE: XIVE: pushes INTID to per-CPU event queue;
XICS: read XIRR via H_XIRR hcall (pseries) or ICP MMIO (powernv)
- s390x: implicit (PSW swap is the claim); read SCHID from lowcore I/O
interruption code area, translate (SCHID, ISC) → SwIrq
- LoongArch64: read EIOINTC pending status register to get vector number
4. Look up IrqDescriptor: IRQ_TABLE[sw_irq].
5. desc.chip.ack(&desc.chip_data)
Edge-triggered: clears pending interrupt at the controller.
Level-triggered: no-op here; level is cleared by the device itself.
6. Call desc.handler(sw_irq, desc.handler_data):
→ Handled: proceed to EOI (step 7)
→ NotHandled: spurious interrupt; emit a debug-level log, skip EOI for this handler
→ WakeThread: wake irq/{sw_irq} kthread (SCHED_FIFO, priority 50), proceed to EOI
7. desc.chip.eoi(&desc.chip_data)
Sends End-Of-Interrupt; allows next interrupt from this line.
8. Return from exception → scheduler preemption check (if preempt_count == 0).
Threaded IRQ flow (when handler returns WakeThread):
The irq/{sw_irq} kthread (created at IRQ registration time) blocks on a
WaitQueueHead. On WakeThread, the scheduler wakes the kthread, which:
1. Runs the threaded handler function in process context (may sleep, may allocate)
2. Calls desc.chip.unmask(&desc.chip_data) after the threaded handler returns
3. Returns to blocking on the WaitQueueHead
The primary handler is stored in IrqDescriptor.handler; the threaded handler in
IrqDescriptor.threaded_handler. The IrqActionFlags::THREADED flag indicates that
a threaded handler is registered. When both are present, the primary handler is called
in hardirq context and returns WakeThread; the kthread then invokes the threaded
handler, which runs the actual device service routine in process context.
Per-registration handler record (IrqAction): the inlined handler /
threaded_handler fields above cover the common single-owner line. A line
requested with IrqActionFlags::SHARED may have several device drivers
registered on it; each request_irq() produces one IrqAction record, and the
descriptor owns a chain of them, walked in registration order until one returns
IrqReturn::Handled. This mirrors Linux's struct irqaction list attached to its
interrupt descriptor. The threaded-IRQ kthread (IrqThread,
Section 8.5) holds an Arc<IrqAction> naming the exact
registration it services, so each shared-line handler gets its own thread and
its own handler_data cookie.
/// One interrupt-handler registration — the record produced by a single
/// `request_irq()` call. For a non-shared line the same handlers are also
/// inlined in `IrqDescriptor` (the hardirq fast path reads them without
/// chasing a pointer); for a shared line the descriptor owns a chain of these
/// records, one per device sharing the line.
pub struct IrqAction {
/// Primary (hardirq-context) handler. Same signature and safety invariant
/// as `IrqDescriptor.handler`. `None` for a pure threaded registration
/// whose hardirq half is the default "mask + wake the thread" stub.
pub handler: Option<fn(SwIrq, *mut ()) -> IrqReturn>,
/// Threaded (kthread-context) handler, run when the primary returns
/// `IrqReturn::WakeThread`. `None` for a hardirq-only registration.
pub threaded_handler: Option<fn(SwIrq, *mut ()) -> IrqReturn>,
/// Opaque per-registration cookie (the `dev_id` argument of
/// `request_irq()`), passed to both handlers so a shared-line handler can
/// recover its device. Type erasure via `*mut ()` is sound because the IRQ
/// core only ever pairs this cookie with the handlers registered with it.
pub handler_data: *mut (),
/// Registration flags (`SHARED`, `THREADED`, `ONESHOT`, `NO_SUSPEND`).
pub flags: IrqActionFlags,
/// Owner name (from `request_irq`), shown in `/proc/interrupts` and used to
/// name the `irq/{n}/{name}` kthread.
pub name: Arc<str>,
/// Next registration sharing the same line; `None` = end of chain.
/// Populated only when `flags` contains `IrqActionFlags::SHARED`.
pub next: Option<Arc<IrqAction>>,
}
RISC-V APLIC MSI mode: level-sensitive re-assertion (errata RiscvErrata::APLIC_LEVEL_MSI):
RISC-V APLIC (Advanced Platform-Level Interrupt Controller) operates in two modes: - Direct mode: wired interrupt delivery, similar to PLIC. Level-sensitive sources remain pending while asserted — no special handling needed. - MSI mode: interrupts are delivered as MSI writes to an IMSIC (Incoming MSI Controller). This is the preferred mode for scalable multi-hart systems.
In MSI mode, level-sensitive interrupts have a fundamental semantic mismatch: an MSI is an edge event (a single write), but a level-sensitive source stays asserted until the device deasserts. If the ISR services the device (clearing the level) but the device re-asserts before the ISR completes the EOI sequence, the re-assertion is lost — the APLIC already delivered the MSI and will not send another until the level drops and rises again. This causes permanent interrupt loss for the affected source.
Mandatory workaround: After the ISR clears the device's interrupt condition, the
EOI handler must re-read the APLIC source's ip[n] (interrupt pending) bit. If the
source has re-asserted between device clear and EOI, the APLIC IrqChip::eoi()
implementation must re-trigger the MSI manually by writing the APLIC setipnum
register:
EOI sequence for APLIC MSI mode (level-sensitive sources):
1. ISR services device → device deasserts interrupt line
2. Read APLIC sourcecfg[n].sm to confirm level-sensitive
3. Read APLIC ip[n] bit → if set, source has re-asserted
4. If re-asserted: write n to APLIC setipnum → triggers new MSI to IMSIC
5. Complete EOI
This re-assertion check is mandatory for ALL level-sensitive sources when APLIC
operates in MSI mode. Without it, any level-sensitive device (GPIO edge-to-level
converters, I2C interrupt expanders, legacy PCI INTx) can permanently lose interrupts.
The errata flag RiscvErrata::APLIC_LEVEL_MSI gates this workaround (set on all
APLIC implementations that support MSI mode, since this is a specification-level
issue, not a silicon bug).
3.13 Collection Usage Policy¶
Kernel code must choose collection types based on access path criticality:
| Path Class | Examples | Allowed Collections | Heap Allocation |
|---|---|---|---|
| Hot (per-syscall, per-packet, scheduler tick, IRQ) | runqueue, routing lookup, page fault handler | ArrayVec, static arrays, slab objects, per-CPU pools, XArray/radix tree (integer-keyed) |
Forbidden (XArray uses pre-allocated nodes from slab) |
| Warm (per-operation, bounded frequency) | driver init, mount, cgroup create, device probe | XArray (integer-keyed), BTreeMap (non-integer ordered keys), ArrayVec, bounded Vec, Idr |
Bounded (max N known at design time) |
| Cold (boot, config load, debug, admin) | ACPI parsing, module load, sysfs population | HashMap, Vec, BTreeMap, String |
Acceptable |
| RCU read-heavy | routing table, capability cache, module registry, page cache | RcuHashMap, RcuIdr, RcuList, XArray (RCU-protected) |
Writers allocate; readers lock-free |
Radix tree / XArray vs BTreeMap selection:
| Criterion | XArray / Radix Tree | BTreeMap |
|---|---|---|
| Key type | Integer only (u64, page index, PID) |
Any Ord type |
| Lookup complexity | O(1) — fixed depth (10 levels for 64-bit, 6 bits/level) | O(log N) |
| RCU-compatible reads | Yes — slot-level RCU, lock-free reads | No — requires external RCU wrapper |
| Cache behavior | Excellent — 64-way fanout, dense subtrees collapse | Good — B-tree nodes are cache-line-friendly |
| Sparse keys | Efficient — empty subtrees not allocated | Efficient — only present keys stored |
| Ordered iteration | Yes (by integer key) | Yes (by Ord key) |
| Hot-path suitability | Yes — all paths | Non-integer keys or range queries only |
Rule: For all integer-keyed mappings — hot, warm, or cold — use XArray or
Idr (which is built on XArray). There is no performance reason to use BTreeMap or
HashMap with integer keys on any path: XArray is O(1) with better cache behavior,
native RCU read support, and ordered iteration. BTreeMap is reserved for:
- Non-integer keys (String, [u8; N], enum types, composite structs)
- Composite keys where ordered iteration by the composite is needed
(e.g., (deadline, task_id) for deadline trees)
- Range queries that require BTreeMap::range() (e.g., IOVA containment lookup,
I/O elevator seek-distance merge)
HashMap is reserved for non-integer keys on cold paths or RCU-protected writer paths.
For integer keys, HashMap is never acceptable — use XArray.
Rules: 1. Hot-path structs must have O(1) or O(log N) access with bounded N. 2. No heap allocation under spinlock or with IRQs disabled. 3. Integer key → XArray. Always. No exceptions for "warm" or "cold" paths. 4. HashMap is only acceptable in cold paths with non-integer keys, or RCU writer paths. 5. Vec is acceptable when maximum size is known and documented. 6. BTreeMap is for non-integer ordered keys or integer keys requiring range queries.
Documented exemptions:
- BTreeMap<u64, IommuMapping> in IOVA management — requires range(..=addr) for
containment lookup. Hardware IOMMU page table handles the hot-path DMA translation.
- BTreeMap<Lba, IoRequest> in I/O elevator — requires ordered iteration and range
merge queries for seek-distance minimisation on rotational media.
3.13.1 Bounded LRU Cache¶
LruCache<K, V> is the canonical fixed-capacity least-recently-used cache. It is a
bounded structure: at most capacity entries are ever live, and inserting into a
full cache evicts the least-recently-used entry. Because the entry count is capped at
construction, its internal key index never grows without bound — the integer-key →
XArray rule (which targets unbounded primary mappings) does not apply to the
capped index. Keys need only Hash + Eq (they are not required to be Ord, so
composite non-ordered keys such as the nfsd DRC key are supported). Recency order is
kept in an intrusive doubly-linked list over a pre-allocated slot slab, so get,
put, and eviction are all O(1) amortized with no per-operation allocation once the
capacity slots are populated.
Consumers: the nfsd duplicate-reply cache (Section 15.12) and the verified-boot measurement cache (Section 9.3).
/// Fixed-capacity least-recently-used cache. Evicts the least-recently-used
/// entry when a `put` would exceed `capacity`. Keys are hashed (`K: Hash + Eq`);
/// recency is tracked by an intrusive index list over a pre-allocated slot slab.
pub struct LruCache<K: Hash + Eq, V> {
/// Maximum number of live entries; a `put` into a full cache evicts the LRU.
capacity: usize,
/// Slot slab: each slot holds one entry plus its recency links. Allocated
/// once at construction (`capacity` slots); reused via the free list.
slots: Vec<LruSlot<K, V>>,
/// Key → slot index. Bounded by `capacity` (never an unbounded map).
index: HashMap<K, usize>,
/// Most-recently-used slot index, or `usize::MAX` when empty.
head: usize,
/// Least-recently-used slot index, or `usize::MAX` when empty.
tail: usize,
/// Free-slot list head, or `usize::MAX` when every slot is occupied.
free: usize,
}
/// One entry in an `LruCache`'s intrusive recency list.
struct LruSlot<K, V> {
key: K,
value: V,
/// More-recently-used neighbour (toward `head`), or `usize::MAX`.
prev: usize,
/// Less-recently-used neighbour (toward `tail`), or `usize::MAX`.
next: usize,
}
impl<K: Hash + Eq + Clone, V> LruCache<K, V> {
/// Create an empty cache holding at most `capacity` entries. Allocates the
/// slot slab and index once, up front.
pub fn with_capacity(capacity: usize) -> Self;
/// Look up `key`, promoting it to most-recently-used on hit.
pub fn get(&mut self, key: &K) -> Option<&V>;
/// Insert or overwrite `key`, promoting it to most-recently-used and
/// evicting the least-recently-used entry if the cache was full. Returns
/// the evicted `(key, value)`, if one was displaced.
pub fn put(&mut self, key: K, value: V) -> Option<(K, V)>;
/// Remove `key`, returning its value if present.
pub fn remove(&mut self, key: &K) -> Option<V>;
/// Number of live entries.
pub fn len(&self) -> usize;
}
3.13.2 Bounded Associative Array¶
ArrayMap<K, V, N> is the canonical fixed-capacity associative map for small
tables that must be const-constructible (usable in static initializers) and
allocation-free. It is backed by an inline ArrayVec<(K, V), N>: at most N
entries are ever live, and lookup/insert/remove are a linear scan — O(N), which
is optimal for the tiny tables it targets (a hashed map would cost more in
constant factors and forbid const fn new()). Keys need only K: Eq; the
Borrow<Q> query bound lets callers look up by a borrowed form (e.g. an
ArrayString<16> key queried by &str), mirroring HashMap.
Because the entry count is capped at N, its key set never grows without
bound, so the integer-key → XArray rule (which targets unbounded mappings)
does not apply. Distinct from LruCache: ArrayMap never evicts — a full-map
insert of a new key is a caller error (callers that may fill the map check
len() < N first).
Consumer: the TCP congestion-control name → slot-id table (Section 16.10).
/// Fixed-capacity associative map backed by an inline array — no heap, no
/// hashing. Holds at most `N` `(K, V)` entries; lookup/insert/remove are a
/// linear scan (O(N)). `const fn new()` yields an empty map usable in `static`
/// initializers.
pub struct ArrayMap<K: Eq, V, const N: usize> {
/// Live entries, packed in the occupied prefix of the inline array.
entries: ArrayVec<(K, V), N>,
}
impl<K: Eq, V, const N: usize> ArrayMap<K, V, N> {
/// Create an empty map. `const` so it can initialize `static` tables.
pub const fn new() -> Self;
/// Look up `key` (by a borrowed query form), returning a reference to its
/// value if present.
pub fn get<Q>(&self, key: &Q) -> Option<&V>
where
K: Borrow<Q>,
Q: Eq + ?Sized;
/// Whether `key` is present.
pub fn contains_key<Q>(&self, key: &Q) -> bool
where
K: Borrow<Q>,
Q: Eq + ?Sized;
/// Insert or overwrite `key`. Returns the previous value if the key was
/// already present. The map must have room for a new key — callers that
/// may fill it check `len() < N` first.
pub fn insert(&mut self, key: K, value: V) -> Option<V>;
/// Remove `key`, returning its value if present.
pub fn remove<Q>(&mut self, key: &Q) -> Option<V>
where
K: Borrow<Q>,
Q: Eq + ?Sized;
/// Number of live entries.
pub fn len(&self) -> usize;
}
3.13.3 Dynamically-Sized Bitmap¶
DynBitmap is a heap-backed bitmap whose bit count is fixed at construction but
chosen at runtime (not compile time), for allocation pools whose size is a
runtime parameter — e.g. devpts PTY indices (max= mount option, up to 2^20,
Section 21.1) and evdev minor numbers
(Section 21.3). Backing store is a boxed [u64] slice. Operations are
non-atomic; callers needing concurrent access wrap it in a lock
(SpinLock<DynBitmap>). Construction is a warm/cold path (allocates once); the
per-bit operations allocate nothing.
/// Heap-backed, runtime-sized bitmap. Bit `i` lives in word `i / 64`, bit
/// `i % 64`. Non-atomic; wrap in a `SpinLock`/`Mutex` for shared use. The
/// backing store is rounded up to a whole `u64` word, so bits in
/// `[nbits, words.len() * 64)` are padding — they read clear, and callers that
/// size the pool below the word boundary must bound their own index against the
/// logical maximum after `find_first_zero()`.
pub struct DynBitmap {
/// Backing words. Length is `ceil(nbits / 64)`.
words: Box<[u64]>,
/// Number of logical bits (`<= words.len() * 64`).
nbits: usize,
/// Count of set bits — makes `is_full()` and free-slot fast paths O(1).
set_count: usize,
}
impl DynBitmap {
/// Create a bitmap of `nbits` bits, all clear. Allocates
/// `ceil(nbits / 64)` `u64` words.
pub fn new(nbits: usize) -> Self;
/// Number of logical bits.
pub fn len(&self) -> usize;
/// Test bit `i` (debug-panics if `i >= words.len() * 64`).
pub fn get(&self, i: usize) -> bool;
/// Set bit `i`; returns the previous value.
pub fn set(&mut self, i: usize) -> bool;
/// Clear bit `i`; returns the previous value.
pub fn clear(&mut self, i: usize) -> bool;
/// Lowest clear bit index, or `None` if every backing bit is set. Scans
/// 64-bit words; O(len / 64) worst case, O(1) amortized with a cached
/// last-freed hint. May return a padding index `>= len()`; range-bounded
/// callers must re-check against their logical maximum.
pub fn find_first_zero(&self) -> Option<usize>;
/// Whether every logical bit is set.
pub fn is_full(&self) -> bool;
}
3.13.4 Compile-time capacities: scratch hints and validated bounds — never ownership¶
Compile-time capacities (ArrayVec<_, N>, const MAX_*) recur across the
kernel. This section is the binding rule for when they are legal. The two
canonical instances are NUMA_NODES_STACK_CAP (a boot-validated bound) and
DynBitmap (a runtime-scaled owner that a stack scratch may page over) — every
other use MUST fall into one of the two roles below.
A compile-time capacity (ArrayVec<_, N>, const MAX_*) is legal in exactly two
roles:
(a) Restartable scratch / batch hint — exhaustion means CONTINUE
(cursor/next-batch), never DROP. The unprocessed remainder stays owned by an
authoritative RUNTIME-SCALED structure (XArray ledger, runtime-sized table,
runtime bitmap such as DynBitmap). The hint bounds work per pass, never the
representable population.
(b) Boot/admission-validated bound — the NUMA_NODES_STACK_CAP pattern:
validated at boot or at admission, FAIL-CLOSED BEFORE ownership is created
(the operation that would exceed it is rejected with its normal error), never
truncated after.
A compile-time capacity may NEVER bound the total live ownership of a hardware- or load-determined population. Overflow of any scratch is either impossible-by-invariant (with the invariant stated and checkable) or handled by a TERMINAL protocol (canonical teardown/quarantine) — never silent truncation, never wedge-until-operator, never "log FMA and drop".
3.14 Error Handling and Fault Containment¶
3.14.1 Kernel Error Model¶
Linux problem: Kernel functions return negative integers (-ENOMEM, -EINVAL) for errors.
Linux sometimes stuffs those errors into pointers via ERR_PTR(); the type system enforces nothing —
callers can silently ignore errors, confuse pointers with error pointers, or propagate the
wrong errno. Unchecked kmalloc returns and missing error propagation are endemic.
UmkaOS design: All kernel-internal functions return Result<T, KernelError>. The ?
operator propagates errors up the call stack. The Rust type system makes it impossible to
silently ignore an error — no integer error codes, no sentinel values, and no error-valued pointers.
/// Canonical kernel error type. All umka-nucleus and driver-facing
/// kernel functions return `Result<T, KernelError>`.
#[non_exhaustive]
#[repr(u32)]
pub enum KernelError {
OutOfMemory = 1, // Physical or virtual memory exhausted
InvalidCapability = 2, // Capability handle missing or revoked
PermissionDenied = 3, // Capability lacks required permission bits
InvalidArgument = 4, // Syscall argument out of range
DeviceError = 5, // Device error or device in error state
Timeout = 6, // Operation timed out
WouldBlock = 7, // Non-blocking I/O: operation would block (passive wait —
// the caller need not do anything special before retrying;
// the resource will become available on its own, e.g.,
// O_NONBLOCK socket with no data, pipe with full buffer).
NotFound = 8, // Requested object does not exist
AlreadyExists = 9, // Object or resource already exists
Interrupted = 10, // Operation interrupted by signal
IoError = 11, // Generic I/O error (disk, network, DMA)
ResourceBusy = 12, // Resource in use (EBUSY) — distinct from WouldBlock
NoSpace = 13, // No space left on device (ENOSPC) — distinct from OutOfMemory
NotSupported = 14, // Operation not supported (ENOSYS/EOPNOTSUPP)
CrossDevice = 15, // Cross-device link (EXDEV) — needed by VFS rename
TryAgain = 16, // Transient resource pressure: action needed before retry
// (e.g., memory pressure — trigger reclaim then retry;
// congestion — back off then retry). Unlike WouldBlock,
// the caller must DO something before the retry will
// succeed. Both map to EAGAIN for Linux ABI compat.
// Variants 17..=21 carry errnos that have NO near-equivalent among the
// variants above and MUST survive to userspace as a distinct value (aliasing
// them onto a nearby variant would corrupt the ABI-visible errno). Each is
// exposed through an E-named associated const below.
NoSuchDevice = 17, // ENXIO (6): device/address does not exist
InappropriateIoctl = 18, // ENOTTY (25): ioctl not handled by this device
NotConnected = 19, // ENOTCONN (107): endpoint not connected
NoKey = 20, // ENOKEY (126): no matching key in the retention service
ValueOverflow = 21, // EOVERFLOW (75): value too large for its representation
// KERNEL-INTERNAL ONLY — never crosses the syscall boundary (see the
// note below the POSIX table). `ProbeDeferred` is the semantic form of
// Linux's EPROBE_DEFER (517), which Linux reserves in its 512..=531
// kernel-internal errno range with the explicit rule "these should never
// be seen by user programs" (torvalds/linux `include/linux/errno.h`,
// verified master). A resource getter (`clk_get`, `regulator_get`, a
// device `probe`) returns it when a required provider has not registered
// yet; the driver-model deferred-probe machinery
// ([Section 11.4](11-drivers.md#device-registry-and-bus-management)) consumes it and re-queues the
// probe, translating it to `ProbeResult::Deferred`. It is NOT a
// userspace-visible errno and has NO POSIX-table row.
ProbeDeferred = 22, // EPROBE_DEFER (517): driver-model probe-retry; kernel-internal only
// ENODEV (19) is DISTINCT from ENXIO (6, `NoSuchDevice`): POSIX defines
// them as separate errnos and Linux surfaces ENODEV where a named object
// has gone away (dead cgroup, removed input/block device). Aliasing it
// onto `NoSuchDevice` would corrupt the ABI-visible value — e.g. a
// cgroup.procs write to a dying cgroup must return ENODEV, not ENXIO
// (Linux source: torvalds/linux `kernel/cgroup/cgroup.c` `cgroup_procs_write`:
// `if (!dst_cgrp) return -ENODEV;`, verified master).
NoDevice = 23, // ENODEV (19): named object no longer exists (dead cgroup, removed device)
// Extensible: new variants added at end, values are stable.
// #[non_exhaustive] ensures forward compatibility: match sites use
// `_ =>` with `#[allow(non_exhaustive_omitted_patterns, reason = "forward compat")]`.
// KABI error translation happens at the syscall layer — drivers receive
// KernelError but never need exhaustive matching of kernel-internal variants.
}
/// Errno-named spellings and conversions for `KernelError`.
///
/// **Design**: `KernelError`'s variants are *semantic* (the canonical form). The
/// E-named associated consts below are ALIASES onto those semantic variants — a
/// spelling convenience for driver-facing and Linux-shaped code, mirroring the
/// `IoError` associated-const surface ([Section 14.1](14-vfs.md#virtual-filesystem-layer)). They add
/// NO new states: `KernelError::EBUSY` *is* `KernelError::ResourceBusy`. Each
/// alias resolves to the variant whose POSIX mapping (table below) produces the
/// matching errno, so the userspace-visible value is Linux-exact in every case.
impl KernelError {
/// EINVAL (22) — argument out of range.
pub const EINVAL: KernelError = KernelError::InvalidArgument;
/// ENOMEM (12) — out of memory.
pub const ENOMEM: KernelError = KernelError::OutOfMemory;
/// EBUSY (16) — resource in use.
pub const EBUSY: KernelError = KernelError::ResourceBusy;
/// ENOSPC (28) — no space left on device.
pub const ENOSPC: KernelError = KernelError::NoSpace;
/// EEXIST (17) — object already exists.
pub const EEXIST: KernelError = KernelError::AlreadyExists;
/// EIO (5) — generic I/O error.
pub const EIO: KernelError = KernelError::IoError;
/// EOPNOTSUPP (95) — operation not supported.
pub const EOPNOTSUPP: KernelError = KernelError::NotSupported;
/// ENOTTY (25) — inappropriate ioctl for device.
pub const ENOTTY: KernelError = KernelError::InappropriateIoctl;
/// ENXIO (6) — no such device or address.
pub const ENXIO: KernelError = KernelError::NoSuchDevice;
/// ENOTCONN (107) — transport endpoint is not connected.
pub const ENOTCONN: KernelError = KernelError::NotConnected;
/// ENOKEY (126) — required key not available.
pub const ENOKEY: KernelError = KernelError::NoKey;
/// EOVERFLOW (75) — value too large for defined data type.
pub const EOVERFLOW: KernelError = KernelError::ValueOverflow;
/// ENODEV (19) — no such device (named object gone: dead cgroup, removed device).
pub const ENODEV: KernelError = KernelError::NoDevice;
/// Convert a driver-returned NEGATIVE errno (e.g. `-EIO`, the convention C
/// KABI drivers use to report failure) into the semantic `KernelError`.
///
/// Total over the errno set used across this spec; an unrecognized code
/// collapses to `DeviceError` — a driver returning an errno the kernel does
/// not model is, by definition, reporting a device-level fault. Also serves
/// as the single bridge for composing `IoError`-returning calls into a
/// `Result<_, KernelError>` context: `io_res.map_err(|e| KernelError::from_errno(e.to_neg_errno()))`.
pub const fn from_errno(neg_errno: i32) -> KernelError {
match neg_errno {
-12 => KernelError::OutOfMemory,
-22 => KernelError::InvalidArgument,
-16 => KernelError::ResourceBusy,
-28 => KernelError::NoSpace,
-17 => KernelError::AlreadyExists,
-5 => KernelError::IoError,
-95 => KernelError::NotSupported,
-6 => KernelError::NoSuchDevice,
-25 => KernelError::InappropriateIoctl,
-107 => KernelError::NotConnected,
-126 => KernelError::NoKey,
-75 => KernelError::ValueOverflow,
-19 => KernelError::NoDevice,
-517 => KernelError::ProbeDeferred, // EPROBE_DEFER from a C KABI driver probe
_ => KernelError::DeviceError,
}
}
}
POSIX errno mapping: The syscall entry point (Section 19.1) converts
KernelError to POSIX errno values at the syscall boundary — the only place integer
error codes exist:
| KernelError | POSIX errno | Value |
|---|---|---|
| OutOfMemory | ENOMEM | 12 |
| InvalidCapability | EBADF | 9 |
| PermissionDenied | EPERM / EACCES | 1 / 13 |
| InvalidArgument | EINVAL | 22 |
| DeviceError | EIO | 5 |
| Timeout | ETIMEDOUT | 110 |
| WouldBlock | EAGAIN | 11 |
| NotFound | ENOENT | 2 |
| AlreadyExists | EEXIST | 17 |
| Interrupted | EINTR | 4 |
| IoError | EIO | 5 |
| ResourceBusy | EBUSY | 16 |
| NoSpace | ENOSPC | 28 |
| NotSupported | ENOSYS / EOPNOTSUPP | 38 / 95 |
| CrossDevice | EXDEV | 18 |
| TryAgain | EAGAIN | 11 |
| NoSuchDevice | ENXIO | 6 |
| InappropriateIoctl | ENOTTY | 25 |
| NotConnected | ENOTCONN | 107 |
| NoKey | ENOKEY | 126 |
| ValueOverflow | EOVERFLOW | 75 |
| NoDevice | ENODEV | 19 |
Some variants map to different errnos depending on context (PermissionDenied becomes
EPERM for capability operations, EACCES for filesystem operations). The translation
is handled by the syscall dispatch layer, not by the originating subsystem.
ProbeDeferred is deliberately absent from the table above. It is a
kernel-internal control value (Linux EPROBE_DEFER, 517) that must NEVER reach
userspace. It is produced and consumed entirely within the driver model: a
resource getter returns Err(KernelError::ProbeDeferred), and the deferred-probe
machinery (Section 11.4) catches it at the probe()
boundary, re-queues the driver, and translates it to ProbeResult::Deferred — so
it is resolved before any syscall completes and never propagates onto a syscall
return path. If the syscall dispatch layer nonetheless observes ProbeDeferred
(a structural invariant violation — some path leaked an internal control value),
it MUST NOT emit errno 517: a debug_assert! fires, and in release builds the
value is mapped to EIO (5) so userspace sees a generic I/O failure rather than a
reserved kernel-internal errno.
WouldBlock vs TryAgain — advisory semantic distinction:
Both map to EAGAIN (11) for Linux ABI compatibility (EWOULDBLOCK == EAGAIN on all
Linux platforms). The kernel-internal distinction is advisory and enables
subsystem-specific retry logic:
| Variant | Semantic | Caller action | Example |
|---|---|---|---|
WouldBlock |
Passive wait — resource will become available on its own | Poll/epoll/retry without special action | O_NONBLOCK socket with empty receive buffer |
TryAgain |
Action needed — caller must do something before retry succeeds | Back off, trigger reclaim, release lock, etc. | Memory allocation under pressure (trigger reclaim first) |
Mandatory rule: every function that returns TryAgain must document in its doc
comment what action the caller should take before retrying. A bare return Err(TryAgain)
without guidance is a spec/code review violation — the distinction is useless if the
caller does not know what to do differently.
3.14.2 Fault Containment Boundaries¶
UmkaOS has four fault containment domains. A fault in one domain does not propagate to domains above it in the hierarchy:
| Domain | Failure scope | Recovery |
|---|---|---|
| umka-nucleus (Tier 0) | Kernel panic — entire system | Reboot (same as Linux) |
| Tier 1 driver (domain-isolated) | Single driver crash | Automatic restart, device FLR (Section 11.9) |
| Tier 2 driver (process) | Single driver process crash | Automatic restart (Section 11.9) |
| Userspace process | Single process terminated | Application-level recovery |
Linux problem: Linux has exactly one fault domain for the entire kernel. A null dereference in an obscure USB driver is indistinguishable from a bug in the scheduler — both trigger the same kernel panic. The only containment boundary is kernel vs. userspace.
UmkaOS design: The isolation model (Section 11.2) gives each Tier 1 driver its own
isolation domain. When a CPU exception fires (page fault, general protection fault,
divide-by-zero), umka-nucleus's exception handler inspects the faulting context's isolation
domain ID (architecture-specific: PKRU on x86, page table base on ARM/RISC-V — see
arch::current::isolation::current_domain_id()):
- Domain 0 (umka-nucleus): The fault is in the trusted kernel. This is a genuine kernel panic — proceed to the panic handler (Section 3.14).
- Domain 1-N (Tier 1 driver): The fault is in an isolated driver. The exception handler identifies the driver from the faulting domain ID, marks it as crashed, and invokes the crash recovery sequence (Section 11.9). The rest of the kernel continues running.
Tier 2 driver faults are even simpler: the driver runs in a separate address space, so a fault (SIGSEGV, SIGBUS, SIGFPE) terminates the driver process. The driver supervisor detects the exit and restarts it.
3.14.3 Panic Handling¶
A kernel panic means a bug in umka-nucleus itself — the small trusted computing base. This is the only code whose failure is fatal.
Panic sequence:
1. DISABLE INTERRUPTS — local CLI on the faulting CPU.
NMI IPI broadcast to all other CPUs. Each CPU receiving the NMI executes
the NMI panic handler, which:
(a) Saves the current register context to a pre-allocated per-CPU crash buffer
(allocated at boot, never freed, immune to OOM — one 4KB page per CPU).
(b) Disables local interrupts (preventing further preemption or nested exceptions).
(c) Spins on an atomic flag waiting for the panic coordinator (the faulting CPU).
This ensures all CPUs are in a known-safe state before the coordinator reads
system data structures. The NMI handler is NMI-safe — it uses no locks, no
allocation, and no console logging. It writes only to the pre-allocated crash buffer.
Architecture-specific NMI delivery: On x86-64, this uses the APIC NMI delivery
mode. On AArch64 (GICv3.3+), the GIC NMI mechanism (GICD_INMIR) is used; on
older GIC implementations, a highest-priority FIQ is used as a pseudo-NMI (same
technique as Linux's CONFIG_ARM64_PSEUDO_NMI). On RISC-V, sbi_send_ipi() with a
dedicated panic IPI vector is used. **Limitation**: standard RISC-V supervisor
interrupts are maskable — sstatus.SIE=0 blocks all supervisor interrupts
regardless of AIA priority. If the target CPU has interrupts disabled, the IPI
will not be delivered. Mitigation: the panic coordinator uses a 100 ms timeout
per CPU; CPUs that do not respond are marked "unavailable" in the crash dump.
On systems implementing the Smrnmi extension (Resumable Non-Maskable Interrupts),
UmkaOS uses RNMI delivery instead, which is truly non-maskable. On PPC64LE, the OPAL
opal_signal_system_reset() call triggers a system reset interrupt on target CPUs.
2. CAPTURE STATE — faulting CPU registers, stack backtrace (.eh_frame),
per-CPU crash buffers (from step 1), key data structures (process list,
cap table, driver registry, last 64KB klog)
3. SERIAL FLUSH — panic message + backtrace to serial (Tier 0, polled, always works)
4. NOTIFY — run the panic notifier chain (`run_panic_notifiers()`), letting
external agents learn the machine is going down: the IPMI notifier sends an
"OS Critical Stop" Platform Event to the BMC (Section
[Section 13.23](13-device-classes.md#ipmi-intelligent-platform-management-interface--platform-event-panic-notifier)),
pstore records the message, a hypervisor "guest panicked" event is raised.
Each notifier is bounded and best-effort; a failing notifier never blocks the
rest or the halt.
5. CRASH DUMP — if configured, write ELF core dump to reserved memory region;
if NVMe panic-write path registered, polled-mode write to disk (Section 11.7)
6. HALT — default halt (umka.panic=halt), or reboot (umka.panic=reboot)
Panic notifier chain: subsystems register a PanicNotifier so they can act
during the panic window (step 4 above). This is the UmkaOS analogue of Linux's panic_notifier_list:
/// A subsystem hook invoked during the kernel panic sequence — before the final
/// crash dump and halt/reboot — so external agents can be told the machine is
/// going down. The canonical implementor is the IPMI panic notifier
/// ([Section 13.23](13-device-classes.md#ipmi-intelligent-platform-management-interface--platform-event-panic-notifier)),
/// which sends an "OS Critical Stop" Platform Event to the BMC; other examples
/// are the pstore backend and a hypervisor "guest panicked" event.
///
/// **Context restrictions**: `notify_panic` runs on the faulting CPU with
/// interrupts disabled and every other CPU parked in the NMI handler. It MUST
/// NOT allocate, take sleeping locks, or block indefinitely. It may take short
/// raw spinlocks and MUST bound any hardware wait with an explicit timeout (the
/// BMC may be unresponsive). Errors are swallowed — a failing notifier must
/// never prevent the remaining notifiers or the halt/reboot from running.
pub trait PanicNotifier: Send + Sync {
/// Called once during panic with the human-readable panic message.
fn notify_panic(&self, msg: &str);
}
/// Maximum number of registered panic notifiers. Small and bounded: the set is
/// a handful of platform hooks (IPMI, pstore, hypervisor, netconsole), all
/// registered at boot/driver-probe time.
pub const MAX_PANIC_NOTIFIERS: usize = 16;
/// Registered panic notifiers, invoked in registration order by
/// `run_panic_notifiers()`. Notifiers are `&'static` (never freed once
/// registered) so the panic path — which must not allocate or take sleeping
/// locks — can iterate them under a short raw spinlock. Registration is a
/// warm/cold path (boot, driver probe).
static PANIC_NOTIFIERS: SpinLock<ArrayVec<&'static dyn PanicNotifier, MAX_PANIC_NOTIFIERS>> =
SpinLock::new(ArrayVec::new_const());
/// Register a panic notifier. Fails with `Err(())` if `MAX_PANIC_NOTIFIERS` is
/// already reached. Warm/cold path.
pub fn register_panic_notifier(n: &'static dyn PanicNotifier) -> Result<(), ()> {
PANIC_NOTIFIERS.lock().try_push(n).map_err(|_| ())
}
/// Invoke every registered panic notifier once, in registration order. Called
/// from the panic coordinator (step 4). Best-effort: each notifier is bounded
/// and its failure is contained by `panic = "abort"` domain semantics.
pub fn run_panic_notifiers(msg: &str) {
for n in PANIC_NOTIFIERS.lock().iter() {
n.notify_panic(msg);
}
}
Panic-boundary wrapper (panic_catch): a small number of call sites must run
a closure that might panic without letting that panic escalate to the full
kernel panic sequence — most importantly the live-evolution Phase A verifier,
which calls into an as-yet-untrusted new Evolvable image
(Section 13.18). Because the kernel is panic = "abort" (no stack
unwinding), this is NOT catch_unwind; it installs a per-CPU setjmp-style
recovery frame that the panic entry point checks before it broadcasts the NMI
and captures state.
/// A captured panic, returned by `panic_catch` when the guarded closure aborts.
/// In a `panic = "abort"` kernel there is no unwinding, so this carries only the
/// panic-site information the panic entry recorded — no boxed payload.
#[derive(Clone, Copy, Debug)]
pub struct PanicInfo {
/// Static panic message when the panic used a string literal, else `""`.
pub message: &'static str,
/// Source file where the panic originated.
pub file: &'static str,
/// Source line where the panic originated.
pub line: u32,
}
/// Run `f` inside a panic-boundary guard. If `f` (or anything it calls) panics,
/// the panic entry point `longjmp`s to this boundary via the current CPU's
/// recovery frame and this returns `Err(PanicInfo)` instead of proceeding to the
/// kernel panic sequence; otherwise it returns `Ok(f())`.
///
/// This is a per-CPU `setjmp`/`longjmp` recovery frame, NOT stack unwinding: on
/// entry it saves SP and the callee-saved registers into a per-CPU frame and
/// pushes it; the panic handler, before step 1 of the panic sequence, checks for
/// an active frame on the faulting CPU and restores it, skipping the guarded
/// closure's drop glue. The sole sanctioned use is sandboxing calls into an
/// untrusted new Evolvable image during evolution Phase A verification.
///
/// # Safety
///
/// The guarded closure MUST NOT hold any lock, own a resource whose `Drop` is
/// required for correctness, or mutate global state that the skipped unwind
/// would leave inconsistent (the recovery jumps over all of it). The caller
/// guarantees this; the framework cannot.
pub unsafe fn panic_catch<F, R>(f: F) -> Result<R, PanicInfo>
where
F: FnOnce() -> R;
Driver panic vs. kernel panic: A panic!() inside Tier 1 driver code does NOT
panic the kernel. The kernel is compiled with panic = "abort", so there is no stack
unwinding. Instead, panic!() calls abort(), which executes an illegal instruction
(ud2 on x86-64, udf on AArch64/ARMv7, unimp on RISC-V, trap on PPC). This
triggers a CPU exception (invalid opcode / undefined instruction) within the driver's
isolation domain. The exception handler identifies the faulting domain as a non-core
driver domain and routes the fault to driver crash recovery (Section 11.9) — not to the
kernel panic path.
OOM policy: When physical memory is exhausted, UmkaOS applies pressure in stages:
- Reclaim page cache: Clean pages are evicted immediately (no I/O cost). Dirty pages are written back and then evicted.
- Compress to CompressPool: Inactive anonymous pages are compressed and moved to the in-kernel compression tier (Section 4.12), reducing physical memory usage 2-3x.
- Swap to disk: If a swap device is configured, compressed pages that haven't been accessed spill to disk.
- OOM killer: If all of the above fail to free enough memory, the OOM killer selects
a process to terminate. Heuristic: largest RSS, not marked
OOM_SCORE_ADJ=-1000, not system-critical, not recently started. The selected process receivesSIGKILL.
umka-nucleus itself is never OOM-killed. Core kernel allocations draw from a reserved memory pool (configured at boot, default 64MB) that is excluded from the general-purpose allocator. If the reserved pool is exhausted — a symptom of a kernel memory leak — this is a kernel panic, not an OOM kill.
3.14.4 Error Reporting to Userspace¶
Syscall error returns: Standard Linux ABI — negative errno in the return register
(rax on x86-64, x0 on AArch64, a0 on RISC-V). Applications, glibc, and musl all
work unmodified.
Extended error information: For complex failures where a single errno is insufficient (e.g., "which capability was invalid?" or "which device returned an error?"), UmkaOS provides a per-thread extended error buffer:
/// Per-thread extended error context, populated on syscall failure.
#[repr(C)]
pub struct ExtendedError {
pub errno: i32, // POSIX errno (same as syscall return) offset 0
pub subsystem: u32, // Kernel subsystem that generated the error offset 4
pub detail_code: u32, // Subsystem-specific detail code offset 8
pub _pad: [u8; 4], // Explicit padding for u64 alignment of object_id (offset 12)
pub object_id: u64, // Related capability/device/inode ID (0 = N/A) offset 16
}
// Layout: 4 + 4 + 4 + 4(pad) + 8 = 24 bytes. Padding made explicit per CLAUDE.md rule 11.
const_assert!(size_of::<ExtendedError>() == 24);
Queried via prctl(PR_GET_EXTENDED_ERROR, &buf) — entirely optional. Applications that
don't use it see standard errno behavior with zero overhead (the buffer is only written on
error). The subsystem and detail_code fields are stable, allowing diagnostic tools to
produce messages like "capability 0x3f revoked by generation advance" instead of "EBADF".
Subsystem ID registry (stable values, never renumbered):
subsystem |
Name | Example detail_code values |
|---|---|---|
| 0 | GENERIC |
Generic errno, no subsystem-specific detail |
| 1 | CAPABILITY |
1=revoked, 2=generation_mismatch, 3=delegation_depth_exceeded, 4=type_mismatch |
| 2 | MEMORY |
1=oom_killed, 2=cgroup_limit, 3=mlock_limit, 4=huge_page_unavailable |
| 3 | VFS |
1=dentry_negative, 2=mount_readonly, 3=quota_exceeded, 4=xattr_limit |
| 4 | SCHEDULER |
1=affinity_conflict, 2=rt_bandwidth_exhausted, 3=cbs_throttled |
| 5 | NETWORK |
1=route_unreachable, 2=socket_buffer_full, 3=congestion_drop |
| 6 | STORAGE |
1=device_error, 2=dm_path_failed, 3=journal_aborted |
| 7 | DRIVER |
1=domain_crashed, 2=timeout, 3=probe_failed, 4=tier_demotion |
| 8 | SECURITY |
1=lsm_denied, 2=ima_appraisal_failed, 3=evm_mismatch |
| 9 | IPC |
1=ring_full, 2=peer_disconnected, 3=message_too_large |
| 10 | KABI |
1=version_mismatch, 2=service_unavailable, 3=signature_invalid |
| 11 | DISTRIBUTED |
1=node_unreachable, 2=dlm_deadlock, 3=dsm_coherence_timeout |
| 12 | CRYPTO |
1=key_expired, 2=algorithm_unavailable, 3=rng_reseed_needed |
| 13-255 | Reserved | For future subsystems. Values >= 256 are available for out-of-tree use. |
Kernel log messages: Errors are logged to the kernel ring buffer (dmesg) with
structured fields. The stable tracepoint ABI (Section 20.2) exposes these as
machine-parseable events for external monitoring tools.
3.14.5 Error Escalation Paths¶
Errors escalate through a five-level hierarchy. Each level is attempted before moving to the next:
-
Retry: Transient hardware errors (bus timeout, CRC mismatch, link retrain) are retried with exponential backoff. Maximum retries are configured per error class (default: 3 retries, 1ms / 10ms / 100ms backoff). If the retry succeeds, no further escalation occurs — the event is logged at
DEBUGlevel for trending. -
Log: Persistent errors that survive retries are logged to the kernel ring buffer and recorded as stable tracepoint events (Section 20.2). The Fault Management engine (Section 20.1) ingests these events for threshold-based diagnosis. No state change yet — the subsystem continues operating.
-
Degrade: Repeated errors from the same subsystem trigger graceful degradation. Examples: storage path failover to a redundant controller, NIC fallback from hardware offload to software path, memory controller marking a DIMM rank as degraded (Section 20.1 RetirePages action). The subsystem continues at reduced capacity. Degradation is reported via uevent to userspace.
-
Isolate: A misbehaving driver is crashed and restarted via the recovery sequence (Section 11.9). If the same driver crashes repeatedly — 3 times within 60 seconds — it is demoted to Tier 2 (full process isolation). If it continues crashing at Tier 2 (5 crashes within 300 seconds), the driver is disabled entirely and its device is marked offline in the device registry (Section 11.4).
-
Panic: Reserved for corrupted umka-nucleus state where continued operation risks data loss or silent corruption. Examples: invalid page table entries in kernel mappings, corrupted capability table metadata, scheduler invariant violations. Any state that cannot be recovered by isolating a single driver triggers a kernel panic (Section 3.14).
See also: Section 11.9 (Crash Recovery) for the full driver restart sequence. Section 20.1 (Fault Management) for proactive, telemetry-driven error handling before faults occur. Section 20.2 (Stable Tracepoints) for the machine-parseable event format used at escalation levels 2-4.
FMA integration: Escalation levels 3 (Degrade) and 4 (Isolate) automatically
emit a FaultEvent to the FMA subsystem (Section 20.1).
Level 2 (Log) emits FaultEvent only if the fma_warn_events_enabled sysctl is
set (default: false, to avoid flooding). Level 1 (Retry) and Level 0 (pre-retry
transient) never emit FaultEvent — they are logged via tracepoints only.