Ledger balances, per-CPU settlement, and limit enforcement

Runtime · Time and accounting · Recount · AST flavors · Bank charging

A ledger is an owned set of resource entries. Clients credit/debit entries; selected features retain historical debits, peaks, periodic allowances or callback thresholds. Scalable entries can defer updates in a per-CPU tab, so a stored balance is not necessarily the final balance visible after settlement. Limit handling publishes AST work rather than invoking every policy callback inside the original accounting operation. This chapter follows common and ARM64 paths in xnu-12377.121.6.

1. Templates select representation and behavior

ledger_template describes entry names/groups/units, callbacks, allocation size, optional per-CPU tabs, and a lookup from tab counters to encoded entry IDs. ledger_template_finalize() assigns offsets/counter indices/features, determines storage size, and sets up the allocation backend. An entry ID is therefore not simply its ordinal in the template. A ledger instance contains an ID, template pointer, atomic reference count and variable entry storage. Template definition, finalization, instance layout.

The header documents feature compatibility; finalization normalizes selected options, removes scalable behavior for non-scalable templates, and adds debit for refill. Do not assume every documented incompatible pair has a separate runtime rejection: inspect the finalization checks when introducing a template. Feature contracts, normalization and allocation.

The small and large entries share initial fields but have different storage. Large entries add debit, limit, and a union for refill timing or maxima. ledger_instantiate() allocates zeroed storage, initializes references and defaults, including initially unlimited limits. Allocation/free may use the protected backend selected by the template/configuration; do not infer a universal ordinary-zone allocation for pmap ledgers. Layouts, instantiate/free.

2. Direct updates versus an open per-CPU tab

ledger_credit()/ledger_debit() validate the ledger/entry and require a positive amount; an invalid entry or nonpositive amount is a no-op in these void interfaces. Direct credit adds to the main credit field atomically. Direct debit either adds historical debit or subtracts from balance. The updates then evaluate applicable limit/callback state. Public update path, direct arithmetic.

A template can instead maintain a ledger_tab per CPU. Each tab has one lt_open ledger pointer, a bitmask identifying dirty counters, a local array of credit/debit pairs, a reentrancy flag and a force-settle acknowledgement flag. The pair layout permits merging either a history-preserving debit or a net balance into the main entry. Tab and pair representation.

Mermaid source

flowchart TD
    Update["Credit or debit a valid entry"] --> Scale{"Scalable entry and matching open tab?"}
    Scale -->|No| Main["Atomic update of main ledger entry"]
    Scale -->|Yes| Local["Preemption disabled: update local credit/debit pair"]
    Local --> Check{"Early settlement required?"}
    Check -->|No| Dirty["Keep counter marked dirty in tab"]
    Check -->|Yes| Merge["Merge pair into main entry and clear local pair"]
    Merge --> Main
    Dirty --> Settle["Ledger switch, explicit read or maintenance IPI"]
    Settle --> Merge
    Main --> Policy["Evaluate limit and publish AST if needed"]

For an eligible scalable update, preemption is disabled so the tab stays on the same CPU. If that tab is open for another ledger, the operation falls back to direct accounting rather than changing its owner implicitly. lt_lock protects against maintenance-IPI reentrancy while a local pair is being changed. If a force-settle request arrives during that region, the updater notices the pending acknowledgement after dropping the local guard and settles. Local credit/debit paths.

le_should_settle() can discard a zero-net pair when no historical debit is needed. For maxima/callback entries it estimates pressure using main balance plus local balance multiplied by the CPU count. That expression is a trigger to flush; it is not a measured sum of all CPUs' pending tabs. Crossing a projected maximum, critical/warning transition or applicable diagnostic threshold can force settlement. Early-flush decisions.

3. Opening, flushing, and remote acknowledgements

ledger_tab_open() requires preemption disabled. When changing to a different ledger it first settles a dirty tab, then publishes the new borrowed pointer. Pmap switches use this mechanism; the SPTM pmap path opens the corresponding pmap ledger. The tab does not itself retain the ledger: the caller must keep the ledger alive while it remains open. Tab switch, explicit lifetime contract, ARM64 SPTM pmap integration.

Local settlement walks dirty bits, resolves each bit through the template's counter lookup, copies and clears the pair, and merges it into the owning ledger. It intentionally processes even a zero amount because merging must re-evaluate state. Only after the stores are ordered does it clear the dirty mask and acknowledgement/reentrancy state.

osfmk/kern/ledger.c, lines 1040–1064:

le_tab_settle(ledger_template_t lt, ledger_tab_t tab)
{
	ledger_t ledger = tab->lt_open;

	assert(!tab->lt_lock);
	os_atomic_store(&tab->lt_lock, true, compiler_acquire);

	for (uint64_t mask = tab->lt_mask; mask; mask &= mask - 1) {
		uint32_t      idx   = __builtin_ffsll(mask) - 1;
		uint32_t      entry = lt->lt_counter_lut[idx];
		ledger_pair_t pair  = tab->lt_pairs[idx];

		/*
		 * do -not- skip amount == 0, le_tab_{credit,debit}()
		 * relies on this performing the check again.
		 */

		tab->lt_pairs[idx].lp_pair = 0;
		le_tab_merge(ledger, entry, pair __ppl_only(, false));
	}

	os_atomic_thread_fence(release);
	tab->lt_mask = 0;
	tab->lt_ipi_and_lock = 0;
}

ledger_tab_settle() is a coordinated flush, not a lock around all future credits. In normal operation it requires interrupts enabled so the caller can itself service other CPUs' settlement requests. It disables preemption while scanning template tabs, flushes its own matching tab, and sends maintenance IPIs to remote matching dirty tabs. It then re-enables preemption before waiting for acknowledgements and finishes with an acquire fence. Remote settlement.

A receiver settles immediately only when an acknowledgement is pending and its local reentrancy guard is clear. Otherwise the active updater services the request when safe. The public IPI acknowledgement routine here services the task-ledger template. These details matter when tracing a pmap update that cannot perform arbitrary policy work inline. Acknowledgement handling, deferred updater flush.

ledger_get_entries() normally settles a scalable entry before reading it; LEO_NO_SETTLE deliberately skips that step. The read rejects an invalid entry and restricts settling reads from interrupt context. Separate main credit/debit loads are not a Recount-style sequence transaction. New concurrent credits remain possible after the flush, so this operation must not be described as freezing all resource consumers. Get entries/balance.

4. A concrete delayed-balance example

The local unit test constructs a scalable template and disables preemption while manipulating its tab. Its values expose the contract:

This is the inspected test sequence, not a performance benchmark. The unit test was not run here. Test body.

5. Limits, refill, and publication of AST work

Changing a limit does not reset current consumption. ledger_set_limit() updates the threshold/warning state, clears callback/warning flags and calls the wake helper, which wakes recorded sleepers only when the balance satisfies the new limit. Changing to unlimited also disables automatic refill and blocking; changing a finite limit while refill is scheduled resets its last-refill anchor to zero. ledger_set_period() requires a finite limit, converts its nanosecond input to Mach ticks, resets current balance without decreasing historical totals, and schedules refill. Limit mutation, period configuration.

ledger_refill() serializes competing refills with LF_REFILL_INPROGRESS. It counts completed periods, computes the allowance to restore, caps that restoration to the outstanding balance, and advances the historical debit rather than overwriting the credit history. It updates the time anchor, clears callback state where appropriate, and wakes recorded sleepers when the balance is at or below the limit. Refilling an allowance is therefore an accounting debit, not a backwards movement of the execution counter. Refill.

The update/check path requests AST_LEDGER when limit/warning state requires service. Normal current-thread publication also propagates to the CPU AST state. Scheduler accounting has a special override: ledger_credit_sched() temporarily identifies the thread whose time is being charged, so the AST belongs to that thread even while another execution context performs the update. The protected-monitor path can defer publication through the tab's PPL AST field. These are separate implementation branches, not an implication that all ARM64 ledgers run inside that monitor. AST producer, scheduler override, tab deferred publication.

Mermaid source

sequenceDiagram
    participant C as Accounting caller
    participant L as Ledger entry
    participant A as Thread AST handler
    participant W as Wait queue and timer
    C->>L: Credit or merge pending balance
    L->>A: Publish AST_LEDGER when state requires service
    A->>L: Reconcile CPU-limit requests and inspect limits
    opt Callback action eligible
        A->>L: Mark callback state and invoke registered callback
    end
    opt Refillable blocking limit exceeded
        A->>W: Register interruptible deadline wait
        W->>A: Wake, timeout or interruption
        A->>L: Reevaluate when blocking helper requests restart
    end

ledger_ast() snapshots policy inputs under the relevant locks, applies/removes per-thread CPU limits, checks whether the thread remains active, then evaluates thread/task/coalition ledger state. It retains and releases the coalition ledger while using it. The callback scan considers critical, warning and diagnostic state; already-reported flags suppress redundant delivery. The actual blocking helper is applied to thread/task ledgers in this path. Do not infer a coalition wait loop merely because its limits are checked. AST body, callback scan.

Blocking is restricted to eligible refill entries with a blocking action and an exceeded limit. It registers an interruptible wait ending at last_refill + refill_period, marks LF_WAKE_NEEDED, and blocks with reason AST_LEDGER. A THREAD_AWAKENED result requests a full AST restart because policy, other resources or thread lifetime may have changed. Other results take the helper's continue path; it is inaccurate to claim every wake/timeout restarts identically. Blocking decisions. The AST flavor chapter explains where AST_LEDGER is consumed on the return-to-user route.

6. Ownership and configuration boundaries

A thread's cached task ledger receives a reference during creation and loses it during destruction; task and bank owners have their own references. That reference lifetime is distinct from template lifetime and from the borrowed pointer in an open tab. The final ledger dereference frees instance storage through its selected backend. Thread acquisition, thread release, ledger reference/free.

This review follows template setup, direct/scalable updates, settlement, period/limit mutations, selected AST policy and final release. Client callbacks decide the actual resource policy; a generic ledger does not itself define a process memory budget, jetsam policy, every coalition limit, or all scheduler CPU-limit interfaces.