ASTs: pending work, safe service points, and return-to-user control

An AST is deferred work represented by reason bits, not a new hardware exception vector or an allocated callback object. Producers record state and request that execution visit a suitable service point. Machine return paths and preemption-enable paths inspect pending bits; common code decides whether to schedule, invoke a subsystem handler, suspend, terminate, or resume the return sequence. The namespace also contains scheduler reason codes that are not independently dispatched callbacks. ast.h contract and namespace, ast_taken_kernel(), ast_taken_user().

Read the complete flavor reference for all 22 active bits, composites, retired positions, and subsidiary payload flags. This chapter follows their common transport and execution contract. ARM64 context switching explains the stack/continuation machinery, while ARM64 exception return follows register restoration after AST work is finished.

1. There are two pending masks, plus subsystem payloads

The thread atomic helpers are in ast.h:195–198, local mask operations in ast.c:465–532, ARM's storage accessor in cpu_common.c:839–842, and scheduler request-mask updates in sched_prim.c:4590–4631. Payload examples are kevent and telemetry producers and Mach exception recording.

ast_context() runs when selecting an incoming thread. It removes the outgoing CPU's AST_PER_THREAD subset, then ORs the entire incoming thread->ast value. ast_propagate() only ORs the thread value into local CPU state. These operations are intentionally different:

ast.c, lines 519–535

void
ast_context(thread_t thread)
{
	ast_t *pending_ast = ast_pending();

	*pending_ast = (*pending_ast & ~AST_PER_THREAD) | thread_ast_get(thread);
}

/*
 * Propagate ASTs set on a thread to the current processor
 * Called at splsched
 */
void
ast_propagate(thread_t thread)
{
	ast_on(thread_ast_get(thread));
}

The mask is an implementation classification, not an assurance that every bit stored in a thread belongs to AST_PER_THREAD. AST_KPERF is set through act_set_ast(thread, AST_KPERF) but excluded from that mask. AST_DTRACE and AST_SFI are also not per-thread members. The user handler explicitly questions the DTrace/Kperf classification; SFI has its own comment that it is currently per-processor. Preserve that nuance instead of inventing a strict partition the code does not implement. Mask definition, Kperf setter, classification comments.

Mermaid source

flowchart LR
    P[Producer updates subsystem payload] --> T[Atomic set in thread AST mask]
    T --> C[Current thread: propagate to local CPU]
    T --> R[Running remote thread: request AST-check IPI]
    T --> L[Async or off-CPU: wait for later propagation]
    R --> A[Target CPU ast_check]
    A --> C
    L --> C
    C --> Q[CPU pending AST mask]
    Q --> K[Eligible kernel preemption point]
    Q --> U[Return-to-user service loop]
    K --> S[Urgent scheduler work]
    U --> H[Subsystem handlers and scheduling]

2. Setting work and interrupting a remote CPU are separate actions

act_set_ast() raises scheduler interrupt level. For the current thread it atomically sets the bit and propagates immediately. For a different thread it takes the thread scheduling lock, sets the bit, then checks that last_processor is running this thread before requesting an AST check there. If the target is not running, no remote interrupt is needed merely to preserve the bit: later context selection propagates its thread state. The thread lock protects that running-target decision; a remembered CPU pointer by itself is insufficient. Setter body, incoming context update.

act_set_ast_async() deliberately omits the remote IPI and thread lock. It still propagates for the current thread; remote targets may notice only at a later switch or quantum expiry. Kevent and telemetry use this path. Quantum expiry explicitly propagates thread bits, and the user-return handler forces another propagation before consuming its callback snapshot. Thus “async” here is a notification-latency choice, not a different callback execution context. Async setter, quantum propagation, user propagation.

On ARM64, cause_ast_check() sends SIGPast only to another CPU. The signal handler clears that signal and calls ast_check(current_processor()). On x86 the corresponding request is i386_signal_cpu(..., MP_AST, ASYNC). The IPI causes a local check; it is not execution of every user-return callback inside the interrupt handler. ARM request, ARM receive, x86 request.

ast_check() ignores a CPU not in PROCESSOR_RUNNING, locks its active thread, acknowledges restartable-range IPI state, propagates thread bits, and runs csw_check(). Scheduling reasons returned by that check are ORed into the local mask. It also updates urgency/performance-control information when needed. A maintenance IPI is different: maintenance_ack_ipi() handles SMR and ledger-tab acknowledgments, not the AST flavor dispatcher. AST-check body, maintenance handler.

3. Kernel service means eligible urgent preemption

ast_taken_kernel() enters with interrupts disabled and must return to its caller. Idle threads consume preemption bits and handle scheduling themselves. A stale urgent check is harmless: the function returns if AST_URGENT has already disappeared. If the thread is preparing to block (waitq_wait_possible() is false), it consumes only URGENT so the caller does not repeatedly invoke it, then returns. Otherwise it consumes AST_PREEMPTION, requires PREEMPT to accompany the request, and blocks with a null continuation. That preserves the interrupted kernel call chain for eventual resumption. Kernel handler, kernel preemption frame.

ast.c, lines 143–157

	if (waitq_wait_possible(thread) == FALSE) {
		/* Consume AST_URGENT or the interrupt will call us again */
		ast_consume(AST_URGENT);
		return;
	}

	/* TODO: Should we csw_check again to notice if conditions have changed? */

	ast_t urgent_reason = ast_consume(AST_PREEMPTION);

	assert(urgent_reason & AST_PREEMPT);

	/* We've decided to try context switching */
	thread_preempted_in_kernel(urgent_reason);
}

This is why the mere presence of BSD, ledger, or exception-reporting bits does not run those callbacks whenever a kernel lock is released. Their general dispatcher is the user-return path. Urgent scheduling can occur at several machine-controlled points:

• ARM64 kernel exception return checks the interrupted IRQ mask, then the thread preemption count and CPU URGENT bit; it switches to the thread stack before calling ast_taken_kernel() (locore.s:1747–1765).
• Re-enabling preemption to count zero checks URGENT. The slow path first verifies interrupts are unmasked, masks them, and reloads the CPU bit to handle a possible intervening CPU switch (preemption_disable.c:65–106).
• Re-enabling interrupts with preemption enabled loops over pending URGENT work before unmasking (machine_routines_common.c:1100–1128).
• x86's T_PREEMPT trap case calls the same common kernel handler with interrupts off (trap.c:729–742).

csw_check() is allowed to decide there is no longer a reason to switch. Nonurgent preemption can receive a bounded grace interval while running in kernel space; the timer rechecks with URGENT at expiry. When the nonurgent timer policy is enabled and no timer is already pending, a new nonurgent request that interrupted userspace is upgraded to URGENT instead of arming that timer. The function's disabled-timer, idle-thread, and already-pending branches return earlier. This timing policy is independent of whether a subsystem callback has work pending. Nonurgent policy, expiry recheck, priority and quantum guide.

4. User-return dispatch consumes a snapshot, not the thread's entire future

ast_taken_user() begins with interrupts disabled, no pending wait, and a non-idle thread. If URGENT is pending, it consumes preemption bits and blocks before running ordinary callbacks. Its continuation is thread_preempted, which calls thread_exception_return(); there is no requirement to preserve the current AST-handler call chain. Urgent user path, continuation.

Otherwise the handler propagates thread bits, consumes a local-CPU snapshot of AST_PER_THREAD | AST_KPERF | AST_DTRACE, and enables interrupts. It then processes callbacks in a fixed order. The thread-level bit is generally cleared immediately before its handler, not wholesale when the CPU snapshot is consumed. If an earlier callback abandons the call chain through a continuation, unserviced thread bits can therefore be re-propagated on the next switch/return. The source also imposes a contract: a handler must not jump straight to exception return without propagation or a context switch that reinstates the unserviced CPU work. Snapshot and contract.

The order in this checkout is DTRACE → BSD → MACF → ARCADE → APC → SYNTHESIZE_MACH → LARGE_ENTER_TELEMETRY → MACH_EXCEPTION → LEDGER → KPERF → RESET_PCS → KEVENT → PROC_RESOURCE → TELEMETRY → DEBUG_ASSERT, skipping compiled-out branches. It is the ordering of tests in handle_user_asts_interrupts_enabled(), not the numeric order of the bits. A callback can block, report an exception, or terminate, so reaching later entries in the same invocation is not guaranteed. Dispatcher body.

After those callbacks, the function returns to scheduler interrupt level, consumes SFI work if configured, and then consumes nonurgent preemption reasons. If PREEMPT was present, it re-runs csw_check() under the thread lock, preserving the QUANTUM reason for that decision. It can service a newly required SFI wait, block through thread_preempted, or retain the current thread and clear its pending nonurgent timer. Finally it restores interrupt state, performs configured SPTM redzone-stack cleanup, and checks return-to-user invariants. Final scheduling and checks.

Mermaid source

flowchart TD
    A[User return notices pending CPU AST] --> U{URGENT?}
    U -->|yes| P[Consume preemption; block with thread_preempted]
    P --> A
    U -->|no| C[Propagate thread bits; consume CPU callback snapshot]
    C --> E[Enable interrupts; run ordered callbacks]
    E --> S[At splsched: SFI and nonurgent scheduling recheck]
    S -->|switch required| P
    S -->|continue current thread| R[Restore interrupt state and verify invariants]
    R --> X[Assembly masks interrupts and checks pending bits again]
    X -->|more work| A
    X -->|none or eligible PFZ deferral| Z[Restore user registers and return]

5. Assembly rechecks, PFZ deferral, and the eventual privilege transition

ARM64 return_to_user masks exceptions and loads CPU_PENDING_AST. With no work it proceeds toward restoration. When the caller requests a preemption-free-zone check and the saved user PC lies in that commpage range, the code writes 1 to saved user x15, leaves the ASTs pending, and proceeds as if there were none for this return. The commpage contract makes that register available as a request to reenter after leaving the protected userspace sequence. This is a controlled deferral of AST work, not consumption or suppression of its producers. Initial check, PFZ decision, PFZ/return guide.

The non-PFZ route calls the C handler and branches back to the check. Saved state must still be restored afterward; calling an AST handler is not itself a user-mode transition:

locore.s, lines 2128–2132

user_take_ast:
	PUSH_FRAME
	bl		EXT(ast_taken_user)							// Handle all ASTs, may return via continuation
	POP_FRAME_WITHOUT_LR
	b		check_user_asts								// Now try again

x86 follows the same broad common-code contract with different saved registers and assembly. return_from_trap finds the active thread's PCB, tests pending bits, optionally marks saved EBX/RBX for a PFZ return, calls ast_taken_user(), disables interrupts, reloads the saved-state location, and loops. This does not imply that x86 and ARM have identical stack or interrupt-mask implementation. x86 return loop.

AST_LAZY is defined as DEBUG_ASSERT, but no executable reference to that mask was found in this checkout outside its definition. These assembly loops test pending bits directly. Therefore the header's statement that lazy ASTs “can be skipped” must not be converted into a claim that these shown return paths actually skip them. Definition, ARM pending test, x86 pending test.

6. Selected asynchronous workers have an explicit service loop

AIO workers can process a limited AST subset while doing work without completing an ordinary return to userspace. ast_check_async_thread() requires interrupts enabled, repeatedly raises scheduler level to consume AST_MACH_EXCEPTION | AST_DTRACE | AST_TELEMETRY | AST_KPERF | AST_DEBUG_ASSERT, and invokes the shared enabled-interrupt dispatcher. It does not consume every flavor and does not call ast_propagate() itself. The AIO loop calls it after processing an entry and before reacquiring the AIO process lock. Restricted loop, AIO caller.

This is an explicit exception to the convenient phrase “all callbacks run on user return.” Kernel-thread self-termination is another targeted exception: thread_terminate() directly clears APC and calls thread_apc_ast() because kernel threads never pass through the user-return AST loop. Neither case makes arbitrary per-thread AST bits a general kernel-worker callback queue. Self-termination.

7. Clearing a request and detaching its payload are different operations

The local reasons snapshot exists only in the current handler's execution. Consuming it from cpu_pending_ast does not atomically clear thread->ast or a subsystem payload. Those operations happen separately at each branch. For KEVENT the implementation first clears the thread bit, then exchanges kevent_ast_bits to zero, and calls the subsystem only for a nonzero payload. TELEMETRY uses the same broad pattern through act_clear_telemetry_ast(). Dispatcher, telemetry clear.

ast.c, lines 379–385

	if (reasons & AST_KEVENT) {
		thread_ast_clear(thread, AST_KEVENT);
		uint16_t bits = atomic_exchange(&thread->kevent_ast_bits, 0);
		if (bits) {
			kevent_ast(thread, bits);
		}
	}

Consider an illustrative interleaving with a previously pending kevent request. A second producer may OR its subsidiary reason before the exchange, so the current callback observes both reasons. If it adds a reason after the exchange, that reason remains in the thread payload for a later pass. The producer also posts the top-level bit; a repeated top-level request can therefore lead to a later handler invocation whose payload is already empty. The if (bits) check accommodates that case. This explains the reason-set representation without treating it as a count, promising one invocation per producer, or replacing the subsystem's synchronization rules with a claim about all possible races. Producer order, consumer order.

The atomic thread-mask helpers use relaxed ordering. Their job is mask manipulation; they do not acquire a reference to a port, pin a VM map, or publish an arbitrary callback closure. Payload protection must come from the producer/consumer protocol. Examples differ substantially: Mach exception state is current-thread-owned; restartable synchronization holds a thread reference while dropping the task lock; Arcade explicitly owns send rights and VM copies. Atomic helpers, exception ownership, restartable reference, Arcade ownership.

Callback state can also be discarded deliberately. Kperf clears its payload after invoking its handlers, including when the sampling handler returns early for an exec transition. Telemetry buffer processing can fail because the buffer has been deallocated or a record is too large. The top-level dispatcher has no generic success code to inspect and no automatic retry queue: it calls void handlers and continues. Any retry or persistent pending state must be implemented by the subsystem. Kperf order, exec skip, telemetry failures.

8. Three distinct ways a callback can block

APC: abandon the old call chain and recheck suspension

APC registration is serialized with the thread mutex and scheduling lock. The consumer enters with no locks held, takes the mutex, clears aborted scheduling flags under the thread lock, and checks whether the thread remains active. Inactivity transfers to thread_terminate_self(). Suspension sets suspend_parked, asserts an abort-safe wait on suspend_count, drops the mutex, and blocks with the non-null thread_suspended continuation. The old AST-handler call chain is not the continuation's return address. Producer locking, consumer.

thread_act.c, lines 1418–1427

	/* If we're suspended, go to sleep and wait for someone to wake us up. */
	if (thread->suspend_count > 0) {
		thread->suspend_parked = TRUE;
		assert_wait(&thread->suspend_count,
		    THREAD_ABORTSAFE | THREAD_WAIT_NOREPORT_USER);
		thread_mtx_unlock(thread);

		thread_block(thread_suspended);
		/*NOTREACHED*/
	}

On wake, thread_suspended() reacquires the mutex, accounts for an interrupted wait, and checks suspend_count again. If a new or continuing suspension remains, it rearms APC; then it unlocks and calls thread_exception_return(). The recheck handles a changing suspension state rather than assuming the original wake grants permanent permission to return to userspace. Unserviced per-thread flavors from the abandoned AST pass remain available for propagation. Continuation, common reinstatement contract.

Ledger: preserve the call chain and explicitly retry

ledger_ast() first reconciles CPU-limit policy, snapshots task parameters under the task lock, and rejects work for a missing ledger or dying task/thread. It checks the thread and task ledgers, temporarily obtains the coalition ledger, checks it, and drops that reference. It can invoke callbacks or identify a need to block. This is accounting policy executed at a safe point, not a passive reporting-only callback. Policy and lifetime checks.

ledger_perform_blocking() asserts a deadline wait on a deficit entry, marks LF_WAKE_NEEDED, and blocks with a null continuation and reason LEDGER. A normal THREAD_AWAKENED return becomes KERN_FAILURE, whose meaning here is “restart evaluation”; the caller goes back to its top label and resnapshots policy. Other wait results return KERN_SUCCESS to continue. This deliberately inverted-sounding local status convention is worth reading in the function body before inferring that failure means resource loss. Blocking body, caller retry.

SFI: a CPU request becomes a class wait

SFI is consumed outside the ordinary callback snapshot. At scheduler interrupt level, sfi_ast() takes sfi_lock, returns if the facility is disabled, and reclassifies the thread while also holding its scheduling lock. If the resulting class is in an off phase, it asserts an interruptible wait on that class's wait queue, records the class continuation, drops locks, and blocks with reason SFI. An interrupted wait is expected for a thread already exiting; other unexpected assertion outcomes are checked. Reclassification under the lock prevents an old CPU AST from blindly applying an obsolete class decision. SFI body and race explanation, common service locations.

These paths show why “AST handler can block” is incomplete documentation. APC uses a continuation and rearms as needed; ledger retains its stack and retries; SFI uses a class-specific continuation. The common dispatcher supports these protocols but does not supply their wake event, priority inheritance, object references, or condition recheck. Continuation machinery, wait and turnstile protocols.

9. Restartable ranges: request delivery is not completion

AST_RESET_PCS makes the distinction especially visible. A caller of task_restartable_ranges_synchronize() wants other threads to stop depending on a registered userspace instruction range before memory can be reclaimed. The routine accepts only the current task, succeeds immediately without registered ranges, starts a temporary priority floor, and serializes requests with task_rr_in_flight under the task lock. Concurrent synchronization returns KERN_ALREADY_WAITING. It then executes a release fence and posts requests to the other threads. Synchronization setup.

The posting decision is not just “set a bit on all threads.” For an inactive target, no new request is installed. A running remote target gets trr_ipi_ack_pending, the bit, and an IPI. An off-CPU target gets a bit only if its saved PC lies in a restartable range; if it is faulting, that fault becomes TRR_FAULT_OBSERVED and requires waiting. Current-thread signaling takes a simpler local set/propagate route. Precise branches.

At IPI or context-switch time, thread_reset_pcs_ack_IPI() runs under the thread lock, clears the acknowledgment flag, and changes an in-flight fault from PENDING to OBSERVED. That is distinct from the eventual AST handler changing the saved PC. A synchronizer may therefore have received acknowledgment while still needing to wait for an already-started memory fault to complete. Acknowledgment, scheduler call site, fault safety requirement.

Mermaid source

sequenceDiagram
    participant S as Synchronizing thread
    participant T as Target CPU/thread
    participant F as Fault/exception completion
    S->>S: task lock#59; serialize#59; release fence
    S->>T: set RESET_PCS#59; mark IPI acknowledgment pending
    T->>T: IPI or context switch acknowledges request
    T->>T: outstanding fault becomes OBSERVED
    S->>S: retain target#59; wait for acknowledgment
    S->>S: if fault OBSERVED, sleep with target as inheritor
    F->>F: clear completed fault state
    F->>S: wake synchronizer if waiting
    S->>S: reacquire task lock#59; restart enumeration if dropped
    T->>T: user-return AST rewrites saved PC if still in range

thread_rr_wait_if_needed() references the target before it drops the task lock. It waits for the IPI flag to change, then can atomically install trr_sync_waiting and sleep with the target as turnstile inheritor for an observed fault. It ultimately releases that reference. When a wait drops the task lock, the outer loop restarts task-thread enumeration rather than continuing through stale list state. Fault completion or termination clears fault state and wakes the waiter. These references and acknowledgments, not the mere AST bit, establish the reclamation boundary. Wait/reference protocol, outer restart, fault completion.

The AST callback itself is much smaller: look up the saved PC, reset it if the thread is active and the registered range contains it, and assert that fault/wait state is resolved. Range data can be read without a lock because registration is restricted to the task's single-threaded state. The callback does not perform the synchronization loop above. Consumer, registration assumptions.

10. Exception records and reporting failures have their own ownership rules

AST_MACH_EXCEPTION uses a single thread record. thread_ast_mach_exception() asserts the current thread, skips kernel_task, and refuses to overwrite an existing sticky exception or replace a non-sticky pending exception with another non-sticky one. It stores the reason/type/code/subcode before posting. The consumer snapshots these fields, zeros the record, and then routes guard reasons or fatal handling. A lingering CPU bit with no valid reason does nothing. This is neither a general FIFO of exceptions nor proof that every synchronous exception first becomes an AST. Producer, consumer, Mach exception delivery.

AST_SYNTHESIZE_MACH instead handles a map-level fault potentially reported asynchronously while kernel work borrows an address space. A task-wide request can wake several threads, but vm_map_synthesize_guard_exception() uses compare/exchange to replace the map record with ALREADY_REPORTED. Threads observing no record, the sentinel, or a lost compare/exchange return. One winning thread performs reporting, with the configured MTE soft-mode alternative. The task fan-out checks task activity and locks each remote thread while deciding whether to send an IPI. Fan-out, map claim.

vm_map.c, lines 7774–7783

	async_fault = *(volatile vm_map_async_fault_t *)&map->async_fault;

	if (async_fault.code == 0 ||
	    async_fault.code == VM_ASYNC_TAG_FAULT_ALREADY_REPORTED ||
	    !os_atomic_cmpxchg(&map->async_fault, async_fault, clear_fault, relaxed)) {
		/*
		 * Someone handled that fault already, just return
		 */
		return;
	}

AST_PROC_RESOURCE similarly consumes the need to evaluate current limits, not an immutable report for the allocation that first set the bit. Descriptor/workloop evaluation uses sentinels so racing threads do not repeat the same limit action; descriptors are checked first. Port-space reporting can exempt PID 0/1, report a full system-sized space as fatal, or allocate a fatal-task send right for a configured hard limit. The reporting routine releases that right after sending, logs notification errors, and uses SIGKILL if the required token cannot be allocated. These are explicit failure branches; generic AST dispatch does not rescue a failed report. FD/workloop arbitration, port reporting and cleanup.

Arcade demonstrates a callback that owns transport resources and retries. It copies a send right under arcade_upcall_mutex, refreshes the service port if invalid, builds a vm_map_copy_t containing the executable path, sends the validation upcall, and releases the send right. MACH_SEND_INVALID_DEST discards the unsent VM copy before retry; MIG_SERVER_DIED also retries. A successful rejection or other failure terminates the task. The ten-second deadline calculated by the handler is passed to the port-refresh wait; it must not be advertised as a proven end-to-end bound on every external upcall or the refresh-in-progress wait, which has no deadline argument. Callback, refresh and waits.

Sources inspected and limits

The source search distinguished live constants, historical comments, per-subsystem payloads, and similarly named trace reasons. It does not establish that all producers execute in every supported configuration or that the source contains the external services. The complete numeric/configuration reference is AST flavors; the common service paths above are the foundation for following any additional producer.