VM pages and resident storage

A vm_page is a descriptor for resident-page state. Its address identifies metadata; its physical-page number identifies bytes; its object/offset pair identifies those bytes in a VM object's namespace. This companion follows how those identities are constructed, published, accessed, moved and released in xnu-12377.121.6. Return to the structure guide, or read alongside VM objects and backing and pmap translation storage.

For full operations, follow fault resolution, reclaim and pressure, UPL gathering and completion, and MTE page lifecycle. This page concentrates on the descriptors and the guarantees those operations must establish.

Inspect the live vm_page layout for the selected source profile.

Four different identities surround one resident page

The structure has no general-purpose ref_count. A busy bit, a wire count, object ownership and queue membership serve different purposes; none is a substitute for understanding how the caller stabilizes the page. vmp_reference is a page-use hint, not a retain count.

Mermaid source

flowchart TD
    O["VM object and aligned object offset"] -->|"resident lookup"| M["vm_page descriptor
kernel metadata address"]
    M -->|"VM_PAGE_GET_PHYS_PAGE"| N["Physical page number"]
    N -->|"multiply by PAGE_SIZE"| B["Physical page bytes"]
    V["Virtual address in a pmap"] -->|"pmap translation"| B
    M -.->|"unpack vmp_object"| C["Borrowed association<br/>to the same object"]

The arrows show lookup/identity relationships, not interchangeable pointers. Several virtual mappings can reach one frame. A descriptor can also exist without mappable content, such as a guard or an absent page awaiting data. See physical identity accessors, status fields, and the pmap companion.

Where descriptors come from

The boot array describes physical memory

vm_page_bootstrap reserves the vm_pages descriptor array with pmap_steal_freeable_memory. The calculation accounts for both physical-page storage and per-page metadata (PAGE_SIZE + sizeof(struct vm_page)). Under XNU_VM_HAS_LINEAR_PAGES_ARRAY, it uses the physical span; other builds use the available-page count. This is early VM/bootstrap allocation, before normal page grabs consume the resulting free capacity.

The initialization loop gets physical page numbers from pmap_next_page, initializes descriptors, records the first page number for a linear array, and publishes eligible pages through startup release. Delayed-page, high-memory, retired-page and MTE branches affect the boot path. The array is accessed through vm_page_get, not by assuming that every caller can directly index a globally exposed array pointer.

vm_page_init initializes metadata. The selected excerpt sets offset to an invalid sentinel, clears queue/membership state and begins busy; it does not fill the physical page:

osfmk/vm/vm_resident.c lines 4213–4223:

	assert(VM_PAGE_NOT_ON_Q == 0);
	assert(sizeof(*mem) % sizeof(uintptr_t) == 0);
	*mem = (struct vm_page) {
		.vmp_offset      = (vm_object_offset_t)-1,
		.vmp_q_state     = VM_PAGE_NOT_ON_Q,
		.vmp_on_specialq = VM_PAGE_SPECIAL_Q_EMPTY,
		.vmp_canonical   = vm_page_in_array(mem),
		.vmp_busy        = true,
	};

	VM_PAGE_INIT_PHYS_PAGE(mem, phys_page);

The optional bootstrap fillPage call is separate. So are later page-zeroing and pager-input operations. vm_page_init even contains a disabled #if 0 ref/mod clearing block; that historical block must not be described as an executed hardware-state reset. Surrounding initialization.

The descriptor zone supplies additional categories

The writable vm pages zone is a second source of metadata. vm_page_module_init rounds sizeof(struct vm_page_with_ppnum) to 64-byte packed-pointer alignment, creates the zone with ZC_ALIGNMENT_REQUIRED | ZC_VM, and sets a reserve of ten elements. It is not an RO zone, and allocating an element does not by itself allocate a physical data page.

vm_page_create allocates that element and initializes it. The callers determine its category:

Sources: additional canonical pages, constructors/category tests, private transition, fictitious release, free preparation and split.

“Outside the boot array” therefore does not mean “fictitious.” The canonical flag is an independent distinction, and vm_page_is_private is defined as neither canonical nor fictitious. A private page descriptor is also distinct from the private flag on a VM object and from a private user mapping.

Packed links and physical-page accessors

The descriptor contains three list roles: vmp_listq links pages in one object's resident set; vmp_pageq links free/reclaim queues or shares storage with the temporary vmp_snext list pointer; vmp_next_m links the object/offset hash bucket. vmp_specialq is an additional special-queue link. Those fields can make the same descriptor reachable through several indexes without allocating another physical copy. Link layout.

Page and object pointers are packed, but not in the same way. A page pointer into vm_pages becomes an array index with bit 31 set. Other supported pointers use a 31-bit relative representation with a six-bit shift and VM_MIN_KERNEL_AND_KEXT_ADDRESS base. The non-array representation is why object heads and zone descriptors require 64-byte alignment. VM_PAGE_OBJECT unpacks vmp_object using the object form; it does not perform a zone lookup, retain or liveness check. Packing geometry, pack/unpack bodies.

osfmk/vm/vm_page.h lines 620–632:

static inline vm_page_packed_t
vm_page_pack_ptr(uintptr_t p)
{
	if (vm_page_in_array(__unsafe_forge_single(vm_page_t, p))) {
		ptrdiff_t diff = (vm_page_t)p - vm_pages_array_internal();
		assert((vm_page_t)p == vm_page_get((uint32_t)diff));
		return (vm_page_packed_t)(diff | VM_PAGE_PACKED_FROM_ARRAY);
	}

	VM_ASSERT_POINTER_PACKABLE(p, VM_PAGE_PACKED_PTR);
	vm_offset_t packed = VM_PACK_POINTER(p, VM_PAGE_PACKED_PTR);
	return CAST_DOWN_EXPLICIT(vm_page_packed_t, packed);
}

For physical identity, linear-array descriptors derive the page number from their array index. Zone descriptors in those builds have an extended vm_page_with_ppnum; non-linear builds store the physical number in the descriptor itself. Always follow the accessor rather than assuming m->vmp_phys_page exists in every build:

osfmk/vm/vm_page.h lines 549–560:

static inline ppnum_t
VM_PAGE_GET_PHYS_PAGE(const struct vm_page *m)
{
#if XNU_VM_HAS_LINEAR_PAGES_ARRAY
	if (vm_page_in_array(m)) {
		uintptr_t index = (uintptr_t)(m - vm_pages_array_internal());

		return (ppnum_t)(vm_pages_first_pnum + index);
	}
#endif /* XNU_VM_HAS_LINEAR_PAGES_ARRAY */
	return ((const struct vm_page_with_ppnum *)m)->vmp_phys_page;
}

VM_PAGE_SET_PHYS_PAGE asserts that the descriptor is neither in the array nor canonical. Reassigning an alias/fictitious descriptor's physical number is supported; rewriting the physical identity of an ordinary canonical descriptor is not this helper's contract. Setter.

Worked coordinates with a 16 KiB VM page

Assume a linear array, PAGE_SIZE = 0x4000, vm_pages_first_pnum = 0x80000, and a valid descriptor at array index 0x1234. These are illustrative values, not addresses measured from a device:

The packed value 0x80001234 is not a physical page number or a CPU pointer. The object offset 0x8120 is not a virtual address. A virtual address requires a map entry and translation; see maps and entries. The source formulas are the page-link encoding and physical-number accessor.

Access protection follows the state being changed

The header's O and P annotations mean object-lock and page-queue protection, with some association fields requiring both for changes and either for reads. The header immediately qualifies that shorthand: queue-state protection depends on the current queue, and separate bitfield structures prevent compiler coalescing across different locking domains. A single claim that “all page fields use the queue lock” would contradict the definition. Annotation contract, queue-word qualification.

Sources: all flag groups, hash mutation, queue state enumeration, wire/unwire. The fault chapter and pmap mapping lifecycle connect these metadata bits to enforcement.

Two consequences are easy to miss. A newly initialized free descriptor is busy, so busy alone does not mean that it is attached to an object or performing disk I/O. And a successful lookup returns a borrowed descriptor pointer under the caller's existing protection; it does not retain, wire, busy or validate the page contents automatically. Initializer, lookup return.

Acquiring a page and publishing its resident identity

vm_page_grab_options first disables preemption to inspect the current CPU's page magazine, including separate tagged/zeroed paths where configured. After reenabling preemption it finalizes a successful grab. A miss can reach delayed pages, global free queues or a secluded pool. A privileged thread can block and retry unless VM_PAGE_GRAB_NOPAGEWAIT was requested; other exhausted paths return null. This is consumption of available physical capacity, not a fresh zalloc of the canonical descriptor.

vm_page_grab_finalize checks clean descriptor state in assertion builds, sets VM_PAGE_NOT_ON_Q, clears queue linkage, and applies selected accounting/tag rules. VM_PAGE_GRAB_ZERO_FILL causes physical zeroing unless the selected CPU path already satisfied it. Without such preparation the caller must supply valid bytes through its own zero/copy/I/O path. Zero option, prezeroed CPU page selection.

Mermaid source

flowchart TD
    F["Canonical descriptor on free capacity"] -->|"vm_page_grab_options"| G["Private unqueued descriptor
busy, no object association"]
    G -->|"caller locks object and chooses offset"| I["vm_page_insert_internal"]
    I -->|"set packed object and offset"| H["Publish hash membership
unless explicitly delayed"]
    H -->|"link memq and update accounting"| R["Resident descriptor
tabled does not guarantee valid contents"]
    R -->|"caller completes bytes and state"| U["Usable page under operation's protocol"]
    F -.->|"no capacity"| W["Wait/retry or return null"]

This is a relationship/publication sketch, not a universal ordering of zeroing versus insertion; individual callers can fill before or after insertion while maintaining their required exclusion.

Object, hash and resident-list publication are separate steps

vm_page_insert_internal asserts aligned offset and exclusive object locking. VMPI_Q_LOCKED requires the queue lock; otherwise the queue lock must not be held and the page must not be pageable. Tagged-object/page compatibility and wired-page accounting tags are checked before publication.

osfmk/vm/vm_resident.c lines 3461–3477:

	/*
	 *	Record the object/offset pair in this page
	 */
	mem->vmp_object = VM_PAGE_PACK_OBJECT(object);
	mem->vmp_offset = offset;

	if (flags & VMPI_REPLACE) {
		assert((flags & VMPI_DELAY_HASH) == 0);
		vm_page_hash_replace(mem, object, offset);
	} else if (flags & VMPI_DELAY_HASH) {
#if MACH_ASSERT
		assert(!object->delayed_page_insert);
		object->delayed_page_insert = true;
#endif /* MACH_ASSERT */
	} else {
		vm_page_hash_insert(mem, object, offset);
	}

The ordinary vm_page_hash_insert checks internal-object bounds, verifies the page's object/offset association, locks the selected bucket, asserts that the key is absent, and inserts it. Other flags can replace a bucket entry or defer hashing. vmp_tabled and vmp_hashed consequently describe related but distinct membership states; delayed insertion must finish before object unlock. Unlock assertion.

The object resident-list publication is visible separately:

osfmk/vm/vm_resident.c lines 3516–3527:

	/*
	 *	Now link into the object's list of backed pages.
	 */
	vm_page_queue_enter(&object->memq, mem, vmp_listq);
	object->memq_hint = mem;
	mem->vmp_tabled = TRUE;

	/*
	 *	Show that the object has one more resident page.
	 */

	object->resident_page_count++;

The remaining insertion body updates wired/shared-cache/internal/external/purgeable/ledger counts and cache attributes as appropriate. Publishing only the packed pointer and offset would omit both index membership and accounting. Remaining insertion.

A real caller: gathering an initially empty object into a UPL

vm_object_iopl_wire_empty is a useful bounded example because its preconditions exclude pager, shadow, copy and volatile/empty purgeable state. With a 16 KiB VM page, an illustrative two-page object range [0, 0x8000) produces descriptors at offsets 0 and 0x4000; each UPL slot records a physical page number, not either offset.

The helper grabs pages in a loop. On shortage it waits with the selected interruptibility; interruption exits with MACH_SEND_INTERRUPTED. For each acquired page it records absent/reference/dirty state. In the ordinary zero-fill path it establishes IOPL wire state, clears busy under the held object lock, inserts the page, then zeroes the physical bytes before completing the operation. In the no-zero-fill path it retains absent/busy state and does not claim completed contents. The bitmap and physical-page output record which pages were actually obtained. Grab/wait, state, insertion and zeroing, output and partial-progress accounting.

The done path flushes accumulated insertion accounting and reports the acquired count even after interruption. Earlier acquired pages do not vanish because the later grab failed. The caller and UPL completion/abort paths must resolve the recorded partial progress; see UPL gathering and completion. This example is why a structure walkthrough must keep “inserted,” “busy cleared,” “bytes initialized” and “returned to the consumer” separate.

Lookup, busy waits and reassociation

vm_page_lookup requires an object lock and page-aligned offset. It tries the object's hint, adjacent resident-list entries, an empty-bucket rejection, then either a short resident-list scan or the locked global hash bucket. The small-object threshold is ten resident pages in this snapshot. For the hash path it uses hw_lck_ticket_unlock_after_lookup, whose ordering is deliberately narrower than a normal release barrier. It updates memq_hint; the older comment “No side effects” should not be read as “no metadata writes.” Threshold and old comment, hint writes, hash ordering.

A descriptor found by lookup can be busy or cleaning. vm_page_sleep sets wanted and sleeps using the descriptor address as event while coordinating release/reacquisition of the object lock. PAGE_SLEEP_WITH_INHERITOR selects the inheritor path; other builds call lck_rw_sleep. A wakeup says that the caller can retry its state check. It is not a promise that the same descriptor still names the same object offset.

A concrete caller makes that rule explicit:

osfmk/vm/vm_object.c lines 7480–7493:

				if (dst_page->vmp_busy || dst_page->vmp_cleaning) {
					/*
					 * someone else is playing with the
					 * page, we will have to wait
					 */
					vm_page_sleep(object, dst_page, THREAD_UNINT, LCK_SLEEP_DEFAULT);
					/*
					 * need to relook the page up since it's
					 * state may have changed while we slept
					 * it might even belong to a different object
					 * at this point
					 */
					continue;
				}

Here vm_object_range_op redoes the lookup after sleeping; it does not trust its saved dst_page. vm_page_wakeup_done requires exclusive object locking, clears busy and invokes wakeup. Plain vm_page_wakeup clears wanted and wakes waiters; it does not itself clear busy, and allows an already detached page with a null object argument.

Rename changes identity without copying bytes

vm_page_rename takes queue protection, normalizes a local active-queue state when necessary, removes the page from the old object's indexes/accounting and inserts it into the new object at the supplied offset. Its called removal and insertion require the relevant objects exclusively locked. Internal/external pageable counts are adjusted when crossing that boundary. The physical identity and bytes need not change.

With the previous 16 KiB example, renaming (O, 0x8000) to (N, 0x4000) keeps physical page 0x81234; byte displacement 0x120 moves from logical O:0x8120 to N:0x4120. It is not a memcpy and does not itself change the CPU virtual addresses that map the frame. Collapse and related callers must establish that the reassociation preserves their logical contents; see collapse/bypass.

Wiring, unmapping and freeing are different transitions

Ordinary vm_page_wire requires queue locking and, for an associated page, exclusive object locking. The first wire removes pageable queue membership, establishes wired state and updates object/global/ledger accounting; each wire increments the count, with overflow checked. vm_page_unwire decrements it and only on the final wire removes wired accounting and optionally queues the page again. Neither operation is equivalent to installing/removing a PTE. IOPL wire state has additional rules, including the special absent/busy preparation described in the UPL chapter.

Physical reuse also requires the pmap side to stop referring to the frame. A simple visible path is vm_object_range_op with UPL_ROP_DUMP: it waits/relooks up busy or cleaning pages, steals pending laundry if necessary, disconnects prior mappings when vmp_pmapped is set, and only then calls VM_PAGE_FREE. The flag is a conservative history bit; the disconnect operation is what removes translations.

For batched purge, vm_object_reap_pages skips wired or ineligible pages, disconnects with deferred-flush options, and accumulates pages. It performs pmap_flush before releasing the final list, with the same ordering for intermediate batches. Intermediate batch flush. The exact mapping locks, reverse maps, TLB operations and backend boundary belong to pmap translation storage and TLB/cache maintenance.

Mermaid source

flowchart TD
    R["Resident page selected for release"] -->|"caller resolves busy, I/O and wire obligations"| D["Disconnect mappings when required"]
    D -->|"complete required TLB flush"| Q["vm_page_free_prepare_queues
remove queues and wire accounting"]
    Q -->|"vm_page_free_prepare_object"| O["Remove object/hash identity
wake wanted waiters"]
    O -->|"canonical descriptor"| C["pmap_recycle_page
reset reusable metadata"]
    C -->|"vm_page_release"| F["Publish canonical physical capacity
descriptor survives"]
    O -->|"private or fictitious descriptor"| P["Reset private alias to fictitious
release zone descriptor"]

This diagrams the obligations around the free helpers; it does not claim that vm_page_free itself performs the earlier disconnect or every wait. Caller-specific bulk and UPL paths can arrange those steps differently while preserving the required completion before reuse.

Object detachment and descriptor release

vm_page_free_prepare_queues requires queue ownership and exclusive locking of an associated object. It removes laundry and queue membership, adjusts wire/gobble/MTE/accounting state, and leaves the page out of those queues. vm_page_free_prepare_object then removes tabled membership, wakes waiters, resets private identity, and checks/resets canonical pages:

osfmk/vm/vm_resident.c lines 5881–5901:

vm_page_free_prepare_object(vm_page_t mem)
{
	if (mem->vmp_tabled) {
		vm_page_remove(mem);  /* clears tabled, object, offset */
	}
	vm_page_wakeup(VM_OBJECT_NULL, mem);               /* clears wanted */

	if (vm_page_is_private(mem)) {
		vm_page_reset_private(mem);
	}
	if (vm_page_is_canonical(mem)) {
		assert(mem->vmp_pageq.next == 0 && mem->vmp_pageq.prev == 0 &&
		    mem->vmp_listq.next == 0 && mem->vmp_listq.prev == 0 &&
		    mem->vmp_specialq.next == 0 && mem->vmp_specialq.prev == 0 &&
		    mem->vmp_next_m == 0);

		pmap_recycle_page(VM_PAGE_GET_PHYS_PAGE(mem));

		vm_page_reset_canonical(mem);
	}
}

Removal is more than setting vmp_object = 0: vm_page_remove removes hash and resident-list links, updates page/ledger/wire/reusable counts, may remove an emptied external object from its cache, resets cache attributes where needed, and finally clears tabled/object/offset. vm_page_remove_internal also repairs memq_hint when it pointed at the departing page.

On the inspected ARM backends, pmap_recycle_page checks that the frame is no longer referenced; it is not the general unmap-all operation. Legacy/PPL-side code verifies free state and panics otherwise. SPTM-side code additionally checks frame type and retypes selected frame categories to XNU_DEFAULT. The secure monitor's own implementation is outside this source boundary. Legacy/PPL recycle, SPTM recycle.

vm_page_reset_canonical restores unassociated/busy metadata while retaining physical identity and selected immutable/configuration state. vm_page_release checks canonical/unqueued state, optionally repeats the pmap free check, clears no-encrypt state and publishes the frame to free capacity. It does not zfree the canonical descriptor or unconditionally zero the data bytes.

The final vm_page_free branch sends canonical pages to vm_page_release, and noncanonical descriptors to vm_page_release_fictitious. The latter releases zone metadata and guard accounting. vm_page_free_unlocked acquires/releases queue protection internally, but does not mean “the associated object can be unlocked.” In this snapshot VM_PAGE_FREE expands directly to that wrapper.

Configuration boundaries and reading route

XNU_VM_HAS_LINEAR_PAGES_ARRAY changes physical identity storage; HAS_MTE changes tagged pools, descriptor state and recycle obligations; SPTM changes protected-frame ownership operations; delayed, low-page and secluded pools change free-capacity selection. The small descriptor is therefore not a complete description of all physical-memory metadata. MTE tag-storage cells, pmap reverse mappings and protected frame types have separate representations and synchronization. Follow MTE storage and pmap structures before drawing conclusions from one flag word.

When following a page through a source operation, write down its category, physical page number, object/offset pair, queue state, busy/absent/cleaning flags and wire count. At each lock drop or callback, identify what keeps the descriptor and association stable or where the code redoes the lookup. At release, separately find data preservation/discard, translation removal and flush completion, object/hash detachment, then physical-capacity or metadata-zone publication. This chapter documents those local source contracts; it does not simulate hardware or missing provider bodies.