DMA preparation, mapping ownership, and failure cleanup

Drivers · Memory introduction · VM wiring · UPLs

A pageable DMA transfer has at least three lifetimes: VM backing must remain available, the device must have suitable addresses, and any CPU mapping must remain valid for its users. IOGeneralMemoryDescriptor, IODMACommand, and IOMemoryMap implement these separately. A successful CPU mapping does not substitute for page preparation; a DMA command does not directly establish every page hold; a command's mapping may be cached by its descriptor instead of released at that command's completion. descriptor preparation, command setup, mapping cache, CPU map cleanup.

The selected trace here starts with virtual memory already described by a descriptor, acquires its UPL-backed preparation, prepares device segments, and unwinds command and descriptor state. Physical, externally supplied UPL, remote, mapper-less, and noncoherent branches are called out where they change that contract. Hardware command rings and a concrete IOMMU backend are outside this generic code's implementation.

For the generation algorithm itself, continue with the DMA segment walkthrough: descriptor dispatch, mapped versus physical coordinates, coalescing and splitting, bounded continuation, output formatting, and errors that do not roll back internal state.

1. Read the state as several ownership ledgers

The public command contract explicitly allows preparation to block and warns against automatic preparation while holding a workloop gate. It also says prepare() is not thread-safe for simultaneous use of one command. These are caller obligations; retaining a descriptor does not make command mutation concurrent-safe. IODMACommand.h, gate execution.

2. Descriptor prepare: direction, UPL construction, and rollback

IOGeneralMemoryDescriptor::prepare() returns success immediately for physical descriptors. It rejects the local path for remote descriptors. For virtual/UIO/vnode types, it optionally acquires _prepareLock, handles the avoid-throttling option, calls wireVirtual(), and increments _wireCount only on success. First successful preparation performs the additional encryption/trace work; another successful preparation extends the interval rather than independently rebuilding all pages. IOMemoryDescriptor.cpp.

IOMemoryDescriptor.cpp, lines 5050–5061

		error = wireVirtual(forDirection);
	}

	if (kIOReturnSuccess == error) {
		if (1 == ++_wireCount) {
			if (kIOMemoryClearEncrypt & _flags) {
				performOperation(kIOMemoryClearEncrypted, 0, _length);
			}

			ktraceEmitPhysicalSegments();
		}
	}

wireVirtual() derives VM and device permissions from the direction. For kIODirectionOut, the device reads host memory, so the code requests UPL_COPYOUT_FROM and marks read-only DMA preparation. Device-input direction requires writable handling. A nested attempt cannot silently upgrade an already read-only preparation to writable. Flags additionally select no-fault/no-zero-fill/noncoherent behavior and low-address requirements, including the no-mapper physical-address limit branch. IOMemoryDescriptor.cpp.

For a first preparation, the routine arranges storage for page information and chooses the source map or memory reference. It walks the descriptor's ranges and splits them into UPL-sized requests, preserving the offset of useful bytes inside the first page. Depending on the representation it uses memory_object_iopl_request(), the static-kernel UPL helper, or vm_map_create_upl(). An ioPLBlock records the UPL, page-list position, descriptor offset, and page offset; device-memory handling may represent a physically contiguous range differently from ordinary page information. The highest observed page feeds later addressing decisions. IOMemoryDescriptor.cpp.

Failures must undo both the just-created request and previously recorded work. If recording a newly obtained UPL fails, the code aborts/deallocates that UPL. The common abort path then visits prior blocks, aborts/deallocates their UPLs, truncates the descriptor's memory-entry storage, and translates VM failures to IOKit statuses. The outer prepare() does not increment _wireCount for that failed attempt. This is the relevant contrast with the command's later activity counter, whose error behavior is different. IOMemoryDescriptor.cpp, success-only increment.

3. Final descriptor completion owns the backing teardown

complete() handles physical and remote cases separately and uses the same optional preparation lock. It records completion error state, handles the final encryption transition, and decrements _wireCount. An intermediate completion can mark data valid when requested while leaving the shared preparation intact. On final completion, active DMA references are an invariant failure: backing pages must not be released while a command still accounts for device use. IOMemoryDescriptor.cpp.

The final path removes a descriptor-cached device mapping, then handles UPL ownership by descriptor type. Created virtual/UIO/vnode UPLs are aborted when completion reported error or committed otherwise, and deallocated. An external kIOMemoryTypeUPL instead clears the UPL's referenced state; this branch does not claim ownership of creating and destroying that external UPL. The descriptor then truncates temporary memory-entry storage, resets its preparation ID, and clears read-only-prepared state. IOMemoryDescriptor.cpp.

IOMemoryDescriptor.cpp, lines 5151–5165

				// Only complete iopls that we created which are for TypeVirtual
				if (kIOMemoryTypeVirtual == type || kIOMemoryTypeVirtual64 == type || kIOMemoryTypeUIO == type || kIOMemoryTypeVnode == type) {
					for (ind = 0; ind < count; ind++) {
						if (ioplList[ind].fIOPL) {
							if (dataP->fCompletionError) {
								upl_abort(ioplList[ind].fIOPL, 0 /*!UPL_ABORT_DUMP_PAGES*/);
							} else {
								upl_commit(ioplList[ind].fIOPL, NULL, 0);
							}
							upl_deallocate(ioplList[ind].fIOPL);
						}
					}
				} else if (kIOMemoryTypeUPL == type) {
					upl_set_referenced(ioplList[0].fIOPL, false);
				}

A conceptual successful transfer therefore stops the device first, completes and clears the command's DMA use, then completes the backing descriptor. This ordering is derived from the command's DMA-reference adjustments and the descriptor's final-reference assertion; it is not a measurement of a particular controller. A device-specific abort/reset path must establish that the device has stopped using those addresses before entering this generic resource release sequence. command clear, DMA unmap accounting, descriptor final completion.

4. Attaching a descriptor to a reusable command

setMemoryDescriptor() refuses to replace a different descriptor while fActive is nonzero. With the same descriptor, autoPrepare == false drains outstanding command activity and returns; with autoPrepare == true, that same-descriptor branch returns without preparing again. For a new descriptor, it first queries characteristics, computes whether physical-address checking is needed, retains the descriptor, and records a new-source flag. Thus this setter has more semantics than an unconditional assignment followed by an unconditional prepare. IODMACommand.cpp.

When there is no mapper, the setter records DMA-active usage immediately through kIOMDSetDMAActive, before optional command preparation. clearMemoryDescriptor() drains fActive with complete(), clears that no-mapper activity record, and releases the descriptor. If asked not to auto-complete while active it returns kIOReturnNotReady. Mapper-backed command activity is instead accounted through map/unmap records. IODMACommand.cpp, descriptor DMA accounting operations.

IODMACommand.cpp, lines 437–448

		fInternalState->fNewMD = true;
		fMemory.reset(const_cast<IOMemoryDescriptor *>(mem), OSRetain);
		fInternalState->fSetActiveNoMapper = (!fMapper);
		if (fInternalState->fSetActiveNoMapper) {
			mem->dmaCommandOperation(kIOMDSetDMAActive, this, 0);
		}
		if (autoPrepare) {
			err = prepare();
			if (err) {
				clearMemoryDescriptor();
			}
		}

Automatic preparation failure calls clearMemoryDescriptor() in this wrapper. A direct caller of prepare() must not assume every error leaves the command unchanged: after checking output format and maximum transfer size, the body evaluates fActive++. A nested mismatched range returns NotReady after incrementing it, and the first-use length-alignment failure also occurs after the increment. Later resource/setup errors have no single rollback label restoring an initial inactive state. fPrepared is only set on the successful final path. This is an implementation observation, not a recommendation to use a partially prepared command. IODMACommand.cpp, prepare completion, automatic cleanup.

IODMACommand.cpp, lines 855–863

	if (fActive++) {
		if ((state->fPreparedOffset != offset)
		    || (state->fPreparedLength != length)) {
			ret = kIOReturnNotReady;
		}
	} else {
		if (fAlignMaskLength & length) {
			return kIOReturnNotAligned;
		}

The descriptor's source segment walker separately enforces that nonphysical backing is wired. Command preparation is not an implicit replacement for IOGeneralMemoryDescriptor::prepare(). A caller evaluating a failure must track which preparation stage was entered and which wrapper owns cleanup, rather than use one undifferentiated “prepared” boolean for both objects. IOMemoryDescriptor.cpp, IODMACommand contracts.

5. Direct iteration, sparse bounce pages, and full copying

First command preparation records its requested range and clears old copy/mapping state. It derives a source-alignment mask, relaxing the relevant part when a mapper can supply aligned device addresses. fCursor selects the iterate-only/direct path when requested or when addressing/alignment evidence allows it. Otherwise walkAll() preflights the source, adding synchronization-out work when requested. This decision depends on physical/address constraints and source alignment, not solely on whether a mapper pointer is null. IODMACommand.cpp.

segmentOp() detects address and alignment violations and counts pages requiring substitution. Misalignment or a forced copy can select full double buffering. For address-limited memory without a mapper, walkAll() first tries to allocate a sparse low-memory page list with nonwaiting page allocation. If it can do that, it records original physical pages in those temporary page records and uses them to substitute only the affected portions. If it cannot, or full buffering is required, it creates a copy-buffer memory descriptor for the prepared range. Resource failure reports NoResources; this is not guaranteed to succeed just because the original descriptor is wired. IODMACommand.cpp, walkAll.

Synchronization-out copies host data into bounce storage for applicable output direction; synchronization-in copies results back for applicable input direction. Sparse pages use segment/page copying, while a full copy descriptor uses readBytes()/writeBytes() and checks the returned byte count. Completion releases sparse pages and resets the copy descriptor. Noncoherent command preparation additionally performs a store/cache operation on the copy descriptor or original range; final command completion can perform the corresponding flush/invalidate operation before copy-back. These generic calls do not specify every hardware cache instruction emitted by a platform. IODMACommand.cpp, prepare cache work, complete.

6. Device mapping can belong to the command or the descriptor

The command's mapper branch constructs IOMDDMAMapArgs with device identity, alignment, address-bit constraint, requested length, and either the original prepared offset or zero into a copy descriptor. It invokes the descriptor's kIOMDDMAMap operation. When an allocation covers a different layout/length from the requested byte range, the command can build a segment array accounting for mapper page padding across source segments. That array has its own allocation-failure path. Successful local mapping state records the allocation, length, and validity for final completion. IODMACommand.cpp.

On the descriptor side, the map operation accumulates stricter address/alignment requirements and distinguishes a reusable full-range mapping through the descriptor's system mapper from a command-owned mapping. Under its preparation lock where applicable, it can reuse a cached mapping or create a new one. For a new cached mapping it transfers allocation ownership to descriptor state and reports zero allocation length to the command; zero length does not mean no mapping or no DMA accounting exists. IOMemoryDescriptor.cpp.

IOGeneralMemoryDescriptor::dmaMap() handles host-only and remote branches, chooses the general fallback for physical/subrange cases, and otherwise builds IODMAMapPageList from prepared page information. The flags include whether pages are externally supplied, whether the range occupies full pages, read-only preparation, and contiguous device-memory representation. It then calls mapper->iovmMapMemory() and records successful mapping/accounting. The concrete mapper decides the device translation implementation; the generic descriptor is not itself an IOMMU page-table backend. IOMemoryDescriptor.cpp, prepared page-list mapping.

dmaUnmap() decrements command DMA references before its zero-allocation-length early return. A command using a descriptor-owned cached mapping can therefore release its active-use accounting without freeing the shared mapping. The descriptor's final completion later removes that mapping with a null command argument. For a nonzero local allocation, unmapping also invokes the mapper and adjusts allocation accounting. IOMemoryDescriptor.cpp, cached mapping release.

The return-value path also deserves literal reading: the generic kIOMDDMAUnmap dispatch calls dmaUnmap() but returns success in that dispatch branch. Likewise, command preparation can assign a later mapping result after earlier work. These bodies should not be summarized as an abstract transaction that automatically propagates every intermediate failure and rolls back every stage. Follow the actual branch and ownership fields when diagnosing a failure. IOMemoryDescriptor.cpp, command preparation.

7. Segment generation advances a bounded cursor

genIOVMSegments() requires active command state and valid output/offset/count arguments. Its input offset is relative to the prepared subrange; internally it adds fPreparedOffset. It rejects exhaustion and resets the saved cursor when the requested position is discontinuous or the descriptor changed. It can source addresses from a local mapper allocation, an allocation's segment array, a copy descriptor, or the original descriptor's DMA segment walker. Device-visible segment generation is therefore not simply virtual-to-physical translation of one CPU pointer. IODMACommand.cpp.

The loop coalesces contiguous input where allowed, handles addressability and sparse substitution, and splits output to satisfy maximum segment size, segment-length alignment, and following-segment alignment. Cursor-only operation detects alignment it cannot repair and reports NotAligned. The output callback can also reject a segment. Only on success does the function commit its saved offset and caller's output offset/count. This permits repeated bounded calls with a small output array while preserving explicit error reporting. IODMACommand.cpp.

IODMACommand.cpp, lines 1626–1631

	if (kIOReturnSuccess == ret) {
		state->fOffset = offset;
		*offsetP       = offset - internalState->fPreparedOffset;
		*numSegmentsP  = segIndex;
	}
	return ret;

A conceptual example is an 8-KiB prepared subrange split into two device segments because of a configured segment-size limit. A two-element output array may describe the range in one call; a one-element array requires another call with the returned relative offset. The example is illustrative rather than observed output: real segmentation also depends on physical continuity, mapper allocation, alignments, and output function. The source's offset/count updates, rather than assumed physical contiguity, define how the caller continues. IODMACommand.cpp.

8. Command completion and CPU maps remain separate

Final IODMACommand::complete() first decrements fActive; nested completions do not immediately tear down shared command resources. At zero it saves the copy descriptor, performs configured noncoherent invalidation and synchronization-in/copy cleanup, unmaps valid command-owned mapping state, clears allocation fields and any segment array, and clears fPrepared. Saving the copy descriptor preserves the object needed for subsequent unmapping even if walkAll(kWalkComplete) clears the command's copy-buffer field. IODMACommand.cpp.

Mermaid source

sequenceDiagram
    participant D as Driver
    participant M as Descriptor
    participant V as VM UPLs
    participant C as DMA command
    participant I as Mapper or bounce storage
    D->>M: prepare pageable source
    M->>V: acquire and record UPLs
    D->>C: setMemoryDescriptor and prepare
    C->>M: characteristics and DMA map operation
    M->>I: map pages or reuse cached mapping
    C-->>D: constrained device segments
    Note over D,I: Driver runs and stops the hardware transfer
    D->>C: complete and clearMemoryDescriptor
    C->>I: synchronize, release local resources
    C->>M: release command DMA references
    D->>M: complete final backing preparation
    M->>I: release descriptor-cached mapping
    M->>V: commit or abort and release owned UPLs

A CPU IOMemoryMap has an independent lifecycle. makeMapping() runs under the global recursive memory lock, can reuse a compatible map, create a submap retaining a supermap, or call doMap() for a new mapping. Compatibility includes target map/task, options, range containment, and address constraints; it is not merely descriptor-pointer equality. For a new general mapping, doMap() can reuse the original same-task range as a static mapping under specific options or create a memory reference and VM mapping. The copy-on-write memory-reference setup is attempted before the global mapping lock because it may allocate pages. IOMemoryDescriptor.cpp, compatibility, makeMapping.

unmap() removes an actual VM mapping only when this map owns that operation: an address/map must exist, and it must not be a supermap-backed or static mapping. It still drops its VM-map reference. taskDied() follows its own invalidation path. The class's release threshold accounts for the descriptor's mappings-set reference; free() unmaps, removes the map from that set, releases descriptor/supermap ownership, and completes any redirection UPL. Removing a CPU view does not itself complete a DMA command or prove a device has stopped reading its IOVA. IOMemoryDescriptor.cpp, VM mapping ownership.

Sources inspected and limits

Deep review covered virtual descriptor preparation and UPL error rollback, nested/final completion, external-UPL distinctions, command descriptor replacement/automatic failure cleanup, direct/copy preflight, selected sparse/full bounce paths, mapper ownership transfer and DMA accounting, segment splitting/cursor updates, final command cleanup, and CPU map reuse/unmap/reference release. The concrete IOMapper backend, every constructor/redirected-map path, all physical contiguity/page-size combinations, and controller stop/error semantics remain boundaries. No DMA transfer or IOMMU experiment was run.

For source reading, begin with one successful virtual preparation and one failed UPL recording, then compare automatic command-prepare failure against direct nested prepare failure. Track _wireCount, fActive, fPrepared, _dmaReferences, and mapping ownership independently at each return; those fields explain more than the shared method name prepare.