One Skywalk channel operation: descriptors, mappings, packets and readiness

Execution paths · Completion and failure · Networking · IOKit

A channel operation crosses BSD descriptor allocation, Skywalk topology, IOKit memory mapping, shared packet metadata and event delivery. This walkthrough follows one concrete user-pipe scenario through those boundaries, keeping the same objects visible throughout. The pipe implementation chapter supplies the full peer-reference and ring algorithms; this chapter explains how a caller reaches them and which result each boundary establishes.

The source example is skt_fullupipe_main() in xnu-12377.121.6. It opens both endpoints in the same test process, fills the sender twice, and demonstrates receiver-driven progress after backpressure. Its initialization selects an anonymous user-pipe nexus, one TX and one RX ring per endpoint, 100 slots per ring, and reject-on-close. It exercises slot movement rather than checking a meaningful network payload. No NIC, IP route, TCP connection or device DMA is involved in this selected transport. Configuration, endpoint creation.

1. Name every identity before tracing the first packet

The test helper creates C, registers P, allocates N and reads back negotiated geometry; it passes N's UUID to the channel test. The provider registration and instance allocation are separate operations, and their errors precede any channel transfer. Anonymous mode changes binding admission; it does not remove controller/channel policy checks. See setup, provider construction, controller wrappers, and controller authority.

Mermaid source

flowchart TB
    U["Process: channel A and channel B handles"] -->|guarded fds| F["BSD fileproc / fileglob"]
    F -->|fg_data| C["Two kern_channel objects"]
    C -->|nexus reference| N["Nexus N: user-pipe instance"]
    C -->|selected rings| R["TX A paired with RX B"]
    R -->|packet attachments| M["Arena M: metadata and buffers"]
    C -->|mapping record| I["IOSKMapper and region IOMemoryMaps"]
    I -->|task virtual mappings| M

The diagram shows relationships, not a reference-count equation. The channel, adapter, parent and arena own different references. Use the pipe object graph when following their exact release counts.

2. Open publishes several things before the application gets a handle

os_channel_create_extended() constructs ch_init with N's UUID, endpoint port, ring selection and requested modes. It calls __channel_open(), then __channel_get_info(), and finally allocates/initializes its userspace handle. Consequently a successful kernel open can still be followed by a failed userspace construction. On that failure the wrapper attempts to close the already-opened guarded fd and discards the close result. Wrapper construction, wrapper cleanup.

Inside the open syscall:

1. Copy/validate ch_init, including version, nexus UUID and consistency of the optional key pointer/length pair.
2. Reserve a guarded fd using falloc_guarded(). The guard restricts close, duplication, socket IPC, fileport and write operations. Only afterward does the body check the key-length limit, allocate temporary key storage and copy the key bytes.
3. ch_open() resolves N and the endpoint, admits the requested ring selection, obtains an adapter and connects the channel. The first pipe endpoint constructs its peer and the parent arena.
4. Common adapter connection maps the arena into proc_task(p) and sets the schema's CSM_ACTIVE.
5. Copy the generated guard and normalized initialization record back to userspace.
6. Install channel_ops, attach the channel to the fileglob, release the fd reservation and return the fd.

Guard allocation and key copying, open/copyout/publication, adapter map/active publication, pipe construction.

The fd has channel-specific operations: ordinary read, write and ioctl do not perform packet transfer. select and kqueue filtering do have channel implementations. Transfer enters through the channel synchronization syscall and mapped ring fields. Operation table.

If initialization copyout fails, the syscall destroys the provisional channel and frees the reserved fileproc. Thus the task mapping and active schema may have existed before the open ultimately reports failure. This cleanup differs from the later wrapper failure, which closes an already-published fd. Kernel rollback.

3. Skywalk reaches VM through an actual IOKit mapping implementation

skmem_arena_mmap() holds the arena lock, checks that it is active and not defunct, calls IOSKMapperCreate(ar->ar_ar, proc_task(p)), obtains the mapped base/extent and records the mapping in the arena's list. The successful record retains the arena and increments its map count. This is a CPU task mapping; the word “mapper” here does not imply an IOMMU device-address mapping. Arena mapping.

The implementation is present in iokit/bsddev/skywalk/IOSkywalkSupport.cpp. IOSKMapper reserves a task virtual interval with mach_vm_map_kernel(), VM_PROT_NONE and VM_INHERIT_NONE. Region mappers overlay populated segment buffers with createMappingInTask() and kIOMapOverwrite; buffers marked not user-writable add kIOMapReadOnly. An arena's total reserved extent therefore need not be populated or uniformly writable. Reservation, segment mapping.

The address conversion is explicit in the region mapper. Exact source, IOSkywalkSupport.cpp lines 387–396:

	if (buffer->fSpec.user_writable == FALSE) {
		options |= kIOMapReadOnly;
	}

	offset = fRegionOffset + (segIndex * fRegion->getSegmentSize());
	assert((offset + fRegion->getSegmentSize()) <= fMapper->fMapSize);
	addr = fMapper->fMapAddr + offset;

	map = buffer->createMappingInTask(fMapper->fTask, addr, options);
	fMemoryMaps[segIndex] = map;

Userspace receives cinfo_mem_base and cinfo_schema_offset; their sum locates this channel's schema. Ring and slot-table offsets are relative to that schema. Buffer/metadata offsets in a ring are relative to the ring. These bases must not be interchanged. Schema construction, ring initialization.

For an illustrative address calculation, let mapping base B = 0x100000000, schema offset S = 0x4000, and a schema-relative ring offset R = 0x8000. The schema is at 0x100004000 and that ring at 0x10000c000. A ring-relative buffer-base offset D = 0x20000 gives 0x10002c000. These are chosen virtual byte addresses, not reported test values or physical addresses; no PTE/page-size calculation is implied. Another mapping of M can have a different B and a different channel schema. Packet identities/offsets, rather than a pointer copied from A, establish the receiver's view.

For page and descriptor ownership below this bridge, continue with Skywalk arena mapping and IOKit CPU mappings. The separate DMA operation follows device-visible addresses.

4. Publishing the user head does not yet transfer a slot

The test iterates TX A's 99 initially available slots, obtains/sets slot properties, and advances through the final slot. os_channel_advance_slot() writes only the mapped ring_head. It does not execute the kernel TX prologue. Test preparation, head update.

os_channel_available_slot_count() calculates TX availability using the mapped ring_head, published ring_khead and ring_num_slots; RX availability uses ring_tail - ring_head, modulo the slot count. It returns zero for a defunct schema. The same zero result therefore needs context: full TX, empty RX and defunct operation are different conditions. Actual calculations.

os_channel_sync(A, CHANNEL_SYNC_TX) enters __channel_sync(). The kernel takes the ring synchronization right, validates/imports user submissions, invokes ckr_na_sync, and finalizes the mapped TX state only on callback success. The pipe callback sees validated ckr_rhead, not arbitrary newly written user state. Wrapper, TX dispatch.

For this pipe, the callback bounds transfer by pending TX slots and free RX capacity. It swaps the kernel/user slot descriptors and fixes packet back-pointers through SD_SWAP; bytes remain in existing arena buffers. A memory fence orders those attachment updates before publishing the kernel RX tail. Capacity and transfer, actual swap macro.

The publication boundary is visible in this excerpt. Exact source, nx_user_pipe.c lines 956–964:

	/*
	 * Make sure the slots are updated before ckr_ktail reach global
	 * visibility, since we are not holding rx ring's kr_enter().
	 */
	os_atomic_thread_fence(seq_cst);

	rxkring->ckr_ktail = j;
	txkring->ckr_khead = k;
	txkring->ckr_ktail = SLOT_PREV(k, lim_tx);

This moves Q from TX A's attachment to RX B's attachment, with the prior RX packet becoming reusable TX storage. It does not instantly refresh every userspace ring view. Generic RX synchronization later publishes B's readable tail and externalizes the metadata. See RX finalize and the pipe publication stages.

5. Follow the test's two batches through backpressure

The table follows the source's assertions with N = 100 and usable capacity N - 1 = 99. “Visible” means what the userspace helper returns at that point, not a count read directly from kernel krings.

First submission and RX-before-TX, second submission and full receiver, receiver-driven transfer and final TX view.

The RX callback first reclaims released slots, computes free capacity and reads the peer's validated pending count. With both pending work and space, it takes the peer TX synchronization right and calls nx_upipe_na_txsync_locked() itself. Reclaim and pending check, internal TX call and notification.

Mermaid source

sequenceDiagram
    participant A as Sender A
    participant T as Kernel TX A
    participant R as Kernel RX B
    participant B as Receiver B
    A->>T: TX sync validates first batch
    T->>R: Swap 99 packet attachments
    T-->>A: Publish recovered TX capacity
    A->>T: TX sync validates second batch
    T-->>A: Success with no RX capacity
    B->>R: RX sync publishes first batch
    R-->>B: 99 readable slots
    B->>R: Advance consumed slots and RX sync
    R->>T: Reclaim capacity and request internal TX
    T->>R: Transfer validated second batch
    R-->>B: Publish next 99 readable slots
    A->>T: Later TX sync
    T-->>A: Publish current TX capacity

The test's comment after the first TX sync calls the RX queue “full,” while its assertion still expects zero visible RX slots. Those statements refer to different representations. Reading the comment alone loses the central point of the test. Exact assertions, skt_fullupipe.c lines 142–146:

	/* The txqueue is now empty and the rx queue is full */
	txavail0 = os_channel_available_slot_count(txring0);
	assert(txavail0 == NSLOTS - 1);
	rxavail1 = os_channel_available_slot_count(rxring1);
	assert(rxavail1 == 0);

6. Readiness can perform synchronization

The explicit-sync test above does not register a kqueue. A caller using readiness reaches the same ring machinery through the descriptor's chop_select() or chop_kqfilter() instead. This is a separate source trace, not an extra assertion attributed to skt_fullupipe. BSD dispatch.

ch_event() can run RX prologue, the domain sync callback and finalize while checking readiness. If nothing is found, the select/poll path registers its wait interest and retries the scan to cover the intervening arrival. Consequently checking readiness can publish newly arrived slots, reclaim released capacity and trigger the receiver-driven transfer above. A wait registration by itself does not copy packets. RX scan and retry.

Notification reaches csi_selwakeup(): it wakes select waiters when requested and posts knotes when present, avoiding a recursive knote post when already inside kevent handling. The kqueue process callback then runs ch_event() under the channel lock and applies the effective slot/byte low-water mark. Error readiness is reported using EV_EOF in this callback. Wake/post, process and threshold.

A readable event therefore reports an observed condition after synchronization; it does not reserve Q for the notified thread or prove that thread has run. Track scheduler admission separately using wait/wakeup protocols and kqueue processing.

7. Close unwinds the fd, mapping and peer graph separately

os_channel_destroy() attempts guarded close, ignores its result and frees its userspace handle/metadata. Its void return is not a report of guarded-close success. On the successful kernel close route, chop_close() clears the fileglob's data pointer before invoking ch_dtor(). Wrapper, fileglob handoff.

Channel destruction marks CHANF_CLOSING, clears its fd identity and disconnects it. Common adapter unbinding decrements channel users and relinquishes ring-use rights. It invokes adapter teardown on the last channel, frees this channel's schema, destroys its task mapping and releases its adapter reference. The pipe-specific ring teardown decides whether peer-held rings must survive; it cannot be replaced with an unconditional “close frees the arena.” Channel close, unbind ordering, peer retention and ring teardown.

Mapping destruction crosses back into IOKit: IOSKMapperDestroy() removes the mapper from its arena and releases the object; IOSKMapper::free() releases region mappings and calls mach_vm_deallocate() for the reserved task virtual interval. It discards that call's result before clearing its bookkeeping, so destruction supplies no separate successful-unmap acknowledgment. The remaining endpoint and parent/arena references have their own lifetime. Mapper destroy, mapper free, Skywalk map-count/reference removal.

Mermaid source

flowchart TB
    C["Successful guarded fd close"] --> F["Clear fileglob channel pointer"]
    F --> D["Mark closing and disconnect channel"]
    D --> R["Release ring use and channel schema"]
    R --> M["Destroy this task mapping"]
    M --> A["Drop adapter and nexus references"]
    A --> P{"Other owners remain?"}
    P -->|yes| K["Retain required rings / arena backing"]
    P -->|no| X["Final destructors release backing"]

This is a resource summary; domain teardown is called during unbind before the final mapping/reference drops. The common and pipe-specific bodies cited above determine the exact interleaving.

The test closes both channel handles; its separate cleanup then frees N, deregisters P and destroys C. The peer-closure variant checks a rejected later sync when reject-on-close is selected. Logical rejection, mapping removal and final storage release are therefore separate observations. Test close, peer-closure case, control cleanup.

8. Failure checkpoints and evidence limits

The example joins actual local test, wrapper, syscall, IOKit and pipe bodies. Numeric virtual addresses are illustrative; the 100-slot count and visible-availability sequence come from source assertions. No test was executed. SKYWALK/CONFIG_NEXUS_USER_PIPE, admission policy and the Darwin test environment remain prerequisites. This operation does not establish NIC/DMA behavior, flowswitch/IP delivery, cross-process memory isolation, payload integrity or all teardown races. Use the diagnostics/test chapter for harness limits, and compare the mapped-file operation and DMA operation for different VM and completion contracts.