Skywalk user and kernel pipes

Networking · Controllers, providers and ports · Channel ABI and readiness · Packet pools · Memory arenas

A Skywalk user pipe transports packets between two user-facing adapters by exchanging slot descriptors inside a shared memory arena. A kernel pipe presents a user channel to callbacks supplied by a kernel provider. The two domains use the same channel ABI but implement different topology, packet movement and teardown. This chapter traces those mechanisms in xnu-12377.121.6, including a ring-index example, provider callbacks and a local utun consumer.

These are packet channels, separate from BSD byte-stream pipes and UNIX sockets. Neither pipe domain intrinsically performs IP routing, transport reliability, encryption or flowswitch classification. A provider can connect a kernel pipe to such machinery, but that behavior comes from its callback bodies.

IPsec interface lifecycle follows another concrete consumer through interface control, netif/kernel-pipe packet movement, transforms and disconnect barriers. It distinguishes those provider operations from the generic pipe protocol explained here.

1. Endpoints, rings and memory are different objects

Sources: port constants, user-pipe state, kernel-pipe adapter, user connection selection, kernel connection selection.

The user-pipe fields that implement peer lifetime are small enough to inspect directly: nx_user_pipe.h:70–78.

	uint32_t        pna_id;                 /* pipe identifier */
	ch_endpoint_t   pna_role;               /* master or slave */

	struct nexus_adapter *pna_parent; /* adapter that owns the memory */
	struct nexus_upipe_adapter *pna_peer; /* the other end of the pipe */
	boolean_t pna_peer_ref; /* 1 iff we are holding a ref to the peer */

	uint32_t pna_parent_slot; /* index in the parent pipe array */
};

pna_peer establishes identity; pna_peer_ref says whether this endpoint currently owns a reference keeping the other endpoint alive. They are not interchangeable. Likewise, a task's mapped ring, the kernel's __kern_channel_ring, a slot descriptor and the packet attached to that slot are distinct objects. The channel ABI chapter describes their mapped layouts.

2. User-pipe creation fixes geometry and admits each side

Provider parameters and the one-pair connection path

The domain defaults to one TX and one RX ring per endpoint, 128 slots per ring and 2 KiB buffers. Its bounds allow 2–4096 slots and 1–16 KiB buffers. User-pipe parameter adjustment requires equal ring counts in the two directions; it doubles temporary counts used to size the arena for both endpoints. It does not double the nxp_tx_rings/nxp_rx_rings exposed by each endpoint. The common adjustment code first copies those fields into local geometry variables and passes pointers to those locals to the domain adjustment callback. Defaults/bounds, domain adjustment, temporary geometry.

TX and RX slot counts need not be identical: the transfer loop uses independent limits and modulo arithmetic for the two rings. A request for more rings does not create more public ports. nx_upipe_dom_connect() selects the role from the port, forces RING_SET_DEFAULT, and sets cr_pipe_id = 0. The parent adapter's generic pipe array can hold up to 64 pipe identifiers, but that capacity is not evidence that this domain connection API lets a user choose 64 independent pairs. Connection normalization, generic pipe array, maximum.

The channel selects all rings or one ring ID through the common na_interp_ringid() path. Multiple channels can refer to the same endpoint adapter through distinct selected ring IDs. The earlier ch_open() gate rejects overlapping {port, ring_id} ownership, including an all-rings request overlapping any existing selection. na_krings_use() also checks per-ring users and exclusive flags. Thus “two endpoints” does not mean exactly two descriptors, nor does opening another descriptor manufacture independent ring indices. Conflicting admission returns EBUSY. Ring selection, port/ring ownership, adapter ring admission.

Bindings and simultaneous endpoint construction

A named user-pipe provider needs the appropriate client/server binding and a matching supplied nxbind. nx_upipe_dom_bind_port() moves the prepared binding into nup_cli_nxb or nup_srv_nxb; duplicate binding returns EEXIST, and an unsupported port returns EDOM. Unbind removes that binding independently of adapter lifetime. This domain's two private binding pointers should not be mistaken for the generic flowswitch port-table association described in the controller chapter. Bind/unbind, credential comparison.

Anonymous providers bypass this binding comparison, while the surrounding nexus/channel admission checks still apply. The binding identity rules explain PID-derived unique identity, executable UUID and key matching; a port number alone is not that authority.

Under SK_LOCK, nx_upipe_na_find() performs the following construction:

1. Find or create the pseudo parent adapter and its arena, keeping the nexus-private parent reference.
2. Search the parent's array by pipe ID. If a pair exists, select its master or peer by requested role and retain it.
3. Otherwise allocate both endpoints, install the user-pipe synchronization/lifetime callbacks, borrow and retain the parent's arena with NAF_MEM_LOANED, and copy the normalized ring geometry.
4. Store the master in the parent's array, link the endpoints, and give each endpoint a parent reference. The requested endpoint retains its initially unopened peer and sets pna_peer_ref.
5. Return a retained requested adapter to common channel binding. nup_pna_users counts the two endpoint objects, not open descriptors.

See parent lookup, master/arena setup, peer and reference publication, and returned reference/error cleanup. Either public side can open first: “master” identifies the server role, not the first process to open a channel.

Common na_bind_channel() creates rings on the first channel, checks ring usability, creates that descriptor's schema and activates the adapter. na_connect() then maps the arena into the task and publishes CSM_ACTIVE; syscall fd publication happens later. A failure during ring setup cleans up initialized rings; failure after schema creation uses na_disconnect() to unmap/release the provisional channel. Binding and rollback, mapping and rollback, fd publication.

3. Paired rings exchange attachments within one arena

At the first open, nx_upipe_na_rings_create() allocates both endpoints' krings, user rings and slot storage, saves the user-ring pointers in ckr_save_ring, then cross-links corresponding directions. Failure setting up the second side tears down the first. The two sides do not share a single producer/consumer index structure. Setup and rollback.

The exact cross-link is nx_user_pipe.c:1131–1141:

	/* cross link the krings */
	for_rx_tx(t) {
		/* swap NR_TX <-> NR_RX (skip host ring) */
		enum txrx r = sk_txrx_swap(t);
		for (i = 0; i < na_get_nrings(na, t); i++) {
			NAKR(na, t)[i].ckr_pipe =
			    NAKR(&pna->pna_peer->pna_up, r) + i;
			NAKR(&pna->pna_peer->pna_up, r)[i].ckr_pipe =
			    NAKR(na, t) + i;
		}
	}

Mermaid source

flowchart TB
    P["Pseudo parent adapter"] -->|owns arena| A["Shared arena: packets, buffers, rings and metadata"]
    S["Server adapter: master"] -->|retains and borrows| A
    C["Client adapter: slave"] -->|retains and borrows| A
    S -->|contains| ST["Server TX ring i"]
    S -->|contains| SR["Server RX ring i"]
    C -->|contains| CT["Client TX ring i"]
    C -->|contains| CR["Client RX ring i"]
    ST <-->|ckr_pipe identity link| CR
    CT <-->|ckr_pipe identity link| SR

The links are kernel pointers. They do not transfer packets by themselves. Separate channel schemas and task virtual mappings describe views into the common arena; a packet buffer's address in one task is not a transferable pointer into the other task. The domain creates its arena without storing pools on the nexus because user pipes are not exposed as external kernel-provider clients. Arena construction, borrowed arena references, per-channel schema offsets.

User-pipe ring setup prepopulates both TX and RX slots with packets. In contrast, ordinary kernel-pipe RX slots start without attached packets; the population helper explicitly skips them. The user-pipe TX prologue validates and internalizes submitted metadata; its TX-specific finalize callback is deliberately NULL. Its RX prologue is the no-detach variant, and RX finalize externalizes newly arrived metadata. The ordinary kernel-pipe path instead uses the normal detach/replenish callbacks, or the UPP variants. Population call, domain-specific population, callback selection, TX metadata validation, RX no-detach.

For each accepted user-pipe transmission, nx_upipe_na_txsync_locked() swaps the source TX and destination RX kernel and user slot descriptor contents. Packet data bytes stay in their existing arena buffer. The destination now refers to the transmitted packet; the sender gets the previously attached destination packet as reusable storage. No allocation-ring ownership handoff or payload copy is performed by this swap. Transfer loop.

The actual SD_SWAP definition repairs each moved quantum's qum_ksd back-pointer. The transfer site's older comment mentions sd_user; the current macro is the evidence for what is updated: channel_var.h:653–661.

#define SD_SWAP(_ksd1, _usd1, _ksd2, _usd2) do {                        \
	_USD_SWAP(_usd1, _usd2);                                        \
	_KSD_SWAP(_ksd1, _ksd2);                                        \
	/* swap packet attachment */                                    \
	*(struct __kern_slot_desc **)(uintptr_t)&(_ksd1)->sd_qum->qum_ksd = \
	    (_ksd1); \
	*(struct __kern_slot_desc **)(uintptr_t)&(_ksd2)->sd_qum->qum_ksd = \
	    (_ksd2); \
} while (0)

This is a statement about this local transfer, not an end-to-end zero-copy guarantee for arbitrary Skywalk providers. Slot attachment also expresses a protocol ownership boundary; it does not establish per-packet VM isolation between tasks mapping the arena.

4. Submission, transfer and user-visible completion occur separately

Index meanings and the one-empty-slot rule

A ring with N slots has usable capacity N-1. Initialization sets ckr_khead = ckr_rhead = 0; TX tails start at N-1, RX tails at zero. Initialization.

os_channel_advance_slot() only moves the mapped ring_head to the position after the last supplied slot. os_channel_sync() enters __channel_sync(), which acquires the ring's sync right, validates/internalizes new submissions, calls the domain callback and, if successful, finalizes the mapped view. The ring is protected against concurrent sync by kr_enter()/kr_exit(), while channel and adapter lifetime use their separate locks/references. User wrappers, sync wrapper, syscall dispatch, sync-right implementation.

The source's “lock” terminology includes this sleepable per-ring ownership right; it is not a spin lock held across the entire callback. kr_enter(kr, TRUE) can wait for another owner. User-pipe RX's comment about skipping when it cannot get the peer lock should not be read as the implemented policy: the body passes TRUE and waits. RX-assisted transfer.

The bounded transfer and publication fence

Under the source TX ring's ownership right, the transfer routine first checks KR_DROP on both rings. Then it computes pending TX slots and destination capacity using wrap-aware arithmetic. Its count is the smaller of those two values. A full destination moves zero packets and returns success; remaining submitted packets stay pending in TX. Drop check and capacity.

The capacity calculation is nx_user_pipe.c:882–899:

	/* # of new tx slots */
	n = txkring->ckr_rhead - txkring->ckr_khead;
	if (n < 0) {
		n += txkring->ckr_num_slots;
	}
	limit = n;

	/* # of rx busy (unclaimed) slots */
	b = j - rxkring->ckr_khead;
	if (b < 0) {
		b += rxkring->ckr_num_slots;
	}

	/* # of rx avail free slots (subtract busy from max) */
	m = lim_rx - b;
	if (m < limit) {
		limit = m;
	}

The loop starts at destination ckr_ktail and source ckr_khead, swaps one descriptor pair per packet and advances each index modulo its own ring size. It requires the source packet to have both QUM_F_INTERNALIZED and QUM_F_FINALIZED. After the swaps, a fence precedes publication of the receiver's new tail because the sender does not hold the receiver's ring ownership right. Publication:

	/*
	 * Make sure the slots are updated before ckr_ktail reach global
	 * visibility, since we are not holding rx ring's kr_enter().
	 */
	os_atomic_thread_fence(seq_cst);

	rxkring->ckr_ktail = j;
	txkring->ckr_khead = k;
	txkring->ckr_ktail = SLOT_PREV(k, lim_tx);

Successful transfer notifies the destination RX ring, but the sender's callback does not run the destination's generic RX finalizer. A receiving sync/readiness pass externalizes arrivals and publishes the new user ring_tail. Conversely, RX-assisted movement may advance the sender's kernel head before the sender runs a finalizer that publishes its user ring_khead. Sender notification, TX publication, RX externalization/publication, RX finalizer.

Receiver-driven progress and readiness

RX sync first calls kr_reclaim(), advancing RX ckr_khead to the validated ckr_rhead with a visibility fence. It then checks destination room and already-submitted work in the peer TX kring. If both exist, it acquires the peer TX right and runs the same bounded transfer routine. This is how consumption releases pressure and can immediately pull pending packets. It does not read a newly advanced peer user ring_head and internalize unsynchronized submissions: the pending count uses the peer's kernel ckr_rhead. Reclaim, RX calculations and fence.

The decisive branch is nx_user_pipe.c:1064–1070:

	if (__probable(n != 0 && m != 0)) {
		(void) kr_enter(txkring, TRUE);
		n = nx_upipe_na_txsync_locked(txkring, p, flags, &ret, TRUE);
		kr_exit(txkring);
	} else {
		n = 0;
	}

If RX reclaimed or transferred anything, it notifies the peer TX ring. The default notifier posts channel readiness through na_post_event(); select/kqueue processing can perform synchronization before evaluating readable/writable low-water counts. A notification is a reason to re-evaluate state, not a payload, a completed user copy or a guarantee of room after another thread runs. The existing readiness chapter follows these filters. RX notification, default notifier, readiness sync.

Mermaid source

sequenceDiagram
    participant A as Sender application
    participant T as Sender TX kring
    participant R as Peer RX kring
    participant B as Receiver application
    A->>A: Fill packets and advance mapped head
    A->>T: TX sync validates and internalizes
    T->>R: Swap attachments up to available capacity
    T->>A: Finalize writable view
    T-->>B: Readiness notification
    B->>R: RX sync externalizes arrivals
    R->>B: Publish readable tail
    B->>B: Consume packets and advance head
    B->>R: RX sync reclaims released slots
    R->>T: Acquire TX right and pull pending submitted packets
    R-->>A: Notify writable progress
    A->>T: Later sync publishes updated TX view

This diagram omits error branches and the reverse direction. Every arrow naming a notification is asynchronous; arrows naming sync are calls in the calling thread.

5. Worked example: an eight-slot pair fills and recovers

Assume one direction, client TX to server RX, with eight slots on each ring, lim = 7, and no competing channels. Packet letters below are illustrative identities, not values observed from a running kernel. Start with TX khead = rhead = 0, ktail = 7; RX khead = ktail = 0.

For the first swap, suppose TX0 names packet A and RX0 names reusable packet B. Afterwards RX0 names A and TX0 names B; both quantum back-pointers are repaired. A's payload buffer has not moved. Reading A does not free A from the pool: the no-detach RX prologue leaves it attached until a later swap or teardown. This separates slot position, packet identity, buffer backing, kernel progress and published user progress.

The arithmetic follows the transfer loop, RX reclaim and the wrapper's available-slot calculation. The in-tree skt_fullupipe test exercises the same distinction with 100 slots: RX cannot pull unsynchronized submissions, then can pull already-submitted packets after reclaiming space. The test was read, not executed.

6. Close, absent peers and defunct are separate transitions

References keep the paired objects alive

Both endpoint adapters and both ring sets can exist before the second endpoint has any channel. This is why TX/RX callbacks can dereference ckr_pipe without discovering a live peer process on every call. An unopened peer can accumulate packets up to its RX capacity, after which submissions remain pending. Object existence does not guarantee a receiver is running. Pair construction, transfer capacity.

The reference handoff is:

The last-channel condition comes from na_unbind_channel(), not merely the user-pipe callback's comments. Common last-channel teardown, activation handoff.

The close-side reference is taken before the channel releases its own reference: nx_user_pipe.c:1313–1320.

	case NA_ACTIVATE_MODE_OFF:
		SK_DF(SK_VERB_USER_PIPE,
		    "%p: case 2.b, grab peer", SK_KVA(na));
		if (!pna->pna_peer->pna_peer_ref) {
			na_retain_locked(na);
			pna->pna_peer->pna_peer_ref = TRUE;
		}
		break;

Mermaid source

stateDiagram-v2
    [*] --> OneOpen: First open creates both adapters and rings
    OneOpen --> BothOpen: Peer opens and takes channel ownership
    BothOpen --> OneOpen: Last channel closes and survivor retains peer
    OneOpen --> Released: Last survivor closes and deletes both ring sets
    BothOpen --> Defunct: Eligible shared mapping revocation
    OneOpen --> Defunct: Eligible shared mapping revocation
    Defunct --> Released: Final close releases remaining references
    Released --> [*]

The states summarize endpoint activity, not exact adapter refcounts or every channel. The source saves/restores user-ring pointers during ring creation/reopening/deletion. Although surrounding historical comments describe “hidden” rings, the activation body shown above primarily changes activity and references; it does not itself clear a ring pointer. Reopen path, ring deletion.

The master destructor removes the pair's array entry using swap-with-last compaction and repairs the moved entry's parent index. Both endpoints drop parent references; the final decrement of nup_pna_users also releases the nexus-private parent reference and clears nup_pna. Array removal, endpoint destructor.

Reject-on-close changes admission, not the pairing representation

nx_upipe_dom_disconnect() calls common disconnect, then sets nexus-wide NXF_REJECT if nxp_reject_on_close is enabled. Subsequent channel activity can fail the common permissiveness check with ENXIO. This is broader than releasing one ring and should not be described as automatic BSD EOF semantics. With that policy disabled, adapter/ring retention still applies; eventual capacity exhaustion is separate from rejection. Policy branch, sync rejection.

The local peer-closure regression expects ENXIO with rejectonclose = 1 in its initializer. Its name alone does not prove every user pipe rejects activity whenever a peer descriptor closes. Test setup, close assertion.

Defunct revokes views before final storage destruction

For an eligible user-pipe defunct operation, the domain locks the two relevant TX directions in a fixed master-first order, including the TX kring reached from its own RX ring. It marks channel rings defunct while holding the same rights the sender and RX-assisted transfer use, then releases those two rights in reverse order for each ring index. Both transfer directions therefore observe the drop state rather than continuing into revoked peer mappings. Defunct serialization.

The shared-arena mapping lifecycle matters here: finalization is called after the arena has no valid mappings for these channels. It invokes na_defunct() for both endpoint adapters, then for the pseudo parent that owns the arena. Borrowed adapters release slots without independently destroying the shared owner's memory. The generic lifecycle chapter covers eligibility, mapping redirection and opportunistic skip conditions. User-pipe finalization, owner-versus-borrower teardown, defunct foundation.

7. Kernel pipes delegate packet decisions to a provider

Registration, connection and pool variants

User registration cannot create a kernel-pipe provider. The kernel registers a provider with versioned callbacks, including required TX/RX sync and connection callbacks. The default kernel-pipe domain then supplies adapter construction and channel mapping. nx_kpipe_na_find() checks the optional client binding and creates a new adapter on its creation path; it does not search for a retained user-pipe peer pair. Provider-specific connection policy can impose additional constraints. Registration admission, required callbacks, kernel-pipe find/create.

The domain permits NXPCAP_USER_PACKET_POOL in addition to user-channel capability. With UPP enabled it sizes allocation/free rings to 128 slots, permits user writes to the relevant slot-descriptor regions, and marks the adapter NAF_USER_PKT_POOL. na_bind_channel() rejects a mismatch between that adapter flag and CHMODE_USER_PACKET_POOL. The ownership hash, allocation/free sync and metadata validation are the packet-pool mechanisms, not user-pipe SD_SWAP. Parameter adjustment, adapter UPP setup, mode match.

Kernel-pipe arena creation supplies nx_tx_pp and nx_rx_pp, allowing the provider to obtain its pools during pre-connect. The constructor requests ring/slot context storage with na_rings_mem_setup(na, TRUE, ch). A nearby comment assumes IOSkywalkFamily consumers, but the local loopback and utun implementations below demonstrate that the actual callback boundary is also used by C providers. Pool storage, pool lookup, context allocation.

Connection proceeds through pre-connect, initialized rings/slots and connected; disconnect follows only the successfully completed stages. The wrapper drops SK_LOCK, retains the channel and reacquires ch_lock around external callbacks. A pre-connect failure does not receive disconnected; a later failure after successful pre-connect does, while pre-disconnect requires the fully connected flag. See callback stages and partial failure and the controller chapter.

Sync callbacks and the kernel ring API

The kernel-pipe adapter does almost no packet work in its TX/RX sync bodies. It maps force-reclaim/force-read into nx_sync_tx()/nx_sync_rx(), which call the registered nxpi_sync_* with KERN_NEXUS_SYNCF_COMMIT when appropriate. Whether packets are copied, attached elsewhere, queued or dropped belongs to the provider. Adapter dispatch.

The actual TX dispatch is nexus_var.h:606–618:

nx_sync_tx(struct __kern_channel_ring *kring, boolean_t commit)
{
	struct kern_nexus_provider *nxprov = NX_PROV(KRNA(kring)->na_nx);

	ASSERT(kring->ckr_tx == NR_TX);
	if (nxprov->nxprov_ext.nxpi_sync_tx != NULL) {
		return nxprov->nxprov_ext.nxpi_sync_tx(nxprov,
		           KRNA(kring)->na_nx, kring,
		           (commit ? KERN_NEXUS_SYNCF_COMMIT : 0));
	} else {
		return 0;
	}
}

The API's directions are from the user channel's perspective:

See available and next-slot helpers, advance, reclaim implementation, and attach/detach. Slot iteration/advance require Skywalk sync protection. They are not a license to keep an unretained ring pointer and call it from arbitrary contexts.

Mermaid source

flowchart TD
    U["User channel TX submission"] -->|validated prologue| T["Kernel-pipe TX kring"]
    T -->|nx_sync_tx call| P["Provider nxpi_sync_tx"]
    P -->|provider decides ownership| Q["Queue, interface input, packet move or drop"]
    E["Provider data ready"] -->|provider fills or schedules RX| R["Kernel-pipe RX kring"]
    R -->|notify| N["Channel readiness"]
    N -->|RX sync and provider nxpi_sync_rx| F["Reclaim, fill and externalize"]
    F -->|published user tail| V["User consumes RX packets"]

This diagram intentionally does not connect TX to RX automatically. A loopback provider supplies that relationship; a tunnel provider supplies a different one.

A complete local example: the development loopback provider

nx_kpipe_loopback.c is compiled for DEVELOPMENT || DEBUG. Its startup registers an external kernel-pipe domain provider, creates a controller, registers the callback vector and creates an instance. Ring/slot initialization records and checks context values; pre-connect obtains the built-in pools. The default setup uses the ordinary slot-attached mode. Build guard and state, callback registration, instance creation.

Its TX callback checks both available TX submissions and RX capacity. With no pair of slots it returns success without advancing either side. Otherwise it reads and validates the TX packet/buflet, exercises metadata setters, detaches the packet from TX and attaches that same handle to RX. This uses the packet API, unlike user-pipe descriptor swapping. Packet movement:

		tph = kern_channel_slot_get_packet(ring, ts);
		VERIFY(tph != 0);
		VERIFY(kern_channel_slot_detach_packet(txkring, ts, tph) == 0);
		VERIFY(kern_packet_finalize(tph) == 0);
		VERIFY(kern_channel_slot_attach_packet(rxkring, rs, tph) == 0);

It advances each ring through the last processed slot, records statistics, advises TX reclaim and notifies RX. Generic finalization later replenishes empty non-UPP TX slots and externalizes arrivals. Its RX callback reclaims consumed slots, enters the TX ring, runs the TX prologue/callback/finalizer, exits and notifies TX on the normal path. Loop bounds and no-progress return, advance/notify, RX-assisted loopback, non-UPP refill.

The test provider deliberately injects errors at registration/connection/ring/slot/sync stages. In particular error injection 9 occurs after packet movement, advancement and notification: an error does not undo that progress. At the syscall layer a nonzero sync callback becomes EIO and skips that ring's generic finalizer; a bad prologue produces EFAULT and can trigger process termination for malformed shared state. This debug provider is evidence for callback/API behavior, not proof that all its exceptional paths or every production provider are correct. Injection macro, post-progress injection, syscall error mapping, bad metadata termination.

8. A real consumer: kernel-pipe directions through utun

The local utun provider registers utun_kpipe_sync_tx and utun_kpipe_sync_rx with connection/ring callbacks; optional UPP is selected by its provider attributes. Ring initialization matches the channel's nexus UUID against the PCB's pipe-instance array and stores RX/TX ring pointers plus an index context. Ring finalization clears the matching pointer. Registration, ring context.

The same word “TX” changes meaning across its two channel interfaces:

Mermaid source

flowchart TB
    H["Host / flowswitch traffic"] -->|netif TX| NT["utun netif TX ring"]
    NT -->|utun_kpipe_sync_rx copies and adds tunnel header| KR["Kernel-pipe RX ring"]
    KR -->|user reads| U["Userspace tunnel endpoint"]
    U -->|user writes| KT["Kernel-pipe TX ring"]
    KT -->|netif mode: notify input service| NR["utun netif RX ring"]
    KT -->|non-netif mode: copy to mbufs| M["ifnet_input_extended"]
    NR -->|host receive| H

For user-to-kernel TX, utun_kpipe_sync_tx() first enters the PCB data-movement lifetime gate. If the pipe is no longer allocated it returns without consuming work. In netif mode it only notifies the netif RX ring when there is submitted TX data; that callback does not itself copy the packet. In the non-netif branch it copies usable packet bytes into an mbuf, converts the leading address-family word from network order, queues the mbufs, advances the visited TX slots and passes the mbuf list to ifnet_input_extended(). A skipped/disabled packet can still have its TX slot consumed. Entry/notification, mbuf conversion and consumption.

For kernel-to-user RX, utun_kpipe_sync_rx() reclaims released RX slots and obtains the corresponding netif TX ring. It drops the PCB shared lock before entering that ring, reacquires the lock and revalidates that the saved ring is still the PCB's current one. This is an explicit lock-drop/revalidation boundary, not a stable-pointer assumption. Reclaim and revalidation.

It then detaches each source packet, allocates a destination packet from the kernel-pipe RX pool, builds the tunnel header, copies payload bytes, finalizes the new packet and attaches it to the RX slot. It accumulates detached source packets for later freeing and advances both rings for the progress actually made. Source detachment/selection occurs before destination allocation; if allocation fails, the selected source is still in the cleanup chain and its source slot is included in the consumed prefix. The source is not automatically preserved for retry merely because no destination packet was produced. Detach/allocation ordering, header, copy and attachment, cleanup and progress.

These local branches demonstrate why kernel-pipe movement must be traced through the provider: loopback moves a packet handle, user pipe swaps descriptors, and this utun RX path allocates and copies into another pool. The tunnel's userspace protocol implementation and every IPsec/utun mode are outside this chapter.

9. Kernel-pipe defunct preserves provider-facing storage until disconnect

After na_connect() succeeds, nx_kpipe_dom_connect() increments the arena's kernel-slot-descriptor no-idle assertion. A provider may still be using kernel ring/slot APIs when userspace mappings are defuncted. Consequently, channel defunct and final descriptor destruction have different deadlines. The assertion:

	err = na_connect(nx, ch, chr, nxb, p);
	if (err == 0) {
		/*
		 * Mark the kernel slot descriptor region as busy; this
		 * prevents it from being torn-down at channel defunct
		 * time, as the (external) nexus owner may be calling
		 * KPIs that require accessing the slots.
		 */
		skmem_arena_nexus_sd_set_noidle(
			skmem_arena_nexus(ch->ch_na->na_arena), 1);
	}

Defunct acquires the selected TX/RX ring rights before marking them unusable; this includes providers such as utun that enter rings outside a channel sync syscall. Generic ring teardown depopulates attached packets and revokes user-ring state, but retains kernel slot-descriptor fields while the region is busy. Final teardown requires that busy assertion to be gone. nx_kpipe_dom_disconnect() decrements it before common disconnect. Retaining descriptor storage therefore does not promise its packets or user mapping remain usable. Domain defunct, kernel slot storage gate, disconnect decrement.

Provider callbacks must also stop their own asynchronous work. utun_nexus_pre_disconnect() calls utun_wait_data_move_drain(): it clears data-path readiness, then waits for the in-flight data-movement count to reach zero. utun_data_move_begin() increments only while ready; utun_data_move_end() decrements and wakes drainers. On the normal successful connection path, connected/disconnected balance an ifnet I/O reference, and ring-fini removes the PCB's ring pointers. The failed ifnet_get_ioref() branch returns ENXIO; this normal-path description does not claim every callback failure has identical reference effects. This lifetime gate complements ring serialization and the arena no-idle count; none substitutes for the others. Connection/disconnection, data movement gate/drain.

The loopback provider similarly distinguishes disabling its public instance from final domain-provider release: stop closes the instance and provider/controller, deregisters the domain provider, and uses final kplo_dom_fini() to clear busy state and wake the waiter. Existing references can delay finalization. Stop, final callback.

10. Configuration, test evidence and limits

Both domains require their respective CONFIG_NEXUS_USER_PIPE/CONFIG_NEXUS_KERNEL_PIPE attachment under Skywalk. Domain presence, controller privilege, provider capabilities, named binding and channel mode are separate checks. Both reject event-ring and low-latency channel requests on these paths. The user-pipe domain always rejects UPP; kernel-pipe capability can enable it. Build/domain selection, domain attachment, user rejected modes, kernel rejected modes.

The review follows the two domain implementations, shared adapter/ring helpers, local channel wrappers, the development loopback provider, and selected utun callback/cleanup bodies. No kernel or in-tree test was built or run. External IOSkywalkFamily/provider bodies, userspace tunnel services, every provider callback implementation and a complete concurrency proof remain outside the available trace.