IPsec interfaces, packet representations, and staged disconnect

Networking guide · IPsec policy and transforms · Skywalk pipes · Packet and interface lifetime

An ipsecN interface supplies an interface identity for IPsec traffic and a lifetime shared by a kernel-control connection, an ifnet, association bindings and, optionally, Skywalk nexuses. Creating it does not negotiate keys or install a usable security association. This chapter follows the local interface implementation in xnu-12377.121.6, from control admission through output, input and asynchronous destruction. The IPsec companion explains policy publication, association generations and transform internals.

1. One interface implementation has three packet paths

The build selects if_ipsec.c with ipsec. Within that file, SKYWALK defines IPSEC_NEXUS; this enables additional code, while each PCB's ipsec_use_netif and ipsec_kpipe_count select its runtime path. Binding initializes ipsec_use_netif to false. Requesting kernel pipes without enabling the native netif fails with ENOTSUP at connect. Build selection, nexus guard, bind defaults, connect selection.

Sources: legacy start, native TX, pipe RX, native RX, mbuf input.

TX and RX are relative to the endpoint: packets leaving the network stack through netif TX become packets the channel client receives on kernel-pipe RX. A channel client's TX becomes netif RX. The third path performs its ESP transforms in the kernel. The external client supplies transport and channel participation; this source does not supply that client's implementation.

Mermaid source

flowchart TD
    S["Network stack selects ipsecN"] --> M{"PCB runtime mode"}
    M -->|legacy| A["ifnet queue: mbuf"]
    M -->|native without pipe| B["netif TX: packet"]
    B -->|copy bytes and policy metadata| A
    A --> C["ipsec_output: SA lookup and transform"]
    C -->|mbuf ownership transferred| D["BSD IPv4 or IPv6 output"]
    M -->|native with pipe| E["netif TX: plaintext packet"]
    E -->|allocate destination and ESP transform| F["kernel-pipe RX: encrypted packet"]
    F --> G["External channel client and transport"]

The diagram shows successful ownership transfers; allocation, policy, association and transform failures can consume/drop the source packet before reaching the next destination.

2. Control admission creates identities before publishing an interface

ipsec_register_control() registers com.apple.net.ipsec_control, its setup/bind/connect/disconnect callbacks and IPv4/IPv6 protocol plumbers. The control is privileged and uses provider-managed setup. Generic kernel-control admission requires the appropriate socket type and a superuser credential for this flag; the provider is not itself the complete admission check. Control name, registration and plumber unwind, generic admission.

Registered lifecycle callbacks — if_ipsec.c, lines 369–382.

	bzero(&kern_ctl, sizeof(kern_ctl));
	strlcpy(kern_ctl.ctl_name, IPSEC_CONTROL_NAME, sizeof(kern_ctl.ctl_name));
	kern_ctl.ctl_name[sizeof(kern_ctl.ctl_name) - 1] = 0;
	kern_ctl.ctl_flags = CTL_FLAG_PRIVILEGED | CTL_FLAG_REG_SETUP; /* Require root */
	kern_ctl.ctl_sendsize = 64 * 1024;
	kern_ctl.ctl_recvsize = 64 * 1024;
	kern_ctl.ctl_setup = ipsec_ctl_setup;
	kern_ctl.ctl_bind = ipsec_ctl_bind;
	kern_ctl.ctl_connect = ipsec_ctl_connect;
	kern_ctl.ctl_disconnect = ipsec_ctl_disconnect;
	kern_ctl.ctl_send = ipsec_ctl_send;
	kern_ctl.ctl_setopt = ipsec_ctl_setopt;
	kern_ctl.ctl_getopt = ipsec_ctl_getopt;

ipsec_ctl_setup() holds the global ipsec_lock while reserving a control unit, choosing a separate unique ID, allocating a zeroed PCB and inserting it into ipsec_head. An explicit occupied unit fails with EBUSY; zero asks the provider to choose one. Setup publishes the PCB as control unitinfo, not a working network interface. Bind records the control reference and initializes the PCB locks and runtime defaults. Connect calls bind itself when necessary. Unit and PCB allocation, bind, connect entry.

Sources: PCB fields, names and ifnet callbacks, data gate.

Configuration is staged around interface publication. IPSEC_OPT_ENABLE_NETIF, channel count and channel PID/UUID binding require ipsec_ifp == NULL. Channel count accepts zero, one or the WMM ring count. Actual channel allocation additionally checks PRIV_SKYWALK_REGISTER_KERNEL_PIPE, then binds the client port to the configured executable UUID or PID, defaulting to the current process when appropriate. This binding is separate from the privilege needed to open the control. Configuration checks, allocation privilege and binding.

Other settings deliberately require an existing interface: delegated interface, external statistics, output traffic class and DSCP mapping. Several also repeat the superuser check. In particular, the header describes DSCP mapping as pre-connect, but its setter rejects a null ipsec_ifp; the function body establishes the behavior described here. Option declarations, privileged options and connected settings, DSCP setter.

The control socket is not a packet injection stream. Its send callback simply frees the supplied mbuf and returns zero. Packet traffic follows the ifnet or channel paths above. Control send.

3. Interface publication changes who owns final cleanup

Legacy connect fills ifnet_init_eparams with the PCB, ipsec_start, protocol callbacks, ioctl handler and ipsec_detached. It allocates an ifnet, applies attributes, then attaches it. The attributes include MTU 1500, point-to-point/multicast/up flags and IFEF_NOAUTOIPV6LL. BPF uses DLT_NULL in this path; native mode uses DLT_RAW. Connect sets IFF_RUNNING after the selected setup succeeds. Attributes, legacy publication.

Native connect first allocates requested kernel-pipe instances, then registers a NET_IF provider and allocates its instance, which supplies the ifnet. The netif uses explicit nexus attachment (IFNET_INIT_NX_NOAUTO), a virtual-device/user-accessible packet pool with one fragment per packet, and provider callbacks carrying the PCB as context. Context retain/release callbacks are null here: they are not an additional generic PCB reference-count protocol. A flowswitch provider/instance is subsequently attached to the netif instance. Its agent UUID is recorded for the option that advertises flowswitch service. NET_IF provider and pool, flowswitch creation and attachment, service option.

The kernel-pipe provider is shared: ipsec_ncd_refcount counts provider users under ipsec_lock, and only the first registration constructs its controller/provider and attributes. Each PCB still has its own instances and packet pool. A later caller reuses that provider; its registration does not replace the shared provider geometry. Shared registration, last-user destruction.

Failure cleanup must be read at the stage that failed:

Sources: native helper unwind, flowswitch unwind, connect errors.

This table describes the shown local actions, not a transactional rollback of every previously created resource. For example, optional channels are allocated before native ifnet attachment, whereas the native-attach error arm calls ipsec_free_pcb() directly; that destructor frees queued mbufs and locks, not channel instances or pools. Keep the channel helper's own allocation unwind separate from later connect stages. The registration entry also calls ipsec_register_nexus() without using its return value, so control registration success alone is not proof that native provider setup succeeded. Ordering, PCB destructor, channel helper unwind, nexus registration call.

The generic control layer drops the socket lock around the provider's connect callback. If connect fails, it calls the disconnect callback, ignores that callback's return and clears the association. Several IPsec error arms have already set unitinfo to null, and IPsec disconnect returns EINVAL for that null value. Thus the generic fallback does not imply that the IPsec disconnect body ran through its normal teardown sequence. Generic connect and failure, null disconnect.

4. Interface output selects an association and consumes the mbuf

Interface-associated policy and SA records are installed separately through PF_KEY. The IPsec-interface extension can resolve an interface name to an ifnet; outgoing-interface selection resolves another name to an index. These are different roles: ipsecN identifies the transform interface, while an outgoing interface constrains transport after transformation. PF_KEY interface resolution, SA-head interface ownership.

ipsec_start() dequeues mbufs and passes each to ipsec_output(), stopping on a nonzero result. The latter consumes the supplied packet through success and error. It first rejects a packet whose NECP last-interface index is already this interface, then marks the packet with the current interface for subsequent policy evaluation. This is a specific same-interface loop check, not a record of every interface the packet has ever visited. Dequeue, output and cleanup.

Loop check and interface mark — if_ipsec.c, lines 3639–3646.

	// Make sure this packet isn't looping through the interface
	if (necp_get_last_interface_index_from_packet(data) == interface->if_index) {
		error = EINVAL;
		goto ipsec_output_err;
	}

	// Mark the interface so NECP can evaluate tunnel policy
	necp_mark_packet_from_interface(data, interface);

After basic IP header validation, the IPv4 or IPv6 adapter constructs endpoint selectors and calls key_alloc_outbound_sav_for_interface(). Under sadb_mutex, that lookup skips dead SA heads, requires the matching ipsec_if and outbound direction, and accepts an IPv4/IPv6 request. Its first pass checks transport-mode endpoint equality; a second pass relaxes that check. Within the chosen head it applies the configured old/new SA-state preference and acquires a referenced SA. Do not turn the first-pass comment into a guarantee of exact endpoint matching on both passes, or into a separate equality check against the SA head's address family. Lookup body.

The adapters call the existing IPv4/IPv6 transform routines and release their SA reference afterward. The mbuf may be replaced, consumed on error, or already submitted by a cross-family tunnel helper. For a same-family result, ipsec_output() sets traffic-class/flow metadata and output arguments, then calls BSD IP output. IPv4 uses IP_NOIPSEC on this reinjection. Both families use the returned flow advisory to disable interface output on flow-control/suspension and report ENOBUFS; they do not propagate the IP output function's return value directly. IPv4 adapter, IPv6 adapter, IPv4 reinjection, IPv6 reinjection and exit.

A zero return does not prove transmission. In the mbuf adapters, missing SA lookup reaches a cleanup arm that frees state->m, clears it and can return the initially zero error. ipsec_output() observes the null packet and has nothing left to submit. The packet ownership result is therefore more informative than an isolated status check. The kernel-pipe adapters have their own error contract and explicitly return ENOENT for a missing SA. Mbuf missing-SA path, pipe missing-SA path.

5. Native TX changes representation before ordinary IPsec output

With no kernel pipe, ipsec_netif_sync_tx() begins a PCB data-movement operation and takes the PCB shared lock. It detaches each source packet from its TX slot and adds that packet to a local chain for later pool release. A new mbuf receives at most MIN(packet_length, ipsec_slot_size) bytes from the first buflet's data address plus offset. This is a copy into distinct storage; the mbuf and original Skywalk packet have separate disposal paths. TX detach and copy, source-chain release.

The adapter transfers policy and skip-policy IDs from packet metadata into the mbuf. ipsec_netif_check_policy() accepts an existing valid IP-layer policy mark; otherwise it evaluates IPv4/IPv6 policy bound to this interface and rejects DROP or SOCKET_DIVERT. The permitted mbuf then enters ipsec_output() and its same-interface loop check. Allocation/copy/policy failures free the new mbuf as appropriate; source packets already detached still reach the local pool cleanup. The callback itself returns zero after this processing, including drop branches. Policy transfer and output, policy helper.

Worked packet: native netif without a kernel pipe

Assume a native interface with a 2,048-byte slot, no kernel pipe, a valid 1,000-byte IPv4 packet wholly within its one buflet, data offset 32, a policy allowing output, and an installed same-family ESP SA. These are illustrative inputs, not a claim about a live interface.

1. The TX slot holds source packet P, whose bytes begin at buflet_address + 32. Sync detaches P and remembers it on its local release chain.
2. mbuf_gethdr() and mbuf_copyback() create mbuf M with 1,000 bytes from that address. Metadata policy IDs are copied separately; copying payload bytes alone would not transfer those IDs.
3. ipsec_output() owns M. It marks the interface, selects a referenced SA, transforms the mbuf chain and hands the resulting mbuf to ip_output(). Its local pointer is cleared after that handoff. ESP overhead depends on the selected mode/algorithm; it is not a fixed addition to the illustrative 1,000 bytes.
4. TX sync frees P through kern_pbufpool_free_chain() and advances/reclaims the consumed source slots. P's storage is not held until remote delivery. The mbuf output path has its own ownership after the copy.
5. For a separate inbound 1,000-byte decrypted packet I, successful native injection transfers I into the PCB input queue. RX sync allocates destination packet R before removing I, copies I into R, sets offset zero and length 1,000, finalizes and attaches R to the RX slot, then frees I. A failed destination allocation leaves I queued; lack of an RX slot also leaves queued input pending.

Sources: TX copy and source release, IP handoff, input ownership, RX allocation/dequeue, RX copy/publication/free.

Mermaid source

sequenceDiagram
    participant TX as Netif TX ring
    participant CB as IPsec callbacks
    participant IP as BSD IP output
    participant Q as PCB input queue
    participant RX as Netif RX ring
    TX->>CB: Detach source packet P
    CB->>CB: Allocate mbuf M and copy P bytes
    CB->>IP: Transform and transfer M
    CB->>CB: Free P through its packet pool
    Note over IP,Q: A separate inbound packet is decrypted
    IP->>Q: Transfer decrypted mbuf I on successful injection
    Q->>CB: Remove I after destination allocation succeeds
    CB->>RX: Copy, finalize and attach packet R
    CB->>CB: Free I

This sequence selects successful same-family processing and input small enough to avoid fragmentation. It omits notification and slot bookkeeping, not additional payload sharing.

6. Kernel pipes keep ESP processing in packet form

With allocated kernel pipes, native TX sync merely selects the corresponding pipe RX ring and notifies it after releasing the PCB lock. The pipe's RX callback performs the work: reclaim user-released RX slots, obtain the corresponding netif TX ring, allocate a destination packet, detach the source packet, validate its size, encrypt, finalize and attach the destination. Source packets accumulate on a local chain and are freed after the loop; failed destinations are returned to their pool. Native notification branch, pipe receive loop.

Ring entry can wait, so the callback drops the PCB shared lock before kr_enter(), reacquires it afterward and verifies that the ring pointer still matches the PCB's published ring. The active data-movement operation spans this interval. Publication and lifetime are handled by distinct mechanisms; a pointer obtained under a lock is not treated as permanently valid after the lock drop. Lock drop and revalidation, ring init/fini.

Unlock, enter, revalidate — if_ipsec.c, lines 792–808.

	// Unlock ipsec before entering ring
	lck_rw_unlock_shared(&pcb->ipsec_pcb_lock);

	(void)kr_enter(tx_ring, TRUE);

	// Lock again after entering and validate
	lck_rw_lock_shared(&pcb->ipsec_pcb_lock);
	if (tx_ring != pcb->ipsec_netif_txring[ring_idx]) {
		// Ring no longer valid
		// Unlock first, then exit ring
		lck_rw_unlock_shared(&pcb->ipsec_pcb_lock);
		kr_exit(tx_ring);
		os_log_error(OS_LOG_DEFAULT, "%s: %s ring %s index %d bad netif_txring 2\n", __func__,
		    pcb->ipsec_if_xname, rx_ring->ckr_name, ring_idx);
		ipsec_data_move_end(pcb);
		return 0;
	}

ipsec_encrypt_kpipe_pkt() dispatches by IP version. The IPv4/IPv6 packet adapters use the same interface-SA lookup but deliberately support a narrower transform set: transport mode, ESP, MATURE/DYING SA state, and rejection of the listed old/derived/NAT-T/cyclic-sequence/padding flag combinations. They copy the IP header into the new destination and call esp_kpipe_output(). AH returns EPROTONOSUPPORT; tunnel mode returns EINVAL. Thus enabling a pipe is not a representation-only switch for every SA supported by the mbuf implementation. Version dispatch, IPv4 packet adapter, IPv6 packet adapter.

On the reverse path, pipe TX sync notifies netif RX. Native RX first services queued decrypted mbufs, then examines each pipe TX ring. For each source packet it allocates a separate netif destination and calls ipsec_transform_kpipe_pkt_to_netif_pkt(). That helper validates lengths/version, copies the IP header, serializes decrypt work with ipsec_kpipe_decrypt_lock, calls esp_kpipe_input(), transfers the recognized wake flag and finalizes destination metadata. The ESP helper looks up an inbound SA with the interface and SPI, and applies its own transport/flag/state checks. Successful destinations attach to netif RX; failed destinations are freed. The source remains attached to its TX slot here and is handled by advance/reclaim rather than the outbound callback's explicit detached-packet chain. Pipe TX notification, packet decrypt adapter, RX pipe loop and reclaim, inbound SA checks.

The encrypt/decrypt mutexes serialize those operations per PCB; they do not replace sadb_mutex, SA references, PCB locking or ring entry. The Skywalk pipe chapter supplies the surrounding channel/ring contract.

7. Inbound injection transfers ownership before delivery finishes

In the ordinary ESP mbuf path, the association head can supply ipsec_if. ESP obtains an ifnet reference under sadb_mutex, drops that lock, validates/repairs the decrypted IP packet and injects it through the interface. It releases its temporary ifnet reference after the injection call. On success it clears its local mbuf pointer; on failure it retains responsibility for the mbuf and reaches its error cleanup. A matching explicit interface argument can instead return the decrypted mbuf directly to that caller. ESP interface handoff.

Legacy injection demultiplexes by IP version, sets the receive interface and invokes proto_input(). A protocol-input error increments the error statistic and frees the packet; the wrapper still returns zero because it consumed the packet. Native injection first obtains PCB data-movement admission, then takes the PCB shared lock and input-chain mutex. It appends the entire m_nextpkt list and notifies netif RX outside those locks. A missing RX ring does not undo the queue ownership transfer. Legacy demux, legacy protocol ownership, native queue and notify.

The queue admission predicate checks whether the existing count is already greater than if_ipsec_max_pending_input; it neither uses >= nor reserves space for the incoming list's full length. Treat that setting as this source's admission threshold, not a strict maximum queue length. ENXIO before admission and ENOSPC at this check leave ownership with the caller. Queue admission.

Input queue rejection leaves the packet untouched — if_ipsec.c, lines 4108–4115.

		lck_mtx_lock(&pcb->ipsec_input_chain_lock);

		if (pcb->ipsec_input_chain_count > (u_int32_t)if_ipsec_max_pending_input) {
			lck_mtx_unlock(&pcb->ipsec_input_chain_lock);
			lck_rw_unlock_shared(&pcb->ipsec_pcb_lock);
			ipsec_data_move_end(pcb);
			return ENOSPC;
		}

Native RX allocates a destination before unlinking the queued mbuf, then removes one packet under the input-chain mutex. Oversized decrypted packets can be fragmented to the pool-buffer size or configured input-fragment size, with resulting fragments appended back onto the queue; the newly allocated but unused destination packet is freed. Otherwise it copies/finalizes/attaches as in the worked example and frees the mbuf. This is asynchronous queue-to-ring delivery, not a synchronous guarantee from ipsec_inject_inbound_packet(). Allocation and dequeue, fragmentation and requeue, normal copy and free.

8. Disconnect crosses several independent lifetime barriers

Four mechanisms must remain distinct:

Sources: PCB gate, nexus stop wrapper, ifnet suspend/drain/resume, connected/disconnected callbacks, generic ifnet lifetime.

The common nexus connected callback obtains an ifnet I/O reference before setting PCB ready. Both netif and pipe provider tables use it. Their pre-disconnect callbacks clear ready and drain active PCB operations; disconnected clears the stored netif nexus pointer when it matches and releases the I/O reference. Ring fini clears the corresponding ring pointer. These callbacks explain why closing the control and destroying every channel-visible object are not a single instantaneous event. Callback bodies, netif table, pipe table.

Admission counts active operations — if_ipsec.c, lines 4227–4239.

ipsec_data_move_begin(struct ipsec_pcb *pcb)
{
	boolean_t ret = 0;

	lck_mtx_lock_spin(&pcb->ipsec_pcb_data_move_lock);
	if ((ret = IPSEC_IS_DATA_PATH_READY(pcb))) {
		pcb->ipsec_pcb_data_move++;
	}
	lck_mtx_unlock(&pcb->ipsec_pcb_data_move_lock);

	return ret;
}

ipsec_ctl_disconnect() first clears PCB ready and waits for active operations, then stops the netif nexus rings when a stored nexus exists. It takes the PCB exclusive lock, moves allocated channel UUIDs/pool into a temporary detached-resources record, clears their PCB publication and clears ipsec_ctlref. The resources are freed later outside that lock. The mode branch uses the actual stored ipsec_netif_nexus, not just the configured ipsec_use_netif bit. Disconnect entry, channel unpublication, channel freeing.

For the native-nexus branch, the order is if_down(), ifnet suspend-and-drain, ifnet_detach(), association/policy unlinking and address/protocol cleanup. It then drops the PCB lock, frees detached channels, tears down flowswitch/netif providers and pools, and calls ifnet_datamov_resume() to release the suspension reference. Resume balances that reference during teardown; it does not mean the already-detaching interface becomes available again. Native disconnect, nexus detach, resume reference release.

Without a stored native nexus, disconnect drops the PCB lock first, frees any detached channel record, unlinks policies/associations, cleans IPv4/IPv6 and finally calls ifnet_detach(). If only bound and never connected, there is no ifnet: it directly frees the PCB. Native detach failure panics in this body; the other branch logs detach failure and nevertheless returns zero. Other disconnect branches.

Mermaid source

flowchart TD
    A["Control disconnect"] --> B["Clear PCB ready and drain active packet work"]
    B --> C["Stop stored netif nexus rings"]
    C --> D["Under PCB lock: unpublish channels and control reference"]
    D --> E{"Stored native nexus?"}
    E -->|yes| F["if_down, suspend ifnet movement, request detach"]
    F --> G["Unlink SA/policy bindings and clean protocol addresses"]
    G --> H["Unlock PCB, free channels and nexuses, balance suspension"]
    E -->|no, connected| I["Unlock PCB, free channels, clean bindings and protocols"]
    I --> J["Request ifnet detach"]
    H --> K["Generic detach and I/O-reference draining"]
    J --> K
    K --> L["ipsec_detached callback"]
    L --> M["Release ifnet, free PCB queue and locks, dispose interface"]

The diagram omits the bound-only direct-free arm and error branches. It separates teardown requests from the later final callback.

key_delsp_for_ipsec_if() works under sadb_mutex: matching policies and SA heads release their interface reference and clear their pointer; policies and selected SA generations are marked dead and references released. Outstanding users can still hold association references, so removal from interface selection is not equivalent to all association storage being freed immediately. Protocol cleanup attempts detach, removes addresses when the protocol reports EBUSY, then retries; it logs failures and closes its temporary privileged socket. Binding removal, protocol cleanup.

Final cleanup belongs to ipsec_detached(): release the interface, hold the global IPsec lock while removing/freeing its PCB, and dispose the interface. ipsec_free_pcb() frees any residual input mbuf list, destroys locks, removes the PCB from the global list and frees its allocation. It is not the routine that stops rings or destroys providers; those obligations occur earlier. Final callback, PCB destructor.

Final callback — if_ipsec.c, lines 4010–4020.

static void
ipsec_detached(ifnet_t interface)
{
	struct ipsec_pcb *__single pcb = ifnet_softc(interface);

	(void)ifnet_release(interface);
	lck_mtx_lock(&ipsec_lock);
	ipsec_free_pcb(pcb, true);
	(void)ifnet_dispose(interface);
	lck_mtx_unlock(&ipsec_lock);
}

9. Scope and external boundaries

This review covers the local IPsec interface's control callbacks, runtime mode selection, mbuf/native/pipe adapters, association-interface selection, input queue and staged disconnect. It follows representative IPv4/IPv6 handoffs into the existing transforms; the companion chapter remains the detailed route through PF_KEY, replay, ESP construction and rekeying. It does not claim that every control option, ioctl, traffic-class ring combination or allocation-failure interleaving has an independent end-to-end walkthrough.

The user process that creates and configures the interface, negotiates or provisions keys, chooses transport policy and services a kernel-pipe channel is outside these implementation bodies. Hardware drivers, remote peers and physical transmission remain downstream boundaries. Native provider declarations and callback registration are evidence of the local kernel contract, not evidence that a particular shipped product enables every mode. The source paths above are shared/configuration-conditioned rather than tied here to a macOS-only debugging interface. No runtime network or kernel test is implied by this source review.