Skywalk namespaces, flow identity, and packet lookup

Networking · Skywalk overview · Registration and teardown · Classification, routes, and packet delivery

A flowswitch registration needs several different identities. It reserves resources that might otherwise belong to BSD or PF, creates an entry for packet lookup, assigns a control-plane UUID, and supplies a compact identifier to packet scheduling. These operations have different keys, locks, owners, and release points. This chapter follows their implementation in xnu-12377.121.6; the lifecycle chapter remains the entry point for channel creation and flow registration.

1. Four namespaces and a destination port

Definitions: ns and ns_token, protons_token, flow_key, flow_entry, and flowidns domains.

Mermaid source

flowchart TD
    R["nx_flow_req: addresses, protocol, UUID, owner"] --> P["Prepare address and route"]
    P --> N["Reserve local port, IP protocol, or custom IPsec namespace"]
    N --> E["flow_entry"]
    E --> K["fe_key<br/>Packet lookup"]
    E --> U["fe_uuid<br/>Owner UUID tree"]
    E --> I["fe_flowid<br/>Scheduling identity"]
    E --> D["fe_nx_port<br/>Channel destination"]
    E --> T["Reservation token<br/>Owned or borrowed"]

These are kernel resource namespaces, not isolated network-stack containers. In particular, assigning nt_ifp does not add an interface to the reservation key. The test source explicitly checks that a wildcard binding on one flowswitch prevents the corresponding binding on another. See namespace representation and cross-flowswitch reservation test.

2. Preparation chooses the reservation mechanism

flow_req_prepare() validates address families, lengths, local-address availability, multicast exclusions, ports, and child-flow demultiplexing parameters. For TCP/UDP, destination address and destination port must either both be present or both be absent. An absent destination selects listener mode. Other IP protocols require zero ports. A connected request obtains a flow_route; when no local address was supplied, the request adopts the route's selected source address before reserving a local endpoint. These operations can fail before a flow entry exists. See request validation and route then namespace preparation.

flow_req_prepare_namespace() then selects:

• TCP/UDP: flow_namespace_create() and netns.
• ESP/AH: key_reserve_custom_ipsec(), with a separate IPsec reservation token.
• Other protocols: protons_reserve().
• A child flow: no independent namespace reservation; its parent supplies the shared endpoint.

See selection predicates, reservation dispatch, and child exclusion. The IPsec helper is a different ownership boundary, not an alternate netns owner.

flow_namespace_create() maps listener/reuse/wake flags, validates TCP or UDP and IPv4 or IPv6, then chooses fixed-port or ephemeral allocation:

Source: bsd/skywalk/nexus/flowswitch/flow/flow_namespace.c, lines 104–113.

	}

	/* Assign an ephemeral port, if no port was specified */
	if (laddr->sin.sin_port == 0) {
		err = netns_reserve_ephemeral(token, addr, addr_len, protocol,
		    &laddr->sin.sin_port, netns_rsv_flags, nfi);
	} else {
		err = netns_reserve(token, addr, addr_len, protocol,
		    laddr->sin.sin_port, netns_rsv_flags, nfi);
	}

The public netns APIs take network-byte-order ports and addresses; ns_reservation.nsr_port and ns_token.nt_port hold the port in host byte order. The conversion happens at the common reservation entry. Applying ntohs() again to the stored port would produce the wrong lookup value. See API contract, stored port, and common entry.

3. Port reservation objects and publication

netns_lock serializes namespace trees, reservation counts, token lists, and interface association. Paths can acquire the mutex in spin mode and convert it before allocation or other operations requiring the ordinary mutex mode. This is one shared allocator/arbitrator; per-interface lists do not supply independent collision protection. See lock macros and common reserve.

See object layouts, namespace cleanup, and reference-only aggregate.

For a new concrete-address reservation, _netns_reserve_kpi_common() first reserves in the specific namespace, then increments the corresponding aggregate non-wildcard namespace. If the second step fails, it releases the first. Only afterward does it create the token, copy optional flow information, associate an interface, and publish the caller's pointer. The aggregate and specific reservations therefore represent the same caller's ownership, not two independently usable endpoint reservations. See reserve and rollback and token publication.

Mermaid source

flowchart TD
    A["Reserve concrete local address and port"] --> S["Increment specific namespace owner count"]
    S --> G{"Increment aggregate count succeeds?"}
    G -->|yes| T["Allocate and initialize token"]
    T --> P["Publish token to caller"]
    G -->|no| R["Release specific reservation and return error"]
    P --> X["Release token later"]
    X --> D["Decrement aggregate and specific counts"]
    D --> F["Remove empty reservation and eligible namespace"]

Passing an existing token is a separate contract. NETNS_PRERESERVED causes assertions that address, protocol, port, and reservation flags match; without it, reusing a valid output token panics. This is an internal API invariant, not a user-facing retry protocol. Existing-token branch.

The disabled-module behavior also differs by layer: low-level netns_reserve() can return success with a null token when netns was not initialized, whereas flow_namespace_create() explicitly rejects a disabled namespace service with ENOTSUP. Do not interpret a low-level zero return alone as proof that an actual reservation exists. Low-level entry, flowswitch guard.

4. Collision decisions: owner counts, wildcards, and reuse

The implementation makes both cross-namespace checks and an owner-specific check. The following table describes the latter; it must be read with the wildcard rules below.

The actual branches are in owner-specific checks. Before them, a Skywalk request checks wildcard reservations; a wildcard Skywalk request examines both IPv4 and IPv6 non-wildcard aggregates. BSD/PF wildcard checks examine the relevant family's non-wildcard Skywalk/listener counts. These are deliberately asymmetric branches, not a generic “wildcard equals every address” comparison. Wildcard checks.

There is another qualification: _netns_is_port_used() ignores the current reservation object and returns false for an existing nsr_reuseport reservation. That bit is established when the reservation object is first allocated. Some comments saying “any entry” therefore describe the broad intent more strictly than the predicate implements. NETNS_REUSEPORT does not erase every owner collision check or grant the complete BSD SO_REUSEPORT policy by itself. See reuse flag initialization and the actual predicate:

Source: bsd/skywalk/namespace/netns.c, lines 534–552.

static boolean_t
_netns_is_port_used(struct ns * gns, struct ns_reservation *curr_res, in_port_t port)
{
	struct ns_reservation *res = NULL;

	if (gns == NULL) {
		return FALSE;
	}

	res = ns_reservation_tree_find(&gns->ns_reservations, port);
	if (res != NULL && res != curr_res) {
		if (!res->nsr_reuseport) {
			return TRUE;
		}
	}

	return FALSE;
}

On success, _netns_reserve_common() increments only the selected owner counter. On failure, it removes a newly inserted empty reservation; an existing reservation and its counts remain. Commit or undo.

The distinction from flow lookup is explicit in the source:

Source: bsd/skywalk/namespace/netns.c, lines 750–754.

			/*
			 * XXX: Duplicate 5-tuple flows under a Skywalk
			 * listener are currently detected by flow manager,
			 * till we implement 5-tuple-aware netns.
			 */

For example, two different peers may legitimately share the server's local TCP port under a listener. netns accounts for that local-port ownership; flow_mgr_find_conflicting_fe() and flow_entry_alloc() reject an already registered matching flow, ordinarily with EEXIST. A port collision commonly yields EADDRINUSE. These are different failed decisions. Conflict lookup, allocation conflict handling.

5. Ephemeral allocation and BSD/PF participation

netns_reserve_ephemeral() uses ipport_firstauto/ipport_lastauto, plus TCP- or UDP-specific randomization policy. It obtains the selected address/protocol namespace, chooses an initial candidate either randomly or from that namespace's remembered cursor, then walks upward or downward with wraparound. Each candidate runs the ordinary collision checks. Restricted ports are skipped; EADDRINUSE advances to another candidate, while other errors stop the search. Exhausting the cycle returns EADDRNOTAVAIL. A single-port range has its own branch, and port zero is rejected. See range setup and UDP headroom and candidate walk.

UDP has an additional guard: it leaves NETNS_NS_UDP_EPHEMERAL_RESERVE—4096 in this snapshot—ports' worth of namespace headroom based on UINT16_MAX - ns_n_reservations. That is not a count of currently free ports inside the configured ephemeral interval. The output port and remembered cursor are updated only after successful reservation. Headroom constant, success publication.

BSD participates without replacing its PCB hash tables. Under SKYWALK, in_pcbinshash() reserves or validates inp_netns_token before linking the PCB into its port/hash lists, then records its last outgoing interface. in_pcbrehash() changes the reservation address or creates a missing token, and asserts success on the assumption that earlier endpoint validation made the transition safe. INP2_EXTERNAL_PORT bypasses these ownership operations. See PCB insertion and rehash; the PCB chapter explains its independent matching and lifetime rules.

PF's port selection also uses NETNS_PF; a representative UDP mapping path reserves before returning its selected translated port. This chapter establishes the shared arbitration point, not the complete PF translation algorithm. PF reservation call.

6. Changing an address, withdrawing, and releasing

netns_change_addr() is BSD-only. It holds netns_lock, reserves the new specific namespace first, then releases the old specific reservation, adjusts aggregate counts, and finally updates the token's address. An ordinary new-address collision leaves the old reservation intact. However, the body has an aggregate-reservation failure branch after the old reservation was released; its comment says this should not fail, and it does not implement a complete rollback. It is therefore too strong to describe every possible error as an atomic no-change result. The PCB caller's VERIFY(err == 0) is another indication of the assumed invariant. Address transition.

netns_set_ifnet() changes reporting/lifetime association: it drops the old interface I/O reference, then tries to acquire one for the new interface and links the token to that interface or to the unbound list. Interface detach moves tokens to the unbound list; it does not release their endpoint reservations. Association and detach, reference release.

Half-close and withdraw only set token state bits. The local-port reporting path excludes withdrawn flows and conditionally excludes half-closed flows when active-only reporting is requested. These operations do not decrement owner counts. netns_release() clears the caller's pointer, releases aggregate and exact reservations, drops the interface association, and frees the token. _netns_release_common() removes reservation storage only after all owner counters become zero. State changes, reporting filters, release, last-owner cleanup.

Externally supplied reservations remain distinguishable throughout flow construction. The preparation path validates available PID/effective-PID flow information; socket tokens can lack that information. It sets NXFLOWREQF_EXT_PORT_RSV or NXFLOWREQF_EXT_PROTO_RSV, and the entry records corresponding FLOWENTF_EXTRL_* flags. Request cleanup and final entry cleanup avoid releasing externally owned reservations. Moving a pointer into fe_port_reservation does not transfer the external owner's release obligation. Preparation validation, pointer/flag transfer, request cleanup, entry cleanup.

7. Protocol tokens and compact flow IDs

protons uses an RB tree keyed by protocol number and one mutex. A new token receives a tree reference and a caller reference. A repeated reservation with the same stored PID/effective-PID pair retains the token; a different pair gets EEXIST. New-token construction normalizes an effective PID of -1 to the owner PID, while existing-token comparisons use their supplied values; callers must respect the actual identity contract rather than assuming every helper performs normalization. Representation and lookup, reservation.

When release leaves only the tree reference, the module removes the token, drops that reference, clears valid/identity fields, and frees storage. Initialization reserves the protocols in both BSD Internet domain protocol lists with PID/effective PID zero. It does not create an unowned protocol merely because no Skywalk flow currently uses it. Release, BSD reservations.

flowidns supplies a separate 32-bit ID. Its top two bits encode one of four domains—IPsec, flowswitch, inpcb, PF—and each domain has its own mutex, RB tree, and allocation/release/collision counters. Allocation draws a random candidate, substitutes the domain bits, and retries collisions against live IDs, with a bounded collision assertion. The stored flow key is metadata associated with the allocated ID; this is not the flowswitch's tuple lookup hash. Release decodes the domain, finds the exact ID, removes its node, and panics if an expected live ID is missing. Domain encoding, allocation/release.

Flows originating from BSD can reuse nfr_inp_flowhash and set FLOWENTF_EXTRL_FLOWID; other entries allocate their own ID. Only internally owned IDs are released by final entry cleanup. During TX, packet metadata switches from the registration UUID to the compact scheduling token. See flow ID selection, final release, and TX metadata.

8. Keys are normalized to local and remote endpoints

Although the fields are named src and dst, a registered key means local and remote. TX packet fields already have this orientation; RX swaps source/destination addresses and ports before lookup. Without this normalization, the incoming half of a connection would hash differently from its registration.

Source: bsd/skywalk/nexus/flowswitch/flow/flow_var.h, lines 802–815.

	if (input) {
		if (flow->flow_ip_ver == IPVERSION) {
			key->fk_src4 = flow->flow_ipv4_dst;
			key->fk_sport = flow->flow_tcp_dst;
			key->fk_dst4 = flow->flow_ipv4_src;
			key->fk_dport = flow->flow_tcp_src;
		} else {
			key->fk_src6 = flow->flow_ipv6_dst;
			key->fk_sport = flow->flow_tcp_dst;
			key->fk_dst6 = flow->flow_ipv6_src;
			key->fk_dport = flow->flow_tcp_src;
		}
	} else {
		if (flow->flow_ip_ver == IPVERSION) {

For an illustrative IPv4 UDP flow registered as local 192.0.2.10:53000, remote 198.51.100.20:443, both packets below produce that same local/remote key:

The addresses are illustrative, not values copied from a running kernel. Packet normalization.

flow_req2key() clears the 48-byte key, sets fields and mask bits only for specified endpoints, and accepts a fixed set of masks:

These names count different logical endpoint components; the 5-tuple mask also includes IP version. A declared supported mask is not proof that every public registration path can construct it: flow_req_prepare() imposes its additional endpoint constraints first. Mask definitions, request conversion.

9. Ordered lookup and retained results

flow_mgr_create() installs the mask order shown above: 5, 4, 3, 2, IPFLOW3, IPFLOW2, IPFLOW1. The manager tracks counts for each mask. Lookup skips unused masks, temporarily places the candidate mask into the packet key, hashes only selected fields, and queries the cuckoo table. The first match wins. A wildcard TCP/UDP listener therefore follows the more specific TCP/UDP masks, and IP-protocol fallback follows those. This is a fixed ordered search, not a general longest-prefix algorithm. Mask order, masked hash.

Source: bsd/skywalk/nexus/flowswitch/flow/flow_manager.c, lines 1150–1163.

	for (int i = 0; i < FKMASK_IDX_MAX; i++) {
		size_t count = fm->fm_flow_hash_count[i];
		uint16_t mask = fm->fm_flow_hash_masks[i];
		if (count == 0 || mask == 0) {
			SK_DF(SK_VERB_FLOW | SK_VERB_LOOKUP,
			    "[%d] mask=%08x count=%zu skiped",
			    i, mask, count);
			continue;
		}
		key->fk_mask = mask;
		hash = flow_key_hash(key);
		node = cuckoo_hashtable_find_with_hash(fm->fm_flow_table, key, hash);
		SK_DF(SK_VERB_FLOW | SK_VERB_LOOKUP,
		    "[%d] mask=%08x hash %08x node %p", i, mask, hash,

Two-tuple matching omits IP version; after a match, the manager explicitly rejects an IPv4-only listener for an IPv6 packet. It releases that result and continues. It restores the caller's original mask before returning. Family guard and completion.

The cuckoo table first examines the primary then alternate bucket under a table resize read lock. Each bucket has its own mutex; a hash match narrows the candidates, and the manager's comparison callback performs masked equality. The result is retained before dropping the bucket lock:

Source: bsd/skywalk/lib/cuckoo_hashtable.c, lines 489–500.

		if (b->_slots[i]._hash == hash) {
			node = b->_slots[i]._node;
			while (node != NULL) {
				if (h->_obj_cmp(node, key) == 0) {
					h->_obj_retain(node);
					goto done;
				}
				node = cuckoo_node_next(node);
			}
		}
	}

This table is not lock-free. A returned flow_entry stays allocated through the retained reference, but can still be logically torn down or unusable; RX/TX callers inspect flags and release rejected results. Insertions try either bucket, then relocation, and can expand/retry on ENOMEM; deletion removes the node and drops the table's reference. The periodic reaper can shrink the table. See lookup locking, insertion and expansion, deletion, and manager callbacks.

lookup_flow_with_pkt() can reuse the preceding retained flow in a batch when its five-tuple matches, avoiding another table search. RX subsequently applies listener destination checks and torn-down/offload checks. TX additionally verifies that the packet's flow UUID equals the selected entry's UUID. Matching packet header bytes alone does not establish the full TX admission decision. Batch shortcut, RX checks, TX checks.

10. UDP child flows add a second lookup stage

Child registration supplies a parent UUID and one or more bounded payload patterns; the preparation path restricts this mechanism to UDP and validates nonzero pattern lengths and the maximum offset-plus-length. Entry allocation requires the conflicting tuple to be the designated parent. The child enters the parent's locked child list instead of the cuckoo table, while also participating in its owner's UUID tree. Request validation, parent selection, publication.

RX first finds the parent by packet key, then walks children under the parent child-list read lock. A child must be viable and have UDP payload. A matching pattern compares (payload bytes, value, mask); any one pattern match succeeds. If the bytes are not available in the first packet buffer, the mbuf-backed path can use m_copydata() into scratch storage; the native short-buffer branch records an error and does not match. The first matching child is retained. TX selects a child by the packet's flow UUID, then still performs the normal TX identity/state checks. Payload matching, RX/TX child lookup.

Mermaid source

flowchart TD
    P["Classified packet"] --> K["Normalize local and remote fields"]
    K --> T["Ordered masked lookup"]
    T --> E{"Parent flow?"}
    E -->|no| V["Validate selected entry"]
    E -->|RX| R["Compare UDP payload patterns in child-list order"]
    R -->|matching child| C["Retain child and release parent result"]
    R -->|none| V
    E -->|TX UUID differs| U["Find child UUID"]
    U -->|found| C
    U -->|absent| X["No usable TX flow"]
    C --> V
    V --> D["Packet route and ring processing"]

This only classifies bytes according to registered patterns. It does not implement a userspace transport's handshake, retransmission, or security protocol.

11. Evidence, tests, and remaining work

The local tests provide useful executable descriptions: skt_flowswitch_ns_reserve.c checks duplicate listeners, cross-flowswitch exclusion, BSD conflicts, and distinct TCP/UDP namespaces; skt_flowlookup.c drives TCP and IP-flow send/receive/listener cases; cuckoo_hashtable_test.c exercises the underlying container. They were read as source evidence, not run as kernel tests for this documentation.

This chapter covers namespace accounting and lookup implementation. The datapath companion explains how parsed packets reach those lookups and where the selected packets go. Full NECP policy evaluation, PF/IPsec internals, payload-demultiplexing concurrency beyond the visible locks, and provider-specific offload behavior remain separate research areas; the presence of a token, pattern, or callback does not establish their complete implementation.