Skywalk netif agents, virtual-port flows and traffic rules

Networking · Netif servicing · Filter reinjection · Controllers and port lifetime · Policy overview

Netif has two related control mechanisms. A network-agent request can assign a process a bound port for an interface filter, custom Ethernet traffic or a low-latency IPv6 connection. A traffic-rule request can associate packet fields with a logical-link queue-set ID. Neither operation itself sends a packet. This chapter follows the local xnu-12377.121.6 bodies from admission through publication, packet lookup and removal, and separates their state from the flowswitch's independent flow manager.

1. Separate control identity, receive ownership and queue selection

Sources: agent and netif state, port and receive-flow structures, rule base, INET subtype.

A netif receive table is also different from the flowswitch's masked flow-key table. Its locally supplied implementations are simple lists keyed by EtherType or an IPv6 address pair. A traffic-rule table is a third lookup structure, organized by interface name and supported field-mask order.

Mermaid source

flowchart TB
    N["NECP nexus request"] -->|"kernel agent callback"| A["netif_agent_flow and port binding"]
    C["Controller flow-add request"] -->|"internalized process request"| A
    A -->|"UUID, port and key returned"| U["User opens a channel"]
    U -->|"filter channel activation"| F["netif_filter callback list"]
    U -->|"virtual-port activation"| V["netif_flow receive table"]
    R["Privileged controller traffic-rule ioctl"] -->|"publish rule"| T["INET or ETH rule table"]
    P["Packet"] -->|"classify for delivery"| V
    P -->|"lookup steering hint"| T
    T -->|"qset ID"| Q["Existing logical-link qset"]

The arrows identify different uses of packet classification. The figure does not imply that a traffic rule authorizes a channel or that every packet goes through all three tables.

2. Agent registration and the local NECP boundary

nx_netif_agent_get_features() advertises interposition when filters are initialized. An initialized ordinary Ethernet flow facility advertises custom-Ethernet support; a low-latency netif advertises nexus-provider support instead. Those features also set network-provider status. If there are no features, agent initialization stops. Feature selection, filter initialization gates, flow initialization gates.

Registration creates a kernel netagent session with nx_netif_netagent_event and nif as context, generates an agent UUID, and advertises the Skywalk / NetIf service. Its nexus data describes link frames, address assignment, unsupported assertions and user-packet-pool support. Low-latency service additionally suggests an event ring. After netagent_register() succeeds, initialization adds the agent UUID to the interface and sets ADDED. Session registration and interface association are distinct stages; failure to add the UUID unregisters the successful session. Registration body, interface association.

There is an implementation wrinkle on registration failure: the local helper destroys the failed session and clears its pointer/UUID, but then sets REGISTERED unconditionally before returning the error. The initializer returns on that error; this is not evidence of a usable registration. Describe the successful path from the actual return value and later ADDED state, rather than treating the flag alone as proof of successful construction. Failure and flag order.

For a kernel agent, network-agent dispatch retains the registration's lock across the callback while releasing the global registration-list lock. The callback must not recursively register or unregister agents. Final registration freeing takes that lock exclusively after list removal to wait out such in-flight callbacks. This is a local lifetime boundary around the netif callback, not merely a convention attached to its function name. Callout locking, final callback drain.

nx_netif_netagent_event() handles insert, remove and abort. Insert chooses interposition first, then custom Ethernet, then low-latency IPv6. Other events return success without work. The add dispatcher also starts with zero error and has no final rejection branch for a request selecting none of those modes; such a call can return zero without an assignment message. The advertised capability and upstream NECP decisions therefore matter. Remove/abort both reach flow deletion; the local deletion helper ignores its abort argument. Event dispatch.

The local handler receives already-prepared necp_client_nexus_parameters. It implements assignment and cleanup, not the entire policy decision that selected this agent. The NECP client/flow companion follows upstream registration publication, synchronous/asynchronous assignment-buffer ownership, result copyout and close; NECP policy evaluation explains how interface/agent choices are formed. Userspace Network Extension policy production and application-side channel reuse are external to these bodies. Flowswitch transport-flow/AOP steering is a separate path from this netif agent.

3. Binding a flow does not activate a packet receiver

The three admitted netif flow forms have different checks:

All require an interface-added agent. Add rejects absence of a supported type. Flag admission.

Custom EtherType assignment rejects IP, ARP, reverse ARP, VLAN, IPv6, PAE and RSN preauthentication, and rejects values at or below ETHERMTU. This is not the same allowed-value set as the Ethernet traffic-rule facility below. The IPv6 validator requires AF_INET6, exact sockaddr length and a nonunspecified address; despite the ULA name, it deliberately does not require a unique-local prefix. EtherType validation, IPv6 validation.

For a nonlistener, nx_netif_netagent_flow_bind() resolves the PID to a referenced process, generates a random UUID key, and sets NXBF_MATCH_UNIQUEID and NXBF_MATCH_KEY. It uses proc_uniqueid(p), not just the numeric PID, to prevent a later process reusing that PID from satisfying the same identity. nxdom_bind_port() chooses a port from NEXUS_PORT_ANY; on success it retains the allocated key/port information. On failure, the caller frees both provisional objects and releases the process reference. The result returns the key and chosen port in nx_flow_req. Binding construction and cleanup.

The critical identity fields are explicit:

nx_netif_netagent.c:287–299:

	nfr->nfr_proc = p;
	uuid_generate_random(uuid_key);
	bzero(&nxb, sizeof(nxb));
	nxb.nxb_flags |= NXBF_MATCH_UNIQUEID;
	nxb.nxb_uniqueid = proc_uniqueid(p);
	nxb.nxb_pid = pid;
	nxb.nxb_flags |= NXBF_MATCH_KEY;
	nxb.nxb_key = sk_alloc_data(sizeof(uuid_key), Z_WAITOK | Z_NOFAIL,
	    skmem_tag_nx_key);
	nxb.nxb_key_len = sizeof(uuid_key);
	bcopy(uuid_key, nxb.nxb_key, nxb.nxb_key_len);

	err = nx_netif_netagent_fill_port_info(nif, nfr, &npi);

The add routine allocates a netif_agent_flow, obtains a MAC address for custom Ethernet/IPv6, then holds nif_agent_lock across binding and list publication. It copies the returned binding identity into the record and inserts it into nif_agent_flow_list. The separate NECP wrapper constructs a nexus-assignment message containing nexus UUID, port and key, plus MAC/endpoints where applicable. Failure to allocate that message invokes flow deletion, so a successful port bind is not the final success boundary for the complete request. Add publication, interpose assignment rollback, custom-Ethernet assignment.

Controller flow-add reaches the same core through nx_netif_ctl_flow_add(). It requires a nonzero flow UUID, masks request flags, checks delegated effective identity where applicable, internalizes the request, clears kernel-only pointers and overwrites nfr_pid with the caller PID. The direct controller route must not be documented as allowing an arbitrary caller-supplied binding PID. General controller authorization precedes this handler. Controller admission/internalization, controller ownership.

Neither binding nor recording the agent flow installs netif_filter or netif_flow. Those appear only when the assigned port's channel activates. A request can therefore own an assignment while there is no active packet consumer.

4. Low-latency address reuse and listener assignments

Before creating a new record, add attempts low-latency duplicate lookup. It compares PID, IPv6-flow flag and the local/remote IPv6 addresses against recorded assignments. It does not compare TCP/UDP ports or a transport protocol. A match returns the existing port/key and creates no new netif_agent_flow for the second request UUID. This supports external channel reuse; it is not a second independently owned port. The lookup and subsequent new binding occur in separate nif_agent_lock intervals, so this function alone is not an atomic check-and-create transaction. Tuple comparison, duplicate lookup, add ordering.

A listener takes a different branch before allocation/binding: obtain the MAC address and generate a local IPv6 address. Generation uses fd as the first byte, five random bytes for the global ID, zero subnet bytes and a monotonically increasing nonzero interface-ID component. The assignment response has a zero nexus UUID, NEXUS_PORT_ANY and no key. There is no listener port or agent-flow record to tear down. Nonlisteners require both supplied endpoint families to be IPv6 before binding. Address generation, listener operation, distinct assignment responses.

For example, two sequential nonlistener requests with different UUIDs but the same PID/address pair can receive the same port/key. A delete lookup is by the UUID stored in the original record; the second UUID is not inserted as an alias. A consumer must preserve the intended shared-channel lifecycle rather than treating every successful assignment response as a new independent record. This is a consequence of the inspected lookup/add/delete bodies, not a claim about how the external library manages every reuse case.

5. Channel activation publishes the actual receive flow

An ordinary netif virtual adapter reads its netif_port_info under the netif write lock. For low latency it first sets up the hardware adapter. It then calls nx_netif_flow_add() with the port's descriptor and netif_deliver_cb. Failure unwinds the hardware setup; success stores the netif_flow pointer and sets active state. A filter channel instead inserts the ordered callback described in filter activation. Virtual-port activation.

The first receive flow creates a table using EtherType operations for an ordinary netif or IPv6-address-pair operations for low latency. Both local table implementations are linked lists. Insert rejects a duplicate port or matching description. Ordinary Ethernet extraction supports hardware VLAN metadata and an inline VLAN header, but accepts only a zero VLAN ID (including priority-tagged traffic). IPv6 extraction uses local=source/remote=destination for outbound validation and reverses them for incoming classification. Table construction and insert, flow add, Ethernet extraction, IPv6 direction.

Classification normally checks the relevant local MAC address: outbound source or inbound destination. Ordinary inbound multicast/broadcast is accepted at this check; outbound multicast/broadcast source is rejected. Low latency rejects multicast/broadcast at the selected address check and skips local-address equality for inbound unicast. A development accept_all mode bypasses ordinary classification and returns the first table entry; it should not be generalized into the production matching contract. MAC checks, test override.

nx_netif_flow_classify() acquires nif_flow_lock, checks enabled/table state, performs lookup and increments the matched flow's callback reference before unlocking. nx_netif_demux() builds same-flow chains, retaining only one reference for a continuing run, calls delivery outside the lock and releases afterward. Unmatched packets are returned through remain when supplied; without that output they are freed. Its zero return is not a per-packet delivery guarantee. Reference acquisition/release, demultiplexing ownership.

The virtual-port delivery callback uses RX ring 0 and separates packet ownership by pool. If the input already belongs to the ring's pool, it finalizes and attaches that same packet. Otherwise nx_netif_pkt_to_pkt() allocates/copies a destination packet and consumes the original on both success and failure; ENOMEM from the wrapper does not return the original to the caller. Ring-disabled delivery frees the chain, while ring exhaustion frees its unprocessed suffix. The loop increments cnt after each attempted copy/attachment, including copy failures; consequently its VP_DELIVERED/VP_LL_DELIVERED counter and notification condition do not prove that exactly cnt slots received packets. Only successful attachments update last_slot, which controls slot publication. Same-pool attachment and consuming copy, ring delivery and counting, copy success/error cleanup.

Receive-flow removal deletes table/list visibility under nif_flow_lock, waits for active callback references, destroys the table after the last flow and frees the record. Virtual-adapter deactivation then clears its pointer, tears down low-latency hardware setup and clears active state. Disable only changes admission; remove performs the reference drain. Flow removal, adapter off, enable state.

6. Agent deletion, channel destruction and interface detach

Agent deletion validates flags, treats explicit listeners as a no-op and searches its list by flow UUID. It takes the nexus port from the found record rather than trusting the request. It attempts unbind, but even an unbind error removes/frees the agent record and returns zero; the code allows for the channel having already closed the port. It does not claim that every unbind attempt succeeded. Deletion.

Interface finalization first clears ADDED and removes the interface's agent UUID, then unregisters/destroys the session and purges its recorded assignments. Purge does not individually unbind each port: the nexus detach cleanup owns that work. The network-agent registration drain described above is part of why the callback context can be released afterward. Finalization, purge ownership.

This produces three separate teardown observations: the agent may no longer advertise service, a port binding may no longer admit new channel opens, and an existing adapter may still need to remove callbacks/reclaim ring packets. The controller/lifetime and channel lifecycle chapters follow those broader resources.

7. Traffic-rule ioctl admission and stored representation

The controller fd's ioctl dispatch recognizes add-INET, add-ETH, remove-by-UUID and get-rules. Root bypasses the additional entitlement test; other callers need com.apple.private.skywalk.traffic_rule.write for mutation or .read for enumeration. The outer ioctl layer handles the fixed argument copyin/copyout; get-rules additionally copies an array through the address in its arguments. File operation, entitlement and dispatch, ioctl layouts.

An add argument supplies interface name, typed traffic descriptor, a steering-action structure and persistence flags; the kernel returns a generated rule UUID. A rule copies the descriptor/action into its INET or ETH subtype and records process name for reporting. The base record has an os_refcnt_t; no PID/unique-ID ownership field is present there. The process-name string is descriptive metadata, not the binding capability used by agent ports. Common/subtype representation, copied immutable rule fields.

Validation and lookup are dispatched through nxctl_traffic_rule_type. A global read/write lock protects type storage and rule publication. A second nxctl_lock protects each controller's nonpersistent cleanup list. Persistent means “not removed on this controller fd's close”; it does not mean written to disk or retained across boot. Type operations, locks, persistent linkage decision.

8. Validation and mask order define rule semantics

Both families validate a syntactically extractable interface name, exact descriptor length, and a nonzero direction flag subset containing only inbound/outbound. They do not require a currently attached interface. Both require the steering-action structure's length; the inspected validators do not check ra_type or prove the requested qset ID exists. Actual qset lookup/provider notification happens later. INET validator, ETH validator.

The INET mask table is an ordered policy, not a sorted-by-popcount calculation. All protocol-bearing masks come before protocol-omitting masks; within each half, local-address selections precede remote-only or port-only selections in the literal table order. First matching list wins. A newer rule is inserted at the head of its mask list, although duplicate detection normally prevents identical selected keys. INET order, lookup order, ETH order.

For example, a protocol+local-address rule can precede a protocol+remote-address+both-ports rule even though the latter selects more fields. If both match the packet, the earlier mask's rule wins. A complete tuple precedes both. “Most specific rule wins” without defining the actual mask ordering would therefore misdescribe this implementation.

The matchers compare the intersection of rule and query masks. Exact duplicate search first restricts traversal to the supplied mask, then compares its selected fields. Ordinary packet-derived INET queries set every field bit, so intersection reduces to the rule's mask. A caller of the lower-level descriptor API supplying fewer bits must not assume the matcher requires every rule bit to be present. Both local matchers ignore itd_flags; direction flags are validated/stored and passed to the provider, but do not partition local software lookup or exact duplicate identity. Thus an otherwise identical inbound-only and outbound-only rule of the same mask collides here. INET matching, ETH matching.

The selected-field test begins with the intersection, not a subset requirement:

nexus_traffic_rule_inet.c:521–531:

	exact = ((flags & NTR_FIND_FLAG_EXACT) != 0);
	mask = tdi0->inet_mask & tdi->inet_mask;
	if (exact) {
		ASSERT(tdi0->inet_mask == tdi->inet_mask);
	}
	ASSERT((mask & IFNET_TRAFFIC_DESCRIPTOR_INET_IPVER) != 0);
	if (tdi0->inet_ipver != tdi->inet_ipver) {
		DTRACE_SKYWALK2(ipver__mismatch,
		    uint8_t, tdi0->inet_ipver, uint8_t, tdi->inet_ipver);
		return FALSE;
	}

9. Publication succeeds before optional provider notification

Under the traffic-rule write lock, add:

1. Selects a known rule type and checks that no other family already has rules for that interface name.
2. Validates descriptor/action and rejects an exact duplicate with EEXIST.
3. Allocates and initializes the copied rule, including its initial reference and generated UUID.
4. Links nonpersistent rules into the requesting controller's cleanup list, taking a reference.
5. Links into per-interface/type storage, taking another reference and updating per-interface/global counts.
6. Takes a temporary reference for its caller and drops the global lock.

The per-interface count is copied into an attached ifnet when available. Netif attachment also retrieves preexisting counts, allowing rules to exist before the interface attaches. Add transaction, INET publication, attachment refresh.

Notification occurs outside the rule lock. Its error is discarded before the add wrapper returns the UUID:

nexus_traffic_rule.c:470–478:

	err = add_traffic_rule(nxctl, ifname, td, ra, flags, &ntr);
	if (err != 0) {
		return err;
	}
	(void) notify_traffic_rule(ntr, NTR_NOTIFY_FLAG_ADD);
	uuid_copy(*uuid, ntr->ntr_uuid);
	release_traffic_rule(ntr);
	return 0;
}

The notifier obtains an interface I/O reference, requires a logical-link provider, finds/retains a qset and calls the configured qset-extension qe_notify_steering_info with the copied descriptor and add/remove boolean. INET explicitly rejects a null or not-externally-initialized qset; ETH's notifier does not repeat that check. The shared callback wrapper checks the capability, then dereferences the qset to provide its driver context. The notifiers retain TODOs about detach/attach safety. These differences are implementation limits, not evidence that every valid rule was installed in hardware. INET notification, ETH notification, provider callback.

Mermaid source

sequenceDiagram
    participant C as Controller
    participant R as Rule storage
    participant I as Interface and qset
    participant D as Provider
    C->>R: Validate under write lock
    R->>R: Link rule and take storage references
    R->>C: Retained rule after unlock
    C->>I: Find interface and retained qset
    I->>D: Notify steering add if supported
    D-->>I: Success or failure
    I-->>C: Notification result
    C->>C: Discard notification error
    C-->>C: Return rule UUID and success

The final self-message represents the wrapper's ioctl result, not transmission. Software publication remains even when the interface was absent or notification failed; there is no automatic rollback of the rule for that failure.

10. From packet bytes to a retained qset

INET packet lookup parses accessible bytes beginning after pkt_l2_len. It checks IPv4 header/total lengths, or the fixed IPv6 header/length, then accepts only a direct TCP or UDP transport header. It does not walk IPv6 extension headers or perform IP-fragment reassembly. Source and destination are copied into inet_laddr and inet_raddr; ports are copied in their packet/network byte order. This packet helper follows the output-oriented source/destination interpretation, whereas other descriptor callers must construct the intended query themselves. INET parser.

The query sets all INET mask bits and both direction flags. Under the traffic-rule read lock, lookup finds the first match and copies only ras_qset_id to its output. It does not return a rule pointer that survives unlocking. This makes immutable rule storage plus the read lock sufficient for that lookup's rule lifetime. Query normalization, lookup output.

The ETH helper extracts the header EtherType and destination MAC, then searches EtherType lists before destination-address lists. It is a narrower parser than the netif receive-flow VLAN parser above: do not borrow that parser's VLAN normalization or full-header validation contract for this one. For example, a PAE frame matching both a PAE rule and a destination-MAC rule selects the PAE rule first. ETH query/parser.

nx_netif_find_qset_with_pkt() uses these rules only for logical-link providers. ETH rules may instead use a previously stored PKT_F_PRIV_HAS_QSET_ID/pkt_priv hint; otherwise it parses the packet. INET rules parse when their count is nonzero. The selected integer is then resolved by nx_netif_find_qset(), which holds link-list protection and returns a retained qset/parent link, with ordinary fallback to the default qset for an unusable hint. A copied qset ID is not a lifetime reference; this second lookup supplies that protection. Rule/hint selection, qset lookup and lifetime.

Consequently, publishing a rule for ID X does not prove that X remains valid, that every lookup returns exactly X, or that a driver honored the steering notification. Software lookup, qset fallback and external steering are distinct decisions.

11. Removal, controller close and enumeration

Explicit removal searches all rule types by UUID under the global write lock. It takes a temporary reference for later notification, unlinks controller membership for nonpersistent rules, removes type/interface membership and its reference, releases the rule's initial reference, then unlocks. Notification runs afterward and its error is again ignored. The temporary reference prevents freeing the copied descriptor while the notifier uses it; its final release invokes the type destructor if no other references remain. Removal, ioctl notification, type unlink and storage destruction, final release.

The local removal body uses the passed controller for nonpersistent cleanup linkage; the generic record has no explicit owner pointer or identity check. This is not a capability scheme equivalent to the unique-ID/key-bound agent port. The supported lifetime trace here is removal using the controller that owns the nonpersistent list, plus removal of persistent rules. Do not infer safe arbitrary cross-controller nonpersistent removal merely from global UUID lookup.

Controller close detaches the whole cleanup list under nxctl_lock, destroys its list container, then unlocks before global removal. It clears ON_NXCTL_LIST on each record to avoid a second unlink, removes/optionally notifies, and releases the retained list reference. Persistent rules were never linked there. This avoids holding the controller lock throughout the global-removal loop. Close cleanup, controller destructor call.

Enumeration holds the traffic-rule read lock while the selected type checks exact record size, verifies supplied capacity and copies records to userspace. Too-small capacity returns ENOBUFS; this body does not update the count to the required size on that failure. A copyout fault may leave an already-copied prefix in user memory even though the call fails. Only successful completion writes the final total count. Enumeration wrapper, INET copyout, ETH copyout.

12. Worked lifetimes and verification boundaries

For an interface filter, a successful agent/controller add produces port P and key K. Opening P with the matching process identity and K creates the adapter; activation inserts the callback. Packets then traverse that callback until adapter deactivation removes it and waits for executing callbacks. Agent deletion removes the assignment record and attempts unbind; it is not itself the entire ring/packet destructor. The filter companion follows the retained packet objects through this lifecycle.

For a nonpersistent steering rule, assume a syntactically valid interface name and descriptor but no attached interface. Add can publish the rule and return UUID R even though notification cannot find the interface. Subsequent attachment refreshes the rule count, and packet software lookup can then use the rule. Closing its controller removes the rule through the detached-list cleanup. A persistent R skips that close cleanup and requires explicit removal. The example does not promise a later replay of the original external notification.

The in-tree skt_steering.c exercises family exclusion, duplicate rules, invalid Ethernet masks/values and removal, while skt_filter.c demonstrates direct controller flow assignment and a key-bound filter channel. These were inspected as source examples, not executed. Steering tests, filter assignment example.

This chapter establishes the supplied netif control, receive-table and steering-rule bodies. It does not implement external NECP policy, provider queue programming, hardware steering, product entitlements or every race/failure permutation. Network-agent callout lifetime was followed only far enough to establish the callback/drain boundary; the complete agent protocol and socket-policy engine remain separate research areas.