Skywalk fragment ordering, reassembly queues, and flow tracking

Networking · Flowswitch datapath · Flow lifetimes · TCP aggregation and segmentation

The flowswitch needs enough transport context to select a channel, observe a connection, and release its namespace safely. Two mechanisms supply that context: a per-interface fragment manager collects an ordered set of IP fragments, and per-flow tracking observes TCP/UDP traffic. Neither object is a BSD socket or tcpcb. This chapter follows their implementation in xnu-12377.121.6, including retained packets, incomplete input, overlap, expiry, close notification, and generated abort traffic.

1. Where fragments enter the receive path

_fsw_receive() demultiplexes the link header and calls flow_pkt_classify() before rx_process_ip_frag(). Classification extracts fragment identity and offset state; a noninitial fragment has no independently usable transport ports. The manager can wait for the first fragment and return an ordered chain whose first packet supplies the lookup key. See RX dispatch, fragment parsing, and key normalization.

rx_process_ip_frag() has three modes. Forced-off returns the input immediately; forced-on invokes the manager; automatic mode invokes it only when the flowswitch netagent is enabled and the flow manager reports at least one flow. IPv6 support here assumes the Fragment header immediately follows the base IPv6 header. Atomic IPv6 fragments are expected to have been classified as nonfragmented traffic. This is not a general extension-header reassembly engine. Mode and call sites, IPv6 wrapper.

The result is a combined error/pointer ownership contract:

Documented contract and actual publication:

Source: bsd/skywalk/nexus/flowswitch/fsw_ip_frag.c, lines 779–787.

done:
	/* ipfm take ownership of, or return assembled packet, if no error */
	if (err == 0) {
		/* reass'ed packet if done; NULL otherwise */
		*pkt_ptr = pkt_reassed;
	}
	ipfq_sched_timeout(mgr, FALSE);
	lck_mtx_unlock(&mgr->ipfm_lock);
	return err;

A NULL from the outer RX helper therefore means either retained input or an input already freed on error. It is not an instruction to retry or reclaim the former packet handle.

2. The manager owns queue wrappers and original packets

fsw_setup_ifp() creates fsw_ip_frag_mgr and stores it in fsw_ipfm. Its caller supplies a fragment limit appropriate to the device-pool capacity. The manager records the interface and flowswitch statistics, owns a queue-list sentinel, counts both queues and fragment wrappers, and allocates a one-shot thread call for timeout work. Attachment, manager creation.

Definitions, key comparison.

IPv4 copies eight address bytes and preserves the identification field's representation consistently between insertion and lookup. IPv6 copies 32 address bytes and its 32-bit identification. The family distinction follows ipfk_len; ipf_key_cmp() compares that length, identification, then exactly those address bytes. Lookup walks the manager's queue list rather than using the flow manager's hash table. Key creation, IPv6 key, linear lookup.

Mermaid source

flowchart TD
    M["fsw_ipfm: manager mutex and queue counts"] --> Q["ipfq: addresses + ID, creation time"]
    Q --> F0["ipf offset 0"]
    F0 --> F1["ipf offset 1024"]
    F1 --> F2["ipf offset 2048, final"]
    F0 --> P0["Original packet 0 with IP header"]
    F1 --> P1["Original packet 1 with IP header"]
    F2 --> P2["Original packet 2 with IP header"]
    Q -->|complete coverage| C["Release wrappers and queue"]
    C --> R["Return packets linked by pkt_nextpkt"]
    Q -->|timeout, pressure, or overlap| D["Free stored packets and wrappers"]

The fragment links and returned packet links are different fields. A queue wrapper can disappear while its packet remains alive in the returned chain.

3. Admission, ordering, overlap, and completion

ipf_process() performs these decisions under ipfm_lock after its initial length check:

1. Require every fragment with the more-fragments bit to contain a nonzero multiple of eight fragmentable bytes.
2. Find the queue by key. A dirty queue rejects this input. A miss allocates and inserts a new queue, initializes unknown unfragmentable length to -1, and records creation time.
3. Apply the per-queue fragment-count check. When offset zero arrives, establish the unfragmentable length and recheck already held fragments against the maximum IP size.
4. Allocate an ipf and locate the first stored fragment whose offset is greater than the new offset.
5. Compare the new interval with its neighbors. Overlap can free all stored fragments and mark the queue dirty; the incoming packet remains caller-owned on the error return.
6. Insert in offset order. Starting with next = 0, require every fragment's offset to equal the preceding cumulative length and require the final fragment's more-fragments bit to be clear.
7. Link original packets with pkt_nextpkt, free wrappers and the queue, and return the first packet.

Admission and initial state, first-fragment recheck, ordering and overlap, completion.

The complete-coverage test is small enough to show directly:

Source: bsd/skywalk/nexus/flowswitch/fsw_ip_frag.c, lines 739–752.

	ipf_enq(f_new, f->ipf_up);
	q->ipfq_nfrag++;
	next = 0;
	for (f = q->ipfq_down; f != (struct ipf *)q; f = f->ipf_down) {
		/* there is a hole */
		if (f->ipf_off != next) {
			goto done;
		}
		next += f->ipf_len;
	}
	/* we haven't got last frag yet */
	if (f->ipf_up->ipf_mff) {
		goto done;
	}

The overlap code is stricter than a general “identical offset/length is always accepted” rule. After detecting overlap, it tests offset/length equality against each existing neighboring node; a differing next neighbor can select the dirty-queue branch even when the predecessor matches the new interval. It does not compare payload bytes to establish byte-for-byte duplicate identity. Its duplicate branch discards the incoming wrapper and returns ERANGE for the caller to free the incoming packet. Avoid replacing this implementation with a generic IPv4 or IPv6 overlap policy. Exact predicate.

There are other comment/body differences worth preserving during upgrades. The insertion comment says to move an active queue to the front, but that block calls only ipf_enq() for the fragment, not ipfq_remque()/ipfq_insque() for its queue. The timestamp is set at queue creation and is not refreshed for each fragment. Thus the inspected code does not implement a sliding inactivity timeout or queue-level LRU update on every insertion. Creation, insertion.

4. What “reassembled” means here

The success block joins the original __kern_packet objects. It does not concatenate payload into one allocation, clear the fragment bits in their wire headers, or rewrite the first IP header into a full datagram. The wrappers recompute pkt_flow_ulen from the first packet's header, not from a newly synthesized aggregate IP header. The outer helper uses the returned pointer/count and does not consume the tlen output. Packet chaining, IPv4 metadata, IPv6 metadata, caller.

rx_flow_batch_packets() enqueues the list with KPKTQ_ENQUEUE_LIST(). Its first member supplies flow lookup context, while the original fragments remain individual packets for the destination path. fe_rx_frag_count causes the worker to pass FLOW_PROC_FLAG_FRAGMENTS; flow_rx_agg_tcp() then explicitly chooses default RX instead of TCP aggregation. Default channel RX copies each fragment's L3 representation into the channel's pool; the host branch hands fragments to BSD mbuf delivery. List enqueue, worker flag, aggregation bypass, default delivery.

For example, consider IPv4 fragmentable intervals [0,1024), [1024,2048), and [2048,2308), with the first two marked “more.” If the middle fragment arrives first, the queue has a hole at zero. If the final fragment arrives next, the hole remains. Arrival of offset zero completes the interval scan and returns three ordered packets. The receiving channel gets the fragments, not a fourth packet containing a flattened 2308-byte payload. The first fragment can contain the UDP header; subsequent ones do not independently provide UDP ports.

This is why fragment ordering and TCP receive aggregation need separate explanations.

5. Resource limits, expiry, ICMP, and destruction

The defaults are 128 fragments per queue, at most 1024 queues subject to the manager's fragment capacity, a 60-second TTL, and a one-second timeout-call interval. Creation computes the queue limit as MIN(1024, fragment_limit / 2). Runtime controls can change several values; the per-queue fragment control is development/debug only. Defaults and controls, capacity.

These are thresholds implemented by specific predicates, not exact immutable ceilings:

Per-queue check, allocation thresholds, fragment allocation, timer predicates.

Pressure reaping removes queues from the tail until only half remain; draining targets zero. Dirty queues suppress the pressure-reap ICMP callback. Timeout iteration removes expired queues and rearms a delayed call only while queues remain. The interval is clamped to at least one second. The timer is independent of the flowswitch flow/route reaper. Reap/drain, timer and rescheduling.

The timeout body contains two details that should not be paraphrased as stronger guarantees: it overwrites n_freed on each expired queue rather than accumulating it, and its dirty predicate reads the advanced iterator q while freeing q->ipfq_prev. These describe this snapshot's accounting/branch expressions, not a corrected algorithm. Timeout loop.

Parameter and timeout errors here generate IPv6 ICMP errors; IPv4 returns without generating one. Timeout notification applies only to offset-zero fragments. The code obtains an mbuf representation for ICMP processing, and uses sk_async_transmit_protect() around icmp6_error_flag() because this can run with ipfm_lock held: entering a driver workloop synchronously could deadlock. IPv6 first-fragment validation also calls ip6_pkt_has_ulp() before insertion. ICMP conversion, async output protection, upper-layer-header check.

Destruction drops the manager mutex before thread_call_cancel_wait() and thread_call_free(), reacquires it, drains all packet queues, then destroys the mutex and storage. Waiting with the manager mutex held would prevent a running timeout callback from finishing its own locked work. This path relies on the enclosing flowswitch lifecycle to have stopped new users. Destroy, flowswitch teardown.

6. Transmit fragments use a different rule

Ordinary channel TX does not feed fragments to this receive manager. A first fragment resolves a normal flow, and tx_flow_batch_packet() records fe_tx_is_cont_frag plus its fragment ID. A continuation is accepted through tx_process_continuous_ip_frag() only if it has a nonzero ID, a previous flow is available, that flow is marked as expecting continuation, and the ID matches. The selected flow gets another reference before batching. A mismatch drops the continuation. Batch state, continuation predicate, caller.

This helper does not perform an offset-coverage or overlap scan. It uses the current batch's previous-flow state; after processing the batch, the flow's fragment-continuation fields are cleared. It must not be described as a persistent TX reassembly table. GSO-marked IP fragments are rejected by the separate segmentation path. Batch reset, GSO rejection.

7. Tracking records observations in each direction

flow_entry contains fe_ltrack and fe_rtrack. Each flow_track has sequence bounds, last FIN position, largest advertised window, window scale, state, SYN counters and sampled RTT state. Input selects remote tracking as src and local tracking as dst; output reverses those roles. They describe the direction of the observed packet, not two separate socket structures. Tracking definitions, direction and dispatch.

Source: bsd/skywalk/nexus/flowswitch/flow/flow_track.c, lines 716–730.

	if (in) {
		src = &fe->fe_rtrack;
		dst = &fe->fe_ltrack;
	} else {
		src = &fe->fe_ltrack;
		dst = &fe->fe_rtrack;
	}

	flow_track_stats(fe, (pkt->pkt_length - pkt->pkt_l2_len), 1,
	    (pkt->pkt_flow_ulen != 0), in);

	/* skip flow state tracking on non-initial fragments */
	if (pkt->pkt_flow_ip_is_frag && !pkt->pkt_flow_ip_is_first_frag) {
		return 0;
	}

Statistics count the packet's L3 bytes and activity before noninitial fragments skip state tracking. TCP/UDP state updates require FLOWENTF_TRACK; flow construction sets it only for eligible nonlistener requests. Host aggregation instead records raw packet statistics in a batch and asserts that host traffic is not transport-tracked. Tracking admission, host accounting, host completion.

The tracker does not acquire a private mutex in flow_pkt_track(). It runs inside the datapath's lifetime protection and direction-specific servicing: RX uses the per-flow worker protocol, and TX is reached from protected channel/ring processing. Those mechanisms should not be relabeled as one exclusive lock covering both directions; the functions update observational fields directly and use atomic operations for selected shared flags. RX worker, TX call, tracking body.

8. TCP window observation and close transitions

On first use, flow_track_tcp_init() initializes the observed source as SYN_SENT, records sequence bounds and window scale from options, initializes the other side as closed, and selects linger time from 2 * tcp_msl / TCP_RETRANSHZ, clamped to 1–120 seconds. Later first traffic from the opposite side can establish state for a connection picked up midstream. This permits observation without demanding that the kernel saw a complete handshake. Initialization, opposite-side initialization.

The main comparison accounts for payload length, SYN/FIN consuming sequence space, scaled windows, retransmission allowance and ACK skew. It uses wrap-aware SEQ_* comparisons for the main sequence bounds and a MAXACKWINDOW allowance of 0xffff + 1500. A reset additionally needs an exact or adjacent observed sequence in the main branch. Empty packets relax the sequence check; packets without ACK use the peer's observed sequence for the ACK comparison. Normalization, main predicate.

The result of that predicate is permission to advance tracking state, not a universal packet-validation verdict. The final mismatch branch can reset provisional SYN bookkeeping and still return zero. A looser branch handles pickup and close-edge traffic. The actual explicit errors include ENETRESET for inbound traffic after both sides reached at least FIN_WAIT_2, and EPROTO for selected outbound SYN rate excess. Alternative branches, closed pair, rate limiting.

State decisions, flag publication.

Mermaid source

flowchart TD
    P["Observed TCP packet"] --> W["Sequence and window observation"]
    W --> S["Update local and remote tracking fields"]
    S --> H["FIN acknowledged: HALF_CLOSED"]
    S --> C["Both closed or established reset: want_withdraw"]
    H --> R["Flowswitch reaper under owner-bucket protection"]
    C --> R
    R --> N["Half-close or withdraw namespace reservation"]
    R --> T["Teardown and deferred close/nonviable notification"]
    T --> D["Destroy lookup visibility"]
    D --> L["WAIT_CLOSE may retain entry in linger list"]
    L --> E["Expiry releases remaining ownership"]

The reaper converts these hints into namespace operations; flow_entry_teardown() withdraws the reservation and marks the entry torn down. Destruction removes lookup membership and either releases the entry or places it on the linger list. Linger expiry and optional TCP abort are separate work. Reaper state consumption, teardown, destroy/linger choice, linger processing.

The outbound SYN limiter increments an endpoint counter and rejects after 20 qualifying observations. The counter resets when the integer uptime difference is greater than one, so “exactly 20 per sliding second” would overstate the implementation. FTF_RST_RLIM is set in several branches, but the final code contains only a SYN limiter; its name does not establish an implemented reset-rate limiter.

RTT sampling is also observational. A payload starts a sample on the opposite endpoint after a two-second sampling interval; an ACK meeting the recorded beginning-sequence check completes it. The estimate uses (15 * old + sample) / 16, is published in microseconds, and does not distinguish retransmission ACKs. Local RTT approximates local processing; remote RTT includes driver/network/remote delay. RTT state and sampling.

9. UDP observations and generated abort packets

UDP tracking is deliberately small: initialize both directions to NO_TRAFFIC; observing a direction moves it to SINGLE; observing return traffic promotes the opposite direction from SINGLE to MULTIPLE. Repeating packets in one direction alone does not execute that promotion for the sending side. It neither retransmits nor maintains TCP-like windows. UDP implementation.

flow_track_tcp_want_abort() accepts only a non-aborted, non-AOP TCP entry with at least one established side. It changes both states to TIME_WAIT and sets FLOWENTF_ABORTED before asking the caller to generate a reset. This marks the one-shot attempt even if the subsequent nonblocking mbuf allocation fails. Abort admission, allocation failure.

flow_track_abort_tcp() has three input cases: an inbound packet triggering a reply, a supplied listener reset template, or no packet and only stored tracking state. It constructs a new minimal TCP/IP mbuf, computes software checksums, binds output to the flowswitch interface/source, and calls BSD ip_output() or ip6_output(). It does not put the reset back into a user channel's TX ring. Header construction, TCP selection, BSD output.

This snapshot has a consequential literal expression in the inbound case:

Source: bsd/skywalk/nexus/flowswitch/flow/flow_track.c, lines 829–837.

	if (in_pkt != NULL) {
		th->th_sport = in_pkt->pkt_flow_tcp_dst;
		th->th_dport = in_pkt->pkt_flow_tcp_src;
		if (__probable(in_pkt->pkt_flow_tcp_flags | TH_ACK)) {
			/* <SEQ=SEG.ACK><CTL=RST> */
			th->th_seq = in_pkt->pkt_flow_tcp_ack;
			th->th_ack = 0;
			th->th_flags = TH_RST;
		} else {

Because TH_ACK is nonzero, the bitwise OR condition is always true. The inspected body chooses SEQ = incoming ACK, no ACK field, and RST for this case; the alternate RST|ACK block is not selected by testing the incoming ACK bit. The documentation records the actual body, not the apparent intended test, and does not modify kernel source.

QUIC close has a separate bounded mechanism. flow_entry_destroy() can supply a stateless-reset token when close parameters are present and nolinger is false. flow_track_abort_quic() ignores an all-zero token, allocates a new UDP/IP packet, creates 30 random prefix bytes with the required first-byte bit pattern, appends the fixed-size token, computes a checksum and uses BSD output. This is a reset-construction path; it is not a QUIC implementation or token-generation policy. Close dispatch, reset construction, checksum/output.

10. Evidence, tests, and remaining boundaries

The supplied skt_reass.c exercises channel UDP fragment delivery with IPv4/IPv6 lengths including 2000/2001 and 8000/8001 bytes. It has selected ICMPv6 timeout, bad fragment length, atomic-fragment and queue-limit scenarios. Some default-mode tests skip when netagent/reassembly support is absent. Data cases, scenario registration.

Useful review invariants are: an incomplete successful insertion consumes the caller's packet; an error does not return old retained fragments; completion preserves a chain of original packets; a fragment-containing batch bypasses TCP aggregation; statistics precede state-tracking skips; and tracking state changes do not themselves remove the entry from lookup tables. No kernel/network tests were run for this documentation change.

This chapter reviews the local fragment manager and the TCP/UDP observer, including caller consequences. It does not establish full RFC conformance, exhaustively verify every race or malformed-header combination, or reconstruct a userspace transport. BSD IP reassembly and BSD TCP timers are separate implementations. AOP provider behavior, policy decisions and device completion are addressed only at the local boundaries in the offload companion.