Routing control and IPv4 ARP

Networking · Route references and interfaces · IPv6 output and neighbor discovery · Flowswitch routes

A routing socket changes the kernel's route table; ARP supplies a link-layer destination for a directly reachable IPv4 neighbor. Both use rtentry, but they solve different problems. A route to a remote network can name an IPv4 gateway, while the route to that gateway carries its Ethernet address and the packets waiting for resolution. This chapter follows control-message admission, route publication, pending-packet ownership, reply processing and expiration in xnu-12377.121.6. The route-reference chapter remains the prerequisite for cached-route references, lock order and final release.

1. Objects: a request, a route, a next hop and a held packet

Sources: routing message layout, route layout, temporary address table, routing socket attach, and ARP state/lifetime contract.

The mutable ARP fields show why one generic “ARP timeout” is insufficient. From in_arp.c:132–143:

	class_queue_t la_holdq;         /* packets awaiting resolution */
	struct  if_llreach *la_llreach; /* link-layer reachability record */
	u_int64_t la_lastused;          /* last used timestamp */
	u_int32_t la_asked;             /* # of requests sent */
	u_int32_t la_maxtries;          /* retry limit */
	u_int64_t la_probeexp;          /* probe deadline timestamp */
	u_int32_t la_prbreq_cnt;        /* probe request count */
	u_int32_t la_flags;
#define LLINFO_RTRFAIL_EVTSENT         0x1 /* sent an ARP event */
#define LLINFO_PROBING                 0x2 /* waiting for an ARP reply */
};

rnh_lock protects route-tree membership and the global llinfo_arp list. Each entry's rt_lock protects its ARP queue, probe state and link-reachability pointer. Operations needing both take rnh_lock before rt_lock; a reference protects storage while locks are dropped. The diagram uses an illustrative remote destination 198.51.100.9, gateway 192.0.2.1 and connected network 192.0.2.0/24; these are logical relationships, not a claim that every destination necessarily receives a host clone. Lock domains, ARP locking.

Mermaid source

flowchart TB
    C["PCB cached route"] -->|ro_rt reference| R["Route for remote destination<br/>gateway 192.0.2.1"]
    R -->|rt_gwroute reference| H["Direct host route<br/>key 192.0.2.1"]
    H -->|rt_parent| N["Connected network route<br/>192.0.2.0/24"]
    H -->|rt_gateway| L["sockaddr_dl<br/>Ethernet address or sdl_alen zero"]
    H -->|rt_llinfo owns storage| A["llinfo_arp"]
    A -->|la_holdq owns packets| M["Mbuf packet list awaiting resolution"]
    A -->|la_llreach reference| Q["Shared link-reachability record"]

2. Routing socket admission and message decoding

The registered routesw connects pr_output to route_output() and pr_usrreqs to route_usrreqs. rts_send() enters the common raw send implementation, which eventually dispatches through that output operation. rts_attach() deliberately bypasses the raw socket attach handler's SS_PRIV check, attaches its own rawcb, marks the socket connected and enables SO_USELOOPBACK. Permission to create this endpoint does not establish permission to mutate routes: route_output() separately permits RTM_GET and requires SS_PRIV for other operations. Registration, send/operations table, raw output dispatch, attach, mutation gate.

route_output() starts with the socket locked, then unlocks it while keeping the socket reference and takes rnh_lock. It validates packet length against rtm_msglen, allocates contiguous kernel request storage, copies the mbuf chain and checks RTM_VERSION. The temporary buffer stays available through route processing and response construction. The input mbuf will become the response carrier or be freed; it is not installed in the routing table. Entry and validation.

rt_xaddrs() walks RTAX_* indices whose bits appear in rtm_addrs. It checks each address's length before treating the bytes as a sockaddr, rejects lengths beyond sockaddr_storage or the available buffer, and copies undersized but accepted addresses into zeroed tiny_sa_storage. Advancement uses ROUNDUP32, including for a 64-bit process. A zero-length address intentionally selects sa_zero and returns success for compatibility; the parser is not an assertion that every advertised slot contains a complete ordinary address. The operation subsequently requires a valid destination and checks IPv4/IPv6 lengths. Parser, family/size checks.

Scope is explicit: RTF_IFSCOPE makes rtm_index the interface constraint. IPv4's user-provided embedded scope is cleared so bytes in sin_zero cannot silently select an interface. The route module later constructs its internal scoped key/mask. The handler also rejects user RTF_PROXY, disallows the selected internal-coprocessor scope, and applies the configured management-interface entitlement gate. These checks do not substitute for later route/neighbor validation. Scope and restricted interfaces, normalization, internal scoped keys.

RTM_GET_SILENT is converted to RTM_GET after requiring loopback delivery; its response is appended only to the requesting socket. The ordinary route protocol uses the destination's address family as the raw-socket protocol selector when known. Readers correlate request/reply metadata such as rtm_seq; the message stream can also contain unrelated asynchronous route/interface events. Silent request, reply dispatch, asynchronous messages.

3. Mutation, scoped lookup and host clones

These branches are in add/delete, lookup/report, and change/metric locks.

Route construction and publication

For RTM_ADD, rtrequest_common_locked() selects and references a suitable interface address, allocates and locks an entry, builds gateway/key storage, installs the interface reference and inserts the radix nodes. A duplicate may be an existing protocol clone: that entry can be deleted and insertion retried. If insertion still fails, the new gateway/address/key/entry state is unwound and EEXIST returned. This retry can already have removed the older clone; treating the operation as an all-or-nothing transaction would hide that branch. Allocation through insertion, duplicate replacement and cleanup.

Insertion is not the end of construction. While rnh_lock still prevents other lookups, cloning copies metrics and retains the parent; the interface address's ifa_rtrequest callback adds protocol-specific state. The function then updates default-route scope and configured NECP clients, increments the family's route generation, synchronizes the entry, and adjusts affected clones. A supplied ret_nrt receives one reference. The route can otherwise remain cached in the table with a zero external reference count. Post-insertion setup, zero-reference cache/final release.

A regular output lookup uses rtalloc1_common_locked(). After obtaining a route, the report argument and RTF_CLONING/RTF_PRCLONING flags decide whether to call RTM_RESOLVE. A resolve derives flags, gateway, address and metrics from the parent, removes cloning/static flags, marks RTF_WASCLONED, usually creates a host key and scopes the clone to the parent's interface. IPv4 link-local/proxy exceptions remain unscoped. A failed clone can leave the original parent as the returned lookup result; subsequent ARP validation may reject it for lacking usable link-layer state. Lookup/clone dispatch, resolve construction, ARP route validation.

Scoped selection first considers an unscoped candidate, derives a primary interface when none was explicitly requested, and may search for a more specific route within that scope. An incompatible explicit interface can discard the unscoped result. The comparison retains a host over a scoped network route, or a nondefault route over a scoped default where allowed; a final non-scoped default must match the selected interface. It is therefore insufficient to model lookup as just a single destination longest-prefix search. Scoped selection and fallback.

Errors after a state change

A concrete partial-progress example is RTM_CHANGE with a valid new gateway plus an invalid RTV_REFRESH_HOST combination. The gateway update and rt_setif() occur before rt_setmetrics() rejects a mixed refresh request, a static/non-LLINFO route, or missing refresh support. There is no compensating restoration here. RTM_ADD, in contrast, explicitly discards the return from rt_setmetrics(). Gateway mutation, refresh validation, add's ignored return.

The ordering is visible in rtsock.c:793–802:

			rt_setif(rt,
			    info.rti_info[RTAX_IFP], info.rti_info[RTAX_IFA],
			    info.rti_info[RTAX_GATEWAY], ifscope);

			if ((error = rt_setmetrics(RTM->rtm_inits,
			    &RTM->rtm_rmx, rt))) {
				int tmp = error;
				RT_UNLOCK(rt);
				senderr(tmp);
			}

Likewise, a successful RTM_DELETE reaches report allocation only after removal. Failure to allocate that report returns an error without reinstalling the route. The response buffer can later fail to fit back into its mbuf, or a receiver can lack socket-buffer space. Mutation and observation are distinct outcomes. Delete-to-report, report allocation failure, mbuf/receive delivery.

Mermaid source

sequenceDiagram
    participant U as Routing socket caller
    participant K as route_output
    participant T as Route tree and entry
    participant W as Network work queue
    participant S as Receiving sockets
    U->>K: Serialized request mbuf
    K->>K: Unlock socket#59; acquire rnh_lock#59; validate
    K->>T: Look up or mutate route
    T-->>K: Referenced result or errno
    opt Route refresh or deletion event
        K->>W: Queue retained route and event data
    end
    K->>K: Build report#59; release route and routing lock
    K->>S: Append response or broadcast through raw_input
    Note over T,S: A later report or delivery failure does not undo mutation
    W->>W: Invoke route callbacks#59; release event reference
    K-->>U: Return operation or selected delivery error

The sequence shows one possible ordering for deferred callbacks; route_output() does not wait for those callbacks before returning.

4. Notifications and route deletion have their own lifetimes

Routing-socket response/event delivery and in-kernel route event callbacks are separate facilities. raw_input() filters rawcb endpoints, copies messages for all but the final recipient, and appends to their receive sockbufs. Copy/admission failures do not return a route-transaction failure to the producer; the routine is void. SO_USELOOPBACK controls whether the sender receives its own ordinary message. The silent-get path stores the operation error in rtm_errno, then resets the local error before sbappendaddr() and checks that delivery error. Thus its syscall return and the response header need not report the same outcome. Raw fanout, loopback and silent delivery.

route_event_enqueue_nwk_wq_entry() instead allocates a network-work item and retains the affected route. It snapshots a gateway socket address for the callback argument; gwrt itself gets no additional reference. The callback later invokes handlers using the retained route and frees that reference. Deregistration is special: it transfers the existing registration reference into queued cleanup instead of taking another. The event therefore preserves route lifetime without providing an immutable snapshot of every route field; for example, callback flags are read at callback time. Queued event construction, callback/deregistration.

The lifetime transfer is explicit in route.c:4645–4657:

	/*
	 * If the intent is to de-register, don't take
	 * reference, route event registration already takes
	 * a reference on route.
	 */
	if (route_event_code != ROUTE_EVHDLR_DEREGISTER) {
		/* The reference is released by route_event_callback */
		if (rt_locked) {
			RT_ADDREF_LOCKED(rt);
		} else {
			RT_ADDREF(rt);
		}
	}

Deletion removes the radix node, clears RTF_UP, takes a temporary/reference-to-return, sets RTF_CONDEMNED, and optionally queues deletion events. It removes affected child routes, detaches its gateway reference, calls the interface address's protocol deletion callback, updates default-route/NECP state and increments the route generation. arp_rtrequest(RTM_DELETE) unlinks its llinfo_arp, clears RTF_LLINFO, purges reachability state and flushes held packets, but leaves rt_llinfo/la_rt connected until final route reclamation. Existing PCB and work-item references thus preserve an unusable route object, not an active ARP queue. Route removal, ARP deletion, ARP final free.

5. IPv4 Ethernet output: use a link address or transfer packet ownership

arp_ifinit() installs arp_rtrequest as the interface address callback and requests cloning. The callback ignores indirect gateway routes; a connected cloning route receives an AF_LINK gateway template, while a resolved/direct host route receives llinfo_arp, callbacks and a place in the global ARP list. This is how route construction becomes neighbor state. Interface setup, cloning template and LLINFO creation.

For an ordinary IPv4 Ethernet packet, DLIL's protocol pre_output operation is ether_inet_pre_output(). It invokes arp_lookup_ip() with the packet, network destination and optional route hint. A successful lookup copies the link address and selects the Ethernet IP type; the caller can then frame/output the same mbuf. With a gateway hint, route_to_gwroute() obtains a referenced, locked route to the next hop. For a gateway route, ip_output() changes the separate dst socket-address pointer passed to DLIL to the route's gateway. The IP destination bytes in the packet remain the remote host; arp_lookup_ip() therefore asks about the next-hop IP, and the frame destination becomes that gateway's link address. Separate next-hop destination, DLIL arguments. Ethernet pre-output, next-hop lookup, gateway-route contract.

Broadcast/multicast use direct interface/link mapping. For ordinary unicast, missing link-layer state can trigger scoped host-route creation. IFF_NOARP, no route, unsuitable gateway family and invalid interface state return errors. A cached address is usable only when its expiration permits use and its sockaddr_dl has nonzero address length; an optional reachability record can still trigger probing. Special destinations and route creation, usable predicate.

Sources: DLIL pre-output result handling, retry exhaustion/tail removal, and this exact ownership decision from in_arp.c:1594–1600:

release_just_return:
	/* The packet is now held inside la_holdq or dropped */
	result = EJUSTRETURN;
	if (packet != NULL && !enqueued) {
		m_freem(packet);
		packet = NULL;
	}

Queue pressure and retries

The queue is initialized with the configured per-entry arp_maxhold (default 16; zero becomes an effectively unbounded 32-bit queue limit). A global arpstat.held check against arp_maxhold_total (default 1024) can refuse the new packet. That check uses global atomic accounting without globally serializing all entry queues, so it is pressure control rather than proof of an exact cross-CPU hard cap. At a full per-entry queue, the code first evicts a packet below the control service class if available, otherwise the head; then it adds the new packet. This is not simply “retain only the last packet.” Defaults, global limit, initialization/admission.

The unresolved path marks a probe deadline when the first retained packet or packetless probe request arrives. For an expirable route, it permits a broadcast attempt when la_asked == 0 or the saved expiration differs from current uptime seconds, then increments la_asked against la_maxtries. Repeated packets within the same second normally queue without another broadcast. With the default five tries, the first five permitted attempts send requests; the next permitted attempt marks RTF_REJECT, extends the down deadline by arpt_down, and returns the current packet to the caller via an error. Queue/probe bookkeeping, request admission and transmission, exhaustion.

dlil_send_arp() is called after retaining the interface address and dropping rt_lock; its return is not used here as evidence that a frame reached the wire. Later sends and received ARP traffic drive progress. A route hint's gateway validation rejects an unexpired RTF_REJECT entry as EHOSTDOWN or EHOSTUNREACH. The probe timer described below expires retained work; it does not emit one broadcast retransmission at each tick. Unlocked ARP send, reject gate.

6. Reply processing and a worked unresolved-neighbor flow

The Ethernet receive adapter validates ARP hardware/protocol types, lengths and a nonbroadcast sender MAC, builds socket-address representations, updates link reachability and calls arp_ip_handle_input(). It always frees the incoming ARP mbuf after the handler; that mbuf is separate from the IP packets retained for resolution. Ethernet input validation and cleanup.

arp_ip_handle_input() retains a matching local interface address or a fallback address for the interface. It ignores its own hardware address, reports duplicate local-IP conflicts, and looks up the sender IP's neighbor route. Ordinary creation requires that this host be the ARP target and the sender IP be nonzero; configured gratuitous-announcement/proxy/link-local branches alter that policy. The common path rejects a mapping learned on the wrong interface and refuses to change an already populated permanent entry. It can learn from accepted requests as well as replies. Address selection/conflicts, lookup, announcement/interface checks, permanent mapping check.

Accepted sender hardware bytes replace the direct route's sockaddr_dl address. Dynamic expiration moves to now + arpt_keep; rejection is cleared; link reachability is refreshed; asked/probing counters reset. Queued route events notify dependent clients. Then the entire held packet list is detached under rt_lock, accounting is reduced, and DLIL is called after unlocking, while the route reference still protects the hint/key. The call's return is ignored here: resolution led to an output attempt, not a guarantee that the device transmitted every held packet. Update and events.

The ownership transfer is visible in in_arp.c:2164–2176:

		_getq_all(&llinfo->la_holdq, &pkt, NULL, &held, NULL);
		m0 = pkt.cp_mbuf;
		if (arp_verbose) {
			log(LOG_DEBUG, "%s: sending %u held packets\n",
			    __func__, held);
		}
		os_atomic_add(&arpstat.held, -held, relaxed);
		VERIFY(qempty(&llinfo->la_holdq));
		RT_UNLOCK(route);
		dlil_output(ifp, PF_INET, m0, (caddr_t)route,
		    rt_key(route), 0, NULL);
		RT_REMREF(route);
		route = NULL;

Consider two IP packets P1 and P2 for 198.51.100.9 through unresolved gateway 192.0.2.1. Assume the route and interface remain usable, queue capacity is available, and P2 arrives within the same uptime second. This is an illustrative trace through the branches above, not an executed packet capture:

Mermaid source

sequenceDiagram
    participant O as IPv4 output and DLIL
    participant A as ARP route and hold queue
    participant D as Ethernet provider
    participant I as ARP input
    O->>A: Resolve P1 using gateway route
    A->>A: Retain P1#59; set deadline
    A->>D: Send ARP request without rt_lock
    A-->>O: EJUSTRETURN
    O->>A: Resolve P2 in same second
    A->>A: Retain P2#59; suppress immediate retry
    A-->>O: EJUSTRETURN
    D->>I: ARP reply packet
    I->>A: Validate and install sender mapping
    A->>A: Detach P1 and P2#59; clear probe state
    I->>O: Reenter dlil_output with held list, unlocked
    O->>D: Frame and submit resolved IP packets
    I->>I: Free ARP reply packet#59; drop temporary references

The nonordinary IPv4 link-local interface-movement branch illustrates necessary revalidation: retain the route, drop rt_lock, take rnh_lock, relock the route, reject a now-resolved parent-bound clone or condemned route, update interface/address/MTU, then drop/reacquire and check RTF_UP again. The gateway pointer is refreshed after those drops. Storage surviving on a reference does not make the earlier interface/gateway decision remain valid. Lock transition and checks.

7. Three independent aging decisions

Sources: defaults, reachability predicate, unicast probe while permitting output, and aging/probe walker.

Both timer handlers walk the global ARP list under rnh_lock, taking each entry lock inside arptfree(). The aging timer reschedules if it found aging entries; the probe timer reschedules if it found retained packets or packetless requests. Deadlines are checked by these walkers, so a configured interval is not an exact hardware-timer guarantee for every entry. The probe expiry branch flushes pending work and clears the link address but does not delete the route or itself clear LLINFO_PROBING. Later resolution/new work or teardown governs that state. Aging scheduling, probe scheduling, expired-probe actions.

For an expired entry with outstanding references, ordinary aging resets link address length, asked count and rejection so its next use resolves again. Without references, a nonstatic route can be deleted through RTM_DELETE. in_arpdrain() uses the same walker with draining set, protecting referenced/permanent state instead of indiscriminately freeing all neighbor objects. Static and expiration-zero properties differ: the body treats an expiration-zero entry as permanent, whereas a static entry with an expiration can still participate in parts of aging. Reference-sensitive aging/drain.

8. Test evidence and remaining boundaries

These tests were read, not executed. The chapter researched the named routing/ARP bodies and their direct Ethernet/raw-socket adapters. It does not claim all radix algorithms, every routing sysctl/metric/address normalization variant, bridge/proxy/link-local races, multicast membership, IPv6 router/prefix discovery, or real driver behavior are covered. ARP providers and the wire environment determine whether a submitted request or data frame is actually transmitted. IPv6/ND and IP fragmentation, reassembly and ICMP continue the common-IP route; PF/IPsec and transport overlays remain separate mechanisms.