Sockets and protocol attachments

A socket holds endpoint state, two embedded buffers, connection queues and the association to a protocol implementation. Its descriptor, protocol control blocks and queued data have separate allocations and lifetimes. This chapter follows the socket record through construction, reference acquisition, listener publication and final disposal in xnu-12377.121.6, commit ac9718fb1af618d5ce8678d0dc6e8a58f252216f.

Key structures · Networking · Socket buffers · Internet PCBs · Fileprocs and fileglobs

Use socket lifetime and I/O for send/receive algorithms, TCP connections for handshake/timer transitions, and content filtering for deferred policy ownership. Here the emphasis is on what a socket pointer owns, which lock a caller actually acquires, and why detach, close and storage release are different events.

1. One allocation, several separately owned associations

struct socket embeds so_rcv and so_snd; it contains no universal socket mutex. The protocol's pr_getlock/pr_lock/pr_unlock callbacks select synchronization, or the generic path uses the domain mutex. Protocol-private state is reached through so_pcb, while so_proto identifies the shared protocol descriptor. Layout, lock selection.

Mermaid source

flowchart TD
    F["Fileglob or kernel socket owner"] -->|"endpoint association"| S["socket allocation<br/>state and use references"]
    S -->|"embedded"| B["so_rcv and so_snd<br/>sockbuf headers"]
    B -->|"queued records"| M["Mbufs and<br/>external storage"]
    S -->|"so_proto"| P["Shared protosw<br/>operations and lock callbacks"]
    P -->|"pr_domain"| D["Domain registry<br/>fallback mutex"]
    S -->|"so_pcb"| I["Protocol control block"]
    I -->|"ordinary Internet PCB"| L["inpcb mutex<br/>socket back-pointer"]
    I -->|"TCP protocol state"| T["tcpcb attachment"]

The Internet/TCP branch is one implementation, not the meaning of every so_pcb. UNIX-domain sockets use unpcb; other protocols have their own attachments. The fileglob arrow is one ownership association even when several descriptor fileprocs share that fileglob. A kernel-created socket can have no user descriptor at all. UNIX conversion, kernel creation, shared file ownership.

Allocation and the historical zone field

This snapshot declares one typed socket_zone, with ZC_ZFREE_CLEARMEM. soalloc() obtains zeroed storage with Z_WAITOK_ZERO, returns null on allocation failure, assigns a new generation and increments allocation statistics. It does not initialize a protocol mutex, attach a PCB or create a file descriptor. Zone, allocator.

Exact allocation, uipc_socket.c lines 453–468:

soalloc(void)
{
	struct socket *__single so;

	so = zalloc_flags(socket_zone, Z_WAITOK_ZERO);
	if (so != NULL) {
		so->so_gencnt = OSIncrementAtomic64((SInt64 *)&so_gencnt);

		/*
		 * Increment the socket allocation statistics
		 */
		INC_ATOMIC_INT64_LIM(net_api_stats.nas_socket_alloc_total);
	}

	return so;
}

so_zone is still declared with the comment “zone we were allocated from,” but the inspected allocation/free bodies do not select a zone from that field: sodealloc() returns every such record to socket_zone. There is no separate socket-cache reuse branch in these bodies. Do not import an older release's socket/PCB cache layout from the field name or from the allocator's historical comment. Internet PCB and mbuf allocator variants are separate subjects in their companions. Field, allocator context, free.

The record is writable zone storage, not an RO-zone object. so_gencnt is changed on allocation and again at deallocation; it supports identity/accounting uses, but merely saving that number does not retain the allocation or provide a safe raw-pointer acquisition protocol. Generation updates, deallocation.

2. A field and ownership map

The field map deliberately distinguishes identity values, owning references, embedded state and borrowed associations. Acquiring a fileproc I/O reference protects the open-file association; it does not itself lock socket state. Acquiring the socket mutex does not keep the endpoint alive after releasing it unless the caller also owns the relevant use/reference.

3. Protocol selection and attachment happen before external publication

socreate_internal() first nulls its output. A specified protocol uses pffindproto(domain, protocol, type); otherwise it uses pffindtype(). Missing domains, missing protocols and a protocol/type mismatch produce different errors. It also checks the protocol memory hard limit before allocating. These checks precede any endpoint returned to the caller. Selection/admission.

After allocation, construction initializes listener queue heads, type/family/protocol, owner values, referenced credential, protocol pointer, buffer back-pointers and protocol memory accounting. It increments the initial socket use count before calling pru_attach, because attach can create the protocol's lock and publish the PCB. Setup and attach.

Exact attach ordering, uipc_socket.c lines 599–606:

	/*
	 * Attachment will create the per pcb lock if necessary and
	 * increase refcount for creation, make sure it's done before
	 * socket is inserted in lists.
	 */
	so->so_usecount++;

	error = (*prp->pr_usrreqs->pru_attach)(so, proto, p);

On success, the function invokes any owner-update callback, increments the domain reference count, initializes socket filters and default traffic class, applies selected background/defunct policy, then stores the result through *aso. A successful allocation is therefore earlier than a completed attach, and a completed attach is earlier than return/publication to the creating layer. Success tail.

Mermaid source

flowchart TD
    R["Select domain and protocol"] -->|"admitted"| A["Allocate zeroed socket"]
    A -->|"initialize identity, credential and buffers"| U["Initial socket use reference"]
    U -->|"pru_attach"| P["Attempt protocol attachment"]
    P -->|"success"| O["Association ready<br/>return socket to creator"]
    P -->|"attach error"| E["Set no-owner and clearing flags<br/>drop construction use"]
    E -->|"cleanup body"| X["sofreelastref with deallocation requested"]
    O -->|"user socket syscall"| F["Set fileglob data<br/>publish reserved descriptor"]
    O -->|"kernel KPI"| K["Return socket handle<br/>configure optional upcall"]

The diagram omits pre-allocation errors and assumes the protocol attach handler fulfills its unwind contract. Attach failure sets SS_NOFDREF and SOF_PCBCLEARING, drops the initial use and calls sofreelastref(so, 1). The body explicitly diagnoses a non-null so_pcb after attach error; generic cleanup is not a replacement for protocol-specific partial-allocation cleanup. Failure branch.

For ordinary IPv4 TCP, the registered protosw selects tcp_usrreqs and the TCP lock callbacks. tcp_usr_attach() calls tcp_attach(), which reserves buffer limits, calls in_pcballoc(), initializes IP/autosizing state and creates the TCP control block. in_pcballoc() establishes both inp_socket and so_pcb, initializes inpcb_mtx for PR_PCBLOCK, then publishes into the PCB list under its table lock. Registration, TCP attach, construction, PCB associations/publication.

The retained tcp_newtcpcb()-returns-null branch in tcp_attach() temporarily clears SS_NOFDREF around IPv4/IPv6 detach, then restores it, preserving the enclosing constructor's socket ownership. In this snapshot that is a defensive branch: tcp_newtcpcb() selects the tcpcb embedded in struct inp_tp and returns that address. Its optional bandwidth-measurement setup also retains a null-result check that clears the feature flag; the called allocator uses waiting storage and immediately initializes the result, without an explicit null-return path. Follow the PCB companion for the combined PCB allocation and disposal. Defensive unwind, embedded selection, optional setup check, bandwidth allocator.

4. Descriptor access and kernel handles reach the same record differently

socket_common() obtains a reserved fileproc/fileglob first, installs socketops, then calls socreate() or its delegated variant. On creation error it uses fp_free() for unpublished open-file cleanup. On success it assigns the socket through fp_set_data(), clears the descriptor reservation under the fd mutex and drops the constructor's extra fileproc I/O hold. Syscall construction.

Exact descriptor publication, uipc_syscalls.c lines 311–326:

	if (error) {
		fp_free(p, fd, fp);
	} else {
		fp_set_data(fp, so);

		proc_fdlock(p);
		procfdtbl_releasefd(p, fd, NULL);

		if (ENTR_SHOULDTRACE) {
			KERNEL_ENERGYTRACE(kEnTrActKernSocket, DBG_FUNC_START,
			    fd, 0, (int64_t)VM_KERNEL_ADDRPERM(so));
		}
		fp_drop(p, fd, fp, 1);
		proc_fdunlock(p);

		*retval = fd;

file_socket() performs a typed fileproc lookup and returns its socket data pointer while retaining fileproc I/O. It does not also increment so_usecount; the caller owes the matching file-I/O drop. Keeping the raw socket beyond that file hold requires acquiring an independent socket use/reference while the association is valid. The helper's contract explicitly warns about this boundary. Contract/body.

Kernel creation sets SOF1_IN_KERNEL_SOCKET, clears the last-process identity convention and optionally installs callbacks. The architecture condition in the legacy sock_setupcall() body is __arm64__: that branch installs both read and write callbacks; the other branch installs only read. Its comment describes historical embedded/desktop behavior, but the compiled guard is architectural. sock_setupcalls() exposes separate callback arguments. Kernel setup, callback setup.

5. Locking and references must be read together

socket_getlock() returns the selected mutex without acquiring it. socket_lock(so, 1) locks and requests a use increment; socket_unlock(so, 1) unlocks and requests a decrement. Passing zero changes locking without changing that ownership. The generic domain-lock implementation performs these operations itself; callback implementations can add their own lifecycle rules. Get/lock, unlock.

Exact independent retain helper, uipc_socket.c lines 7341–7352:

soreference(struct socket *so)
{
	socket_lock(so, 1);     /* locks & take one reference on socket */
	socket_unlock(so, 0);   /* unlock only */
}

void
sodereference(struct socket *so)
{
	socket_lock(so, 0);
	socket_unlock(so, 1);
}

For ordinary TCP, tcp_lock() obtains inpcb_mtx, then increments so_usecount if requested. MPTCP subflows redirect through the parent socket and recheck whether the relationship changed while waiting for the mutex. tcp_getlock() similarly returns either the PCB mutex or parent-selected mutex. This is why a saved mutex address is not a universal socket-lock accessor. TCP lock, mutex accessor.

The generic domain unlock calls sofreelastref(so, 1) when its decrement reaches zero. tcp_unlock() does not perform that final socket deallocation: it decrements and releases the applicable mutex, leaving PCB disposal to the Internet/TCP lifecycle. A zero use count is consequently not a universal instruction to zfree(socket_zone, so). Generic final decrement, TCP unlock, PCB disposal.

A count example with explicit assumptions

Select a successfully created non-UNIX socket with initial so_usecount == 1, no extra protocol/filter/work references, and one kernel retain. The table describes deltas through the selected wrappers, not all TCP states or connected UNIX peers.

The explicit-retain body increments both counters. sock_release() decrements so_retaincnt, then chooses full close or just dropping the retained use according to count, family, connected state and SS_NOFDREF/MPTCP conditions. Connected UNIX peers have an explicit count-three case because a peer connection supplies another use. Retain/release body, retained original close.

This ledger is distinct from fileproc I/O and fileglob.fg_count. Duplicating an fd normally increments the shared fileglob count, not the socket use count for every duplicate; only the final fileglob close dispatches soo_close(). Fileglob ownership, socket close callback.

6. Listener queues publish children before accept publishes descriptors

sonewconn_internal() runs from a locked listener. Admission uses queue counts, backlog tunables and protocol memory pressure. It allocates a separate socket, inherits selected identity/credential/options and buffer limits, starts the child with one use, and sets SS_NOFDREF. That flag describes a child with no accepted descriptor, not an allocation ready to free. Admission/construction.

Attachment can drop the listener's per-socket lock. On reacquisition the constructor rechecks SO_ACCEPTCONN; if the listener closed, it closes the newly attached child instead of adding it to a dead listener. The successful path takes accept-list serialization, sets so_head, marks setup in progress and links the child into incomplete or completed state. Attach/recheck, queue publication.

Exact queue accounting, uipc_socket2.c lines 470–480:

	if (connstatus) {
		TAILQ_INSERT_TAIL(&head->so_comp, so, so_list);
		so->so_state |= SS_COMP;
	} else {
		TAILQ_INSERT_TAIL(&head->so_incomp, so, so_list);
		so->so_state |= SS_INCOMP;
		head->so_incqlen++;
	}
	head->so_qlen++;

	so_release_accept_list(head);

so_acquire_accept_list() is a flag/wait protocol for per-socket-lock families. It can release a child lock, sleep using the listener mutex, then restore child/listener lock order. Callers must revalidate relationships after it returns. The domain-lock case needs no separate flag acquisition. Accept-list helpers.

Mermaid source

flowchart TD
    N["New child<br/>SS_NOFDREF and one use"] -->|"attach and queue publication"| I["Incomplete queue<br/>SS_INCOMP and so_head"]
    I -->|"connected and verdict permits"| C["Completed queue<br/>SS_COMP and same so_head"]
    C -->|"accept removes child under serialization"| A["Detached candidate<br/>no published fd yet"]
    A -->|"falloc and accept callback"| F["Fileglob owns socket<br/>descriptor published"]
    A -->|"allocation or policy failure"| X["Close detached child"]
    I -->|"listener closes or protocol aborts"| X
    C -->|"listener explicitly removes and aborts"| X

For a concrete queue example, a listener with so_qlen = 5 and so_incqlen = 3 has two completed children. Promoting one incomplete child yields total 5, incomplete 2 and completed 3. Accepting one completed child yields total 4 and incomplete 2. soisconnected() decrements only the incomplete subset when moving queues; accept decrements the total after removal. Admission's optional completed-only calculation changes which value is compared, not how these stored counts are maintained. Promotion, accept removal, admission calculation.

Accept removes the candidate before allocating its new descriptor because falloc() may block. It clears so_head/SS_COMP, keeps listener ownership across unlocked work, then creates a fileglob and sets its socket data. soacceptlock() clears SS_NOFDREF and invokes pru_accept; it does not perform the handshake. Descriptor-allocation failure closes the detached candidate rather than returning it to the queue. Dequeue and policy, new descriptor/failure, accept callback, publication.

A completed child is specially protected by final-free logic: sofreelastref() refuses to decommission an object still on so_comp, even when ordinary owner/protocol flags permit cleanup. Listener close explicitly removes completed children before aborting them. Queue membership is thus an additional lifecycle gate, not merely a list pointer that can be discarded at count zero. Completed-child gate, listener close.

7. Upcalls have a separate close handshake

Each sockbuf stores callback and argument fields; SB_UPCALL enables dispatch and SB_UPCALL_LOCK requests invocation while retaining the socket mutex. sowakeup() snapshots the callback/argument and increments so_upcallusecount before optionally unlocking with refcount zero. After invoking it, the code restores locking, decrements the upcall count and wakes a close waiter when the last callback ends. UNIX peer callbacks additionally protect peer-disconnect state while both locks are temporarily released. Callback dispatch.

Exact upcall preparation, uipc_socket2.c lines 649–657:

	if (sb->sb_flags & SB_UPCALL) {
		void (*sb_upcall)(struct socket *, void *, int);
		caddr_t __single sb_upcallarg;
		int lock = !(sb->sb_flags & SB_UPCALL_LOCK);

		sb_upcall = sb->sb_upcall;
		sb_upcallarg = sb->sb_upcallarg;
		/* Let close know that we're about to do an upcall */
		so->so_upcallusecount++;

An upcall counter increment is not the same operation as a socket use increment. The surrounding caller must already provide socket lifetime. Close waiting is also conditional: soclose_wait_locked() returns immediately if the count is zero or SOF_UPCALLCLOSEWAIT is absent. When enabled it disables new buffer upcalls, marks SOF_CLOSEWAIT and sleeps on the upcall counter with the selected mutex. Do not describe all socket closes as unconditionally joining every callback. Wait gate/body, option update.

Changing an upcall with sock_setupcall() does not wait for an already-running callback; the source explicitly documents that fact. An argument pointer's lifetime therefore comes from the callback owner's synchronization, not merely from replacing the stored pointer. Setup contract.

8. Close, detach, buffer cleanup and final storage release

soo_close() reads the socket from its fileglob, clears the fileglob association first and calls soclose(). Fileproc drain is earlier and different: soo_drain() sets SS_DRAINING and wakes connection/buffer/event waiters so in-progress descriptor I/O can finish. Neither operation can be replaced by freeing a raw socket pointer. File callbacks.

Without explicit kernel retains, soclose() takes a temporary socket use and calls soclose_locked(). That routine notifies filters, optionally waits for upcalls and content filtering, stops new listener children, drains/aborts queued children, disconnects a connected endpoint, applies linger and invokes pru_detach if the PCB is present and not already clearing. Protocol detach may arrange later completion. Close front, listener cleanup, disconnect/detach.

The tail sets SS_NOFDREF, notifies socket knotes, drops the domain reference and original socket use, and calls sofree(). The wrapper later drops its temporary use. Errors from disconnect/linger/detach can be returned after owner state changed; they do not republish the fileglob or reconstruct a descriptor. Close tail/wrapper.

sofree() has a misleading nearby comment saying it will unlock the socket. The body obtains the selected mutex for an ownership assertion and calls sofreelastref(so, 0); it does not itself unlock or request socket-storage deallocation. Its caller must follow the actual protocol lock/release path. Body.

Exact cleanup gate, uipc_socket.c lines 1012–1020:

	if (!(so->so_flags & SOF_PCBCLEARING) || !(so->so_state & SS_NOFDREF)) {
		selthreadclear(&so->so_snd.sb_sel);
		selthreadclear(&so->so_rcv.sb_sel);
		so->so_rcv.sb_flags &= ~(SB_SEL | SB_UPCALL);
		so->so_snd.sb_flags &= ~(SB_SEL | SB_UPCALL);
		so->so_event = sonullevent;
		return;
	}

sofreelastref() performs selected policy/flow detach calls, then requires both SOF_PCBCLEARING and SS_NOFDREF before full buffer cleanup. If the gate fails, it clears select/upcall/event notification state and returns. It handles incomplete-list removal, preserves completed-list members, flushes send and receive buffers and disables callbacks. It calls sodealloc() only when its dealloc argument requests that final step. Cleanup body.

Mermaid source

flowchart TD
    C["Last fileglob close or kernel-owner close"] -->|"disconnect and detach protocol"| O["Set SS_NOFDREF<br/>drop original owner use"]
    O -->|"cleanup attempt"| G{"PCB clearing and<br/>no owner reference?"}
    G -->|"not yet"| W["Retain allocation<br/>protocol cleanup continues"]
    G -->|"yes"| Q{"Still on completed<br/>accept queue?"}
    Q -->|"yes"| W
    Q -->|"no"| B["Flush buffers and disable callbacks"]
    B -->|"generic final-decrement path"| D["sodealloc"]
    B -->|"Internet PCB lifecycle"| P["Dispose PCB when use/want gates permit"]
    P -->|"detach links and destroy mutex"| D
    D -->|"release credential and filters"| Z["Return socket to socket_zone"]

The two final branches are implementation alternatives. in_pcbdispose() requires socket usecount zero and inp_wantcnt == WNT_STOPUSING, with exclusive PCB-table ownership. For PR_PCBLOCK it performs socket cleanup while the PCB mutex is still usable, then unlocks/destroys that mutex, clears the bidirectional association, releases route/PCB storage and calls sodealloc(). Logical in_pcbdetach() precedes this and schedules collection; it need not clear so_pcb immediately. Logical detach, final disposal.

Exact final socket storage release, uipc_socket.c lines 795–807:

sodealloc(struct socket *so)
{
	proto_memacct_sub(so->so_proto, sizeof(struct socket));

	kauth_cred_unref(&so->so_cred);

	/* Remove any filters */
	sflt_termsock(so);

	so->so_gencnt = OSIncrementAtomic64((SInt64 *)&so_gencnt);

	zfree(socket_zone, so);
}

This function subtracts protocol socket-memory accounting, unreferences so_cred, terminates socket filters, advances the generation and frees the zone element. It does not discover and dispose arbitrary protocol-private state on its own. Construction errors, completed accept queues, TCP deferred close and explicit kernel retention must reach it through their respective ownership protocols.

9. Reading boundaries and next steps

For an unfamiliar caller, first identify whether it owns a file I/O hold, initial kernel handle, socket use or listener/PCB association. Then identify the selected protocol lock and whether a lock drop preserves a use reference. Finally separate changes to connection flags, queue membership, policy attachments, protocol-dead state and physical allocation lifetime.

The common socket record and selected IPv4/TCP paths are documented here. IPv6 shares much of the socket construction while selecting different protocol callbacks; UNIX peer accounting and MPTCP lock redirection are explicitly different cases. Optional policy providers and external socket filters retain their own implementation boundaries. This is a source account, not a claim that every protocol has identical close timing or that a source-level reference ledger proves all possible concurrent schedules.

Continue with socket buffers and record queues for data ownership, Internet PCB storage for lookup/transport attachment and deferred disposal, and mbuf storage for the allocations linked from buffers.