Fileprocs and fileglobs: descriptor instances and shared open files

A settled descriptor-table slot points to a fileproc. That record holds per-descriptor flags, guard/select state and references for in-progress I/O. Its fp_glob points to a fileglob, the separately reference-counted open-file state that can be shared by duplicated descriptors, inherited descriptors, UNIX-domain messages and fileports. This chapter follows the allocations, field access and last releases in xnu-12377.121.6 so that a pointer or count in a caller can be matched to the object it actually protects.

Key structures · Descriptor tables · Processes guide · Vnode identity · Credential storage · IPC ports

The existing descriptor operations chapter traces the syscall algorithms. The emphasis here is representation and ownership: why duplication allocates another fileproc, why I/O retention does not increment the fileglob count, how a guard changes a union's interpretation, and why closing a descriptor is different from closing the endpoint.

Inspect the live fileglob layout for the selected source profile.

1. The sharing graph has several independent lifetimes

An ordinary descriptor slot owns a fileproc's base reference. A fileproc owns one shared fileglob reference. The fileglob holds a referenced credential and an endpoint association selected by its operations table. The endpoint may be a vnode, socket, pipe, shared-memory object or another supported type; its own reference/close protocol belongs to that type. Fileproc and fileglob declarations, fileglob credential reference, fileops.

Mermaid source

flowchart TD
    A["Process A: fd 4"] -->|"base fileproc hold"| FP1["fileproc A<br/>flags, guard and I/O count"]
    B["Process A: fd 9"] -->|"base fileproc hold"| FP2["fileproc B<br/>independent descriptor state"]
    C["Process B: inherited fd 4"] -->|"base fileproc hold"| FP3["fileproc C"]
    FP1 -->|"one fg_count reference"| FG["Shared fileglob<br/>offset, status and operations"]
    FP2 -->|"one fg_count reference"| FG
    FP3 -->|"one fg_count reference"| FG
    M["UNIX message or fileport"] -->|"additional shared owner"| FG
    FG -->|"fg_cred retains credential"| CR["Credential"]
    FG -->|"typed fg_data association"| E["Endpoint object<br/>fg_ops selects callbacks"]

The message/fileport arrow does not imply those owners are fileprocs. A queued UNIX-domain rights message stores fileglob pointers; a fileport associates a Mach kobject port with a fileglob. A recipient allocates a new fileproc when it creates a descriptor. UNIX externalization, fileport association, fileport-to-fd.

Conversely, two independent opens of the same vnode normally have separate fileglobs, credentials and offsets, even though both fg_data associations reach that vnode. A duplicate shares the existing fileglob. The distinction is established by fg_alloc_init() in new-file construction versus fg_ref() and assignment in finishdup(). New construction, duplicate construction.

2. Ordinary writable zones and the constructor's extra hold

Fileglobs use the typed fg_zone; fileprocs use the fixed ID ZONE_ID_FILEPROC. Both zone declarations request ZC_ZFREE_CLEARMEM. These are ordinary mutable objects, not the RO-zone companions used by proc/task/thread. The flag asks the allocator to clear storage on free; it is not a field-access locking rule. Zone definitions, RO comparison.

Exact zone declarations, kern_descrip.c lines 150–151:

ZONE_DEFINE_TYPE(fg_zone, "fileglob", struct fileglob, ZC_ZFREE_CLEARMEM);
ZONE_DEFINE_ID(ZONE_ID_FILEPROC, "fileproc", struct fileproc, ZC_ZFREE_CLEARMEM);

fileproc_alloc_init() obtains zeroed no-fail storage and initializes fp_iocount to its base reference. A newly allocated fileproc is private and has no fileglob until its caller assigns one. Allocator.

Exact fileproc initialization, kern_descrip.c lines 1146–1153:

fileproc_alloc_init(void)
{
	struct fileproc *fp;

	fp = zalloc_id(ZONE_ID_FILEPROC, Z_WAITOK | Z_ZERO | Z_NOFAIL);
	os_ref_init(&fp->fp_iocount, &f_refgrp);
	return fp;
}

fg_alloc_init(ctx) requests Z_WAITOK | Z_ZERO, initializes the mutex and atomic shared count, installs uninitops, retains ctx->vc_ucred, and increments the global nfiles count. Unlike the fileproc call, the written allocation flags do not include Z_NOFAIL; the body nevertheless proceeds directly to initialization without an explicit null-return branch. Record the actual flags/body rather than inventing a recoverable allocation return that callers do not handle. Constructor.

Exact shared initialization, kern_descrip.c lines 179–188:

	fg = zalloc_flags(fg_zone, Z_WAITOK | Z_ZERO);
	lck_mtx_init(&fg->fg_lock, &file_lck_grp, LCK_ATTR_NULL);

	os_ref_init_raw(&fg->fg_count, &f_refgrp);
	fg->fg_ops = &uninitops;

	kauth_cred_ref(ctx->vc_ucred);
	fg->fg_cred = ctx->vc_ucred;

	os_atomic_inc(&nfiles, relaxed);

The provisional uninitops object ensures that FILEGLOB_DTYPE() can dereference an operations table before endpoint setup. It is a zero-initialized static table, not a completed endpoint with working callbacks. Normal constructors install the real operations before publication. For a vnode open, open1() writes fg_flag, sets fg_ops = &vnops, then uses fp_set_data() for the vnode pointer. That ordering also supplies the type used by the pointer-authentication accessor. Provisional table, type macro, vnode initialization.

falloc_withinit() first reserves a descriptor, optionally initializes guard state, allocates both records, and increments fp_iocount a second time. Thus its result has a base hold plus a constructor I/O hold. It installs the pointer while the descriptor remains reserved. On successful open, publication clears the reservation and fp_drop() removes the extra hold. Construction, open publication/drop.

For post-construction failure, fp_free() releases the slot, directly calls fg_free(), releases the extra I/O hold and finally frees the base fileproc. It does not call the endpoint's fo_close. Endpoint cleanup already performed by the constructor's error path is therefore part of the contract: late vnode-open failure calls vn_close() and vnode_put() before fp_free(). Earlier failure can free an uninitialized fileglob without dispatching through uninitops. Record rollback, early open errors, late vnode cleanup.

3. Fileproc fields are descriptor-instance state

Exact layout, file_internal.h lines 120–130:

struct fileproc {
	os_refcnt_t      fp_iocount;
	_Atomic fileproc_vflags_t fp_vflags;
	fileproc_flags_t fp_flags;
	uint16_t         fp_guard_attrs;
	struct fileglob *XNU_PTRAUTH_SIGNED_PTR("fileproc.fp_glob") fp_glob;
	union {
		struct select_set     *fp_wset;   /* fp_guard_attrs == 0 */
		struct fileproc_guard *XNU_PTRAUTH_SIGNED_PTR("fileproc.fp_guard") fp_guard;  /* fp_guard_attrs != 0 */
	};
};

The fileproc does not contain a descriptor number or a back-pointer to its owning table. Functions that need to drop I/O or change slot state therefore receive proc and fd separately. fp_drop() can use an explicitly supplied pointer, or recover the pointer by number for existing I/O holders; the latter permits UF_RESERVED only when UF_CLOSING is also set. That exception lets an already admitted operation finish while new lookups are denied. Layout, drop lookup, special existing-I/O access.

4. Fileglob fields describe the shared open state

FILEGLOB_DTYPE(fg) reads fg_ops->fo_type; there is no independent mutable fg_type field. Source files also define aliases such as fp->f_flag, fp->f_ops and fp->f_offset that expand through fp_glob. An assignment to fp->f_offset therefore changes shared state, despite the expression beginning with a fileproc pointer. Type accessor, local aliases.

fd_vn_data contains its own mutex, offset/cookie, temporary directory-entry buffer, valid/consumed/allocation sizes and start/end filesystem offsets. Its constructor initializes fv_lock; its destructor frees the buffer and destroys the mutex. The shared pointer's location is more informative about duplication semantics than the historical structure comment describing “a file descriptor.” Layout, allocation/free.

Confinement is shared state too. F_SETCONFINED checks for exactly one fileglob reference and FP_CLOFORK before setting the immutable FG_CONFINED bit; an already confined fileglob succeeds, a shared one returns EAGAIN, and clearing confinement is rejected. fg_sendable() rejects confined objects and admits only selected types. The fd-lock requirement on fg_ref() is explicitly tied to this admission/count relationship. Setter, sendability, reference contract.

5. Accessing endpoint data and selecting a callback

fg_get_data() is a __pure2 wrapper around fg_get_data_volatile(). The documented distinction concerns compiler hoisting of repeated calls, not reference ownership. The corresponding fp_get_data() forwards through fp_glob. Neither accessor obtains vnode I/O, socket locks or another endpoint-specific hold. Accessor contracts, fileproc forwarding.

On builds with __has_feature(ptrauth_calls), the accessor authenticates the stored pointer using the address of fg_data blended with a discriminator derived from the fileglob type. On other builds the field is returned without that branch. This is why copying raw fg_data bits into another fileglob is different from storing the decoded pointer through its setter. Setter/getter bodies.

Exact authenticated load, kern_descrip.c lines 325–336:

#if __has_feature(ptrauth_calls)
	int type = FILEGLOB_DTYPE(fg);

	if (fg_data) {
		type ^= OS_PTRAUTH_DISCRIMINATOR("fileglob.fg_data");
		fg_data = ptrauth_auth_data(fg_data,
		    ptrauth_key_process_independent_data,
		    ptrauth_blend_discriminator(store, type));
	}
#endif // __has_feature(ptrauth_calls)

The ordinary read path demonstrates the layers. preparefileread() validates the descriptor and access mode while retaining the fileproc. readv_internal() copies the current VFS context but replaces its credential with fg_cred. fo_read() calls the selected operations table. For a vnode, vn_read() obtains a vnode iocount with vnode_getwithref() before filesystem work and puts it afterward. Finally the syscall path drops fileproc I/O. Admission, context/dispatch/drop, dispatch wrapper, vnode access.

The caller's current process, the open-file credential, the fileproc I/O hold and the vnode iocount answer different questions. An inherited descriptor keeps the shared fileglob's credential; it does not automatically replace fg_cred with the inheriting process's current credential. Individual operations can still perform additional checks using an explicitly supplied current credential. Construction, inheritance, read context, MAC duplication check.

The vnode table makes callback ownership visible: read/write/ioctl/select/kqfilter receive a fileproc, while close receives the shared fileglob. Its default drain callback is fo_no_drain, which returns ENOTSUP; fileproc drain ignores that return and waits according to I/O count. The generic drain mechanism therefore cannot be read as a guarantee that every endpoint can cancel every pending operation. Vnode operations table, generic drain callback, drain loop.

6. A guard adds storage without changing the shared fileglob

Guarded and unguarded fileprocs use the same ZONE_ID_FILEPROC allocation. A guarded record additionally owns a struct fileproc_guard allocated through KALLOC_TYPE_DEFINE(fp_guard_zone, ..., KT_DEFAULT) with Z_WAITOK | Z_ZERO | Z_NOFAIL. The guard record stores a guardid_t and the select wait-set pointer that otherwise lives directly in the fileproc union. It is not a separate kind of fileglob or a protected RO allocation. Guard allocation, guard structure.

Mermaid source

flowchart LR
    U["Unguarded fileproc<br/>fp_guard_attrs = 0"] -->|"union fp_wset"| W["Select wait set"]
    G["Guarded fileproc<br/>fp_guard_attrs != 0"] -->|"union fp_guard"| R["Separate guard record<br/>fpg_guard = guard ID"]
    R -->|"fpg_wset"| W
    U -->|"fp_glob"| F["Shared fileglob"]
    G -->|"fp_glob"| F

The two fileprocs in this diagram illustrate the two representations; it does not claim that a guard operation creates a duplicate. Adding a guard to an existing fileproc preserves the existing wait-set pointer by moving it into fpg_wset before switching the union. Removing it saves the guard allocation, clears attributes and restores fp_wset; the old allocation can then be freed. Thus “unguard” does not mean “discard select state.” Installation, removal.

Exact final unguard helper, kern_guarded.c lines 130–138:

guarded_fileproc_unguard(struct fileproc *fp)
{
	struct fileproc_guard *fpg = fp->fp_guard;

	fp->fp_guard_attrs = 0;
	fp->fp_wset = fpg->fpg_wset;

	zfree(fp_guard_zone, fpg);
}

change_fdguard_np() copies user guard values and may preallocate storage before entering the fd-table critical section. It looks up the fileproc with an I/O hold, validates the current ID/attributes, and changes the representation under the fd mutex. An optional old-flags copyout temporarily drops that mutex while retaining the fileproc; subsequent checks use the state under the reacquired lock. Guard addition is restricted to vnode, pipe, socket, kqueue and netpolicy types, sets FP_CLOEXEC, and adds FP_CLOFORK when GUARD_CLOSE is present. Guard removal applies the requested descriptor close flags after restoring the unguarded representation. Change setup/checks, type/flag transitions.

The guard mask governs operations such as close, duplication, UNIX rights transfer, fileport creation and guarded writes. The guard ID is checked by the relevant guarded API; it is not a reference count or a descriptor generation. guarded_fileproc_copy_guard() creates separate storage when exec constructs a shadow process. At final free, exclusive fileproc ownership allows guarded_fileproc_unguard() without the fd mutex. Mask, guarded lookup, shadow copy/final cleanup.

7. The reference ledger: I/O holds are not shared-file owners

The following transitions describe an ordinary new file and its later sharing. A private, newly allocated fileproc starts with the base count even before installation; calling that a “table hold” describes its eventual ownership, not proof that the object is already published.

The relevant bodies are constructors, constructor extra hold, duplication, message retain, externalization, fileport retain and close/free.

fg_ref() uses an atomic reference operation, but its process-associated acquisition contract still requires the fd mutex. Atomic counting alone does not serialize against deciding whether the fileglob can become confined. The debug/development body checks that mutex when a process is supplied. fg_drop_live() specifically requires that the released reference is not the last; it is not a substitute for final fg_drop(). Reference bodies, contracts.

8. Shared offsets: a worked duplicate and independent-open example

Assume a normal regular vnode, no swap-file special case, no concurrent seek and successful reads of the stated lengths. Opening it once yields fileproc A/fileglob G with fg_offset == 0. Duplicating the descriptor yields fileproc B pointing to G. A separate ordinary open of the same vnode yields fileproc C/fileglob H with its own zeroed offset. New initialization, duplicate association.

The read arithmetic follows read_len - uio_resid(uio), not merely the requested byte count or a success/failure boolean. The syscall read wrapper can convert certain interruption/would-block errors into success after partial transfer, and the vnode routine updates the implicit offset by the transferred amount. Vnode position update, partial read result, per-fp close flag, shared status update.

The shared offset is serialized by FG_OFF_LOCKED, with FG_OFF_LOCKWANT recording a waiter. vn_offset_lock() uses fg_lock to inspect/set those bits, sleeps through the mutex if busy, then releases the mutex after setting the ownership bit. The bit remains held across the relevant I/O; the fileglob mutex itself does not remain locked across the entire filesystem call. Offset lock/unlock.

Exact offset lock acquisition, vfs_vnops.c lines 1095–1105:

vn_offset_lock(struct fileglob *fg)
{
	lck_mtx_lock_spin(&fg->fg_lock);
	while (fg->fg_lflags & FG_OFF_LOCKED) {
		fg->fg_lflags |= FG_OFF_LOCKWANT;
		msleep(&fg->fg_lflags, &fg->fg_lock, PVFS | PSPIN,
		    "fg_offset_lock_wait", 0);
	}
	fg->fg_lflags |= FG_OFF_LOCKED;
	lck_mtx_unlock(&fg->fg_lock);
}

vn_read() takes that bit lock for implicit-offset access before obtaining the vnode iocount. If vnode acquisition fails, it releases the bit before returning. It drops the bit early for non-regular or swap vnodes, so the regular-file worked example is not a universal endpoint serialization guarantee. Explicit-offset reads skip the shared-position protocol. Read branches.

Directory enumeration has its own fd_vn_data.fv_lock and buffer/cookie state. Sharing the fileglob also shares that storage; do not treat every descriptor number as having an independent directory enumeration cursor merely because the type is named fd_vn_data. Directory storage, fileglob association.

9. UNIX messages and fileports keep fileglobs without original slots

UNIX-domain rights internalization first validates the supplied descriptors while holding the process fd mutex, including sendability and applicable guards. It then replaces descriptor integers with fileglob pointers and takes message references. fg_insertuipc_mark() increments both fg_count and fg_msgcount; first-message insertion is completed separately on the global list. Internalization, message marking/list insertion.

Exact message ownership increment, uipc_usrreq.c lines 2274–2281:

	os_ref_retain_raw(&fg->fg_count, &f_refgrp);
	fg->fg_msgcount++;
	if (fg->fg_msgcount == 1) {
		fg->fg_lflags |= FG_INSMSGQ;
		insert = TRUE;
	}
	lck_mtx_unlock(&fg->fg_lock);
	return insert;

FG_INSMSGQ/FG_RMMSGQ and their waiter bits coordinate the interval between changing local counts and changing the global list. fg_removeuipc_mark() decrements fg_msgcount and marks last-message removal, but does not itself decrement fg_count. That distinction lets successful externalization transfer the existing shared reference into a new fileproc. Discard instead removes message bookkeeping and calls fg_drop(). The discard helper takes a process argument and substitutes current_proc() for null; call sites can pass FG_NOPROC when process lock-release side effects must be avoided. Remove marking, transfer, discard, explicit failure argument.

Mermaid source

flowchart TD
    D["One descriptor owns G<br/>fg_count 1, msgcount 0"] -->|"internalize one rights occurrence"| M["Descriptor plus message<br/>fg_count 2, msgcount 1"]
    M -->|"close sending descriptor"| Q["Message alone owns G<br/>fg_count 1, msgcount 1"]
    Q -->|"receive: transfer reference to new fileproc"| R["Receiving descriptor owns G<br/>fg_count 1, msgcount 0"]
    Q -->|"discard: remove message count and drop reference"| Z["No shared owners<br/>endpoint close and fg free"]

This example has one non-cyclic rights occurrence and omits other temporary owners. fg_msgcount is a subset of shared references, not an extra count to add to fg_count. Queued socket cycles require the UNIX garbage-collection algorithm described with the local IPC paths.

Fileports use a different ownership container. sys_fileport_makeport() retains source fileproc I/O, checks sendability/guard restrictions, then takes a fileglob reference before dropping the fd mutex. fileport_alloc() creates an IKOT_FILEPORT kobject port associated with that reference. Port-allocation failure drops the extra reference with fg_drop_live() because the held source fileproc still owns G. Failed port-name insertion or final user copyout uses the port/right cleanup path; the syscall also drops its source I/O hold. Creation and unwind, port allocator.

The no-senders callback deallocates the kobject port and releases its fileglob through fileport_releasefg(), which calls fg_drop(FG_NOPROC, fg). Copying a Mach send right does not create another fileproc or independently call fg_ref() for each right. fileport_makefd() obtains the port's fileglob association, allocates a new fileproc, reserves a local descriptor, takes a new shared reference under that process's fd mutex and publishes it. The fileport association continues to own its original reference. No-senders, release bridge, new descriptor.

The caller's valid fileport/right lifetime is part of borrowing that association: fileport_port_to_fileglob() does not itself retain a fileglob. Follow its supplied port/right contract before moving the returned pointer across another lifetime boundary. Conversion, syscall port copyin/release.

One inspection caller deliberately uses the structure outside the ordinary allocation protocol. proc_fileport_info() zeroes a stack-local fileproc and assigns the fileport's fileglob so selected detail helpers can reuse their fileproc-shaped interface. It does not initialize a base I/O reference, install a descriptor slot or own a new fileglob reference. fileport_invoke() holds the typed send reference across the callback, which supplies the actual lifetime. Passing PROC_NULL to vnode detail omits descriptor-specific close flags. This adapter must not be mistaken for a zone-allocated fileproc that should receive fp_drop()/fileproc_free(). Adapter, callback lifetime, conditional descriptor information. See fileport inspection.

10. Drain the fileproc, then drop the shared open file

fp_close_and_unlock() hides the descriptor with UF_RESERVED | UF_CLOSING before any I/O drain. Notification/AIO work can drop and reacquire the fd mutex while that reservation prevents ordinary lookup. After descriptor knotes are removed, fileproc_drain() sets FPV_DRAIN, invokes the drain callback and wakes select waiters through the correct guarded/unguarded union arm. It sleeps on the process drain event while fp_iocount > 1, reacquiring the fd mutex and checking again. Close setup, drain.

An existing I/O holder calls fp_drop() under the same fd-lock protocol. Reaching count one can clear select-conflict state and wake process drainers. The closing caller must not retain an extra I/O hold on itself: guarded close explicitly drops the hold acquired by guarded lookup before entering fp_close_and_unlock(). Drop/wakeup, guarded close ordering.

When only the base count remains, close detaches the slot, releases the fd mutex and calls fileproc_free(). This consumes the final fileproc reference, frees any guard storage and asserts no select wait-set association remains. The function does not drop fp_glob; its caller saved that pointer and separately calls fg_drop() after the fileproc is gone. Close completion.

Exact fileproc final free, kern_descrip.c lines 1157–1165:

fileproc_free(struct fileproc *fp)
{
	os_ref_release_last(&fp->fp_iocount);
	if (fp->fp_guard_attrs) {
		guarded_fileproc_unguard(fp);
	}
	assert(fp->fp_wset == NULL);
	zfree_id(ZONE_ID_FILEPROC, fp);
}

Mermaid source

flowchart TD
    C["Close one fileproc"] -->|"hide slot and wait for I/O count 1"| D["No remaining I/O users"]
    D -->|"detach slot and unlock"| F["Consume base count<br/>free guard and fileproc"]
    F -->|"fg_drop with saved fileglob"| P["Apply process POSIX-lock cleanup if applicable"]
    P -->|"decrement shared count"| Q{"Last shared reference?"}
    Q -->|"no"| R["Shared endpoint remains"]
    Q -->|"yes"| E["fo_close using stored credential"]
    E -->|"even if close returns error"| G["Free directory data and credential<br/>destroy mutex and free fileglob"]

fg_drop() first constructs a VFS context using fg_cred; it supplies current_thread() only when the passed process is current, otherwise a null thread. For a vnode and an applicable process, POSIX record-lock cleanup occurs before the shared-count decrement. Consequently closing one duplicate can release that process's POSIX locks even when another duplicate keeps the fileglob alive. This is separate from flock/OFD cleanup associated with final vnode-file close. Context and POSIX cleanup, final vnode lock cleanup.

Only a decrement to zero calls fo_close(fg, &context), then fg_free(fg). The callback's error is returned after storage cleanup; an error does not restore the descriptor or fileglob. fg_free() decrements global nfiles, frees optional directory state, unreferences the credential, destroys the mutex, performs conditional vnode-guard cleanup and returns the allocation to fg_zone. Last release, storage cleanup.

For a vnode, fo_close dispatches to vn_closefile(). It tries to obtain a vnode iocount, releases applicable flock/OFD/lease state, calls vn_close() and puts the iocount. vn_close() has distinct special-device ordering for dropping the open use reference around VNOP_CLOSE; this is endpoint teardown, not a release of fp_iocount. If acquiring the vnode fails, vn_closefile() returns that error and the caller still frees the fileglob. Vnode final-close body, vnode close ordering, generic finalization.

11. Reading an unfamiliar caller without conflating the objects

These are common BSD structures and dispatch paths. Pointer signing is conditional on pointer-authentication support; MAC/vnode-guard, file leases and other endpoint features have explicit guards. This account does not claim every descriptor endpoint uses vnode offsets, vnode credentials or a cancellable drain. It follows the generic record owners and selected concrete vnode/UNIX/fileport bodies; filesystem-provider implementations remain the boundary described by VFS dispatch. For the objects beyond fg_data, continue with vnode identity and UBC/vnode pagers.