1 \documentclass{article}
4 \usepackage[pdfborder={0 0 0}]{hyperref}
7 \title{bcachefs: Principles of Operation}
8 \author{Kent Overstreet}
17 \section{Introduction and overview}
19 Bcachefs is a modern, general purpose, copy on write filesystem descended from
20 bcache, a block layer cache.
22 The internal architecture is very different from most existing filesystems where
23 the inode is central and many data structures hang off of the inode. Instead,
24 bcachefs is architected more like a filesystem on top of a relational database,
25 with tables for the different filesystem data types - extents, inodes, dirents,
28 bcachefs supports almost all of the same features as other modern COW
29 filesystems, such as ZFS and btrfs, but in general with a cleaner, simpler,
30 higher performance design.
32 \subsection{Performance overview}
34 The core of the architecture is a very high performance and very low latency b+
35 tree, which also is not a conventional b+ tree but more of hybrid, taking
36 concepts from compacting data structures: btree nodes are very large, log
37 structured, and compacted (resorted) as necessary in memory. This means our b+
38 trees are very shallow compared to other filesystems.
40 What this means for the end user is that since we require very few seeks or disk
41 reads, filesystem latency is extremely good - especially cache cold filesystem
42 latency, which does not show up in most benchmarks but has a huge impact on real
43 world performance, as well as how fast the system "feels" in normal interactive
44 usage. Latency has been a major focus throughout the codebase - notably, we have
45 assertions that we never hold b+ tree locks while doing IO, and the btree
46 transaction layer makes it easily to aggressively drop and retake locks as
47 needed - one major goal of bcachefs is to be the first general purpose soft
50 Additionally, unlike other COW btrees, btree updates are journalled. This
51 greatly improves our write efficiency on random update workloads, as it means
52 btree writes are only done when we have a large block of updates, or when
53 required by memory reclaim or journal reclaim.
55 \subsection{Bucket based allocation}
57 As mentioned bcachefs is descended from bcache, where the ability to efficiently
58 invalidate cached data and reuse disk space was a core design requirement. To
59 make this possible the allocator divides the disk up into buckets, typically
60 512k to 2M but possibly larger or smaller. Buckets and data pointers have
61 generation numbers: we can reuse a bucket with cached data in it without finding
62 and deleting all the data pointers by incrementing the generation number.
64 In keeping with the copy-on-write theme of avoiding update in place wherever
65 possible, we never rewrite or overwrite data within a bucket - when we allocate
66 a bucket, we write to it sequentially and then we don't write to it again until
67 the bucket has been invalidated and the generation number incremented.
69 This means we require a copying garbage collector to deal with internal
70 fragmentation, when patterns of random writes leave us with many buckets that
71 are partially empty (because the data they contained was overwritten) - copy GC
72 evacuates buckets that are mostly empty by writing the data they contain to new
73 buckets. This also means that we need to reserve space on the device for the
74 copy GC reserve when formatting - typically 8\% or 12\%.
76 There are some advantages to structuring the allocator this way, besides being
77 able to support cached data:
79 \item By maintaining multiple write points that are writing to different buckets,
80 we're able to easily and naturally segregate unrelated IO from different
81 processes, which helps greatly with fragmentation.
83 \item The fast path of the allocator is essentially a simple bump allocator - the
84 disk space allocation is extremely fast
86 \item Fragmentation is generally a non issue unless copygc has to kick
87 in, and it usually doesn't under typical usage patterns. The
88 allocator and copygc are doing essentially the same things as
89 the flash translation layer in SSDs, but within the filesystem
90 we have much greater visibility into where writes are coming
91 from and how to segregate them, as well as which data is
92 actually live - performance is generally more predictable than
93 with SSDs under similar usage patterns.
95 \item The same algorithms will in the future be used for managing SMR
96 hard drives directly, avoiding the translation layer in the hard
97 drive - doing this work within the filesystem should give much
98 better performance and much more predictable latency.
101 \section{Feature overview}
103 \subsection{IO path options}
105 Most options that control the IO path can be set at either the filesystem level
106 or on individual inodes (files and directories). When set on a directory via the
107 \texttt{bcachefs attr} command, they will be automatically applied recursively.
109 \subsubsection{Checksumming}
111 bcachefs supports both metadata and data checksumming - crc32c by default, but
112 stronger checksums are available as well. Enabling data checksumming incurs some
113 performance overhead - besides the checksum calculation, writes have to be
114 bounced for checksum stability (Linux generally cannot guarantee that the buffer
115 being written is not modified in flight), but reads generally do not have to be
118 Checksum granularity in bcachefs is at the level of individual extents, which
119 results in smaller metadata but means we have to read entire extents in order to
120 verify the checksum. By default, checksummed and compressed extents are capped
121 at 64k. For most applications and usage scenarios this is an ideal trade off, but
122 small random \texttt{O\_DIRECT} reads will incur significant overhead. In the
123 future, checksum granularity will be a per-inode option.
125 \subsubsection{Encryption}
127 bcachefs supports authenticated (AEAD style) encryption - ChaCha20/Poly1305.
128 When encryption is enabled, the poly1305 MAC replaces the normal data and
129 metadata checksums. This style of encryption is superior to typical block layer
130 or filesystem level encryption (usually AES-XTS), which only operates on blocks
131 and doesn't have a way to store nonces or MACs. In contrast, we store a nonce
132 and cryptographic MAC alongside data pointers, meaning we have a chain of trust
133 up to the superblock (or journal, in the case of unclean shutdowns) and can
134 definitely tell if metadata has been modified, dropped, or replaced with an
135 earlier version. Therefore, replay attacks are not possible, with the exception
136 of an offline rollback of the entire filesystem to a previous version (but see
139 Encryption can only be specified for the entire filesystem, not per file or
140 directory - this is because metadata blocks do not belong to a particular file.
141 All data and metadata except for the superblock is encrypted, and all data
142 and metadata is authenticated.
144 In the future we'll probably add AES-GCM for platforms that have hardware
145 acceleration for AES, but in the meantime software implementations of ChaCha20
146 are also quite fast on most platforms.
148 \texttt{scrypt} is currently used for the key derivation function (KDF), which
149 converts the user supplied passphrase to an encryption key. This is the same
150 function used by Tarsnap and Qubes OS’s backup support. The key derivation is
151 implemented entirely in user-space, so other means of deriving a key can be used
152 in the future without any kernel changes.
155 To format a filesystem with encryption, use
156 \begin{quote} \begin{verbatim}
157 bcachefs format --encrypted /dev/sda1
158 \end{verbatim} \end{quote}
160 You will be prompted for a passphrase. Then, to use an encrypted filesystem
162 \begin{quote} \begin{verbatim}
163 bcachefs unlock /dev/sda1
164 \end{verbatim} \end{quote}
166 You will be prompted for the passphrase and the encryption key will be added to
167 your in-kernel keyring; mount, fsck and other commands will then work as usual.
169 The passphrase on an existing encrypted filesystem can be changed with the
170 \texttt{bcachefs set-passphrase} command. To permanently unlock an encrypted
171 filesystem, use the \texttt{bcachefs remove-passphrase} command - this can be
172 useful when dumping filesystem metadata for debugging by the developers.
174 There is a \texttt{wide\_macs} option which controls the size of the
175 cryptographic MACs stored on disk. By default, only 80 bits are stored, which
176 should be sufficient security for most applications. With the
177 \texttt{wide\_macs} option enabled we store the full 128 bit MAC, at the cost of
178 making extents 8 bytes bigger. \texttt{wide\_macs} is recommended for cases
179 where an attacker can make repeated attempts at forging a MAC, such as scenarios
180 where the storage device itself is untrusted (but see below).
182 For technical reasons, bcachefs encryption is unsafe if the underlying storage
183 is snapshotted and rolled back to an earlier version. (Using bcachefs's own
184 snapshot functionality \textit{is} safe.) Therefore, one must exercise care
185 when using bcachefs encryption with ``fancy'' storage devices. It is safe to
186 rely on bcachefs encryption if both of the following hold:
189 \item You trust your drives to not be actively malicious. For the
190 internal storage on your laptop or desktop, this is probably a
191 safe assumption, and if it is not, you likely have much worse
192 problems. However, it is not necessarily a safe assumption for
193 e.g. USB drives or network storage. In those cases you will
194 need to decide for yourself if you are worried about this.
196 \item You are not using ``fancy'' storage systems that support snapshots.
197 This includes e.g. LVM, ZFS, and loop devices on reflinked or
198 snapshotted files. Most network storage and/or virtualization
199 solutions also support snapshots.
202 If you \textit{are} using snapshots, you must make sure that you never mount
203 a snapshotted, encrypted volume, except with \texttt{-o nochanges}. If this
204 rule is violated, an attacker might be able to recover sensitive data that
205 the encryption was supposed to protect \footnotemark. Future versions of
206 bcachefs will not have this limitation. In the meantime, one can make this
207 problem much more difficult to exploit by encrypting the volumes on which
208 bcachefs resides using LUKS, provided that LUKS is above anything that could
209 take a snapshot. For instance, if you are using bcachefs on LVM and might
210 take an LVM snapshot, LUKS would need to be between LVM and bcachefs.
212 \footnotetext{Technical details: AEAD algorithms, such as ChaCha20/Poly1305,
213 require that a \textit{nonce} be used for every encryption. This nonce does not
214 need to be kept secret, but one must never encrypt more than one message with
215 the same (key, nonce) pair. In the case of ChaCha20/Poly1305, violating this
216 rule loses confidentiality and integrity for all messages with the reused nonce.
217 Unfortunately, bcachefs currently derives the nonce for data and journal extents
218 from on-disk state. If a volume is snapshotted and the snapshot mounted,
219 bcachefs will use the same keys and nonces for both the original volume and the
220 snapshot. As long at least one of the volumes is strictly read-only, everything
221 is okay, but soon as data is written, bcachefs will use the same nonce to
222 encrypt what is almost certain to be two different messages, which is insecure.
223 Encrypting the volume bcachefs is on makes this much harder to exploit because
224 the attacks rely on observing the XOR of the ChaCha20 ciphertexts, and disk
225 encryption hides this information.}
227 \subsubsection{Compression}
229 bcachefs supports gzip, lz4 and zstd compression. As with data checksumming, we
230 compress entire extents, not individual disk blocks - this gives us better
231 compression ratios than other filesystems, at the cost of reduced small random
234 Data can also be compressed or recompressed with a different algorithm in the
235 background by the rebalance thread, if the \texttt{background\_compression}
238 \subsection{Multiple devices}
240 bcachefs is a multi-device filesystem. Devices need not be the same size: by
241 default, the allocator will stripe across all available devices but biasing in
242 favor of the devices with more free space, so that all devices in the filesystem
243 fill up at the same rate. Devices need not have the same performance
244 characteristics: we track device IO latency and direct reads to the device that
245 is currently fastest.
247 \subsubsection{Replication}
249 bcachefs supports standard RAID1/10 style redundancy with the
250 \texttt{data\_replicas} and \texttt{metadata\_replicas} options. Layout is not
251 fixed as with RAID10: a given extent can be replicated across any set of
252 devices; the \texttt{bcachefs fs usage} command shows how data is replicated
255 \subsubsection{Erasure coding}
257 bcachefs also supports Reed-Solomon erasure coding - the same algorithm used by
258 most RAID5/6 implementations) When enabled with the \texttt{ec} option, the
259 desired redundancy is taken from the \texttt{data\_replicas} option - erasure
260 coding of metadata is not supported.
262 Erasure coding works significantly differently from both conventional RAID
263 implementations and other filesystems with similar features. In conventional
264 RAID, the "write hole" is a significant problem - doing a small write within a
265 stripe requires the P and Q (recovery) blocks to be updated as well, and since
266 those writes cannot be done atomically there is a window where the P and Q
267 blocks are inconsistent - meaning that if the system crashes and recovers with a
268 drive missing, reconstruct reads for unrelated data within that stripe will be
271 ZFS avoids this by fragmenting individual writes so that every write becomes a
272 new stripe - this works, but the fragmentation has a negative effect on
273 performance: metadata becomes bigger, and both read and write requests are
274 excessively fragmented. Btrfs's erasure coding implementation is more
275 conventional, and still subject to the write hole problem.
277 bcachefs's erasure coding takes advantage of our copy on write nature - since
278 updating stripes in place is a problem, we simply don't do that. And since
279 excessively small stripes is a problem for fragmentation, we don't erasure code
280 individual extents, we erasure code entire buckets - taking advantage of bucket
281 based allocation and copying garbage collection.
283 When erasure coding is enabled, writes are initially replicated, but one of the
284 replicas is allocated from a bucket that is queued up to be part of a new
285 stripe. When we finish filling up the new stripe, we write out the P and Q
286 buckets and then drop the extra replicas for all the data within that stripe -
287 the effect is similar to full data journalling, and it means that after erasure
288 coding is done the layout of our data on disk is ideal.
290 Since disks have write caches that are only flushed when we issue a cache flush
291 command - which we only do on journal commit - if we can tweak the allocator so
292 that the buckets used for the extra replicas are reused (and then overwritten
293 again) immediately, this full data journalling should have negligible overhead -
294 this optimization is not implemented yet, however.
296 \subsubsection{Device labels and targets}
298 By default, writes are striped across all devices in a filesystem, but they may
299 be directed to a specific device or set of devices with the various target
300 options. The allocator only prefers to allocate from devices matching the
301 specified target; if those devices are full, it will fall back to allocating
302 from any device in the filesystem.
304 Target options may refer to a device directly, e.g.
305 \texttt{foreground\_target=/dev/sda1}, or they may refer to a device label. A
306 device label is a path delimited by periods - e.g. ssd.ssd1 (and labels need not
307 be unique). This gives us ways of referring to multiple devices in target
308 options: If we specify ssd in a target option, that will refer to all devices
309 with the label ssd or labels that start with ssd. (e.g. ssd.ssd1, ssd.ssd2).
311 Four target options exist. These options all may be set at the filesystem level
312 (at format time, at mount time, or at runtime via sysfs), or on a particular
316 \item \texttt{foreground\_target}: normal foreground data writes, and
317 metadata if \\ \texttt{metadata\_target} is not set
318 \item \texttt{metadata\_target}: btree writes
319 \item \texttt{background\_target}: If set, user data (not metadata) will
320 be moved to this target in the background
321 \item\texttt{promote\_target}: If set, a cached copy will be added to
322 this target on read, if none exists
325 \subsubsection{Caching}
327 When an extent has multiple copies on different devices, some of those copies
328 may be marked as cached. Buckets containing only cached data are discarded as
329 needed by the allocator in LRU order.
331 When data is moved from one device to another according to the \\
332 \texttt{background\_target} option, the original copy is left in place but
333 marked as cached. With the \texttt{promote\_target} option, the original copy is
334 left unchanged and the new copy on the \texttt{promote\_target} device is marked
337 To do writeback caching, set \texttt{foreground\_target} and
338 \texttt{promote\_target} to the cache device, and \texttt{background\_target} to
339 the backing device. To do writearound caching, set \texttt{foreground\_target}
340 to the backing device and \texttt{promote\_target} to the cache device.
342 \subsubsection{Durability}
344 Some devices may be considered to be more reliable than others. For example, we
345 might have a filesystem composed of a hardware RAID array and several NVME flash
346 devices, to be used as cache. We can set replicas=2 so that losing any of the
347 NVME flash devices will not cause us to lose data, and then additionally we can
348 set durability=2 for the hardware RAID device to tell bcachefs that we don't
349 need extra replicas for data on that device - data on that device will count as
350 two replicas, not just one.
352 The durability option can also be used for writethrough caching: by setting
353 durability=0 for a device, it can be used as a cache and only as a cache -
354 bcachefs won't consider copies on that device to count towards the number of
355 replicas we're supposed to keep.
359 bcachefs supports reflink, similarly to other filesystems with the same feature.
360 \texttt{cp --reflink} will create a copy that shares the underlying storage.
361 Reading from that file will become slightly slower - the extent pointing to that
362 data is moved to the reflink btree (with a refcount added) and in the extents
363 btree we leave a key that points to the indirect extent in the reflink btree,
364 meaning that we now have to do two btree lookups to read from that data instead
367 \subsection{Inline data extents}
369 bcachefs supports inline data extents, controlled by the \texttt{inline\_data}
370 option (on by default). When the end of a file is being written and is smaller
371 than half of the filesystem blocksize, it will be written as an inline data
372 extent. Inline data extents can also be reflinked (moved to the reflink btree
373 with a refcount added): as a todo item we also intend to support compressed
376 \subsection{Subvolumes and snapshots}
378 bcachefs supports subvolumes and snapshots with a similar userspace interface as
379 btrfs. A new subvolume may be created empty, or it may be created as a snapshot
380 of another subvolume. Snapshots are writeable and may be snapshotted again,
381 creating a tree of snapshots.
383 Snapshots are very cheap to create: they're not based on cloning of COW btrees
384 as with btrfs, but instead are based on versioning of individual keys in the
385 btrees. Many thousands or millions of snapshots can be created, with the only
386 limitation being disk space.
388 The following subcommands exist for managing subvolumes and snapshots:
390 \item \texttt{bcachefs subvolume create}: Create a new, empty subvolume
391 \item \texttt{bcachefs subvolume destroy}: Delete an existing subvolume
393 \item \texttt{bcachefs subvolume snapshot}: Create a snapshot of an
397 A subvolume can also be deleting with a normal rmdir after deleting all the
398 contents, as with \texttt{rm -rf}. Still to be implemented: read-only snapshots,
399 recursive snapshot creation, and a method for recursively listing subvolumes.
403 bcachefs supports conventional user/group/project quotas. Quotas do not
404 currently apply to snapshot subvolumes, because if a file changes ownership in
405 the snapshot it would be ambiguous as to what quota data within that file
406 should be charged to.
408 When a directory has a project ID set it is inherited automatically by
409 descendants on creation and rename. When renaming a directory would cause the
410 project ID to change we return -EXDEV so that the move is done file by file, so
411 that the project ID is propagated correctly to descendants - thus, project
412 quotas can be used as subdirectory quotas.
416 \subsection{Formatting}
418 To format a new bcachefs filesystem use the subcommand \texttt{bcachefs
419 format}, or \texttt{mkfs.bcachefs}. All persistent filesystem-wide options can
420 be specified at format time. For an example of a multi device filesystem with
421 compression, encryption, replication and writeback caching:
422 \begin{quote} \begin{verbatim}
423 bcachefs format --compression=lz4 \
426 --label=ssd.ssd1 /dev/sda \
427 --label=ssd.ssd2 /dev/sdb \
428 --label=hdd.hdd1 /dev/sdc \
429 --label=hdd.hdd2 /dev/sdd \
430 --label=hdd.hdd3 /dev/sde \
431 --label=hdd.hdd4 /dev/sdf \
432 --foreground_target=ssd \
433 --promote_target=ssd \
434 --background_target=hdd
435 \end{verbatim} \end{quote}
437 \subsection{Mounting}
439 To mount a multi device filesystem, there are two options. You can specify all
440 component devices, separated by colons, e.g.
441 \begin{quote} \begin{verbatim}
442 mount -t bcachefs /dev/sda:/dev/sdb:/dev/sdc /mnt
443 \end{verbatim} \end{quote}
444 Or, use the mount.bcachefs tool to mount by filesystem UUID. Still todo: improve
445 the mount.bcachefs tool to support mounting by filesystem label.
447 No special handling is needed for recovering from unclean shutdown. Journal
448 replay happens automatically, and diagnostic messages in the dmesg log will
449 indicate whether recovery was from clean or unclean shutdown.
451 The \texttt{-o degraded} option will allow a filesystem to be mounted without
452 all the devices, but will fail if data would be missing. The
453 \texttt{-o very\_degraded} can be used to attempt mounting when data would be
456 Also relevant is the \texttt{-o nochanges} option. It disallows any and all
457 writes to the underlying devices, pinning dirty data in memory as necessary if
458 for example journal replay was necessary - think of it as a "super read-only"
459 mode. It can be used for data recovery, and for testing version upgrades.
461 The \texttt{-o verbose} enables additional log output during the mount process.
465 It is possible to run fsck either in userspace with the \texttt{bcachefs fsck}
466 subcommand (also available as \texttt{fsck.bcachefs}, or in the kernel while
467 mounting by specifying the \texttt{-o fsck} mount option. In either case the
468 exact same fsck implementation is being run, only the environment is different.
469 Running fsck in the kernel at mount time has the advantage of somewhat better
470 performance, while running in userspace has the ability to be stopped with
471 ctrl-c and can prompt the user for fixing errors. To fix errors while running
472 fsck in the kernel, use the \texttt{-o fix\_errors} option.
474 The \texttt{-n} option passed to fsck implies the \texttt{-o nochanges} option;
475 \texttt{bcachefs fsck -ny} can be used to test filesystem repair in dry-run
478 \subsection{Status of data}
480 The \texttt{bcachefs fs usage} may be used to display filesystem usage broken
481 out in various ways. Data usage is broken out by type: superblock, journal,
482 btree, data, cached data, and parity, and by which sets of devices extents are
483 replicated across. We also give per-device usage which includes fragmentation
484 due to partially used buckets.
488 The journal has a number of tunables that affect filesystem performance. Journal
489 commits are fairly expensive operations as they require issuing FLUSH and FUA
490 operations to the underlying devices. By default, we issue a journal flush one
491 second after a filesystem update has been done; this is controlled with the
492 \texttt{journal\_flush\_delay} option, which takes a parameter in milliseconds.
494 Filesystem sync and fsync operations issue journal flushes; this can be disabled
495 with the \texttt{journal\_flush\_disabled} option - the
496 \texttt{journal\_flush\_delay} option will still apply, and in the event of a
497 system crash we will never lose more than (by default) one second of work. This
498 option may be useful on a personal workstation or laptop, and perhaps less
499 appropriate on a server.
501 The journal reclaim thread runs in the background, kicking off btree node writes
502 and btree key cache flushes to free up space in the journal. Even in the absence
503 of space pressure it will run slowly in the background: this is controlled by
504 the \texttt{journal\_reclaim\_delay} parameter, with a default of 100
507 The journal should be sized sufficiently that bursts of activity do not fill up
508 the journal too quickly; also, a larger journal mean that we can queue up larger
509 btree writes. The \texttt{bcachefs device resize-journal} can be used for
510 resizing the journal on disk on a particular device - it can be used on a
511 mounted or unmounted filesystem.
513 In the future, we should implement a method to see how much space is currently
514 utilized in the journal.
516 \subsection{Device management}
518 \subsubsection{Filesystem resize}
520 A filesystem can be resized on a particular device with the
521 \texttt{bcachefs device resize} subcommand. Currently only growing is supported,
524 \subsubsection{Device add/removal}
526 The following subcommands exist for adding and removing devices from a mounted
529 \item \texttt{bcachefs device add}: Formats and adds a new device to an
531 \item \texttt{bcachefs device remove}: Permenantly removes a device from
532 an existing filesystem.
533 \item \texttt{bcachefs device online}: Connects a device to a running
534 filesystem that was mounted without it (i.e. in degraded mode)
535 \item \texttt{bcachefs device offline}: Disconnects a device from a
536 mounted filesystem without removing it.
537 \item \texttt{bcachefs device evacuate}: Migrates data off of a
538 particular device to prepare for removal, setting it read-only
540 \item \texttt{bcachefs device set-state}: Changes the state of a member
541 device: one of rw (readwrite), ro (readonly), failed, or spare.
543 A failed device is considered to have 0 durability, and replicas
544 on that device won't be counted towards the number of replicas
545 an extent should have by rereplicate - however, bcachefs will
546 still attempt to read from devices marked as failed.
549 The \texttt{bcachefs device remove}, \texttt{bcachefs device offline} and
550 \texttt{bcachefs device set-state} commands take force options for when they
551 would leave the filesystem degraded or with data missing. Todo: regularize and
552 improve those options.
554 \subsection{Data management}
556 \subsubsection{Data rereplicate}
558 The \texttt{bcachefs data rereplicate} command may be used to scan for extents
559 that have insufficient replicas and write additional replicas, e.g. after a
560 device has been removed from a filesystem or after replication has been enabled
563 \subsubsection{Rebalance}
565 To be implemented: a command for moving data between devices to equalize usage
566 on each device. Not normally required because the allocator attempts to equalize
567 usage across devices as it stripes, but can be necessary in certain scenarios -
568 i.e. when a two-device filesystem with replication enabled that is very full has
569 a third device added.
571 \subsubsection{Scrub}
573 To be implemented: a command for reading all data within a filesystem and
574 ensuring that checksums are valid, fixing bitrot when a valid copy can be found.
578 Most bcachefs options can be set filesystem wide, and a significant subset can
579 also be set on inodes (files and directories), overriding the global defaults.
580 Filesystem wide options may be set when formatting, when mounting, or at runtime
581 via \texttt{/sys/fs/bcachefs/<uuid>/options/}. When set at runtime via sysfs the
582 persistent options in the superblock are updated as well; when options are
583 passed as mount parameters the persistent options are unmodified.
585 \subsection{File and directory options}
587 <say something here about how attrs must be set via bcachefs attr command>
589 Options set on inodes (files and directories) are automatically inherited by
590 their descendants, and inodes also record whether a given option was explicitly
591 set or inherited from their parent. When renaming a directory would cause
592 inherited attributes to change we fail the rename with -EXDEV, causing userspace
593 to do the rename file by file so that inherited attributes stay consistent.
595 Inode options are available as extended attributes. The options that have been
596 explicitly set are available under the \texttt{bcachefs} namespace, and the effective
597 options (explicitly set and inherited options) are available under the
598 \texttt{bcachefs\_effective} namespace. Examples of listing options with the
601 \begin{quote} \begin{verbatim}
602 $ getfattr -d -m '^bcachefs\.' filename
603 $ getfattr -d -m '^bcachefs_effective\.' filename
604 \end{verbatim} \end{quote}
606 Options may be set via the extended attribute interface, but it is preferable to
607 use the \texttt{bcachefs setattr} command as it will correctly propagate options
610 \subsection{Full option list}
613 \hspace{0.2in} \= \kill
614 \texttt{block\_size} \` \textbf{format} \\
615 \> \parbox{4.3in}{Filesystem block size (default 4k)} \\ \\
617 \texttt{btree\_node\_size} \` \textbf{format} \\
618 \> Btree node size, default 256k \\ \\
620 \texttt{errors} \` \textbf{format,mount,runtime} \\
621 \> Action to take on filesystem error \\ \\
623 \texttt{metadata\_replicas} \` \textbf{format,mount,runtime} \\
624 \> Number of replicas for metadata (journal and btree) \\ \\
626 \texttt{data\_replicas} \` \textbf{format,mount,runtime,inode} \\
627 \> Number of replicas for user data \\ \\
629 \texttt{replicas} \` \textbf{format} \\
630 \> Alias for both metadata\_replicas and data\_replicas \\ \\
632 \texttt{metadata\_checksum} \` \textbf{format,mount,runtime} \\
633 \> Checksum type for metadata writes \\ \\
635 \texttt{data\_checksum} \` \textbf{format,mount,runtime,inode} \\
636 \> Checksum type for data writes \\ \\
638 \texttt{compression} \` \textbf{format,mount,runtime,inode} \\
639 \> Compression type \\ \\
641 \texttt{background\_compression} \` \textbf{format,mount,runtime,inode} \\
642 \> Background compression type \\ \\
644 \texttt{str\_hash} \` \textbf{format,mount,runtime,inode} \\
645 \> Hash function for string hash tables (directories and xattrs) \\ \\
647 \texttt{metadata\_target} \` \textbf{format,mount,runtime,inode} \\
648 \> Preferred target for metadata writes \\ \\
650 \texttt{foreground\_target} \` \textbf{format,mount,runtime,inode} \\
651 \> Preferred target for foreground writes \\ \\
653 \texttt{background\_target} \` \textbf{format,mount,runtime,inode} \\
654 \> Target for data to be moved to in the background \\ \\
656 \texttt{promote\_target} \` \textbf{format,mount,runtime,inode} \\
657 \> Target for data to be copied to on read \\ \\
659 \texttt{erasure\_code} \` \textbf{format,mount,runtime,inode} \\
660 \> Enable erasure coding \\ \\
662 \texttt{inodes\_32bit} \` \textbf{format,mount,runtime} \\
663 \> Restrict new inode numbers to 32 bits \\ \\
665 \texttt{shard\_inode\_numbers} \` \textbf{format,mount,runtime} \\
666 \> Use CPU id for high bits of new inode numbers. \\ \\
668 \texttt{wide\_macs} \` \textbf{format,mount,runtime} \\
669 \> Store full 128 bit cryptographic MACs (default 80) \\ \\
671 \texttt{inline\_data} \` \textbf{format,mount,runtime} \\
672 \> Enable inline data extents (default on) \\ \\
674 \texttt{journal\_flush\_delay} \` \textbf{format,mount,runtime} \\
675 \> Delay in milliseconds before automatic journal commit (default 1000) \\ \\
677 \texttt{journal\_flush\_disabled}\`\textbf{format,mount,runtime} \\
678 \> \begin{minipage}{4.3in}Disables journal flush on sync/fsync.
679 \texttt{journal\_flush\_delay} remains in effect, thus with the
680 default setting not more than 1 second of work will be lost.
683 \texttt{journal\_reclaim\_delay}\` \textbf{format,mount,runtime} \\
684 \> Delay in milliseconds before automatic journal reclaim \\ \\
686 \texttt{acl} \` \textbf{format,mount} \\
687 \> Enable POSIX ACLs \\ \\
689 \texttt{usrquota} \` \textbf{format,mount} \\
690 \> Enable user quotas \\ \\
692 \texttt{grpquota} \` \textbf{format,mount} \\
693 \> Enable group quotas \\ \\
695 \texttt{prjquota} \` \textbf{format,mount} \\
696 \> Enable project quotas \\ \\
698 \texttt{degraded} \` \textbf{mount} \\
699 \> Allow mounting with data degraded \\ \\
701 \texttt{very\_degraded} \` \textbf{mount} \\
702 \> Allow mounting with data missing \\ \\
704 \texttt{verbose} \` \textbf{mount} \\
705 \> Extra debugging info during mount/recovery \\ \\
707 \texttt{fsck} \` \textbf{mount} \\
708 \> Run fsck during mount \\ \\
710 \texttt{fix\_errors} \` \textbf{mount} \\
711 \> Fix errors without asking during fsck \\ \\
713 \texttt{ratelimit\_errors} \` \textbf{mount} \\
714 \> Ratelimit error messages during fsck \\ \\
716 \texttt{read\_only} \` \textbf{mount} \\
717 \> Mount in read only mode \\ \\
719 \texttt{nochanges} \` \textbf{mount} \\
720 \> Issue no writes, even for journal replay \\ \\
722 \texttt{norecovery} \` \textbf{mount} \\
723 \> Don't replay the journal (not recommended) \\ \\
725 \texttt{noexcl} \` \textbf{mount} \\
726 \> Don't open devices in exclusive mode \\ \\
728 \texttt{version\_upgrade} \` \textbf{mount} \\
729 \> Upgrade on disk format to latest version \\ \\
731 \texttt{discard} \` \textbf{device} \\
732 \> Enable discard/TRIM support \\ \\
735 \subsection{Error actions}
736 The \texttt{errors} option is used for inconsistencies that indicate some sort
737 of a bug. Valid error actions are:
739 \item[{\tt continue}] Log the error but continue normal operation
740 \item[{\tt ro}] Emergency read only, immediately halting any changes
741 to the filesystem on disk
742 \item[{\tt panic}] Immediately halt the entire machine, printing a
743 backtrace on the system console
746 \subsection{Checksum types}
747 Valid checksum types are:
750 \item[{\tt crc32c}] (default)
754 \subsection{Compression types}
755 Valid compression types are:
757 \item[{\tt none}] (default)
763 \subsection{String hash types}
764 Valid hash types for string hash tables are:
768 \item[{\tt siphash}] (default)
771 \section{Debugging tools}
773 \subsection{Sysfs interface}
775 Mounted filesystems are available in sysfs at \texttt{/sys/fs/bcachefs/<uuid>/}
776 with various options, performance counters and internal debugging aids.
778 \subsubsection{Options}
780 Filesystem options may be viewed and changed via \\
781 \texttt{/sys/fs/bcachefs/<uuid>/options/}, and settings changed via sysfs will
782 be persistently changed in the superblock as well.
784 \subsubsection{Time stats}
786 bcachefs tracks the latency and frequency of various operations and events, with
787 quantiles for latency/duration in the
788 \texttt{/sys/fs/bcachefs/<uuid>/time\_stats/} directory.
791 \item \texttt{blocked\_allocate} \\
792 Tracks when allocating a bucket must wait because none are
793 immediately available, meaning the copygc thread is not keeping
794 up with evacuating mostly empty buckets or the allocator thread
795 is not keeping up with invalidating and discarding buckets.
797 \item \texttt{blocked\_allocate\_open\_bucket} \\
798 Tracks when allocating a bucket must wait because all of our
799 handles for pinning open buckets are in use (we statically
802 \item \texttt{blocked\_journal} \\
803 Tracks when getting a journal reservation must wait, either
804 because journal reclaim isn't keeping up with reclaiming space
805 in the journal, or because journal writes are taking too long to
806 complete and we already have too many in flight.
808 \item \texttt{btree\_gc} \\
809 Tracks when the btree\_gc code must walk the btree at runtime -
810 for recalculating the oldest outstanding generation number of
811 every bucket in the btree.
813 \item \texttt{btree\_lock\_contended\_read}
814 \item \texttt{btree\_lock\_contended\_intent}
815 \item \texttt{btree\_lock\_contended\_write} \\
816 Track when taking a read, intent or write lock on a btree node
819 \item \texttt{btree\_node\_mem\_alloc} \\
820 Tracks the total time to allocate memory in the btree node cache
821 for a new btree node.
823 \item \texttt{btree\_node\_split} \\
824 Tracks btree node splits - when a btree node becomes full and is
825 split into two new nodes
827 \item \texttt{btree\_node\_compact} \\
828 Tracks btree node compactions - when a btree node becomes full
829 and needs to be compacted on disk.
831 \item \texttt{btree\_node\_merge} \\
832 Tracks when two adjacent btree nodes are merged.
834 \item \texttt{btree\_node\_sort} \\
835 Tracks sorting and resorting entire btree nodes in memory,
836 either after reading them in from disk or for compacting prior
837 to creating a new sorted array of keys.
839 \item \texttt{btree\_node\_read} \\
840 Tracks reading in btree nodes from disk.
842 \item \texttt{btree\_interior\_update\_foreground} \\
843 Tracks foreground time for btree updates that change btree
844 topology - i.e. btree node splits, compactions and merges; the
845 duration measured roughly corresponds to lock held time.
847 \item \texttt{btree\_interior\_update\_total} \\
848 Tracks time to completion for topology changing btree updates;
849 first they have a foreground part that updates btree nodes in
850 memory, then after the new nodes are written there is a
851 transaction phase that records an update to an interior node or
852 a new btree root as well as changes to the alloc btree.
854 \item \texttt{data\_read} \\
855 Tracks the core read path - looking up a request in the extents
856 (and possibly also reflink) btree, allocating bounce buffers if
857 necessary, issuing reads, checksumming, decompressing, decrypting,
858 and delivering completions.
860 \item \texttt{data\_write} \\
861 Tracks the core write path - allocating space on disk for a new
862 write, allocating bounce buffers if necessary,
863 compressing, encrypting, checksumming, issuing writes, and
864 updating the extents btree to point to the new data.
866 \item \texttt{data\_promote} \\
867 Tracks promote operations, which happen when a read operation
868 writes an additional cached copy of an extent to
869 \texttt{promote\_target}. This is done asynchronously from the
872 \item \texttt{journal\_flush\_write} \\
873 Tracks writing of flush journal entries to disk, which first
874 issue cache flush operations to the underlying devices then
875 issue the journal writes as FUA writes. Time is tracked starting
876 from after all journal reservations have released their
877 references or the completion of the previous journal write.
879 \item \texttt{journal\_noflush\_write} \\
880 Tracks writing of non-flush journal entries to disk, which do
881 not issue cache flushes or FUA writes.
883 \item \texttt{journal\_flush\_seq} \\
884 Tracks time to flush a journal sequence number to disk by
885 filesystem sync and fsync operations, as well as the allocator
886 prior to reusing buckets when none that do not need flushing are
890 \subsubsection{Internals}
893 \item \texttt{btree\_cache} \\
894 Shows information on the btree node cache: number of cached
895 nodes, number of dirty nodes, and whether the cannibalize lock
896 (for reclaiming cached nodes to allocate new nodes) is held.
898 \item \texttt{dirty\_btree\_nodes} \\
899 Prints information related to the interior btree node update
900 machinery, which is responsible for ensuring dependent btree
901 node writes are ordered correctly.
903 For each dirty btree node, prints:
905 \item Whether the \texttt{need\_write} flag is set
906 \item The level of the btree node
907 \item The number of sectors written
908 \item Whether writing this node is blocked, waiting for
909 other nodes to be written
910 \item Whether it is waiting on a btree\_update to
911 complete and make it reachable on-disk
914 \item \texttt{btree\_key\_cache} \\
915 Prints infromation on the btree key cache: number of freed keys
916 (which must wait for a sRCU barrier to complete before being
917 freed), number of cached keys, and number of dirty keys.
919 \item \texttt{btree\_transactions} \\
920 Lists each running btree transactions that has locks held,
921 listing which nodes they have locked and what type of lock, what
922 node (if any) the process is blocked attempting to lock, and
923 where the btree transaction was invoked from.
925 \item \texttt{btree\_updates} \\
926 Lists outstanding interior btree updates: the mode (nothing
927 updated yet, or updated a btree node, or wrote a new btree root,
928 or was reparented by another btree update), whether its new
929 btree nodes have finished writing, its embedded closure's
930 refcount (while nonzero, the btree update is still waiting), and
931 the pinned journal sequence number.
933 \item \texttt{journal\_debug} \\
934 Prints a variety of internal journal state.
936 \item \texttt{journal\_pins}
937 Lists items pinning journal entries, preventing them from being
940 \item \texttt{new\_stripes} \\
941 Lists new erasure-coded stripes being created.
943 \item \texttt{stripes\_heap} \\
944 Lists erasure-coded stripes that are available to be reused.
946 \item \texttt{open\_buckets} \\
947 Lists buckets currently being written to, along with data type
950 \item \texttt{io\_timers\_read} \\
951 \item \texttt{io\_timers\_write} \\
952 Lists outstanding IO timers - timers that wait on total reads or
953 writes to the filesystem.
955 \item \texttt{trigger\_journal\_flush} \\
956 Echoing to this file triggers a journal commit.
958 \item \texttt{trigger\_gc} \\
959 Echoing to this file causes the GC code to recalculate each
960 bucket's oldest\_gen field.
962 \item \texttt{prune\_cache} \\
963 Echoing to this file prunes the btree node cache.
965 \item \texttt{read\_realloc\_races} \\
966 This counts events where the read path reads an extent and
967 discovers the bucket that was read from has been reused while
968 the IO was in flight, causing the read to be retried.
970 \item \texttt{extent\_migrate\_done} \\
971 This counts extents moved by the core move path, used by copygc
974 \item \texttt{extent\_migrate\_raced} \\
975 This counts extents that the move path attempted to move but no
976 longer existed when doing the final btree update.
979 \subsubsection{Unit and performance tests}
981 Echoing into \texttt{/sys/fs/bcachefs/<uuid>/perf\_test} runs various low level
982 btree tests, some intended as unit tests and others as performance tests. The
984 \begin{quote} \begin{verbatim}
985 echo <test_name> <nr_iterations> <nr_threads> > perf_test
986 \end{verbatim} \end{quote}
988 When complete, the elapsed time will be printed in the dmesg log. The full list
989 of tests that can be run can be found near the bottom of
990 \texttt{fs/bcachefs/tests.c}.
992 \subsection{Debugfs interface}
994 The contents of every btree, as well as various internal per-btree-node
995 information, are available under \texttt{/sys/kernel/debug/bcachefs/<uuid>/}.
997 For every btree, we have the following files:
1000 \item \textit{btree\_name} \\
1001 Entire btree contents, one key per line
1003 \item \textit{btree\_name}\texttt{-formats} \\
1004 Information about each btree node: the size of the packed bkey
1005 format, how full each btree node is, number of packed and
1006 unpacked keys, and number of nodes and failed nodes in the
1007 in-memory search trees.
1009 \item \textit{btree\_name}\texttt{-bfloat-failed} \\
1010 For each sorted set of keys in a btree node, we construct a
1011 binary search tree in eytzinger layout with compressed keys.
1012 Sometimes we aren't able to construct a correct compressed
1013 search key, which results in slower lookups; this file lists the
1014 keys that resulted in these failed nodes.
1017 \subsection{Listing and dumping filesystem metadata}
1019 \subsubsection{bcachefs show-super}
1021 This subcommand is used for examining and printing bcachefs superblocks. It
1022 takes two optional parameters:
1024 \item \texttt{-l}: Print superblock layout, which records the amount of
1025 space reserved for the superblock and the locations of the
1027 \item \texttt{-f, --fields=(fields)}: List of superblock sections to
1028 print, \texttt{all} to print all sections.
1031 \subsubsection{bcachefs list}
1033 This subcommand gives access to the same functionality as the debugfs interface,
1034 listing btree nodes and contents, but for offline filesystems.
1036 \subsubsection{bcachefs list\_journal}
1038 This subcommand lists the contents of the journal, which primarily records btree
1039 updates ordered by when they occured.
1041 \subsubsection{bcachefs dump}
1043 This subcommand can dump all metadata in a filesystem (including multi device
1044 filesystems) as qcow2 images: when encountering issues that \texttt{fsck} can
1045 not recover from and need attention from the developers, this makes it possible
1046 to send the developers only the required metadata. Encrypted filesystems must
1047 first be unlocked with \texttt{bcachefs remove-passphrase}.
1049 \section{ioctl interface}
1051 This section documents bcachefs-specific ioctls:
1054 \item \texttt{BCH\_IOCTL\_QUERY\_UUID} \\
1055 Returs the UUID of the filesystem: used to find the sysfs
1056 directory given a path to a mounted filesystem.
1058 \item \texttt{BCH\_IOCTL\_FS\_USAGE} \\
1059 Queries filesystem usage, returning global counters and a list
1060 of counters by \texttt{bch\_replicas} entry.
1062 \item \texttt{BCH\_IOCTL\_DEV\_USAGE} \\
1063 Queries usage for a particular device, as bucket and sector
1064 counts broken out by data type.
1066 \item \texttt{BCH\_IOCTL\_READ\_SUPER} \\
1067 Returns the filesystem superblock, and optionally the superblock
1068 for a particular device given that device's index.
1070 \item \texttt{BCH\_IOCTL\_DISK\_ADD} \\
1071 Given a path to a device, adds it to a mounted and running
1072 filesystem. The device must already have a bcachefs superblock;
1073 options and parameters are read from the new device's superblock
1074 and added to the member info section of the existing
1075 filesystem's superblock.
1077 \item \texttt{BCH\_IOCTL\_DISK\_REMOVE} \\
1078 Given a path to a device or a device index, attempts to remove
1079 it from a mounted and running filesystem. This operation
1080 requires walking the btree to remove all references to this
1081 device, and may fail if data would become degraded or lost,
1082 unless appropriate force flags are set.
1084 \item \texttt{BCH\_IOCTL\_DISK\_ONLINE} \\
1085 Given a path to a device that is a member of a running
1086 filesystem (in degraded mode), brings it back online.
1088 \item \texttt{BCH\_IOCTL\_DISK\_OFFLINE} \\
1089 Given a path or device index of a device in a multi device
1090 filesystem, attempts to close it without removing it, so that
1091 the device may be re-added later and the contents will still be
1094 \item \texttt{BCH\_IOCTL\_DISK\_SET\_STATE} \\
1095 Given a path or device index of a device in a multi device
1096 filesystem, attempts to set its state to one of read-write,
1097 read-only, failed or spare. Takes flags to force if the
1098 filesystem would become degraded.
1100 \item \texttt{BCH\_IOCTL\_DISK\_GET\_IDX} \\
1101 \item \texttt{BCH\_IOCTL\_DISK\_RESIZE} \\
1102 \item \texttt{BCH\_IOCTL\_DISK\_RESIZE\_JOURNAL} \\
1103 \item \texttt{BCH\_IOCTL\_DATA} \\
1104 Starts a data job, which walks all data and/or metadata in a
1105 filesystem performing, performaing some operation on each btree
1106 node and extent. Returns a file descriptor which can be read
1107 from to get the current status of the job, and closing the file
1108 descriptor (i.e. on process exit stops the data job.
1110 \item \texttt{BCH\_IOCTL\_SUBVOLUME\_CREATE} \\
1111 \item \texttt{BCH\_IOCTL\_SUBVOLUME\_DESTROY} \\
1112 \item \texttt{BCHFS\_IOC\_REINHERIT\_ATTRS} \\
1115 \section{On disk format}
1117 \subsection{Superblock}
1119 The superblock is the first thing to be read when accessing a bcachefs
1120 filesystem. It is located 4kb from the start of the device, with redundant
1121 copies elsewhere - typically one immediately after the first superblock, and one
1122 at the end of the device.
1124 The \texttt{bch\_sb\_layout} records the amount of space reserved for the
1125 superblock as well as the locations of all the superblocks. It is included with
1126 every superblock, and additionally written 3584 bytes from the start of the
1127 device (512 bytes before the first superblock).
1129 Most of the superblock is identical across each device. The exceptions are the
1130 \texttt{dev\_idx} field, and the journal section which gives the location of the
1133 The main section of the superblock contains UUIDs, version numbers, number of
1134 devices within the filesystem and device index, block size, filesystem creation
1135 time, and various options and settings. The superblock also has a number of
1136 variable length sections:
1139 \item \texttt{BCH\_SB\_FIELD\_journal} \\
1140 List of buckets used for the journal on this device.
1142 \item \texttt{BCH\_SB\_FIELD\_members} \\
1143 List of member devices, as well as per-device options and
1144 settings, including bucket size, number of buckets and time when
1147 \item \texttt{BCH\_SB\_FIELD\_crypt} \\
1148 Contains the main chacha20 encryption key, encrypted by the
1149 user's passphrase, as well as key derivation function settings.
1151 \item \texttt{BCH\_SB\_FIELD\_replicas} \\
1152 Contains a list of replica entries, which are lists of devices
1153 that have extents replicated across them.
1155 \item \texttt{BCH\_SB\_FIELD\_quota} \\
1156 Contains timelimit and warnlimit fields for each quota type
1157 (user, group and project) and counter (space, inodes).
1159 \item \texttt{BCH\_SB\_FIELD\_disk\_groups} \\
1160 Formerly referred to as disk groups (and still is throughout the
1161 code); this section contains device label strings and records
1162 the tree structure of label paths, allowing a label once parsed
1163 to be referred to by integer ID by the target options.
1165 \item \texttt{BCH\_SB\_FIELD\_clean} \\
1166 When the filesystem is clean, this section contains a list of
1167 journal entries that are normally written with each journal
1168 write (\texttt{struct jset}): btree roots, as well as filesystem
1169 usage and read/write counters (total amount of data read/written
1170 to this filesystem). This allows reading the journal to be
1171 skipped after clean shutdowns.
1174 \subsection{Journal}
1176 Every journal write (\texttt{struct jset}) contains a list of entries:
1177 \texttt{struct jset\_entry}. Below are listed the various journal entry types.
1180 \item \texttt{BCH\_JSET\_ENTRY\_btree\_key} \\
1181 This entry type is used to record every btree update that
1182 happens. It contains one or more btree keys (\texttt{struct
1183 bkey}), and the \texttt{btree\_id} and \texttt{level} fields of
1184 \texttt{jset\_entry} record the btree ID and level the key
1187 \item \texttt{BCH\_JSET\_ENTRY\_btree\_root} \\
1188 This entry type is used for pointers btree roots. In the current
1189 implementation, every journal write still records every btree
1190 root, although that is subject to change. A btree root is a bkey
1191 of type \texttt{KEY\_TYPE\_btree\_ptr\_v2}, and the btree\_id
1192 and level fields of \texttt{jset\_entry} record the btree ID and
1195 \item \texttt{BCH\_JSET\_ENTRY\_clock} \\
1196 Records IO time, not wall clock time - i.e. the amount of reads
1197 and writes, in 512 byte sectors since the filesystem was
1200 \item \texttt{BCH\_JSET\_ENTRY\_usage} \\
1201 Used for certain persistent counters: number of inodes, current
1202 maximum key version, and sectors of persistent reservations.
1204 \item \texttt{BCH\_JSET\_ENTRY\_data\_usage} \\
1205 Stores replica entries with a usage counter, in sectors.
1207 \item \texttt{BCH\_JSET\_ENTRY\_dev\_usage} \\
1208 Stores usage counters for each device: sectors used and buckets
1209 used, broken out by each data type.
1214 \subsection{Btree keys}
1217 \item \texttt{KEY\_TYPE\_deleted}
1218 \item \texttt{KEY\_TYPE\_whiteout}
1219 \item \texttt{KEY\_TYPE\_error}
1220 \item \texttt{KEY\_TYPE\_cookie}
1221 \item \texttt{KEY\_TYPE\_hash\_whiteout}
1222 \item \texttt{KEY\_TYPE\_btree\_ptr}
1223 \item \texttt{KEY\_TYPE\_extent}
1224 \item \texttt{KEY\_TYPE\_reservation}
1225 \item \texttt{KEY\_TYPE\_inode}
1226 \item \texttt{KEY\_TYPE\_inode\_generation}
1227 \item \texttt{KEY\_TYPE\_dirent}
1228 \item \texttt{KEY\_TYPE\_xattr}
1229 \item \texttt{KEY\_TYPE\_alloc}
1230 \item \texttt{KEY\_TYPE\_quota}
1231 \item \texttt{KEY\_TYPE\_stripe}
1232 \item \texttt{KEY\_TYPE\_reflink\_p}
1233 \item \texttt{KEY\_TYPE\_reflink\_v}
1234 \item \texttt{KEY\_TYPE\_inline\_data}
1235 \item \texttt{KEY\_TYPE\_btree\_ptr\_v2}
1236 \item \texttt{KEY\_TYPE\_indirect\_inline\_data}
1237 \item \texttt{KEY\_TYPE\_alloc\_v2}
1238 \item \texttt{KEY\_TYPE\_subvolume}
1239 \item \texttt{KEY\_TYPE\_snapshot}
1240 \item \texttt{KEY\_TYPE\_inode\_v2}
1241 \item \texttt{KEY\_TYPE\_alloc\_v3}